Log processing method, control apparatus, and computer storage medium

Abstract

The present application relates to a log processing method, a control apparatus, and a computer storage medium. The method comprises: acquiring a plurality of abnormal log files, file names of at least a part of the abnormal log files comprising identifiers compressed on the basis of log information; on the basis of a preset condition, performing a deduplication operation on the abnormal log files to obtain deduplicated abnormal log files and a data set table, the data set table comprising statistical data in the deduplication operation; and, on the basis of the data set table and a preset rule, sending the deduplicated abnormal log files to a specified backend for processing, the specified backend being specified on the basis of the preset rule. In the implementation of the technical solution of the present application, an efficient and automated log deduplication mechanism is provided. Such a method significantly increases the speed and accuracy of log processing and reduces the waste of resources caused by duplicate or highly similar logs.

Description

Log processing method, control device and computer storage medium

[0001] This application claims priority to Chinese patent application No. 202311391159.6, filed on October 24, 2023, with the invention name “Log processing method, control device and computer storage medium”. The entire contents of the above Chinese patent application are incorporated into this application by reference. Technical Field

[0002] The present application relates to the field of log processing, and specifically provides a log processing method, a control device, and a computer storage medium. Background Art

[0003] In existing log management systems, processing massive amounts of exception logs is a major challenge. Due to the complexity of application systems and devices, the frequency and volume of exception logs generated are continuously increasing. The problem is that many logs are actually duplicates or highly similar to older logs, but because of the difference in file names, they are treated as new exceptions.

[0004] Traditional log processing systems often compare and classify these log file names manually or through simple algorithms, which is not only inefficient but also prone to errors.

[0005] Accordingly, a new log processing solution is needed in this field to solve the above problems.

[0006] Summary of the Invention

[0007] In order to overcome the above-mentioned defects, the present application is proposed to provide a solution or at least partially solve the problem in the prior art that log processing is complex and cannot be automated.

[0008] In a first aspect, the present application provides a log processing method, which includes obtaining multiple exception log files, wherein the file names of at least some of the exception log files include identifiers compressed based on log information; performing a deduplication operation on the exception log files based on preset conditions to obtain deduplicated exception log files and a data set table, wherein the data set table includes statistical data in the deduplication operation; and sending the deduplicated exception log files to a designated backend for processing based on the data set table and preset rules, wherein the designated backend is designated based on the preset rules.

[0009] In one of the above-mentioned technical solutions, "deduplicate the exception log file based on preset conditions to obtain a deduplicated exception log file", including: determining whether the exception log file is a non-text exception log file; if the exception log file is a non-text exception log file, parsing the non-text exception log file to obtain a feature value, wherein the feature value is a mapping of a feature problem set in the non-text exception log file; and deduplicating the obtained non-text exception log file based on the feature value.

[0010] In one of the above-mentioned technical solutions, "parsing the non-text type exception log file to obtain a characteristic value" includes: determining whether the file name of the non-text type exception log file can be directly parsed into log information; if the file name of the non-text type exception log file can be directly parsed into log information, parsing the file name of the non-text type exception log file to obtain log information of the non-text type exception log file, wherein the log information includes: module information; based on the module information in the log information, obtaining a parser corresponding to the non-text type exception log file; parsing the corresponding non-text type exception log file based on the parser to obtain a characteristic value.

[0011] In one of the above-mentioned technical solutions, the method also includes: if the file name of the non-text exception log file cannot be directly parsed into log information, obtaining a responsibility chain for processing the non-text exception log file based on the log type of the non-text exception log file, wherein the responsibility chain includes at least two parsers set corresponding to the log type of the non-text exception log file; inputting the non-text exception log file into the responsibility chain to obtain the matching degree given by each parser; using the parser with the highest matching degree among the at least two parsers to parse the corresponding non-text exception log file to obtain a feature value.

[0012] In one of the above-mentioned technical solutions, "sending the deduplicated exception log file to the designated backend for processing based on the data set table and preset rules" includes: obtaining a module information table, wherein the module information table contains a correspondence between module information and the backend; and sending the deduplicated exception log file to the designated backend for processing based on the data set table and the module information table.

[0013] In one of the above-mentioned technical solutions, "based on the module information in the log information, obtaining the parser corresponding to the non-text exception log file" includes: obtaining a mapping table between modules and parsers; based on the mapping table between modules and parsers and the module information in the log information, obtaining the parser corresponding to the non-text exception log file.

[0014] In one of the above technical solutions, the log information also includes event information and a timestamp.

[0015] In one of the above technical solutions, the method further includes: if the exception log file is a text-type exception log file, filtering a specified text-type exception log file set based on a custom condition to perform a deduplication operation.

[0016] In a second aspect, the present application provides a log processing device, which includes a log collection module: obtaining multiple abnormal log files, wherein the file names of at least some of the abnormal log files include unique identifiers compressed based on log information; a log deduplication module: selectively parsing and deduplicating the file names of the abnormal log files; and a log processing module: sending the deduplicated abnormal log files to a designated backend for processing based on the deduplication operation and preset rules.

[0017] In a third aspect, a control device is provided, which includes a processor and a storage device, wherein the storage device is suitable for storing multiple computer programs, and the computer programs are suitable for being loaded and run by the processor to execute the log processing method described in any one of the technical solutions of the above-mentioned log processing method.

[0018] In a fourth aspect, a computer-readable storage medium is provided, which stores a plurality of computer programs, wherein the computer programs are suitable for being loaded and run by a processor to execute the log processing method described in any one of the technical solutions of the above-mentioned log processing method.

[0019] The above one or more technical solutions of this application have at least one or more of the following beneficial effects:

[0020] The technical solution implemented in this application provides an efficient and automated log deduplication mechanism. This method significantly improves the speed and accuracy of log processing and reduces resource waste caused by duplicate or highly similar logs. Overall, this solution not only optimizes the log management process but also ensures efficient utilization of backend resources, thereby greatly improving efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The disclosure of this application will be more easily understood with reference to the accompanying drawings. Those skilled in the art will readily appreciate that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this application. Furthermore, similar numbers in the figures represent similar components, where:

[0022] FIG1 is a flow chart showing the main steps of a log processing method according to an embodiment of the present application;

[0023] FIG2 is a flowchart illustrating the main steps of a log processing method according to an embodiment of the present application;

[0024] FIG3 is a flow chart of a log processing method according to an embodiment of the present application;

[0025] FIG4 is a schematic diagram of a log processing method according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] Some embodiments of the present application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present application and are not intended to limit the scope of protection of the present application.

[0027] In the description of this application, "module" and "processor" may include hardware, software, or a combination of both. A module may include hardware circuitry, various suitable sensors, communication ports, and memory. It may also include software components, such as computer programs, or a combination of software and hardware. A processor may be a central processing unit, a microprocessor, an image processor, a digital signal processor, or any other suitable processor. A processor has data and / or signal processing capabilities. A processor may be implemented in software, hardware, or a combination of both. Non-transitory computer-readable storage media include any suitable medium capable of storing computer programs, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, and the like. The term "A and / or B" refers to all possible combinations of A and B, such as only A, only B, or both A and B. The terms "at least one of A or B" or "at least one of A and B" have similar meanings to "A and / or B" and may include only A, only B, or both A and B. The singular forms "a" and "the" may also include the plural forms.

[0028] Refer to Figure 1, which is a flow chart of the main steps of a log processing method according to an embodiment of the present application. As shown in Figure 1, the log processing method in the embodiment of the present application mainly includes the following steps S10 to S30, and the details are shown in Figures 3 and 4.

[0029] Step S10: Acquire multiple abnormality log files.

[0030] In this embodiment, the abnormality log file is an abnormality log file sent by the terminal.

[0031] In one implementation, these exception log files record issues encountered by the system or application during operation, typically containing error codes, error messages, timestamps, and other relevant contextual information. Given the large-scale system and application environments, these exceptions can generate a large number of log files, requiring efficient management and processing.

[0032] In this embodiment, the collection of abnormal log files comes from multiple sources. These sources can be physical servers, virtual machines, containers, or other computing entities running applications and services. In this embodiment, the collection of abnormal log files comes from multiple terminal mobile phones.

[0033] Preferably, in one embodiment, corresponding log agents or collectors are deployed on the terminal to ensure the integrity and consistency of the logs. These agents or collectors are responsible for monitoring and capturing abnormal events in real time, and then transmitting the relevant log files to a designated storage or processing center.

[0034] In this embodiment, the file names of at least some exception log files will include a unique identifier generated from the compressed log information. In this embodiment, the unique identifier is generated by applying a hash function, such as MD5 or SHA256, to the log content. This not only ensures the uniqueness of the log file name but also provides a method for rapid comparison and retrieval.

[0035] In this embodiment, by applying a hash function to the log file name to generate a unique identifier, these files can be easily managed and processed. The previous data protocol agreement provides clear guidance and standards for the entire process.

[0036] Step S20: performing a deduplication operation on the abnormal log file based on preset conditions to obtain the deduplicated abnormal log file and data set table.

[0037] In this embodiment, the exception log file includes a text-type exception log file and a non-text-type exception log file.

[0038] In one embodiment, different processing methods are generated depending on whether the abnormal log file is a text-type abnormal log file or a non-text-type abnormal log file. Specifically, steps S202-S203 are the methods used to process abnormal log files that are non-text-type abnormal log files, and S204 is the method used to process abnormal log files that are text-type abnormal log files, as shown in Figure 2, specifically as follows:

[0039] Step S201: Determine whether the abnormal log file is a non-text abnormal log file.

[0040] In this embodiment, further judgment is performed by obtaining the log type of the abnormal log file.

[0041] In one embodiment, for an abnormal log file, its log type or extension is first obtained. Taking a UNIX-like system as an example, a command tool such as file is used to obtain the log type. On other platforms, corresponding system APIs or library functions can be used.

[0042] In this embodiment, the text-type exception log files are usually ASCII or UTF-8 encoded, which is easy for humans to read. They have .txt, .log or other common text file extensions.

[0043] Non-text exception log files may be in binary, proprietary, or application-specific formats, such as .bin or .dat. These files are generally not suitable for direct viewing.

[0044] In this embodiment, the log type is the extension of the log file. For example, when the file extension of an exception log file is .txt, it indicates that the log type is a txt file, that is, a text-type exception log file.

[0045] Step S202: If the abnormal log file is a non-text abnormal log file, the non-text abnormal log file is parsed to obtain a characteristic value.

[0046] In this embodiment, the characteristic value is a unique mapping of a characteristic problem set in a non-text abnormal log file.

[0047] In one embodiment, the feature values ​​are described in detail herein. When parsing non-text exception log files, a key goal is to extract all core or key content. This core content may include a detailed description of a specific exception or problem pattern, related parameters, timestamps, etc. Since log files may contain a large amount of information, much of which may be unnecessary for subsequent analysis, a method is needed to summarize this key content.

[0048] In this embodiment, all key contents are processed using digest technology, such as the SHA-256 hash function. This way, regardless of the length or complexity of the key contents, a fixed-length output, i.e., a characteristic value, can be obtained. This characteristic value is unique. In this embodiment, the key contents mentioned here are certain keywords.

[0049] SHA-256 is a secure digest algorithm that converts input of any length into a unique 256-bit output. This simplifies and simplifies subsequent analysis or matching. For example, if two anomaly log files collected at different times or locations have the same key content, their SHA-256 signatures will also be identical. Therefore, only these signatures need to be compared, eliminating the need to compare the entire log file contents.

[0050] Parsing non-text abnormal log files to obtain characteristic values ​​is a process of converting complex, structured data into simple, unique identifiers, providing a solid foundation for subsequent processing.

[0051] In this embodiment, the characteristic value is obtained through steps S202-1 to S202-7, as follows.

[0052] Step S202 - 1 : Determine whether the file name of the non-text abnormality log file can be directly parsed into log information.

[0053] In this embodiment, the file name of the non-text exception log file may be an identifier compressed based on the log information, or may not be an identifier compressed based on the log information. In this embodiment, the identifier is a unique identifier.

[0054] In one implementation, unique identifier naming (hereinafter referred to as standard naming) is used to obtain a quick overview of the log content based solely on the file name, thereby quickly obtaining feature values. However, in actual operation, some special circumstances may arise, resulting in log files not being generated according to the default naming standard. In such cases, additional processing is required for these non-text abnormal log files.

[0055] Step S202 - 2 : If the file name of the non-text abnormality log file can be directly parsed into log information, the file name of the non-text abnormality log file is parsed to obtain the log information of the non-text abnormality log file.

[0056] In this embodiment, the log information includes: module information, event information and timestamp.

[0057] In one embodiment, the hash value of the file name of the non-text exception log file is first extracted. This typically involves identifying the hash code portion of the file name and determining the type of hash algorithm used (e.g., SHA-256, MD5, etc.). Once the hash value and associated algorithm are identified, the next step is to perform a reverse lookup on the hash value.

[0058] Reverse lookup refers to searching for the original input data based on a known hash value. Preferably, in one embodiment, the system has a hash mapping table that records the correspondence between the original log information and its corresponding hash value. Using this mapping table, the log information can be extracted from the unique identifier.

[0059] In this embodiment, the module information, event information and timestamp included in the log information are described in detail.

[0060] Module information typically refers to the specific system part or component that generated the log entry. In complex systems, many modules or components may run in parallel, each responsible for different functions or tasks. By identifying the specific module generating the log entry, you can quickly locate the system part where a problem or anomaly may occur. This helps developers and maintainers quickly find the root cause of the error and implement appropriate fixes or optimizations.

[0061] Event information describes a specific activity or state change that occurred within a system at a specific point in time. This might include errors, warnings, operational status, user activity, or any other significant event. Event information provides developers and system administrators with detailed insight into system behavior. It can help identify potential issues, track user behavior, or ensure that the system is operating as expected.

[0062] A timestamp is the specific date and time a log entry is recorded. It's typically accurate to the second or millisecond, ensuring each log entry has a unique time identifier. Timestamps allow system administrators or developers to review events in chronological order, reconstructing system behavior and state changes. They can also help determine the relationship, frequency, or duration between events, providing critical information for performance tuning or troubleshooting.

[0063] Step S202 - 3 : Based on the module information in the log information, obtain a parser corresponding to the non-text abnormal log file.

[0064] In this embodiment, a mapping table between modules and parsers is preset in the system.

[0065] In one embodiment, once module information is extracted from log information, a predefined mapping or directory can be used to find the specific parser corresponding to that module. For example, logs from a database module might require a parser focused on database queries and operations, while logs from a network module might require a parser focused on network requests and responses. This approach ensures that each log uses the parser best suited to its content and format, thereby ensuring accuracy and efficiency.

[0066] Step S202 - 4 : Parse the corresponding non-text abnormal log file based on the parser to obtain a feature value.

[0067] In one implementation, once the correct parser is determined, the next task is to use it to parse the abnormal log file. The main goal of the parsing process is to extract meaningful information from unstructured or semi-structured log data and convert it into a format that can be further processed or analyzed.

[0068] The work of a parser is usually divided into two stages: first, it identifies key elements in the log, such as events, error codes, parameters, etc., according to predefined rules or patterns; then, it organizes these elements into a structured output, which is usually a data object or dictionary.

[0069] Finally, features are extracted from the parsed structured data. This typically involves hashing or other transformations of key data points to produce a unique identifier. This feature can then be used for subsequent comparisons, pattern recognition, or other analysis to help diagnose the cause or nature of anomalies.

[0070] Step S202 - 5 : If the file name of the non-text exception log file cannot be directly parsed into log information, a responsibility chain for processing the non-text exception log file is obtained based on the log type of the non-text exception log file.

[0071] In this embodiment, the responsibility chain includes at least two parsers set corresponding to the log type of the non-text type exception log file.

[0072] In one embodiment, when the file name of a non-text exception log file cannot be directly parsed into log information, it indicates that there is no direct parser corresponding to it in the system table. In this case, it is necessary to obtain a chain of responsibility for processing the non-text exception log file based on the log type. In other words, in this case, multiple objects are allowed to handle the request, or more specifically, in this scenario, it allows multiple parsers to attempt to parse the non-text exception log file.

[0073] Based on the log type of a non-text exception log file, the system can determine a chain of responsibility associated with that log type. This chain of responsibility is an ordered list of parsers, each designed for that specific log type. For example, if the log type is about database operations, the chain of responsibility might include a SQL parser, a transaction parser, and a connection parser.

[0074] Each parser in the chain of responsibility has a specific priority. Priorities are often based on the parser's specialization or specific application scenarios. More specifically, some parsers may be better suited for processing specific subtypes of logs, while others may have a broader scope of application. By assigning a priority to each parser in the chain of responsibility, the system can ensure that the parser most likely to successfully parse the log file is attempted first.

[0075] In general, this step ensures that even if the file name of the non-text exception log file does not provide clear parsing clues, the system can effectively select and apply the best parser to ensure the correct parsing of the log.

[0076] Step S202-6: Input the non-text exception log file into the responsibility chain to obtain the matching degree given by each parser.

[0077] In this embodiment, the matching degree reflects the matching degree between the non-text abnormality log file and the parser.

[0078] In one embodiment, when a non-text exception log file is input into the responsibility chain, each parser in the chain attempts to parse it. To determine which parser is most suitable for processing the log file, each parser outputs a matching score. The matching score is a value between 0 and 1, where a higher value indicates a better match. The score is determined based on factors such as the parser's recognition of the log file content, the correspondence with the log structure, and possible specific pattern matching.

[0079] Rather than a simple binary output (match / no match), the match score provides a more nuanced metric that allows the system to better judge which parser is most effective at processing a particular log file. For example, if a parser can recognize 80% of the structure and content in a log file, its match score might be 0.8.

[0080] Step S202-7: Use the parser with the highest matching degree among the at least two parsers to parse the corresponding non-text abnormal log file to obtain a feature value.

[0081] In this embodiment, after all parsers attempt to parse the log file and provide their matching scores, the system selects the parser with the highest matching score to perform further parsing operations.

[0082] For example, there are three parsers in the responsibility chain: Parser A, Parser B, and Parser C. After Parser A attempts to parse the abnormal log file, it gives a match degree of 40%. After Parser B attempts to parse the abnormal log file, it gives a match degree of 90%. After Parser C attempts to parse the abnormal log file, it gives a match degree of 50%.

[0083] In this example, parser B has the highest matching degree, so the system will select parser B to perform further parsing operations on the non-text exception log file to obtain the required feature values.

[0084] In this embodiment, this ensures that the log file is processed by the most appropriate tool, thereby increasing the likelihood of parsing accuracy.

[0085] Step S203: performing a deduplication operation on the acquired non-text abnormal log file based on the feature value.

[0086] In one embodiment, when processing a large number of non-text exception log files, duplicate log entries may be encountered. These duplicates may be due to the same error occurring from multiple sources, or due to duplicate uploads during log generation and collection. Regardless of the cause, duplicate log entries may interfere with subsequent analysis and waste storage and processing resources.

[0087] To solve this problem, this implementation uses a deduplication method based on feature values. Specifically, each non-text abnormal log file is converted into a feature value, which is the unique identifier of the log. This ensures that logs with the same key content will receive the same feature value, while logs with different key content will receive different feature values.

[0088] During deduplication, the system creates a database or repository of signature values. When a new non-text anomaly log file arrives, the system first calculates its signature value and then checks whether it already exists in the database. If so, it indicates a duplicate log entry and is marked and deleted. If not, the signature value is added to the database and processing continues on the original log file.

[0089] Furthermore, in this embodiment, deduplication is performed using a Bloom filter.

[0090] In summary, this feature-based deduplication strategy provides an efficient and accurate method for processing large amounts of non-textual anomaly log files. This approach not only reduces data redundancy but also saves storage and computing resources, thereby improving the efficiency of the entire log processing process.

[0091] It should be noted that if the collected exception log files are in compressed format, such as .zip format, you need to decompress the exception log files after completing the deduplication operation.

[0092] Step S204: If the exception log file is a text-type exception log file, a specified text-type exception log file set is filtered based on the custom condition to perform a deduplication operation.

[0093] In one embodiment, when processing a text-type exception log file, it is first necessary to determine the characteristics and structure of its content. Unlike non-text-type logs, text-type logs are generally easier to read and parse.

[0094] When performing deduplication, consider implementing a pattern-matching strategy. This can use regular expressions or other text-matching techniques to identify and extract key information from log entries. For example, if a log entry contains a specific error code or description, you can use this information as a filter.

[0095] Additionally, to ensure accurate filtering, you can use a word segmentation technique to break log entries into individual words or phrases. You can then use word frequency statistics and other text analysis techniques to rank and weight these words or phrases. This allows you to effectively identify recurring patterns even among a large number of similar, but not identical, log entries.

[0096] After possible duplicate log entries have been identified, they can be compared against the existing log base. If a match is found, then these entries already exist and can be safely removed from the processing queue.

[0097] Step S30: Send the deduplicated exception log file to a designated backend for processing based on the data set table and preset rules.

[0098] In one embodiment, two data set tables are generated in response to the deduplication operation. The two tables are:

[0099] Log parsing table: Counts and stores log information in the deduplicated exception log files and further in-depth information, including the model and system version corresponding to the exception log files. The in-depth information is obtained by log parsing.

[0100] Deduplication table: Statistics the correspondence between duplicate logs and feature values.

[0101] Through these two tables, as well as the module information tables in the system, a complete information map can be established, so that any specific log entry can be traced back to its original file, the relevant business module, and the person in charge. The module information table contains the correspondence between module information and the backend.

[0102] In this embodiment, a detailed JIRA ticket can be automatically generated for each specific exception log based on the information in the log parsing table, the deduplication table, and the module information table.

[0103] It should be pointed out that although the various steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effect of the present application, different steps do not have to be performed in such an order. They can be performed simultaneously (in parallel) or in other orders. These changes are within the scope of protection of the present application.

[0104] Furthermore, the present application also provides a log processing device.

[0105] The apparatus includes: a log collection module: acquiring a plurality of abnormal log files, wherein the file names of at least some of the abnormal log files include unique identifiers compressed based on log information;

[0106] Log deduplication module: selectively parses and deduplicates the file names of abnormal log files;

[0107] Log processing module: Based on the deduplication operation and preset rules, the deduplicated abnormal log files are sent to the designated backend for processing. In one embodiment, the description of the specific implementation functions can be found in steps S10-S30.

[0108] The above-mentioned log processing device is used to execute the embodiment of the log processing method shown in Figure 1. The technical principles, technical problems solved, and technical effects produced by the two are similar. Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process and related instructions of the log processing device can refer to the contents described in the embodiment of the log processing method and will not be repeated here.

[0109] It will be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment of the present application can also be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. The computer program includes a computer program, and the computer program can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device, medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory, random access memory, electric carrier signal, telecommunication signal and software distribution medium that can carry the computer program. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electric carrier signals and telecommunication signals.

[0110] Furthermore, the present application also provides a control device. In a control device embodiment according to the present application, the control device includes a processor and a storage device. The storage device can be configured to store a program for executing the log processing method of the above-mentioned method embodiment, and the processor can be configured to execute the program in the storage device, which includes but is not limited to a program for executing the log processing method of the above-mentioned method embodiment. For ease of explanation, only the parts related to the embodiment of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present application. The control device can be a control device device formed by various electronic devices.

[0111] Furthermore, the present application also provides a computer-readable storage medium. In a computer-readable storage medium embodiment according to the present application, the computer-readable storage medium can be configured to store a program for executing the log processing method of the above-mentioned method embodiment, and the program can be loaded and run by the processor to implement the above-mentioned log processing method. For ease of explanation, only the parts related to the embodiment of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present application. The computer-readable storage medium can be a storage device formed by various electronic devices. Optionally, the computer-readable storage medium in the embodiment of the present application is a non-temporary computer-readable storage medium.

[0112] Furthermore, it should be understood that since the configuration of each module is merely for the purpose of illustrating the functional units of the apparatus of the present application, the physical devices corresponding to these modules may be the processor itself, or a portion of the software in the processor, a portion of the hardware, or a combination of software and hardware. Therefore, the number of modules in the figure is merely illustrative.

[0113] Those skilled in the art will appreciate that the various modules in the device can be adaptively split or merged. Such splitting or merging of specific modules will not cause the technical solution to deviate from the principles of this application. Therefore, the technical solutions after splitting or merging will fall within the scope of protection of this application.

[0114] The relevant user personal information that may be involved in the various embodiments of this application is strictly in accordance with the requirements of laws and regulations, following the principles of legality, legitimacy and necessity, and based on the reasonable purposes of business scenarios, to process the personal information that users actively provide during the use of products / services or generated due to the use of products / services, as well as the personal information obtained with the user's authorization.

[0115] The personal information processed by the Applicant will vary depending on the specific product / service scenario and will be based on the specific scenario in which the user uses the product / service. This may involve the user's account information, device information, driving information, vehicle information, or other related information. The Applicant will treat the user's personal information and its processing with a high degree of diligence.

[0116] The Applicant attaches great importance to the security of user personal information and has taken reasonable and feasible security measures that comply with industry standards to protect user information and prevent personal information from being accessed, disclosed, used, modified, damaged or lost without authorization.

[0117] Thus far, the technical solutions of the present application have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is readily understood by those skilled in the art that the scope of protection of the present application is obviously not limited to these specific embodiments. Without departing from the principles of the present application, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present application.

Claims

1. A log processing method, characterized in that: include: Acquire a plurality of abnormal log files, wherein the file names of at least some of the abnormal log files include identifiers compressed based on log information; Performing a deduplication operation on the abnormal log file based on preset conditions to obtain a deduplicated abnormal log file and a data set table, wherein the data set table includes statistical data in the deduplication operation; Based on the data set table and preset rules, the deduplicated abnormal log files are sent to the specified backend for processing, wherein the specified backend is specified based on the preset rules.

2. The log processing method according to claim 1, characterized in that: “Performing a deduplication operation on the abnormal log file based on a preset condition to obtain a deduplicated abnormal log file” includes: Determine whether the abnormal log file is a non-text abnormal log file; If the abnormal log file is a non-text abnormal log file, the non-text abnormal log file is parsed to obtain a characteristic value, wherein the characteristic value is a mapping of a characteristic problem set in the non-text abnormal log file; The obtained non-text abnormal log file is deduplicated based on the characteristic value.

3. The log processing method according to claim 2, characterized in that: "Parse non-text abnormal log files to obtain characteristic values", including: Determine whether the file name of the non-text abnormal log file can be directly parsed into log information; If the file name of the non-text type abnormal log file can be directly parsed into log information, the file name of the non-text type abnormal log file is parsed to obtain the log information of the non-text type abnormal log file, wherein the log information includes: module information; Based on the module information in the log information, obtaining a parser corresponding to the non-text type abnormal log file; Based on the parser, the corresponding non-text abnormal log file is parsed to obtain the abnormal log file. Characteristic value.

4. The log processing method according to claim 3, characterized in that: The method further comprises: If the file name of the non-text type exception log file cannot be directly parsed into log information, obtaining a responsibility chain for processing the non-text type exception log file based on the log type of the non-text type exception log file, wherein the responsibility chain includes at least two parsers set corresponding to the log type of the non-text type exception log file; Input the non-text type exception log file into the responsibility chain to obtain the matching degree given by each parser; The corresponding non-text type abnormal log file is parsed using the parser with the highest matching degree among the at least two parsers to obtain a characteristic value.

5. The log processing method according to any one of claims 1 to 3, characterized in that: "Send the deduplicated abnormal log files to the specified backend for processing based on the data set table and preset rules", including: Obtain a module information table, wherein the module information table contains a correspondence between module information and a backend; Based on the data set table and the module information table, the deduplicated exception log file is sent to a designated backend for processing.

6. The log processing method according to claim 3, characterized in that: "Based on the module information in the log information, obtaining a parser corresponding to the non-text type abnormal log file", including: Get the mapping table between modules and parsers; Based on the mapping table between the modules and the parsers and the module information in the log information, a parser corresponding to the non-text type exception log file is obtained.

7. The log processing method according to any one of claims 3 to 6, characterized in that: The log information also includes event information and a timestamp.

8. The log processing method according to any one of claims 2 to 7, characterized in that: The method further includes: if the exception log file is a text-type exception log file, filtering a specified text-type exception log file set based on a custom condition to perform a deduplication operation.

9. A control device, comprising a processor and a storage device, wherein the storage device is suitable for storing a plurality of computer programs, characterized in that: The computer program is suitable for being loaded and run by the processor to execute the log processing method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a plurality of computer programs, characterized in that: The computer program is suitable for being loaded and run by a processor to execute the log processing method according to any one of claims 1 to 8.

Images