Log analysis method and device, equipment and medium
By generating a target fingerprint and searching for matching rules in the rule knowledge base, and if no match is found, the log data is parsed using a parsing model. This solves the problems of low efficiency and poor accuracy in existing log parsing, and achieves efficient and accurate log parsing.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING YOUTEJIE INFORMATION TECH
- Filing Date
- 2025-12-30
- Publication Date
- 2026-04-21
AI Technical Summary
Existing log parsing methods suffer from low parsing efficiency and poor parsing accuracy, especially in regular expression parsing and large language model parsing schemes. These methods are characterized by high manual maintenance costs, cumbersome operations, large query latency, high resource consumption, long model inference time, and a tendency to generate erroneous results.
The target fingerprint is generated by acquiring target log data. A matching rule is found using a pre-configured rule knowledge base. If no matching rule is found, a pre-configured parsing model is used to parse the log data, generating log parsing results and parsing rules.
It improves the efficiency and accuracy of log parsing, reduces manual maintenance costs, lowers query latency and resource consumption, and ensures the accuracy of parsing results.
Smart Images
Figure CN121901752A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of log processing, and more particularly to a log parsing method, apparatus, device, and medium. Background Technology
[0002] In the field of log parsing, with the rapid development of information technology, the log data generated by various systems is exploding. The information contained in logs, such as system operating status, abnormal alarms, and user behavior, is of great value for system operation and maintenance, fault diagnosis, and business optimization. However, existing log parsing technologies still have many bottlenecks, making it difficult to meet the needs for efficient and accurate parsing.
[0003] Current mainstream log parsing solutions generally suffer from poor parsing efficiency and low accuracy. Firstly, solutions using regular expression parsing during the collection phase require manually writing custom regular expressions for each log type, resulting in extremely high maintenance costs. Different systems and business processes have significantly different log formats; when new log types appear or log fields are updated, technical personnel must rewrite and debug the regular expressions, a tedious and error-prone process. This solution is also very unfriendly to beginners, making it difficult to learn and further limiting parsing efficiency. Secondly, solutions that parse during the query phase suffer from significant latency. Since parsing is triggered during the query, pre-aggregation of log data is impossible, requiring repeated parsing for each query. This not only slows query response but also significantly increases server resource consumption, reducing overall system efficiency. Thirdly, pure Large Language Model (LLM) parsing solutions have costs that increase linearly with the amount of log data, making large-scale application difficult. Furthermore, LLM is susceptible to illusion, easily generating parsing results that do not match the actual log content, leading to decreased accuracy. The model inference process is also time-consuming, resulting in high latency as well.
[0004] In summary, existing log parsing methods suffer from low parsing efficiency and poor accuracy of parsing results. Summary of the Invention
[0005] This invention provides a log parsing method, apparatus, device, and medium that can solve the problems of low log parsing efficiency and poor accuracy of parsing results in existing log parsing methods.
[0006] In a first aspect, embodiments of the present invention provide a log parsing method, the method comprising:
[0007] Acquire target log data and generate a target fingerprint that matches the target log data;
[0008] Based on the target fingerprint, search in a pre-configured rule knowledge base whether there is a target rule that matches the target fingerprint;
[0009] After confirming that there is no target rule in the rule knowledge base that matches the target fingerprint, the target log data is parsed based on the pre-configured parsing model to obtain the log parsing result that matches the target log data and the target parsing rule.
[0010] Secondly, embodiments of the present invention provide a log parsing apparatus, the apparatus comprising:
[0011] A fingerprint generation module is used to acquire target log data and generate a target fingerprint that matches the target log data.
[0012] The rule matching module is used to search in a pre-configured rule knowledge base whether there is a target rule that matches the target fingerprint;
[0013] The result generation module is used to parse the target log data based on a pre-configured parsing model after confirming that there is no target rule in the rule knowledge base that matches the target fingerprint, and to obtain the log parsing result that matches the target log data and the target parsing rule.
[0014] Thirdly, embodiments of the present invention provide an electronic device, the electronic device comprising:
[0015] At least one processor; and
[0016] A memory communicatively connected to the at least one processor; wherein,
[0017] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform a log parsing method according to any embodiment of the present invention.
[0018] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing computer instructions, which are used to cause a processor to execute a log parsing method according to any embodiment of the present invention.
[0019] The technical solution of this invention acquires target log data and generates a target fingerprint that matches the target log data. Then, based on the target fingerprint, it searches in a pre-configured rule knowledge base to see if a target rule matching the target fingerprint exists. After confirming that no target rule matching the target fingerprint exists in the rule knowledge base, it parses the target log data based on a pre-configured parsing model to obtain a log parsing result matching the target log data and a target parsing rule. This solves the problems of low log parsing efficiency and poor accuracy of parsing results in existing log parsing methods, realizes log parsing, improves log parsing efficiency, and ensures the accuracy of log parsing results.
[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description
[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0022] Figure 1 This is a flowchart of a log parsing method provided according to Embodiment 1 of the present invention;
[0023] Figure 2 This is a flowchart of a log parsing method provided according to Embodiment 2 of the present invention;
[0024] Figure 3 This is a schematic diagram of the structure of a log parsing device according to Embodiment 3 of the present invention;
[0025] Figure 4 This is a schematic diagram of the structure of an electronic device that implements a log parsing method according to an embodiment of the present invention. Detailed Implementation
[0026] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, any variations of the terms "comprising" and "having" are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0028] Example 1
[0029] Figure 1 This is a flowchart of a log parsing method provided in Embodiment 1 of the present invention. This embodiment can be applied to the situation of parsing logs to obtain parsing results. The method can be executed by a log parsing device, which can be implemented in hardware and / or software. The log parsing device can be configured in a terminal or server with log parsing function.
[0030] like Figure 1 As shown, the method includes:
[0031] S110. Obtain target log data and generate a target fingerprint that matches the target log data.
[0032] S120. Based on the target fingerprint, search in a pre-configured rule knowledge base whether there is a target rule that matches the target fingerprint.
[0033] The pre-configured rule knowledge base stores at least one log parsing rule. The log parsing rule is a standardized parsing scheme for a specific log format, stored in JSON structure, and includes field data describing the overall log structure, the key information of each key-value pair, the type of each key-value pair, and key log information.
[0034] In this embodiment, the pre-configured rule knowledge base can be a schema knowledge base, which is a framework for describing the data structure within the knowledge base and is used to define the organization method of unstructured data such as documents and question-answer pairs.
[0035] Optionally, after searching in a pre-configured rule knowledge base for a target rule that matches the target fingerprint, the method further includes: if a target rule matching the target fingerprint is found, extracting the target rule; and using the target rule to parse the target log data to obtain a log parsing result that matches the target log data.
[0036] Specifically, if a target rule matches the target fingerprint, the target rule is fully extracted. The extracted content includes the log level definition, the explicit key information of each key-value pair, the key-value pair type, and the data categories of key log information. After extraction, the target rule is directly used to perform structured parsing of the target log data: the severity of the target log is quickly identified based on the log level definition; the corresponding key-value pair content in the log is accurately located according to the key information of each key-value pair, and the data type of the value is automatically verified and confirmed based on the key-value pair type specification to ensure data format consistency; and the core data categories that are not covered by key-value pairs but are of significant importance in the log are extracted by referring to the data categories of key log information.
[0037] In a specific implementation scenario of this embodiment, if the extracted target rule content includes: log level defined as preset categories such as info, error, and warn; key information of key-value pairs including user_id, request_url, response_time, file_size, and attempt_count; key-value pair type of user_id, request_url, response_time, file_size, and attempt_count; data type corresponding to log key information as auth_action and login_result; and target log data as "2025-08-10 14:30:22 ERROR User login failed: invalid credentials for useralice@xxx.com (attempt#5), request_url: / api / login, response_time: 0.8s". During the parsing process, based on the "log level definition" in the target rule, the "ERROR" field in the log is matched to determine the log severity as ERROR. According to the key information "user_id", "request_url", "response_time", "file_size", and "attempt_count", the corresponding content in the log is located respectively. Combined with type specification validation, user_id is extracted as the string "alice@xxx.com", request_url is extracted as the string " / api / login", response_time is extracted as the floating-point number 0.8, file_size is extracted as the value "20KB" with unit, and attempt_count is extracted as the integer 5. Referring to the key information types "auth_action" and "login_result", the core data "login" and "failed" are extracted from the log "Userloginfailed". In summary, the final log parsing result is: {"severity":"ERROR","attributes":{"user_id":"alice@xxx.com","request_url":" / api / login","response_time":0.8,"file_size":"20KB","attempt_count":5,"auth_action":"login","login_result":"failed"}}.
[0038] S130. After confirming that there is no target rule matching the target fingerprint in the rule knowledge base, the target log data is parsed based on the pre-configured parsing model to obtain the log parsing result matching the target log data and the target parsing rule.
[0039] The process of parsing the target log data based on a pre-configured parsing model to obtain log parsing results matching the target log data includes: performing a keyword scanning operation on the target log data based on a preset built-in dictionary to obtain log levels matching the target log data, which serve as the level results of the target log data; the built-in dictionary includes at least one preset keyword and log levels matching the preset keyword; parsing the target log data to obtain at least one target key-value pair, and performing semantic enhancement operations on each key-value pair to obtain standard fields; performing message body pruning operations on the target log data based on each target key-value pair to obtain unprocessed data; inputting the unprocessed data into a pre-trained large language inference model to obtain log key information matching the unprocessed data; and aggregating the log levels, standard fields, and log key information to obtain log parsing results matching the target log data.
[0040] Furthermore, the large language reasoning model is used to search in the target log data according to the data type set by the user to find whether there is log data matching the data type. If the search finds it, the log data and the data type are aggregated and output as key log information.
[0041] For example, the specific process of parsing the target log data based on a pre-configured parsing model to obtain the log parsing results is as follows: First, perform a log level keyword scanning operation. The built-in dictionary pre-stores preset keywords such as info, error, warn, and debug, and each keyword is mapped one-to-one with its corresponding log level, such as "ERROR" corresponding to a critical error level and "INFO" corresponding to a normal message level; Second, use dynamic regular expressions to perform a comprehensive scan of the target log data, extracting all target key-value pairs that conform to the format "key:value" or "key=value"; Third, based on the extracted target... The first step involves labeling key-value pairs and then pruning the message body of the target log data. This automatically removes key-value pairs extracted in the second step, retaining only the parts without extracted key information. The second step involves batch inputting the pruned, unprocessed data into a pre-trained large language inference model. The model searches the unprocessed data for user-defined data retrieval requirements to see if any user-defined data types exist. If they do, they are extracted as key log information; otherwise, an empty set is output. The third step involves aggregating and integrating the previously obtained log levels, standard fields, and key log information extracted by the large model to form a complete log parsing result.
[0042] It should be noted that in this embodiment, the log level mapping and data retrieval requirements of the big oracle inference model pre-stored in the built-in dictionary can be set and adjusted by the user according to the actual implementation scenario, and this embodiment does not impose any restrictions here.
[0043] The process involves parsing the target log data based on a pre-configured parsing model to obtain target parsing rules that match the target log data. This includes: parsing the target log data to obtain the log level, each target key-value pair, and key log information; marking each target key-value pair with its type based on the data type of the value in each target key-value pair to obtain the key-value pair type; and generating target parsing rules that match the target log data based on the log level, the key information of each target key-value pair, the key-value pair type, and the key log information.
[0044] Specifically, the process of parsing the target log data based on a pre-configured parsing model to obtain target parsing rules is as follows: First, summarize the core information obtained during the above parsing process, including log level, each target key-value pair, and key log information; second, intelligently infer and mark the data type of the values in each target key-value pair, for example, determine that "bob@xxx.com" is a string type, " / api / login" is a string type, and "1.2s" is a numeric type with units, thereby obtaining the key-value pair type specification of each target key-value pair; finally, based on the above summarized log level definition, key information of each target key-value pair, corresponding key-value pair type specification (string, numeric with units, etc.), and data type of key log information, generate structured target parsing rules.
[0045] Optionally, after obtaining the log parsing result matching the target log data and the target parsing rule, the method further includes: obtaining the target fingerprint of the target log data, storing the target fingerprint and the target parsing rule in association, and updating the rule knowledge base based on the association storage result.
[0046] Furthermore, updating the rule knowledge base based on the correlation storage result includes: parsing the target log data based on the log level, key information of each target key-value pair, and log key information in the target parsing rule to obtain a verification parsing result; verifying the matching consistency between the verification parsing result and the log parsing result using a pre-trained large language inference model based on the type of each key-value pair in the target log parsing rule and the log key information; if the verification result meets a preset matching threshold, writing the correlation storage result between the target fingerprint and the target parsing rule into the rule knowledge base to complete the update of the rule knowledge base; if the verification result does not meet the preset matching threshold, discarding the target parsing rule.
[0047] The technical solution of this invention involves acquiring target log data and generating a target fingerprint that matches the target log data. Then, based on the target fingerprint, it searches a pre-configured rule knowledge base to see if a target rule matching the target fingerprint exists. After confirming that no target rule matching the target fingerprint exists in the rule knowledge base, it parses the target log data based on a pre-configured parsing model to obtain a log parsing result matching the target log data and the target parsing rule. This achieves log parsing, improves log parsing efficiency, and ensures the accuracy of the log parsing results.
[0048] Example 2
[0049] Figure 2This is a flowchart of a log parsing method provided in Embodiment 2 of the present invention. This embodiment is a refinement based on the above embodiment. Specifically, this embodiment refines the method for generating a target fingerprint that matches the target log data.
[0050] like Figure 2 As shown, the method includes:
[0051] S210. Obtain target log data. Starting from the first byte of the target log data, obtain a truncated log of a preset byte length.
[0052] In this embodiment, the preset byte length can be specifically 50 bits; for example, the original target log data generated by an application system is "2025-08-11 09:20:35 [192.168.1.100:8080] INFO User successfully submitted order No. 123456". After truncating 50 bytes starting from the first byte, the truncated log is "2025-08-11 09:20:35 [192.168.1.100:8080] INFO User successfully submitted".
[0053] S220. Parse the captured log and locate the timestamp data and resource attribute data in the captured log.
[0054] The resource attribute data refers to information in the log that represents fixed configurations of devices, services, networks, etc., such as fields that change dynamically but have a fixed format, like IP address, port number, device ID, and service name.
[0055] S230. Replace the timestamp data and resource attribute data with pre-set placeholders to obtain fingerprint data.
[0056] S240. The fingerprint data is processed using a hash algorithm to obtain a target fingerprint that matches the target log data.
[0057] In this embodiment, the hash algorithm can be specifically the SHA256 hash algorithm, which can be used to convert fingerprint data of any length into a unique hash value.
[0058] S250. Based on the target fingerprint, search in a pre-configured rule knowledge base whether there is a target rule that matches the target fingerprint.
[0059] S260. After confirming that there is no target rule matching the target fingerprint in the rule knowledge base, the target log data is parsed based on the pre-configured parsing model to obtain the log parsing result matching the target log data and the target parsing rule.
[0060] The technical solution of this invention involves acquiring target log data, starting with the first byte of the target log data, obtaining a truncated log of a preset byte length, parsing the truncated log, locating the timestamp data and resource attribute data in the truncated log, and replacing the timestamp data and resource attribute data with pre-set placeholders to obtain fingerprint data. Then, a hash algorithm is used to process the fingerprint data to obtain a target fingerprint that matches the target log data. Next, based on the target fingerprint, a pre-configured rule knowledge base is searched to see if a target rule matching the target fingerprint exists. If no target rule matching the target fingerprint exists in the rule knowledge base, the target log data is parsed based on a pre-configured parsing model to obtain a log parsing result matching the target log data and the target parsing rule. This achieves log parsing, improves log parsing efficiency, and ensures the accuracy of the log parsing results.
[0061] Example 3
[0062] Figure 3 This is a schematic diagram of a log parsing device provided in Embodiment 3 of the present invention. Figure 3 As shown, the device includes:
[0063] The fingerprint generation module 310 is used to acquire target log data and generate a target fingerprint that matches the target log data;
[0064] The rule matching module 320 is used to search in a pre-configured rule knowledge base whether there is a target rule that matches the target fingerprint based on the target fingerprint;
[0065] The result generation module 330 is used to parse the target log data based on a pre-configured parsing model after confirming that there is no target rule in the rule knowledge base that matches the target fingerprint, and to obtain the log parsing result and the target parsing rule that match the target log data.
[0066] The technical solution of this invention involves acquiring target log data and generating a target fingerprint that matches the target log data. Then, based on the target fingerprint, it searches a pre-configured rule knowledge base to see if a target rule matching the target fingerprint exists. After confirming that no target rule matching the target fingerprint exists in the rule knowledge base, it parses the target log data based on a pre-configured parsing model to obtain a log parsing result matching the target log data and the target parsing rule. This achieves log parsing, improves log parsing efficiency, and ensures the accuracy of the log parsing results.
[0067] Based on the above embodiments, the fingerprint generation module 310 includes:
[0068] The log truncation unit is used to extract a truncated log of a preset byte length, starting from the first byte of the target log data.
[0069] The intercepted log parsing unit is used to parse the intercepted log and locate the timestamp data and resource attribute data in the intercepted log;
[0070] The placeholder replacement unit is used to replace the timestamp data and resource attribute data with pre-set placeholders to obtain fingerprint data;
[0071] The hash processing unit is used to process the fingerprint data using a hash algorithm to obtain a target fingerprint that matches the target log data.
[0072] Based on the above embodiments, the result generation module 330 includes:
[0073] The keyword scanning unit is used to perform keyword scanning operations on the target log data based on a preset built-in dictionary to obtain the log level that matches the target log data, which is used as the level result of the target log data. The built-in dictionary includes at least one preset keyword and the log level that matches the preset keyword.
[0074] The key-value pair acquisition unit is used to parse the target log data to obtain at least one target key-value pair of the target log data, and to perform semantic enhancement operations on each key-value pair to obtain each standard field;
[0075] The message body trimming unit is used to perform message body trimming operations on the target log data based on each target key-value pair to obtain unprocessed data.
[0076] The model inference unit is used to input the unprocessed data into a pre-trained large language inference model to obtain log key information that matches the unprocessed data.
[0077] The result aggregation unit is used to aggregate the log level, each standard field, and key log information to obtain log parsing results that match the target log data.
[0078] Based on the above embodiments, the result generation module 330 includes:
[0079] The data parsing unit is used to parse the target log data to obtain the log level, each target key-value pair, and key log information;
[0080] The type marking unit is used to mark the type of each target key value according to the data type of the value in each target key value pair, so as to obtain the key value pair type of each target key value pair;
[0081] The rule generation unit is used to generate target parsing rules that match the target log data based on the log level, the key information of each target key-value pair, the type of each key-value pair, and the log key information.
[0082] Based on the above embodiments, the rule matching module 320 is further configured to: search in a pre-configured rule knowledge base for a target rule that matches the target fingerprint; if a target rule that matches the target fingerprint is found, extract the target rule; and use the target rule to parse the target log data to obtain a log parsing result that matches the target log data.
[0083] Based on the above embodiments, the result generation module 330 is further configured to: after obtaining the log parsing result matching the target log data and the target parsing rule, obtain the target fingerprint of the target log data, store the target fingerprint and the target parsing rule in association, and update the rule knowledge base based on the association storage result.
[0084] Based on the above embodiments, the result generation module 330 is further configured to: parse the target log data based on the log level, key information of each target key-value pair, and log key information in the target parsing rule to obtain a verification parsing result; verify the matching consistency between the verification parsing result and the log parsing result using a pre-trained large language inference model, based on the type of each key-value pair in the target log parsing rule and the log key information; if the verification result meets a preset matching threshold, write the association storage result between the target fingerprint and the target parsing rule into the rule knowledge base to complete the update of the rule knowledge base; if the verification result does not meet the preset matching threshold, discard the target parsing rule.
[0085] The log parsing apparatus provided in this embodiment of the invention can execute the log parsing method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.
[0086] Example 4
[0087] Figure 4 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.
[0088] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded into the RAM 13 from storage unit 18. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.
[0089] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0090] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as a log parsing method.
[0091] Accordingly, the method includes:
[0092] Acquire target log data and generate a target fingerprint that matches the target log data;
[0093] Based on the target fingerprint, search in a pre-configured rule knowledge base whether there is a target rule that matches the target fingerprint;
[0094] After confirming that there is no target rule in the rule knowledge base that matches the target fingerprint, the target log data is parsed based on the pre-configured parsing model to obtain the log parsing result that matches the target log data and the target parsing rule.
[0095] In some embodiments, a log parsing method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the log parsing method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to execute a log parsing method by any other suitable means (e.g., by means of firmware).
[0096] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0097] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0098] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0099] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0100] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.
[0101] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.
[0102] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and no limitation is imposed herein.
Claims
1. A log parsing method, characterized in that, include: Acquire target log data and generate a target fingerprint that matches the target log data; Based on the target fingerprint, search in a pre-configured rule knowledge base whether there is a target rule that matches the target fingerprint; After confirming that there is no target rule in the rule knowledge base that matches the target fingerprint, the target log data is parsed based on the pre-configured parsing model to obtain the log parsing result that matches the target log data and the target parsing rule.
2. The method according to claim 1, characterized in that, Generating a target fingerprint that matches the target log data includes: Starting from the first byte of the target log data, obtain a truncated log of a preset byte length; The captured log is parsed to locate the timestamp data and resource attribute data in the captured log; The fingerprint data is obtained by replacing the timestamp data and resource attribute data with pre-set placeholders. The fingerprint data is processed using a hash algorithm to obtain a target fingerprint that matches the target log data.
3. The method according to claim 1, characterized in that, The target log data is parsed based on a pre-configured parsing model to obtain log parsing results that match the target log data, including: The target log data is scanned for keywords based on a preset built-in dictionary to obtain the log level that matches the target log data. This log level is then used as the level result for the target log data. The built-in dictionary includes at least one preset keyword and the log level that matches the preset keyword. The target log data is parsed to obtain at least one target key-value pair, and semantic enhancement operations are performed on each key-value pair to obtain each standard field; Based on each target key-value pair, the target log data is pruned to obtain unprocessed data. The unprocessed data is input into a pre-trained large language inference model to obtain log key information that matches the unprocessed data; The log level, standard fields, and key log information are aggregated to obtain log parsing results that match the target log data.
4. The method according to any one of claims 1-3, characterized in that, The target log data is parsed based on a pre-configured parsing model to obtain target parsing rules that match the target log data, including: The target log data is parsed to obtain the log level, each target key-value pair, and key log information; The key-value pair type is obtained by marking the data type of each key-value pair according to the data type of the value in each key-value pair. Based on the log level, the key information of each target key-value pair, the type of each key-value pair, and the key information of the log, a target parsing rule is generated to match the target log data.
5. The method according to claim 1, characterized in that, After searching a pre-configured rule knowledge base for a target rule that matches the target fingerprint, the process further includes: If a target rule is found to match the target fingerprint, the target rule is extracted. The target log data is parsed using the target rules to obtain log parsing results that match the target log data.
6. The method according to claim 1, characterized in that, After obtaining the log parsing result matching the target log data and the target parsing rules, the process also includes: Obtain the target fingerprint of the target log data, store the target fingerprint and the target parsing rules in association, and update the rule knowledge base based on the association storage result.
7. The method according to claim 6, characterized in that, Updating the rule knowledge base based on the aforementioned relational storage results includes: Based on the log level, key information of each target key-value pair, and log key information in the target parsing rules, the target log data is parsed to obtain the verification parsing results; Using a pre-trained large language inference model, the consistency between the verification parsing result and the log parsing result is verified based on the key-value pair types in the target log parsing rules and the key information of the log. If the verification result meets the preset matching threshold, the association storage result between the target fingerprint and the target parsing rule is written into the rule knowledge base to complete the update of the rule knowledge base; If the verification result does not meet the preset matching threshold, the target parsing rule is discarded.
8. A log parsing apparatus, characterized in that, include: A fingerprint generation module is used to acquire target log data and generate a target fingerprint that matches the target log data. The rule matching module is used to search in a pre-configured rule knowledge base whether there is a target rule that matches the target fingerprint; The result generation module is used to parse the target log data based on a pre-configured parsing model after confirming that there is no target rule in the rule knowledge base that matches the target fingerprint, and to obtain the log parsing result that matches the target log data and the target parsing rule.
9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform a log parsing method according to any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute a log parsing method according to any one of claims 1-7.