Log analysis method and device based on artificial intelligence, computer equipment and medium

Abstract

The invention belongs to the field of artificial intelligence and the field of financial science and technology, and relates to a log analysis method and device based on artificial intelligence, computer equipment and a storage medium. Judging whether a first log template completely matched with the log exists in the analysis cache or not; if not, screening a specified number of second log templates meeting similar conditions with the log from the analysis cache based on a first similar processing algorithm; generating a cue word based on the log and the second log template; outputting a third log template corresponding to the cue word through the target large language model; calculating template similarities between the third log template and all log templates in the analysis cache, and judging whether the template similarities are smaller than a similarity threshold value or not; and if yes, generating a first analysis result corresponding to the log based on the third log template. In addition, the first analysis result can be stored in the block chain. According to the log analysis method and device, the log analysis efficiency is improved, and the log analysis flexibility and accuracy are improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence development technology and the field of financial technology, and in particular to artificial intelligence-based log parsing methods, devices, computer equipment and storage media. Background Art

[0002] With the rapid development of information technology today, especially in the financial, Internet and big data industries, system logs are becoming increasingly important as an important carrier for recording key information generated during software operation. Log data not only contains detailed records of program execution, but also contains a wealth of key information such as system status, user behavior and potential errors. It is of immeasurable value to enterprises in error diagnosis, troubleshooting, performance monitoring, automated report and warning generation, data insight and decision support.

[0003] At present, log parsing technology has become a hot topic of research and application as a core means to extract valuable information from massive, unstructured log data. Existing log parsing methods usually adopt grammar-based log parsing methods. Such methods mainly rely on preset rule sets or heuristic algorithms to identify specific patterns or keywords in log messages, thereby realizing the structuring of logs. Although this method shows high efficiency in scenarios with clear rules and fixed log formats, its significant limitation is that it is difficult to adapt to the dynamic changes in log formats and complex and changeable actual application scenarios. With system upgrades, the introduction of new functions, or adjustments to log output logic, the original rules may need to be updated frequently, which not only increases maintenance costs, but also reduces the flexibility and accuracy of log parsing. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a log parsing method, apparatus, computer device and storage medium based on artificial intelligence to solve the technical problems of the existing grammar-based log parsing method having high maintenance cost and low flexibility and accuracy of log parsing.

[0005] In order to solve the above technical problems, the embodiment of the present application provides a log parsing method based on artificial intelligence, which adopts the following technical solutions:

[0006] Receive logs to be parsed;

[0007] Determine whether there is a first log template that completely matches the log in a preset parsing cache;

[0008] If there is no first log template that completely matches the log, a specified number of second log templates that meet similar conditions with the log are screened out from the parsing cache based on a preset first similarity processing algorithm;

[0009] Generate a corresponding prompt word based on the log and the second log template;

[0010] Inputting the prompt word into a pre-built target large language model based on a preset parser, and outputting a third log template corresponding to the prompt word through the target large language model;

[0011] Calculating template similarities between the third log template and all log templates included in the parsing cache, and determining whether the template similarities are all less than a preset similarity threshold;

[0012] If the template similarities are all smaller than the similarity threshold, a first parsing result corresponding to the log is generated based on the third log template.

[0013] Furthermore, the step of determining whether there is a first log template that completely matches the log in the preset parsing cache specifically includes:

[0014] Obtaining the dictionary tree corresponding to the log template stored in the parsing cache;

[0015] Get the preset query method;

[0016] The dictionary tree is queried based on the query method and the log to obtain a query result corresponding to the log; wherein the query result includes a first log template that exists in the parsing cache and completely matches the log, or includes a first log template that does not exist in the parsing cache and completely matches the log.

[0017] Furthermore, after the step of determining whether the template similarities are all less than a preset similarity threshold, the method further includes:

[0018] If the template similarities are not all less than the preset similarity threshold, a fourth log template having a similarity greater than the similarity threshold is selected from all template similarities;

[0019] Call the preset merge rules;

[0020] Merge the fourth log template with the third log template based on the merging rule to obtain a corresponding target log template;

[0021] A second parsing result corresponding to the log is generated based on the target log template.

[0022] Further, the step of merging the fourth log template with the third log template based on the merging rule to obtain a corresponding target log template specifically includes:

[0023] Performing difference analysis on the fourth log template and the third log template to determine the same characters and different characters of the fourth log template relative to the third log template;

[0024] Get the preset target symbol;

[0025] Replacing the difference character in the fourth log template with the target symbol to obtain a corresponding fifth log template;

[0026] The fifth log template is used as the target log template.

[0027] Further, after the step of merging the fourth log template with the third log template based on the merging rule to obtain a corresponding target log template, the method further includes:

[0028] Obtaining location information of the fourth log template in the parsing cache;

[0029] Based on the position information, the fourth log template is replaced using the target log template.

[0030] Furthermore, before the step of inputting the prompt word into a pre-built target large language model based on a preset parser and outputting a third log template corresponding to the prompt word through the target large language model, the method further includes:

[0031] Get pre-collected log data;

[0032] Collecting a first log sample from the log data based on a preset sample sampling algorithm;

[0033] Performing labeling processing on the first log sample to obtain a corresponding second log sample;

[0034] Constructing corresponding prompt word sample data based on the second log sample;

[0035] Fine-tune the preset large language model based on the prompt word sample data to obtain a processed large language model;

[0036] The processed large language model is used as the target large language model.

[0037] Furthermore, the step of constructing corresponding prompt word sample data based on the second log sample specifically includes:

[0038] Calculate the similarity between each designated log sample contained in the second log sample and other log samples based on a preset second similarity processing algorithm; wherein the designated log sample is any one log sample among all the second log samples;

[0039] Based on the similarity, a preset number of example log samples with the highest similarity corresponding to the specified log sample are screened out;

[0040] Get the preset prompt word format;

[0041] The designated log sample and the example log sample are processed for data generation based on the prompt word format to obtain the prompt word sample data.

[0042] In order to solve the above technical problems, the embodiment of the present application also provides a log parsing device based on artificial intelligence, which adopts the following technical solution:

[0043] A receiving module, used to receive logs to be parsed;

[0044] A first determination module is used to determine whether there is a first log template that completely matches the log in a preset parsing cache;

[0045] A first screening module, configured to screen out a specified number of second log templates that meet similar conditions with the log from the parsing cache based on a preset first similarity processing algorithm if there is no first log template that completely matches the log;

[0046] A first generating module, configured to generate corresponding prompt words based on the log and the second log template;

[0047] an output module, configured to input the prompt word into a pre-built target large language model based on a preset parser, and output a third log template corresponding to the prompt word through the target large language model;

[0048] A second judgment module is used to calculate the template similarity between the third log template and all log templates included in the parsing cache, and judge whether the template similarities are all less than a preset similarity threshold;

[0049] The second generating module is configured to generate a first parsing result corresponding to the log based on the third log template if the template similarities are all less than the similarity threshold.

[0050] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0051] Receive logs to be parsed;

[0052] Determine whether there is a first log template that completely matches the log in a preset parsing cache;

[0053] If there is no first log template that completely matches the log, a specified number of second log templates that meet similar conditions with the log are screened out from the parsing cache based on a preset first similarity processing algorithm;

[0054] Generate a corresponding prompt word based on the log and the second log template;

[0055] Inputting the prompt word into a pre-built target large language model based on a preset parser, and outputting a third log template corresponding to the prompt word through the target large language model;

[0056] Calculating template similarities between the third log template and all log templates included in the parsing cache, and determining whether the template similarities are all less than a preset similarity threshold;

[0057] If the template similarities are all smaller than the similarity threshold, a first parsing result corresponding to the log is generated based on the third log template.

[0058] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0059] Receive logs to be parsed;

[0060] Determine whether there is a first log template that completely matches the log in a preset parsing cache;

[0061] If there is no first log template that completely matches the log, a specified number of second log templates that meet similar conditions with the log are screened out from the parsing cache based on a preset first similarity processing algorithm;

[0062] Generate a corresponding prompt word based on the log and the second log template;

[0063] Inputting the prompt word into a pre-built target large language model based on a preset parser, and outputting a third log template corresponding to the prompt word through the target large language model;

[0064] Calculating template similarities between the third log template and all log templates included in the parsing cache, and determining whether the template similarities are all less than a preset similarity threshold;

[0065] If the template similarities are all smaller than the similarity threshold, a first parsing result corresponding to the log is generated based on the third log template.

[0066] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0067] The present application first receives the log to be parsed, and determines whether there is a first log template that completely matches the log in the preset parsing cache; if there is no first log template that completely matches the log, then based on the preset first similarity processing algorithm, a specified number of second log templates that meet similar conditions with the log are screened out from the parsing cache; then a corresponding prompt word is generated based on the log and the second log template; then the prompt word is input into a pre-built target large language model based on a preset parser, and a third log template corresponding to the prompt word is output through the target large language model; subsequently, the template similarity between the third log template and all log templates contained in the parsing cache is calculated, and it is determined whether the template similarity is less than a preset similarity threshold; finally, if the template similarity is less than the similarity threshold, a first parsing result corresponding to the log is generated based on the third log template. The present application performs parsing processing on the log to be parsed by combining the parsing cache, similarity processing algorithm, parser and target large language model, which effectively reduces the maintenance cost of log parsing, improves the efficiency of log parsing, and improves the flexibility and accuracy of log parsing. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0069] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0070] Figure 2 A flow chart of an embodiment of a log parsing method based on artificial intelligence according to the present application;

[0071] Figure 3 It is a structural schematic diagram of an embodiment of a log parsing device based on artificial intelligence according to the present application;

[0072] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0073] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0074] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0075] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0076] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0077] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0078] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0079] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0080] It should be noted that the artificial intelligence-based log parsing method provided in the embodiments of the present application is generally executed by a server / terminal device, and accordingly, the artificial intelligence-based log parsing device is generally arranged in a server / terminal device.

[0081] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0082] Continue to refer Figure 2 , showing a flow chart of an embodiment of an artificial intelligence-based log parsing method according to the present application. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted. The artificial intelligence-based log parsing method provided in the embodiment of the present application can be applied to any scenario that requires log parsing, and the artificial intelligence-based log parsing method can be applied to products in these scenarios, for example, log parsing processing in the financial insurance field. The artificial intelligence-based log parsing method comprises the following steps:

[0083] Step S201, receiving the log to be parsed.

[0084] In this embodiment, the electronic device (e.g. Figure 1 The server / terminal device shown in the figure) can obtain the log to be parsed through a wired connection or a wireless connection. It should be noted that the above-mentioned wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future. The log to be parsed can be received in advance from various business systems, applications or devices, and the log to be parsed is a log that needs to be parsed. Exemplarily, a log sample may include: Process s2d3 write to / app / cknes.conf failed. In the field of finance and insurance, the above-mentioned business systems may include transaction systems, payment systems, claims systems, etc. The above-mentioned applications may include transaction applications, payment applications, claims applications, etc. The above-mentioned devices may include transaction devices, payment devices, claims devices, etc.

[0085] Step S202: determine whether there is a first log template that completely matches the log in the preset parsing cache.

[0086] In this embodiment, a parsing cache is pre-set, and the parsing cache is also called an adaptive parsing cache. The parsing cache is used to store pre-built log templates. By using the parsing cache, the efficiency problems that may be encountered in real-time log parsing of a large language model can be solved. The log template format is such as Process*write to*failed., where * represents a character of any length. Among them, the specific implementation process of the above-mentioned judgment whether there is a first log template that completely matches the log in the preset parsing cache will be further described in detail in the subsequent specific embodiments of this application, and will not be elaborated on here.

[0087] Step S203: If there is no first log template that completely matches the log, a specified number of second log templates that meet similar conditions with the log are screened out from the parsing cache based on a preset first similarity processing algorithm.

[0088] In this embodiment, the above-mentioned first similarity processing algorithm can specifically adopt the KNN algorithm or the maximum length character matching algorithm. By using the first similarity processing algorithm, the similarity of the log and all log templates contained in the parsing cache is calculated respectively to obtain the corresponding first similarity, and the above-mentioned first similarity is sorted in order from large to small to obtain a sorted list, and then the second similarities that are the same as the above-mentioned specified number are successively screened out from the sorted list, and then the log target corresponding to the second similarity is extracted from the parsing cache as the above-mentioned second log template. Among them, there is no specific limitation on the selection of the above-mentioned specified number of values, which can be set according to actual business needs. In addition, if there is a first log template that completely matches the log in the parsing cache, the first log template is directly used as the parsing result corresponding to the above-mentioned log and returned.

[0089] Step S204: Generate corresponding prompt words based on the log and the second log template.

[0090] In this embodiment, the log and the second log template are merged, and the obtained merged data is used as the prompt word of the target large language model.

[0091] Step S205: input the prompt word into a pre-built target large language model based on a preset parser, and output a third log template corresponding to the prompt word through the target large language model.

[0092] In this embodiment, the above-mentioned parser can specifically adopt the LLM Log Template Generator (LLTG) enhanced parser. Among them, by predicting the input prompt words through the target large language model, a new log template, that is, the above-mentioned third log template, can be generated. In addition, for the construction process of the above-mentioned target large language model, this application will provide further detailed descriptions in subsequent specific embodiments, which will not be elaborated here. Based on the use of the parser and the target large language model, this embodiment can achieve accurate parsing of log data while maintaining a high parsing efficiency.

[0093] Step S206: Calculate the template similarity between the third log template and all log templates included in the parsing cache, and determine whether the template similarities are all less than a preset similarity threshold.

[0094] In this embodiment, the template similarity may specifically refer to token similarity. There is no specific limitation on the value of the similarity threshold, which can be determined according to the actual similarity setting requirements. Specifically, for the third log template and each log template included in the parsing cache, the token similarity between the third log template and all log templates included in the parsing cache can be calculated by using Jaccard similarity, edit distance, longest common subsequence, custom similarity function and other calculation methods.

[0095] Step S207: If the template similarities are all less than the similarity threshold, a first parsing result corresponding to the log is generated based on the third log template.

[0096] In this embodiment, if the template similarities are all less than the similarity threshold, it means that no log template with a similarity greater than the similarity threshold is found in the above-mentioned parsing cache. Since the generated third log template can accurately describe the structure and information of the log message, thereby realizing the parsing of the log, the third log template is directly used as the first parsing result corresponding to the log. The third log template is inserted into the parsing cache to complete the template update processing for the parsing cache. Among them, the use of the third log template can improve the efficiency and accuracy of log parsing and provide convenience for subsequent processing and analysis.

[0097] The present application first receives the log to be parsed, and determines whether there is a first log template that completely matches the log in the preset parsing cache; if there is no first log template that completely matches the log, then based on the preset first similarity processing algorithm, a specified number of second log templates that meet similar conditions with the log are screened out from the parsing cache; then a corresponding prompt word is generated based on the log and the second log template; then the prompt word is input into a pre-built target large language model based on a preset parser, and a third log template corresponding to the prompt word is output through the target large language model; subsequently, the template similarity between the third log template and all log templates contained in the parsing cache is calculated, and it is determined whether the template similarity is less than a preset similarity threshold; finally, if the template similarity is less than the similarity threshold, a first parsing result corresponding to the log is generated based on the third log template. The present application performs parsing processing on the log to be parsed by combining the parsing cache, similarity processing algorithm, parser and target large language model, which effectively reduces the maintenance cost of log parsing, improves the efficiency of log parsing, and improves the flexibility and accuracy of log parsing.

[0098] In some optional implementations, step S202 includes the following steps:

[0099] Obtain the dictionary tree corresponding to the log template stored in the parsing cache.

[0100] In this embodiment, in the above-mentioned parsing cache, the data structure corresponding to the Trie tree (also called a prefix tree or a dictionary tree) is used in advance to store the log template. The Trie tree is particularly suitable for processing string search, insertion and deletion operations, especially when these operations are mainly concentrated on the prefix of the string. The process of building a dictionary tree in the parsing cache includes: Step 1: Define the Trie tree structure. First, the basic node structure of the Trie tree needs to be defined. Each node contains a pointer to its child node (usually an array or a hash table), and a mark bit to indicate whether the node is the end of a complete template. Step 2: Build the Trie tree. A Trie tree class is used to manage the insertion and query operations of the node. Step 3: Insert the template. After the Trie tree is built, the known log template is inserted into the Trie tree.

[0101] Gets the preset query method.

[0102] In this embodiment, the query method may specifically adopt a search method.

[0103] The dictionary tree is queried based on the query method and the log to obtain a query result corresponding to the log; wherein the query result includes a first log template that exists in the parsing cache and completely matches the log, or includes a first log template that does not exist in the parsing cache and completely matches the log.

[0104] In this embodiment, the above dictionary tree is queried by using the search method to detect whether there is a log template that completely matches the input log to be parsed, and the corresponding query result is obtained. Among them, the above query processing also supports wildcards or fuzzy matching, which can be achieved by modifying the search method. For example, special characters (such as *) can be used as wildcards to ignore these characters or match them to strings of any length during the search.

[0105] The present application obtains the dictionary tree corresponding to the log template stored in the parsing cache; then obtains the preset query method; and subsequently performs query processing on the dictionary tree based on the query method and the log to obtain the query result corresponding to the log. The present application performs query processing on the dictionary tree corresponding to the log template stored in the parsing cache based on the use of the query method and the log to be parsed, so as to quickly and accurately generate the query result corresponding to the log, thereby ensuring the accuracy and intelligence of the query result obtained.

[0106] In some optional implementations of this embodiment, after step S207, the electronic device may further perform the following steps:

[0107] If the template similarities are not all smaller than the preset similarity threshold, a fourth log template with a similarity greater than the similarity threshold is selected from all template similarities.

[0108] In this embodiment, by obtaining the comparison result between the template similarity and the similarity threshold, a template with a similarity greater than the similarity threshold can be screened out from all template similarities and used as the fourth log template.

[0109] Call the preset merge rule.

[0110] In this embodiment, the above merging rule is to retain the same characters of two similar log templates and replace the different characters with the target symbol. There is no specific limitation on the selection of the above target symbol.

[0111] The fourth log template and the third log template are merged based on the merging rule to obtain a corresponding target log template.

[0112] In this embodiment, the specific implementation process of merging the fourth log template and the third log template based on the merging rule to obtain the corresponding target log template will be further described in detail in subsequent specific embodiments of the present application and will not be elaborated here.

[0113] A second parsing result corresponding to the log is generated based on the target log template.

[0114] In this embodiment, the final target log template can be returned as the parsing result. Since the target log template can accurately describe the structure and information of the log message, the log parsing is realized. At the same time, the use of the target log template also improves the efficiency and accuracy of log parsing, and facilitates subsequent processing and analysis.

[0115] If the present application detects that the template similarities are not all less than the preset similarity threshold, the fourth log template greater than the similarity threshold is selected from all template similarities; then the preset merge rule is called; then the fourth log template and the third log template are merged based on the merge rule to obtain the corresponding target log template; and then the second parsing result corresponding to the log is generated based on the target log template. If the present application detects that the template similarities are not all less than the preset similarity threshold, the fourth log template greater than the similarity threshold is intelligently selected from all template similarities, and then the preset merge rule is called to merge the fourth log template and the third log template, thereby realizing the rapid and intelligent generation of the target log template corresponding to the log as the corresponding second parsing result, ensuring the processing efficiency and accuracy of generating the second parsing result. It is beneficial to realize the rapid and accurate parsing of the log according to the second parsing result, thereby improving the processing efficiency and accuracy of log parsing.

[0116] In some optional implementations, the merging the fourth log template with the third log template based on the merging rule to obtain a corresponding target log template includes the following steps:

[0117] A difference analysis is performed on the fourth log template and the third log template to determine the same characters and different characters of the fourth log template relative to the third log template.

[0118] In this embodiment, the fourth log template and the third log template are segmented according to punctuation marks and characters to obtain a first character list corresponding to the fourth log template and a second character list corresponding to the third log template, and then the same parts in the first character list and the second character list are found, that is, the same characters are obtained. And the difference parts in the first character list and the second character list are found, that is, the difference characters are obtained.

[0119] Gets the preset target symbol.

[0120] In this embodiment, there is no specific limitation on the selection of the target symbol, for example, * may be used.

[0121] The difference character in the fourth log template is replaced with the target symbol to obtain a corresponding fifth log template.

[0122] In this embodiment, after the difference characters in the fourth log template are determined, the difference characters are replaced by using the target symbol to obtain the corresponding fifth log template.

[0123] The fifth log template is used as the target log template.

[0124] The present application determines the same characters and different characters of the fourth log template relative to the third log template by performing a difference analysis on the fourth log template and the third log template; then obtains a preset target symbol; then replaces the difference characters in the fourth log template with the target symbol to obtain the corresponding fifth log template; and subsequently uses the fifth log template as the target log template. The present application uses a merge rule to intelligently replace the difference characters in the fourth log template with the preset target symbol after determining the same characters and different characters of the fourth log template relative to the third log template, so as to quickly and accurately generate the corresponding target log template, thereby improving the generation efficiency of the target log template and ensuring the accuracy of the obtained target log template.

[0125] In some optional implementations, after the step of merging the fourth log template with the third log template based on the merging rule to obtain a corresponding target log template, the electronic device may further perform the following steps:

[0126] Obtain location information of the fourth log template in the parsing cache.

[0127] In this embodiment, the above-mentioned location information refers to the storage location of the fourth log template in the parsing cache.

[0128] Based on the position information, the fourth log template is replaced using the target log template.

[0129] In this embodiment, the storage location of the fourth log template in the parsing cache can be determined by the location information, and then the fourth log template is replaced using the target log template to use the target log template as a new cache entry in the above-mentioned parsing cache.

[0130] The present application obtains the location information of the fourth log template in the parsing cache; subsequently, based on the location information, the fourth log template is replaced using the target log template. After merging the fourth log template with the third log template based on the merging rule to obtain the corresponding target log template, the present application will also intelligently replace the fourth log template in the parsing cache with the target log template, so as to intelligently and quickly complete the automatic update of the fourth log template in the parsing cache, thereby improving the efficiency and intelligence of the log template update of the parsing cache.

[0131] In some optional implementations of this embodiment, before step S205, the electronic device may further perform the following steps:

[0132] Get pre-collected log data.

[0133] In this embodiment, log data is collected in advance from various business systems, applications or devices. These data should contain diverse log entries to cover different scenarios and error types. In the field of finance and insurance, the above-mentioned business systems may include transaction systems, payment systems, claims systems, etc. The above-mentioned applications may include transaction applications, payment applications, claims applications, etc. The above-mentioned devices may include transaction devices, payment devices, claims devices, etc. In addition, after obtaining the log data, the log data may also be pre-processed to remove duplicate, incomplete or erroneous data.

[0134] A first log sample is collected from the log data based on a preset sample sampling algorithm.

[0135] In this embodiment, there is no specific limitation on the selection of the sample sampling algorithm, for example, random sampling, stratified sampling and other algorithms can be selected. By using the sample sampling algorithm to select the diversified log data that can cover different types from the log data, the first log sample is obtained.

[0136] The first log sample is labeled to obtain a corresponding second log sample.

[0137] In this embodiment, the above-mentioned annotation processing may adopt a manual annotation processing method or a machine automatic annotation processing method. The annotation information for the first log sample may include information such as the type, level, and key information of the log.

[0138] Corresponding prompt word sample data is constructed based on the second log sample.

[0139] In this embodiment, the specific implementation process of constructing the corresponding prompt word sample data based on the second log sample will be further described in detail in subsequent specific embodiments of the present application, and will not be elaborated on here.

[0140] The preset large language model is fine-tuned based on the prompt word sample data to obtain a processed large language model.

[0141] In this embodiment, the above-mentioned large language model can specifically adopt an LLM model, such as the GPT series, the BERT series, etc. The process of fine-tuning the preset large language model based on the prompt word sample data includes: converting the constructed training data into a format acceptable to the LLM model. Data division: Divide the prompt word sample data into a training set, a validation set, and a test set for model training and evaluation. Model fine-tuning: 1) Training settings. Configure the hyperparameters of model training according to the actual model building requirements, including learning rate, batch size, training rounds, etc. 2) Supervised training. Use the labeled data in the training set to fine-tune the LLM model to optimize the performance of the model on specific log data. 3) Verification and adjustment. Evaluate the model performance of the LLM model on the validation set, and adjust the model parameters or training strategy based on the evaluation results. Evaluation and iteration. 4) Test evaluation. Evaluate the performance of the fine-tuned model on the test set, including indicators such as accuracy, recall rate, and F1 score. 5) Iterative optimization. Perform iterative optimization according to the test results until the model performance meets the requirements, thereby obtaining the final processed large language model.

[0142] The processed large language model is used as the target large language model.

[0143] The present application obtains pre-collected log data; then collects a first log sample from the log data based on a preset sample sampling algorithm; then annotates the first log sample to obtain a corresponding second log sample; and constructs corresponding prompt word sample data based on the second log sample; then fine-tunes the preset large language model based on the prompt word sample data to obtain a processed large language model; and finally uses the processed large language model as the target large language model. The present application obtains a first log sample from pre-collected log data based on a preset sample sampling algorithm, then annotates and constructs the first log sample to obtain prompt word sample data, and then fine-tunes the preset large language model based on the prompt word sample data, thereby realizing the rapid and accurate construction of a target large language model with log parsing function, improving the construction efficiency of the target large language model, and ensuring the model effect of the obtained target large language model.

[0144] In some optional implementations of this embodiment, the constructing corresponding prompt word sample data based on the second log sample includes the following steps:

[0145] The similarity between each designated log sample contained in the second log sample and other log samples is calculated based on a preset second similarity processing algorithm; wherein the designated log sample is any one log sample among all the second log samples.

[0146] In this embodiment, the second similarity processing algorithm may specifically adopt a KNN algorithm. The similarity between each designated log sample included in the second log sample and other log samples may be calculated by using the KNN algorithm.

[0147] Based on the similarity, a preset number of example log samples with the highest similarity corresponding to the designated log sample are screened out.

[0148] In this embodiment, the similarity between each specified log sample included in the calculated second log sample and other log samples can be compared to find the preset number of specified similarities with the highest similarity, and then the matching samples can be screened out according to the specified similarity to obtain the preset number of example log samples with the highest similarity corresponding to the specified log sample. There is no specific limitation on the selection of the numerical value of the above preset number, which can be set according to actual business needs. For example, the preset number is the same as the above specified number.

[0149] Get the preset prompt word format.

[0150] In this embodiment, a corresponding prompt format, that is, a prompt word format, may be pre-set according to actual model building requirements. The prompt word format may include a problem description, context information, and expected output.

[0151] The designated log sample and the example log sample are processed for data generation based on the prompt word format to obtain the prompt word sample data.

[0152] In this embodiment, the designated log sample and the example log sample may be combined and processed according to the above prompt word format to obtain corresponding prompt word sample data.

[0153] The present application calculates the similarity between each designated log sample and other log samples contained in the second log sample based on a preset second similarity processing algorithm; wherein the designated log sample is any one of all the second log samples; then based on the similarity, a preset number of sample log samples with the highest similarity corresponding to the designated log sample are screened out; then a preset prompt word format is obtained; and subsequently, data generation processing is performed on the designated log sample and the sample log sample based on the prompt word format to obtain the prompt word sample data. The present application calculates the similarity between each designated log sample and other log samples contained in the second log sample using a second similarity processing algorithm, and screens out a preset number of sample log samples with the highest similarity corresponding to the designated log sample based on the obtained similarity, and then data generation processing is performed on the designated log sample and the sample log sample based on the preset prompt word format, so that the required prompt word sample data can be constructed quickly and accurately, the construction efficiency of the prompt word sample data is improved, the accuracy of the obtained prompt word sample data is ensured, and it is beneficial to subsequently realize accurate fine-tuning processing of the large language model based on the prompt word sample data and obtain the target large language model.

[0154] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0155] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned first analysis result, the above-mentioned first analysis result can also be stored in a node of a blockchain.

[0156] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0157] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0158] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0159] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0160] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0161] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a log parsing device based on artificial intelligence, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0162] like Figure 3 As shown, the log parsing device 300 based on artificial intelligence described in this embodiment includes: a loading module 301, a calling module 302, a collecting module 303, a generating module 304, a processing module 305, a processing module 306 and a processing module 307. Among them:

[0163] Receiving module 301, used for receiving logs to be parsed;

[0164] A first determination module 302 is used to determine whether there is a first log template that completely matches the log in a preset parsing cache;

[0165] A screening module 303 is used to screen out a specified number of second log templates that meet similar conditions with the log from the parsing cache based on a preset first similarity processing algorithm if there is no first log template that completely matches the log;

[0166] A first generating module 304, configured to generate corresponding prompt words based on the log and the second log template;

[0167] An output module 305 is used to input the prompt word into a pre-built target large language model based on a preset parser, and output a third log template corresponding to the prompt word through the target large language model;

[0168] A second judgment module 306 is used to calculate the template similarity between the third log template and all log templates included in the parsing cache, and determine whether the template similarities are all less than a preset similarity threshold;

[0169] The second generating module 307 is configured to generate a first parsing result corresponding to the log based on the third log template if the template similarities are all less than the similarity threshold.

[0170] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the artificial intelligence-based log parsing method in the aforementioned implementation mode, and are not described in detail here.

[0171] In some optional implementations of this embodiment, the first determining module 302 includes:

[0172] A first acquisition submodule is used to acquire a dictionary tree corresponding to the log template stored in the parsing cache;

[0173] The second acquisition submodule is used to acquire a preset query method;

[0174] The first generation submodule is used to query the dictionary tree based on the query method and the log to obtain a query result corresponding to the log; wherein the query result includes a first log template that exists in the parsing cache and completely matches the log, or includes a first log template that does not exist in the parsing cache and completely matches the log.

[0175] In this embodiment, the operations performed by the above modules or units respectively correspond one-to-one to the steps of the artificial intelligence-based log parsing method in the aforementioned implementation mode, and are not repeated here.

[0176] In some optional implementations of this embodiment, the log parsing device based on artificial intelligence also includes:

[0177] A second screening module, configured to screen out a fourth log template having a similarity greater than the similarity threshold from all template similarities if the template similarities are not all less than the preset similarity threshold;

[0178] A calling module is used to call the preset merging rules;

[0179] a merging module, configured to merge the fourth log template with the third log template based on the merging rule to obtain a corresponding target log template;

[0180] The third generating module is used to generate a second parsing result corresponding to the log based on the target log template.

[0181] In this embodiment, the operations performed by the above modules or units respectively correspond one-to-one to the steps of the artificial intelligence-based log parsing method in the aforementioned implementation mode, and will not be repeated here.

[0182] In some optional implementations of this embodiment, the merging module includes:

[0183] an analysis submodule, configured to perform a difference analysis on the fourth log template and the third log template, and determine the same characters and different characters of the fourth log template relative to the third log template;

[0184] The third acquisition submodule is used to acquire the preset target symbol;

[0185] a replacement submodule, replacing the difference character in the fourth log template with the target symbol to obtain a corresponding fifth log template;

[0186] A submodule is determined to use the fifth log template as the target log template.

[0187] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the artificial intelligence-based log parsing method in the aforementioned implementation mode, and are not described in detail here.

[0188] In some optional implementations of this embodiment, the log parsing device based on artificial intelligence further includes:

[0189] A first acquisition module, used to obtain the location information of the fourth log template in the parsing cache;

[0190] A replacement submodule is used to replace the fourth log template with the target log template based on the location information.

[0191] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the artificial intelligence-based log parsing method in the aforementioned implementation mode, and are not described in detail here.

[0192] In some optional implementations of this embodiment, the log parsing device based on artificial intelligence further includes:

[0193] The second acquisition module is used to acquire the pre-collected log data;

[0194] A collection module, used for collecting a first log sample from the log data based on a preset sample sampling algorithm;

[0195] A labeling module, used to label the first log sample to obtain a corresponding second log sample;

[0196] A construction module, used to construct corresponding prompt word sample data based on the second log sample;

[0197] A fine-tuning module, used for fine-tuning a preset large language model based on the prompt word sample data to obtain a processed large language model;

[0198] A determination module is used to use the processed large language model as the target large language model.

[0199] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the artificial intelligence-based log parsing method in the aforementioned implementation mode, and are not described in detail here.

[0200] In some optional implementations of this embodiment, the building blocks include:

[0201] A calculation submodule, configured to calculate the similarity between each designated log sample contained in the second log sample and other log samples based on a preset second similarity processing algorithm; wherein the designated log sample is any one log sample among all the second log samples;

[0202] A screening submodule, configured to screen out a preset number of example log samples having the highest similarity to the designated log samples based on the similarity;

[0203] The fourth acquisition submodule is used to acquire a preset prompt word format;

[0204] The second generation submodule is used to perform data generation processing on the designated log sample and the example log sample based on the prompt word format to obtain the prompt word sample data.

[0205] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the artificial intelligence-based log parsing method in the aforementioned implementation mode, and are not described in detail here.

[0206] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0207] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components 41-43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0208] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0209] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk equipped on the computer device 4, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of a log parsing method based on artificial intelligence, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0210] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer-readable instructions stored in the memory 41 or process data, such as computer-readable instructions for running the artificial intelligence-based log parsing method.

[0211] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0212] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the above-mentioned artificial intelligence-based log parsing method.

[0213] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0214] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A log parsing method based on artificial intelligence, characterized in that: The steps include: Receive logs to be parsed; Determine whether there is a first log template that completely matches the log in a preset parsing cache; If there is no first log template that completely matches the log, a specified number of second log templates that meet similar conditions with the log are screened out from the parsing cache based on a preset first similarity processing algorithm; Generate a corresponding prompt word based on the log and the second log template; Inputting the prompt word into a pre-built target large language model based on a preset parser, and outputting a third log template corresponding to the prompt word through the target large language model; Calculating template similarities between the third log template and all log templates included in the parsing cache, and determining whether the template similarities are all less than a preset similarity threshold; If the template similarities are all smaller than the similarity threshold, a first parsing result corresponding to the log is generated based on the third log template.

2. The log parsing method based on artificial intelligence according to claim 1, characterized in that: The step of determining whether there is a first log template that completely matches the log in the preset parsing cache specifically includes: Obtaining the dictionary tree corresponding to the log template stored in the parsing cache; Get the preset query method; The dictionary tree is queried based on the query method and the log to obtain a query result corresponding to the log; wherein the query result includes a first log template that exists in the parsing cache and completely matches the log, or includes a first log template that does not exist in the parsing cache and completely matches the log.

3. The log parsing method based on artificial intelligence according to claim 1 is characterized in that: After the step of determining whether the template similarities are all less than a preset similarity threshold, the method further includes: If the template similarities are not all less than the preset similarity threshold, a fourth log template having a similarity greater than the similarity threshold is selected from all template similarities; Call the preset merge rules; Merge the fourth log template with the third log template based on the merging rule to obtain a corresponding target log template; A second parsing result corresponding to the log is generated based on the target log template.

4. The log parsing method based on artificial intelligence according to claim 3 is characterized in that: The step of merging the fourth log template with the third log template based on the merging rule to obtain a corresponding target log template specifically includes: Performing difference analysis on the fourth log template and the third log template to determine the same characters and different characters of the fourth log template relative to the third log template; Get the preset target symbol; Replacing the difference character in the fourth log template with the target symbol to obtain a corresponding fifth log template; The fifth log template is used as the target log template.

5. The log parsing method based on artificial intelligence according to claim 3 is characterized in that: After the step of merging the fourth log template with the third log template based on the merging rule to obtain a corresponding target log template, the method further includes: Obtaining location information of the fourth log template in the parsing cache; Based on the position information, the fourth log template is replaced using the target log template.

6. The log parsing method based on artificial intelligence according to claim 1 is characterized in that: Before the step of inputting the prompt word into a pre-built target large language model based on a preset parser and outputting a third log template corresponding to the prompt word through the target large language model, the method further includes: Get pre-collected log data; Collecting a first log sample from the log data based on a preset sample sampling algorithm; Performing labeling processing on the first log sample to obtain a corresponding second log sample; Constructing corresponding prompt word sample data based on the second log sample; Fine-tune the preset large language model based on the prompt word sample data to obtain a processed large language model; The processed large language model is used as the target large language model.

7. The log parsing method based on artificial intelligence according to claim 6 is characterized in that: The step of constructing corresponding prompt word sample data based on the second log sample specifically includes: Calculate the similarity between each designated log sample contained in the second log sample and other log samples based on a preset second similarity processing algorithm; wherein the designated log sample is any one log sample among all the second log samples; Based on the similarity, a preset number of example log samples with the highest similarity corresponding to the specified log sample are screened out; Get the preset prompt word format; The designated log sample and the example log sample are processed for data generation based on the prompt word format to obtain the prompt word sample data.

8. A log parsing device based on artificial intelligence, characterized in that: include: A receiving module, used to receive logs to be parsed; A first determination module is used to determine whether there is a first log template that completely matches the log in a preset parsing cache; A first screening module, configured to screen out a specified number of second log templates that meet similar conditions with the log from the parsing cache based on a preset first similarity processing algorithm if there is no first log template that completely matches the log; A first generating module, configured to generate corresponding prompt words based on the log and the second log template; an output module, configured to input the prompt word into a pre-built target large language model based on a preset parser, and output a third log template corresponding to the prompt word through the target large language model; A second judgment module is used to calculate the template similarity between the third log template and all log templates included in the parsing cache, and judge whether the template similarities are all less than a preset similarity threshold; The second generating module is configured to generate a first parsing result corresponding to the log based on the third log template if the template similarities are all less than the similarity threshold.

9. A computer device comprising a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the artificial intelligence-based log parsing method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the log parsing method based on artificial intelligence as described in any one of claims 1 to 7 are implemented.

Images