Method and device for log analysis based on fusion model

Abstract

The invention discloses a method and device for log analysis based on a fusion model, and the method comprises the steps: inputting an obtained to-be-processed log into a preset routing network, and obtaining a model probability list; the model probability list comprises probabilities of a plurality of preset basic large models; taking a preset basic large model with the maximum probability in the model probability list as a prediction model; inputting the to-be-processed log into the prediction model to obtain an analysis text; splicing the to-be-processed log with the analysis text to obtain an intermediate analysis log; and re-executing the process of inputting the intermediate analysis log into the preset routing network until a preset condition is achieved, and obtaining a log analysis result. According to the method, a plurality of preset basic large models are fused, advantage intervals of different preset basic large models are fully utilized, and an appropriate preset basic large model is dynamically selected for analysis according to the log content, so that the limitation of a traditional method is solved, the automation of log analysis is realized, and the accuracy and comprehensiveness of log analysis are improved.

Description

Technical Field

[0001] The present invention relates to a method and device for log parsing based on a fusion model. Background Art

[0002] Log parsing is a key link in system monitoring and fault diagnosis. With the continuous development of information technology, various applications and systems generate a large number of logs, recording various events, exceptions and error information in system operation. When a system fails or an error occurs, the operation and maintenance engineer needs to identify the type of error fault based on the meaning of the error log. When encountering unfamiliar logs, it is necessary to query the meaning of the log, understand the type of fault described in the log and the situation that caused the log. However, when a fault occurs, a large number of logs are usually generated. The operation and maintenance engineer usually needs to screen the logs related to the fault from a large number of logs to identify the fault type and locate the root cause of the fault.

[0003] In order to process these log data efficiently, existing technologies usually rely on text classification technology or keyword extraction. Text classification technology refers to classifying logs according to predefined categories, such as error level, source or event type, etc., to help organize and analyze logs. Keyword extraction means that in log screening, operation and maintenance engineers extract the most representative or important words or phrases from log texts based on keywords, such as extracting keywords "ERROR", "Connection", "database", "failed", "timeout", etc. from the log "ERROR: Connection to database failed due to timeout", thereby extracting key logs related to the fault. Summary of the invention

[0004] In order to obtain more accurate and universal log analysis results, an embodiment of the present invention provides a method and device for log analysis based on a fusion model.

[0005] In a first aspect, an embodiment of the present invention provides a method for log parsing based on a fusion model, which may include:

[0006] Input the acquired logs to be processed into a preset routing network to obtain a model probability list; the model probability list includes the probabilities of multiple preset basic large models;

[0007] The preset basic large model with the largest probability in the model probability list is used as the prediction model;

[0008] Inputting the log to be processed into the prediction model to obtain a parsed text;

[0009] Splicing the to-be-processed log with the parsed text to obtain an intermediate parsed log;

[0010] The above process of inputting the intermediate parsing log into the preset routing network is re-executed until the preset conditions are met to obtain the log parsing result.

[0011] In one or some optional implementations of the embodiment of the present application, the preset routing network is obtained in the following manner:

[0012] Obtain a routing network training data set and multiple preset basic models;

[0013] Building an initial routing network based on the multiple preset basic large models;

[0014] The initial routing network is trained based on the routing network training data set to obtain a preset routing network.

[0015] In one or some optional implementations of the embodiments of the present application, the routing network training data set is obtained by:

[0016] Acquire multiple log texts, and for each log text, obtain multiple initial parsing results of the log text based on each preset basic large model;

[0017] Using a preset scoring expert model to score the multiple initial parsing results of the log text respectively, and taking the initial parsing result with the highest score as the best parsing result of the log text;

[0018] Using the preset basic large model corresponding to the best parsing result of the log text as the label of the log text;

[0019] Based on all the log texts and the labels of the log texts, the routing network training data set is obtained.

[0020] In one or some optional implementations of the embodiments of the present application, the plurality of preset basic large models are obtained by:

[0021] Obtain multiple open source large models and open source log QA test sets;

[0022] Based on the open source log QA test set, each open source big model is evaluated using the C-eval method to determine a preset number of open source big models with the best evaluation results as the preset basic big models.

[0023] In one or some optional implementations of the embodiments of the present application, the initial routing network includes a gating network and a plurality of preset basic large models; the routing network training data set includes a plurality of log texts and a label corresponding to each log text;

[0024] The step of training the initial routing network based on the routing network training data set to obtain a preset routing network includes:

[0025] Fixing parameters of a plurality of preset basic large models in the initial routing network;

[0026] Input the log text into the encoding module of each preset basic large model respectively to obtain multiple hidden vectors;

[0027] Inputting the multiple hidden vectors into the inference modules corresponding to the preset basic large models respectively, and obtaining the prediction probability of each preset basic large model;

[0028] Inputting the multiple hidden vectors into the gating network to obtain the prediction weight of each preset basic large model;

[0029] Calculate the loss function according to the label corresponding to the log text and the prediction weight and preset probability of each preset basic large model, and update the parameters of the initial routing network according to the loss function;

[0030] The above initial routing network training process is repeated until a preset stop condition is reached to obtain a preset routing network.

[0031] In one or some optional implementations of the embodiments of the present application, the loss function is calculated according to the label corresponding to the log text and the prediction weight and preset probability of each preset basic large model, and the parameters of the initial routing network are updated according to the loss function, including:

[0032] According to the label corresponding to the log text and the prediction weight and preset probability of each preset basic model, the loss function is calculated based on the following formula:

[0033]

[0034] In the formula, loss is the loss function, n is the number of preset basic large models, i represents the i-th preset basic large model, GatingNetwork represents the gated network, and Tokenizer i represents the encoding module of the i-th preset basic model, X represents the log text, CrossEntropyLoss is the cross entropy function, Model i represents the inference module of the i-th preset basic large model, OneHot represents one-hot encoding, and Y represents the parsed text obtained by the preset basic large model corresponding to the label of the log text;

[0035] In a second aspect, an embodiment of the present invention provides a device for performing log parsing based on a fusion model, which may include:

[0036] The first prediction module is used to input the acquired log to be processed into a preset routing network to obtain a model probability list; the model probability list includes the probabilities of multiple preset basic large models;

[0037] A first processing module is used to use a preset basic large model with the largest probability in the model probability list as a prediction model;

[0038] A second prediction module, used for inputting the log to be processed into the prediction model to obtain a parsed text;

[0039] A first splicing module, used for splicing the log to be processed with the parsed text to obtain an intermediate parsed log;

[0040] The first judgment module is used to judge whether a preset condition is met; if so, obtain the log analysis result; if not, re-execute the first prediction module.

[0041] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program / instruction stored thereon, which, when executed by a processor, implements the method for performing log parsing based on a fusion model as described above.

[0042] In a fourth aspect, an embodiment of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the method for log parsing based on a fusion model as described above.

[0043] In a fifth aspect, an embodiment of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein when the processor executes the computer program, the method for performing log parsing based on the fusion model as described above is implemented.

[0044] The beneficial effects of the above technical solution provided by the embodiment of the present invention include at least:

[0045] The embodiment of the present invention provides a method for log parsing based on a fusion model, which generates a model probability list of multiple preset basic large models by inputting a log to be processed into a preset routing network, then selects the preset basic large model with the largest probability as a prediction model, inputs the log into the prediction model for parsing, obtains a parsed text, splices the log to be processed with the parsed text, generates an intermediate parsed log, and finally repeats the prediction process of inputting the intermediate parsed log into the preset routing network until the preset conditions are met to obtain a log parsing result. This method, by fusing multiple preset basic large models, makes full use of the advantage intervals of different preset basic large models, and dynamically selects the most appropriate preset basic large model for parsing according to the log content, thereby solving the limitations of traditional text classification methods and keyword extraction methods, realizing the automation of log parsing, being able to process log data more efficiently, and improving the accuracy and comprehensiveness of log parsing. At the same time, this method can better cope with diverse and complex log data, has high robustness and applicability, thereby assisting operation and maintenance engineers in troubleshooting system failures and anomalies, and improving the stability and reliability of the system.

[0046] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0047] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0049] Figure 1 A flow chart of a method for log parsing based on a fusion model provided in an embodiment of the present invention;

[0050] Figure 2 A schematic diagram of the structure of a preset routing network provided in an embodiment of the present invention;

[0051] Figure 3 A schematic diagram of the structure of a device for performing log parsing based on a fusion model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0053] The inventors have discovered that the premise of traditional text classification methods is that log semantic types need to be enumerated in advance. However, there are many log semantics and new logs are constantly generated. The use of text classification for log semantic recognition is only applicable to closed scenarios. At the same time, when performing log screening, the keyword extraction method requires the operation and maintenance engineers to have good professional knowledge and be able to give keywords related to the fault type as comprehensively as possible. The knowledge requirements for the operation and maintenance engineers are relatively high. If the keywords are not comprehensive, it is easy to cause the recall logs to be incomplete, affecting the efficiency of fault location. Therefore, how to realize intelligent parsing of logs more accurately and comprehensively has become an important issue to improve the efficiency and accuracy of system diagnosis. Based on this, the inventors have made the present invention after further research and development, providing a method and device for log parsing based on a fusion model.

[0054] Embodiment 1

[0055] In the first embodiment of the present invention, a method for log parsing based on a fusion model is provided. Figure 1 As shown, the method may include the following steps S101-S105:

[0056] S101: Input the acquired log to be processed into a preset routing network to obtain a model probability list, wherein the model probability list includes the probabilities of multiple preset basic large models.

[0057] S102: Using the preset basic large model with the largest probability in the model probability list as the prediction model.

[0058] S103: Input the log to be processed into the prediction model to obtain the parsed text.

[0059] S104: Join the log to be processed with the parsed text to obtain an intermediate parsed log.

[0060] S105: Determine whether the preset condition is met: if so, obtain the log analysis result; if not, re-execute the above steps S101-S104 to input the intermediate analysis log into the preset routing network.

[0061] The embodiment of the present invention provides a method for log parsing based on a fusion model, which generates a model probability list of multiple preset basic large models by inputting a log to be processed into a preset routing network, then selects the preset basic large model with the largest probability as a prediction model, inputs the log into the prediction model for parsing, obtains a parsed text, splices the log to be processed with the parsed text, generates an intermediate parsed log, and finally repeats the prediction process of inputting the intermediate parsed log into the preset routing network until the preset conditions are met to obtain a log parsing result. This method, by fusing multiple preset basic large models, makes full use of the advantage intervals of different preset basic large models, and dynamically selects the most appropriate preset basic large model for parsing according to the log content, thereby solving the limitations of traditional text classification methods and keyword extraction methods, realizing the automation of log parsing, being able to process log data more efficiently, and improving the accuracy and comprehensiveness of log parsing. At the same time, this method can better cope with diverse and complex log data, has high robustness and applicability, thereby assisting operation and maintenance engineers in troubleshooting system failures and anomalies, and improving the stability and reliability of the system.

[0062] In the above step S101, the acquired log to be processed is input into a preset routing network to obtain a model probability list.

[0063] Specifically, the log to be processed may be input into a preset routing network to obtain a model probability list, wherein the model probability list includes the probabilities of multiple preset basic large models.

[0064] In a specific embodiment, the log to be processed is "ERROR: Connection failed due to timeout", and the multiple preset basic large models are xverse-13, qwen-14B and chatGLM3-6B. The log to be processed is input into the preset routing network, and the obtained model probability list is [0.8, 0.1, 0.1]. The three values ​​in the model probability list respectively represent the adaptability of the three preset basic large models, that is, xverse-13 is most suitable for generating the next token of the log to be processed.

[0065] In the embodiment of the present application, the preset routing network in the above step S101 is obtained through the following steps S1011-S1013:

[0066] S1011: Obtain a routing network training data set and a plurality of preset basic large models, wherein the routing network training data set is obtained through the following steps S10111-S10114, and the plurality of preset basic large models are obtained through the following steps S10115-S10116.

[0067] The routing network training data set is obtained through the following steps S10111-S10114:

[0068] S10111: Acquire multiple log texts, and for each log text, obtain multiple initial parsing results of the log text based on each preset basic large model.

[0069] Specifically, the error usage logs in resources such as documents and forums of popular middleware are collected and preprocessed to obtain multiple log texts. Then, for each log text, the first preset prompt statement is used to guide each preset basic large model to parse the log text to obtain multiple initial parsing results.

[0070] Those skilled in the art can set the first preset prompt statement specifically according to the specific preset basic model. For example, if the preset basic model is chatglm2, the first preset prompt statement can be:

[0071] """

[0072] You are a professional operation and maintenance engineer who is familiar with the semantic understanding of various log data. Please explain the semantics of the following log: "Log text"

[0073] """

[0074] S10112: Use a preset scoring expert model to score multiple initial parsing results of the log text respectively, and use the initial parsing result with the highest score as the best parsing result of the log text.

[0075] Specifically, based on the multiple initial parsing results of each log text obtained in the above step S10111, the second preset prompt sentence is used to guide the preset scoring expert model to score the multiple initial parsing results of a log text, and the scores corresponding to the initial parsing results of the log text are obtained. The initial parsing result with the highest score is taken as the best parsing result of the log text.

[0076] Those skilled in the art can set the second preset prompt statement specifically according to the specific preset scoring expert model. For example, if the preset scoring expert model is GPT-4, the second preset prompt statement can be:

[0077] """

[0078] You are an experienced system operations expert responsible for interpreting and troubleshooting system logs.

[0079] Please evaluate the following system log interpretation results based on different models to determine which interpretation is more accurate and complete so that you can take the correct action during troubleshooting.

[0080] When evaluating your interpretation, consider the following:

[0081] Accuracy: Do the interpretations accurately reflect the content and intent of the log?

[0082] Completeness: Does the explanation include all the important information in the log, without missing any critical details?

[0083] Clarity: Are the interpretations clear and understandable, without ambiguity or vagueness?

[0084] Consistency: Is the interpretation consistent with your expertise and experience and does it appear unreasonable?

[0085] Please give a specific rating and reason, with a minimum of 1 point and a maximum of 5 points.

[0086] The logs that need to be parsed are as follows:

[0087] "XX log example"

[0088] The interpretation result of model A on the log: "the interpretation result of model A";

[0089] Model B’s interpretation result of the log: “Model B’s interpretation result”;

[0090] The interpretation result of model C on the log: “the interpretation result of model C”;

[0091] Please rate the interpretation results of these different models according to the above evaluation criteria and explain your judgment in detail in the scoring reason. The scoring results will be used to determine the final interpretation results to guide subsequent troubleshooting and repair work.

[0092] """

[0093] S10113: The preset basic large model corresponding to the best parsing result of the log text is used as the label of the log text. For example, for a certain log text, the corresponding initial parsing results are obtained using the three preset basic large models xverse-13, qwen-14B and chatGLM3-6B, and the scores output by the preset scoring expert model GPT-4 are 4, 3 and 2 respectively, then the xverse-13 with the highest score is used as the label of the log text.

[0094] S10114: Integrate all log texts and log text labels to obtain a routing network training data set.

[0095] Multiple preset basic large models are obtained through the following steps S10115-S10116:

[0096] S10115: Obtain multiple open source large models and open source log QA test sets.

[0097] Specifically, we can select open source large models with excellent performance from the language large model rankings, and pay special attention to models with small memory usage (such as less than 30B), including xverse-13, qwen-14B, and chatGLM3-6B. These open source large models perform well in natural language processing tasks, especially in complex semantic understanding and processing. By selecting these high-performance basic large models, a strong foundation is provided for subsequent fusion models.

[0098] At the same time, in order to verify the actual capabilities of these large models in log parsing tasks, open source operation and maintenance QA test sets are obtained for subsequent evaluation.

[0099] S10116: Based on the open source log QA test set, each open source big model is evaluated using the C-eval method to determine a preset number of open source big models with the best evaluation results as the preset basic big models.

[0100] Specifically, the C-eval (Code Evaluation) test method can be used to evaluate the level of understanding of log semantics of each open source model. This evaluation method can objectively evaluate the accuracy, efficiency and reliability of each open source large model when parsing different types of logs, and further understand its performance in actual operation and maintenance scenarios.

[0101] Finally, according to the evaluation results, a preset number of open source big models with the best performance are selected from all open source big models as the preset basic big models. The preset number can be set to 3 for example.

[0102] In the embodiment of the present application, the above steps S10115-S10116 obtain the preset basic large model through screening, ensuring that the fusion model can perform optimally in the log semantic parsing task, and providing strong support for the subsequent model fusion and reasoning process.

[0103] S1012: Constructing an initial routing network based on multiple preset basic large models.

[0104] Specifically, since the dimensions of the tokenizer layer and the embedding layer of different preset basic large models are different, this method designs an initial routing network with a multi-tower structure to be compatible with the tokenizer and embedding outputs of different large models. In a specific embodiment, the model structure of the initial routing network is as follows: Figure 2As shown, it includes a gating network and three preset basic large models. The gating network includes 3 network and fc (Fully Connected) layers. The three preset basic large models are expert llm1, expert llm2 and expert llm1. Expert llm1 (tokenizer & embedding) represents the encoding module of the preset basic large model expert llm1, and expert llm1 (transformer forward) represents the reasoning module of the preset basic large model expert llm1.

[0105] Here is another explanation in terms of data flow Figure 2 The input log text input text will first be input into the encoding module of each preset basic model to obtain the hidden vector. Then, the hidden vector will be input into the reasoning module and gating unit of the corresponding preset basic model respectively. The loss function loss is calculated based on the output summary of the reasoning model and the gating unit.

[0106] S1013: Train the initial routing network based on the routing network training data set to obtain a preset routing network. Specifically, it includes the following steps S10131-S10136:

[0107] S10131: Fixing parameters of multiple preset basic large models in the initial routing network.

[0108] S10132: Input the log text into the encoding module of each preset basic large model respectively to obtain multiple hidden vectors.

[0109] S10133: Input the multiple hidden vectors into the inference modules corresponding to the preset basic large models respectively to obtain the prediction probability of each preset basic large model.

[0110] S10134: Input multiple hidden vectors into the gating network to obtain the prediction weight of each preset basic large model.

[0111] S10135: Calculate the loss function according to the label corresponding to the log text and the prediction weight and preset probability of each preset basic large model, and update the parameters of the initial routing network according to the loss function.

[0112] Specifically, the loss function may be calculated based on the following formula according to the label corresponding to the log text and the prediction weight and preset probability of each preset basic large model:

[0113]

[0114] In the formula, loss is the loss function, n is the number of preset basic large models, i represents the i-th preset basic large model, GatingNetwork represents the gated network, and Tokenizer i represents the encoding module of the i-th preset basic model, X represents the log text, CrossEntropyLoss is the cross entropy function, Model i It represents the inference module of the i-th preset basic large model, OneHot represents one-hot encoding, and Y represents the parsed text obtained by the preset basic large model corresponding to the label of the log text.

[0115] Then, back propagation is performed according to the calculated loss function to update the parameters of the initial routing network.

[0116] S10136: Determine whether the preset stop condition is met: if so, obtain the preset routing network. If not, re-execute the initial routing network training process described in the above steps S10132-S10135.

[0117] Among them, the preset stop condition can be set to reaching a fixed number of iterations, accuracy reaching a threshold, etc., and is not specifically limited here.

[0118] In the embodiment of the present application, the goal of the initial routing network is to intelligently route the log text to a suitable preset basic large model based on the characteristics and contextual information of the log text, thereby improving parsing efficiency and accuracy.

[0119] In the above step S102, the preset basic large model with the largest probability in the model probability list is used as the prediction model.

[0120] Specifically, the preset basic large model with the largest probability value in the model probability list can be selected to determine the preset basic large model that is most suitable for parsing the log to be processed and generating the next token as the prediction model. For example, if the model probability list is [0.8, 0.1, 0.1], and the corresponding three preset basic large models are xverse-13, qwen-14B, and chatGLM3-6B, then xverse-13 with the largest probability is selected as the prediction model.

[0121] In the above steps S103-S104, the log to be processed is input into the prediction model to obtain a parsed text. Then, the log to be processed and the parsed text are concatenated to obtain an intermediate parsed log.

[0122] In a specific embodiment, the log to be processed in the above steps S103-S104 is "Last_Errno:1580Last_Error:Error'You cannot'ALTER'a log table if logging is enabled'onquery.Default database:'mysql'.", the prediction model is xverse-13, the log to be processed is input into the prediction model, and the next token output by the prediction model is obtained, that is, the parsed text, which is "This log means in MySQL".

[0123] Then, the log to be processed is concatenated with the parsed text to obtain the intermediate parsed log "Last_Errno:1580Last_Error:Error'You cannot'ALTER'a log table if logging is enabled'onquery.Default database:'mysql'. This log means in MySQL".

[0124] In the above step S105, it is determined whether the preset condition is met: if so, the log parsing result is obtained; if not, the above steps S101-S104 are re-executed to input the intermediate parsed log into the preset routing network.

[0125] The preset condition may be that the parsed text is equal to a preset end mark, or that the length of the intermediate parsed log reaches a preset maximum length.

[0126] In order to facilitate those skilled in the art to understand the present solution, the specific implementation process of S101-S105 provided in the embodiment of the present application is described more clearly and completely in the form of code below. The algorithm code is as follows:

[0127] def ENSEMBLE_LLM_CHAT(input_query,routing_network,models):

[0128] eos←False

[0129] while not eos and length(input_query) <max_length do

[0130] model_prob←rouing_network(input_query)

[0131] model_index←argmax(model_prob)

[0132] next_token←models[model_index].PREDICT_NEXT_TOKEN(input_query)

[0133] input_query←input_query+DECODE(next_token)

[0134] eos←True if next_token==eos_token else False

[0135] return input_query

[0136] Among them, when the algorithm starts, input_query is the text to be processed, routing_network is the preset routing network, models are multiple preset basic large models, eos represents the end mark value, which is used to determine whether the preset conditions are met, max_length is the preset maximum length, model_prob is the model probability list, model_index represents the number of the prediction model, next_token is the parsed text, the input_query in the fourth row of the while loop is the intermediate parsing log, and eos_token is the preset end mark.

[0137] In order to facilitate technical personnel in this field to understand the present solution, the following example illustrates the results after the implementation of S101-S105 provided in the embodiment of the present application: the text to be processed is "Last_Errno:1580Last_Error:Error'You cannot'ALTER'a log table iflogging is enabled'on query.Default database:'mysql'.". After the processing of steps S101-S105, the log parsing result obtained is "This log means that a problem occurred when an ALTER operation was performed in the MySQL database. Specifically, MySQL attempted to modify a special table used to record system activities or operations, but because the current logging function is enabled, the ALTER operation was rejected." Among them, the symbol "·" is a separator, indicating that the parsed texts before and after are generated by different preset basic large models.

[0138] In an embodiment of the present application, when parsing a text to be processed, the method will route according to the content of the text to be processed by the preset routing network, determine the most appropriate basic preset large model to generate the next token, and the generated result is composed of at least one token. Therefore, during the generation process, the present invention can adjust the preset basic large model for generating the next token in real time according to the content of the updated intermediate parsing log, make full use of the advantage intervals of different preset basic large models, and dynamically select the most appropriate preset basic large model for parsing according to the log content, thereby improving the accuracy and comprehensiveness of log parsing.

[0139] Embodiment 2

[0140] Based on the same inventive concept, the embodiment of the present invention also provides a device for log parsing based on a fusion model, referring to Figure 3 As shown, the device comprises:

[0141] The first prediction module 101 is used to input the acquired log to be processed into a preset routing network to obtain a model probability list; the model probability list includes the probabilities of multiple preset basic large models;

[0142] A first processing module 102 is used to use a preset basic large model with the largest probability in the model probability list as a prediction model;

[0143] The second prediction module 103 is used to input the log to be processed into the prediction model to obtain a parsed text;

[0144] A first splicing module 104 is used to splice the log to be processed with the parsed text to obtain an intermediate parsed log;

[0145] The first judgment module 105 is used to judge whether a preset condition is met; if so, obtain the log analysis result; if not, re-execute the first prediction module.

[0146] Embodiment 3

[0147] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the method for performing log parsing based on a fusion model as described in the first embodiment above is implemented.

[0148] Embodiment 4

[0149] Based on the same inventive concept, an embodiment of the present invention further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the method for performing log parsing based on a fusion model as described in the first embodiment above.

[0150] Embodiment 5

[0151] Based on the same inventive concept, an embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored in the memory. When the processor executes the computer program, the method for performing log parsing based on a fusion model as described in the first embodiment above is implemented.

[0152] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0153] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0154] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0155] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0156] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for log parsing based on a fusion model, characterized in that: include: Input the acquired logs to be processed into the preset routing network to obtain a model probability list; The model probability list includes the probabilities of multiple preset basic large models; The preset basic large model with the largest probability in the model probability list is used as the prediction model; Inputting the log to be processed into the prediction model to obtain a parsed text; Splicing the to-be-processed log with the parsed text to obtain an intermediate parsed log; The above process of inputting the intermediate parsing log into the preset routing network is re-executed until the preset conditions are met to obtain the log parsing result.

2. The method according to claim 1, characterized in that The preset routing network is obtained in the following manner: Obtain a routing network training data set and multiple preset basic models; Building an initial routing network based on the multiple preset basic large models; The initial routing network is trained based on the routing network training data set to obtain a preset routing network.

3. The method according to claim 2, characterized in that The routing network training data set is obtained in the following manner: Acquire multiple log texts, and for each log text, obtain multiple initial parsing results of the log text based on each preset basic large model; Using a preset scoring expert model to score the multiple initial parsing results of the log text respectively, and taking the initial parsing result with the highest score as the best parsing result of the log text; Using the preset basic large model corresponding to the best parsing result of the log text as the label of the log text; Based on all the log texts and the labels of the log texts, the routing network training data set is obtained.

4. The method according to claim 2, characterized in that: The multiple preset basic large models are obtained in the following manner: Obtain multiple open source large models and open source log QA test sets; Based on the open source log QA test set, each open source big model is evaluated using the C-eval method to determine a preset number of open source big models with the best evaluation results as the preset basic big models.

5. The method according to claim 2, characterized in that: The initial routing network includes a gating network and a plurality of preset basic large models; the routing network training data set includes a plurality of log texts and a label corresponding to each log text; The step of training the initial routing network based on the routing network training data set to obtain a preset routing network includes: Fixing parameters of a plurality of preset basic large models in the initial routing network; Input the log text into the encoding module of each preset basic large model respectively to obtain multiple hidden vectors; Inputting the multiple hidden vectors into the inference modules corresponding to the preset basic large models respectively, and obtaining the prediction probability of each preset basic large model; Inputting the multiple hidden vectors into the gating network to obtain the prediction weight of each preset basic large model; Calculate the loss function according to the label corresponding to the log text and the prediction weight and preset probability of each preset basic large model, and update the parameters of the initial routing network according to the loss function; The above initial routing network training process is repeated until a preset stop condition is reached to obtain a preset routing network.

6. The method according to claim 5, characterized in that The step of calculating the loss function according to the label corresponding to the log text and the prediction weight and the preset probability of each preset basic large model, and updating the parameters of the initial routing network according to the loss function includes: According to the label corresponding to the log text and the prediction weight and preset probability of each preset basic large model, the loss function is calculated based on the following formula: In the formula, loss is the loss function, n is the number of preset basic large models, i represents the i-th preset basic large model, GatingNetwork represents the gated network, and Tokenizer i represents the encoding module of the i-th preset basic model, X represents the log text, CrossEntropyLoss is the cross entropy function, Model i represents the inference module of the i-th preset basic large model, OneHot represents one-hot encoding, and Y represents the parsed text obtained by the preset basic large model corresponding to the label of the log text; The parameters of the initial routing network are updated according to the loss function.

7. A device for log parsing based on a fusion model, characterized in that: include: The first prediction module is used to input the acquired logs to be processed into a preset routing network to obtain a model probability list; The model probability list includes the probabilities of multiple preset basic large models; A first processing module is used to use a preset basic large model with the largest probability in the model probability list as a prediction model; A second prediction module, used for inputting the log to be processed into the prediction model to obtain a parsed text; A first splicing module, used for splicing the log to be processed with the parsed text to obtain an intermediate parsed log; The first judgment module is used to judge whether a preset condition is met; if so, obtain the log analysis result; if not, re-execute the first prediction module.

8. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the method for log parsing based on a fusion model described in any one of claims 1 to 6 is implemented.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the method for log parsing based on a fusion model described in any one of claims 1 to 6 is implemented.

10. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the method for log parsing based on a fusion model as described in any one of claims 1-6.

Images