Log analysis method and device, computer equipment, storage medium and program product

Abstract

The invention relates to a log analysis method and device, computer equipment, a storage medium and a program product. The method comprises the steps of obtaining a log analysis target corresponding to a to-be-analyzed log instance; the format of the log instance to be analyzed is a machine language structure; screening out a target analysis model corresponding to the log analysis target from the plurality of candidate analysis models; and inputting the to-be-analyzed log instance into the target analysis model for analysis to obtain an analysis result of the to-be-analyzed log instance. According to the log analysis target of the to-be-analyzed instance, the target analysis model is matched for the to-be-analyzed log, the accuracy of analyzing the to-be-analyzed log is improved, the target analysis model is a model capable of processing a machine language, and the accuracy of analyzing the to-be-analyzed log is further improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a log analysis method, device, computer equipment, storage medium and program product. Background Art

[0002] In computer devices, log files record various events and operations, such as user activities, system errors, security incidents, etc. By analyzing logs, patterns, trends, and abnormal behaviors can be identified, thereby helping organizations monitor system health, optimize performance, ensure security, and perform troubleshooting.

[0003] In traditional technology, after the logs are collected, they are parsed using preset log rules to obtain key indicators of the logs, and the log analysis results are determined based on the key indicators and preset indicator thresholds.

[0004] However, traditional log analysis methods have the problem of low analysis accuracy. Summary of the invention

[0005] Based on this, it is necessary to provide a log analysis method, apparatus, computer equipment, storage medium and program product that can improve the analysis accuracy in response to the above technical problems.

[0006] In a first aspect, the present application provides a log analysis method, comprising:

[0007] Obtaining a log analysis target corresponding to a log instance to be analyzed; the format of the log instance to be analyzed is a structure of a machine language;

[0008] Filtering out a target analysis model corresponding to the log analysis target from a plurality of candidate analysis models;

[0009] The log instance to be analyzed is input into the target analysis model for analysis to obtain an analysis result of the log instance to be analyzed.

[0010] In one embodiment, the training process of the target analysis model includes:

[0011] Obtain a log template sample and a first log instance sample;

[0012] Performing data enhancement processing on the log template sample to obtain an enhanced log template sample, and annotating the enhanced log template sample to obtain an annotated log template sample;

[0013] A first initial model is trained according to the enhanced log template sample, the annotated log template sample and the first log instance sample to obtain the target analysis model.

[0014] In one embodiment, the training of the first initial model according to the enhanced log template sample, the annotated log template sample and the first log instance sample to obtain the target analysis model includes:

[0015] Inputting the annotated log template sample into a first reference model for analysis to obtain a first output value;

[0016] Inputting the enhanced log template sample into the first initial model for analysis to obtain a second output value;

[0017] Determine a first loss value according to the first reference model and the first initial model, and train the first initial model according to the first loss value to obtain a trained first initial model;

[0018] Inputting the first log instance sample into the trained first initial model for analysis to obtain a new first log instance sample, and labeling the new first log instance sample to obtain a labeled first log instance sample;

[0019] The second initial model is trained according to the new first log instance sample and the labeled first log instance sample to obtain a target analysis model.

[0020] In one embodiment, the training of the second initial model according to the new first log instance sample and the labeled first log instance sample to obtain the target analysis model includes:

[0021] Inputting the labeled first log instance sample into the second reference model for analysis to obtain a third output value;

[0022] Inputting the new first log instance sample into the second initial model for analysis to obtain a fourth output value;

[0023] Determining a second loss value according to the third output value and the fourth output value;

[0024] The second initial model is trained according to the second loss value to obtain the target analysis model.

[0025] In one embodiment, the training of the second initial model according to the second loss value to obtain the target analysis model includes:

[0026] Training the second initial model according to the second loss value to obtain an intermediate analysis model;

[0027] Determine a preset log analysis target, and obtain a second log instance sample corresponding to the preset log analysis target;

[0028] The intermediate analysis model is customized and trained according to the second log instance sample to obtain the target analysis model.

[0029] In one embodiment, the step of inputting the first log instance sample into the trained first initial model for analysis to obtain a new first log instance sample includes:

[0030] Inputting the first log instance sample into the trained first initial model for analysis to obtain a feature vector corresponding to the first log instance sample;

[0031] Performing clustering processing on the feature vectors corresponding to the log instance samples to obtain multiple cluster feature vectors;

[0032] Determine the similarity between each feature vector in each cluster feature vector, and perform deduplication processing on feature vectors whose similarities all meet the preset similarity condition to obtain processed multi-cluster feature vectors;

[0033] The new first log instance sample is generated according to the processed multi-cluster feature vectors.

[0034] In a second aspect, the present application also provides a log analysis device, including:

[0035] A first acquisition module is used to acquire a log analysis target corresponding to a log instance to be analyzed; the format of the log instance to be analyzed is a structure of a machine language;

[0036] A screening module, used to screen out a target analysis model corresponding to the log analysis target from a plurality of candidate analysis models;

[0037] The analysis module is used to input the log instance to be analyzed into the target analysis model for analysis to obtain the analysis result of the log instance to be analyzed.

[0038] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0039] Obtaining a log analysis target corresponding to a log instance to be analyzed; the format of the log instance to be analyzed is a structure of a machine language;

[0040] Filtering out a target analysis model corresponding to the log analysis target from a plurality of candidate analysis models;

[0041] The log instance to be analyzed is input into the target analysis model for analysis to obtain an analysis result of the log instance to be analyzed.

[0042] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0043] Obtaining a log analysis target corresponding to a log instance to be analyzed; the format of the log instance to be analyzed is a structure of a machine language;

[0044] Filtering out a target analysis model corresponding to the log analysis target from a plurality of candidate analysis models;

[0045] The log instance to be analyzed is input into the target analysis model for analysis to obtain an analysis result of the log instance to be analyzed.

[0046] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0047] Obtaining a log analysis target corresponding to a log instance to be analyzed; the format of the log instance to be analyzed is a structure of a machine language;

[0048] Filtering out a target analysis model corresponding to the log analysis target from a plurality of candidate analysis models;

[0049] The log instance to be analyzed is input into the target analysis model for analysis to obtain an analysis result of the log instance to be analyzed.

[0050] The log analysis method, apparatus, computer equipment, storage medium and program product described above obtain a log analysis target corresponding to a log instance to be analyzed; the format of the log instance to be analyzed is a structure of machine language; a target analysis model corresponding to the log analysis target is selected from multiple candidate analysis models; the log instance to be analyzed is input into the target analysis model for analysis, and an analysis result of the log instance to be analyzed is obtained. According to the log analysis target of the instance to be analyzed, the target analysis model is matched to the log to be analyzed, thereby improving the accuracy of analyzing the log to be analyzed, and the target analysis model is a model that can process machine language, further improving the accuracy of analyzing the log to be analyzed. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0052] Figure 1An application environment diagram of a log analysis method in an embodiment;

[0053] Figure 2 A flow chart of a log analysis method in one embodiment;

[0054] Figure 3 A flow chart of a log analysis method according to another embodiment;

[0055] Figure 4 A flow chart of a log analysis method according to another embodiment;

[0056] Figure 5 A flow chart of a log analysis method according to another embodiment;

[0057] Figure 6 A flow chart of a log analysis method according to another embodiment;

[0058] Figure 7 A flow chart of a log analysis method according to another embodiment;

[0059] Figure 8 A flow chart of a log analysis method according to another embodiment;

[0060] Fig. 9 FIG. 4 is a structural block diagram of a log analysis device in an embodiment. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0062] The log analysis method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown in FIG. 1 , the computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 1As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store log analysis data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a log analysis method is implemented.

[0063] Those skilled in the art will understand that Figure 1 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0064] In one embodiment, Figure 2 As shown, a log analysis method is provided, which is applied to Figure 1 The following is an example of a server in the example, including:

[0065] S201, obtaining a log analysis target corresponding to a log instance to be analyzed; the format of the log instance to be analyzed is a structure of a machine language.

[0066] The log to be analyzed is a machine language in coded form. The log analysis objectives may include anomaly detection, log classification, etc.

[0067] In an embodiment of the present application, the server may determine the log analysis target based on the user's input data, or the server may determine the log analysis target to be analyzed based on the storage location or size of the log to be analyzed.

[0068] S202, selecting a target analysis model corresponding to the log analysis target from a plurality of candidate analysis models.

[0069] In an embodiment of the present application, multiple candidate analysis models are pre-trained, and a correspondence between each candidate analysis model and a log analysis target is pre-established, so that a target analysis model corresponding to the log analysis target is screened out from the multiple candidate analysis models based on the log analysis target and the above correspondence.

[0070] S203: input the log instance to be analyzed into the target analysis model for analysis to obtain the analysis result of the log instance to be analyzed.

[0071] Optionally, if the log analysis target is anomaly detection, the log instance to be analyzed is input into the target analysis model for analysis to determine whether the log instance to be analyzed is abnormal, or to determine the abnormality level of the analyzed log instance.

[0072] Optionally, if the log analysis target is log classification, the log instance to be analyzed is input into the target analysis model for analysis to obtain the type of the log instance to be analyzed, for example, operation log record, alarm log record, etc.

[0073] In the above log analysis method, the log analysis target corresponding to the log instance to be analyzed is obtained; the format of the log instance to be analyzed is a structure of machine language; the target analysis model corresponding to the log analysis target is screened out from multiple candidate analysis models; the log instance to be analyzed is input into the target analysis model for analysis, and the analysis result of the log instance to be analyzed is obtained. According to the log analysis target of the instance to be analyzed, the target analysis model is matched for the log to be analyzed, which improves the accuracy of the analysis of the log to be analyzed, and the target analysis model is a model that can process machine language, which further improves the accuracy of the analysis of the log to be analyzed.

[0074] In one embodiment, Figure 3 As shown, the training process of the above target analysis model includes:

[0075] S204, obtaining a log template sample and a log instance sample.

[0076] In the embodiment of the present application, the log target sample is a log template that does not include specific log data, and the historical data of the log is read to obtain the log template sample and the log instance sample from the historical data.

[0077] S205 , performing data enhancement processing on the log template sample to obtain an enhanced log template sample, and annotating the enhanced log template sample to obtain an annotated log template sample.

[0078] In an embodiment of the present application, the target analysis model can be a large language model. The prompt strategy of the large language model is used to perform data enhancement on the log template samples to generate new log templates, thereby enriching the diversity of the training set. In the data enhancement process, an open source pre-trained model that has been trained with a natural language corpus is used, and the parameters of the target analysis model are frozen to ensure the stability of the model during the enhancement process.

[0079] Optionally, the scheme for performing data augmentation on log template samples using a large language model may include:

[0080] 1) Synonym replacement: some words in the log are replaced with their synonyms, such as replacing "insufficient" with "lacking" or "low".

[0081] 2) Structural changes: adjust the structure of the log, such as reversing the order of causes and effects.

[0082] 3) Parameterization enhancement: introduce parameterized placeholders, such as using [error_code] to replace specific numbers.

[0083] 4) Noise injection: adding random noise to the log, such as random characters or blanks.

[0084] 5) Context change, changing the context of the log, such as using similar error messages in different system components or services.

[0085] 6) Mixed logs: Mix multiple log entries together to generate new log entries.

[0086] In an embodiment of the present application, the enhanced log template sample may be annotated according to a preset script to obtain an annotated log template sample; or, the enhanced log template sample may be annotated according to input information to obtain an annotated log template sample.

[0087] S206: Train the first initial model according to the enhanced log template samples, the annotated log template samples and the log instance samples to obtain a target analysis model.

[0088] In an embodiment of the present application, enhanced log template samples, annotated log template samples and log instance samples can be used as training samples of the first initial model, the first initial model can be trained, and the parameters in the first initial model can be iteratively updated to obtain the target analysis model.

[0089] In the above application embodiment, data enhancement is performed on the log template sample, which improves the accuracy of the enhanced template sample and the annotated log sample, thereby improving the accuracy of the analysis to be determined.

[0090] In the embodiment of the present application, an implementation method of S206 is provided, such as Figure 4 As shown, the above “training the first initial model according to the enhanced log template sample, the labeled log template sample and the log instance sample to obtain the target analysis model” includes:

[0091] S301: Input the labeled log template sample into a first reference model for analysis to obtain a first output value.

[0092] The annotated log template sample is a log template annotated as a natural language based on the enhanced log template sample.

[0093] In an embodiment of the present application, the first reference model is the same as the first initial model, that is, the initial parameters of the first reference model and the first initial model are the same. The first reference model is a recognition model for natural language. Therefore, the log template marked as natural language is used as the sample data of the first initial template to obtain the first output value.

[0094] S302: Input the enhanced log template sample into the first initial model for analysis to obtain a second output value.

[0095] In the embodiment of the present application, the data in the enhanced log template sample corresponds to the data in the annotated log template sample one by one, and the first initial model and the first reference model are both large language models. Optionally, for a given enhanced log template and annotated log template sample pair {( , )|i=1,2,...,N}, and input them into the encoder of the corresponding large language model to encode them into feature vectors , , , . represents the encoder of the corresponding large language model, Represents the respective network parameters.

[0096] S303: Determine a first loss value according to the first reference model and the first initial model, and train the first initial model according to the first loss value to obtain a trained first initial model.

[0097] In the present application example, construct ( , ) as a positive example pair, and Same as other descriptions in the same batch Constitute a negative example pair. Using contrastive learning, the training goal is the contrastive loss function: ; Among them, sim() refers to the similarity measurement method that measures the angle between two non-zero vectors. Text analysis generally uses cosine similarity. The calculation formula is: ;in: is the dot product of two eigenvectors; Respectively The Euclidean norm of .

[0098] It should be noted that, since the annotations are generally expressed in the form of natural language, only the parameters of the first initial model are trained and adjusted. Optionally, the large language model includes Bert, Albert or the open source GPT model.

[0099] S304: input the log instance sample into the trained first initial model for analysis to obtain a new first log instance sample, and label the new log instance sample to obtain a labeled first log instance sample.

[0100] In this embodiment, a first log instance sample is obtained, wherein the first log instance sample is a log instance including actual log data, the first log instance sample is input into the trained first initial model for analysis to obtain a new first log instance sample, and a labeled first log instance sample is obtained by labeling the new first log instance sample.

[0101] In an embodiment of the present application, the log instance sample may be annotated according to a preset script to obtain a first annotated log instance sample; or, the log instance sample may be annotated according to input information to obtain a first annotated log instance sample.

[0102] S305: Train the second initial model according to the new first log instance sample and the labeled first log instance sample to obtain a target analysis model.

[0103] In an embodiment of the present application, in an embodiment of the present application, the new first log instance sample and the labeled first log instance sample can be used as training samples of the second initial model, the second initial model can be trained, and the parameters in the second initial model can be iteratively updated to obtain the target analysis model.

[0104] In the above application embodiment, the sample data is labeled twice to obtain sample data and labeled sample data, the labeled data is input into the reference model, and the sample data in machine language is input into the initial model, so that the initial model is trained as an analysis model for machine learning.

[0105] In one embodiment, an implementation of the above S305 is provided, such as Figure 5 As shown, the above “training the second initial model according to the new first log instance sample and the labeled first log instance sample to obtain the target analysis model” includes:

[0106] S401: Input the labeled first log instance sample into the second reference model for analysis to obtain a third output value.

[0107] S402: Input the new first log instance sample into the second initial model for analysis to obtain a fourth output value.

[0108] The annotated log instance sample is a log instance annotated as a natural language based on the new log instance sample.

[0109] In the embodiment of the present application, the data in the new log instance sample corresponds one-to-one to the data in the annotated log instance sample, and the first initial model and the first reference model are both large language models. The annotated first log instance sample is input into the second reference model for analysis to obtain a third output value; and the new log instance sample is input into the second initial model for analysis to obtain a fourth output value. It should be noted that the method for training the second initial model according to the new first log instance sample and the annotated first log instance sample in the embodiment of the present application is the same as S302 above.

[0110] S403: Determine a second loss value according to the third output value and the fourth output value.

[0111] In the present application example, construct ( , ) as a positive example pair, and Same as other descriptions in the same batch Constitute a negative example pair. Using contrastive learning, the training goal is the contrastive loss function: ; Among them, sim() refers to the similarity measurement method that measures the angle between two non-zero vectors. Text analysis generally uses cosine similarity. The calculation formula is: ;in: is the dot product of two eigenvectors; Respectively The Euclidean norm of .

[0112] S404: Train the second initial model according to the second loss value to obtain a target analysis model.

[0113] A second loss value between the initial model and the reference model is determined, and the parameters of the second initial model are adjusted according to the second loss value to obtain a target analysis model.

[0114] In the above application embodiment, the initial model is trained for the second time based on the new log instance samples and the labeled log instance samples. Since the new log instance samples are obtained through the trained first initial model, the accuracy of the new log instance samples is improved, thereby improving the accuracy of the target analysis model.

[0115] In one embodiment, an implementation of the above S404 is provided, such as Figure 6 As shown, the above “training the second initial model according to the second loss value to obtain the target analysis model” includes:

[0116] S501, training the second initial model according to the second loss value to obtain an intermediate analysis model.

[0117] In an embodiment of the present application, after determining the second loss value, the parameters of the second initial model are iteratively adjusted according to the second loss value so that the second loss value is reduced. After the second loss value is reduced to a preset target value, the current second initial model is determined as an intermediate analysis model.

[0118] S502: Determine a preset log analysis target, and obtain a second log instance sample corresponding to the preset log analysis target.

[0119] Among them, log analysis goals can include anomaly detection, log classification, etc.

[0120] In this embodiment, a preset log analysis sample library can be pre-established, and a correspondence between each second log analysis sample and the log analysis target can be established, so that the second log instance sample corresponding to the preset log analysis target can be determined from the log analysis sample library based on the correspondence.

[0121] S503: Customize and train the intermediate analysis model according to the second log instance sample to obtain a target analysis model.

[0122] In the embodiment of the present application, a fully connected layer and a softmax classification layer are added after the encoder of the intermediate analysis model, and the category is finally output. A small number of samples used for risk log identification can be used to fine-tune the parameters of the intermediate analysis model, and the trained model can be deployed in the production environment for real-time or batch log risk identification.

[0123] Optionally, taking risk log identification as an example, the goal of risk log identification is to determine whether the log has an impact on the system. This is a binary classification task for a single log. For example, a log of receiving an invalid data packet will affect the system, while a notification type log will not affect the system. The input of this task is a log, and the output is True for impact, and False for no impact. When performing this downstream task, add a fully connected layer and a softmax classification layer after the encoder of the intermediate analysis model, and finally output the category.

[0124] Alternatively, taking fault prediction as an example, log-based fault prediction is essentially a time series prediction problem, making it more challenging than risk identification tasks. A sequence usually consists of multiple logs over a period of time, i.e., within a short time window, and is considered to contain enough information to predict future faults. Combined with the previously pre-trained large language model dedicated to log analysis, our fault prediction includes three steps:

[0125] (1) Sequence construction: In time series analysis, a sliding window is usually used to move the time window so as to extract features at different time points. Each movement can be a fixed time interval, such as a time window of one hour;

[0126] (2) Sequence representation: N logs sampled in the time window are input into the pre-trained large model in sequence to obtain N feature vectors. These N feature vectors are converted into one feature vector using the maximum pooling method.

[0127] (3) Model prediction: A fully connected layer and a softmax classification layer are added after the pooling layer, and the final output is the category.

[0128] In the above-mentioned application embodiment, the second initial model is trained based on the sample data output by the trained first initial model, so that the sample data is more matched with the second initial model, thereby improving the accuracy of the intermediate analysis model. Furthermore, the intermediate analysis model is customized and trained to obtain the target analysis model, so that the target analysis model is more matched with actual usage requirements.

[0129] In one embodiment, an implementation of the above S502 is provided, such as Figure 7 As shown, the above “inputting the first log instance sample into the trained first initial model for analysis to obtain a new first log instance sample” includes:

[0130] S601: Input a first log instance sample into a trained first initial model for analysis to obtain a feature vector corresponding to the log instance sample.

[0131] The first log instance refers to a specific example of a log file or log record generated in an actual application.

[0132] In an embodiment of the present application, the collected log instances are input into the trained first initial model to obtain feature vectors in preparation for subsequent clustering analysis.

[0133] S602: Perform clustering processing on the feature vector corresponding to the first log instance sample to obtain multiple cluster feature vectors.

[0134] In the embodiment of the present application, the DBSCAN clustering algorithm is used to cluster these vectors, and the logs in the same cluster are given the same label to identify the potential patterns and structures in the log template. Specifically:

[0135] 1) Define parameters: Determine two key parameters: eps is the maximum distance between two eigenvectors so that they are considered neighbors of each other; min_samples is the minimum number of vectors required to form a dense region.

[0136] 2) Select core points: traverse all vectors, and for each point, calculate the number of its neighbors within the eps distance. If the number of neighbors is greater than or equal to min_samples, the vector is considered a core point.

[0137] 3) Construct clusters: For each core point, find all its neighbors within the eps distance, and these neighbors also become part of the cluster. Then, for these newly added vectors, find their neighbors again, and if the number of neighbors meets the conditions, add them to the cluster. This process is recursive until there are no more points to join.

[0138] 4) Processing boundary points: Boundary points are those points that are not core points but are within the eps distance of core points and have at least one core point as a neighbor. They are classified into the cluster where the core points of their neighbors are located.

[0139] 5) Cluster labeling: Each cluster can be considered as a cluster and is assigned a unique identifier. Usually, the first cluster is labeled 0, the second cluster is labeled 1, and so on. If the clustering result is not satisfactory, you can adjust the eps and min_samples parameters and re-execute the algorithm. Optionally, you can also choose a commonly used unsupervised clustering algorithm such as Kmeans.

[0140] S603, determining the similarity between each feature vector in each cluster feature vector, and performing deduplication processing on feature vectors whose similarities all meet a preset similarity condition, to obtain processed multi-cluster feature vectors.

[0141] In the embodiment of the present application, the cosine similarity between the feature vectors corresponding to the two logs is determined. When the similarity exceeds a preset threshold and the cluster labels of the clusters of the two log instances are the same, only one of the vectors is retained to reduce duplication and redundancy in the data set.

[0142] S604: Generate a new first log instance sample according to the processed multi-cluster feature vectors.

[0143] In the embodiment of the present application, the log instance corresponding to each feature vector is determined according to the processed multi-cluster feature vectors, and the determined log instance is encapsulated as a new log instance sample.

[0144] Optionally, generate a new log instance sample and merge it with the generated new log template dataset to form a new sample set. This dataset has been enhanced and cleaned to exclude redundant parts and contains a rich sample volume. It can be used for pre-training of subsequent downstream tasks to improve the performance and generalization ability of the model in practical applications.

[0145] In the above application embodiment, the feature vectors are clustered and the feature vectors are screened according to the similarity between the feature vectors, thereby reducing the number of new log instance samples and improving the training efficiency.

[0146] In one embodiment, a complete log analysis method is provided, such as Figure 8 As shown, the method includes:

[0147] S1, obtaining a log template sample and a first log instance sample.

[0148] S2, performing data enhancement processing on the log template sample to obtain an enhanced log template sample, and labeling the enhanced log template sample to obtain a labeled log template sample.

[0149] S3: Input the labeled log template sample into the first reference model for analysis to obtain a first output value.

[0150] S4, inputting the enhanced log template sample into the first initial model for analysis to obtain a second output value.

[0151] S5, determining a first loss value according to the first reference model and the first initial model, and training the first initial model according to the first loss value to obtain a trained first initial model.

[0152] S6: Input the first log instance sample into the trained first initial model for analysis to obtain a feature vector corresponding to the first log instance sample.

[0153] S7, performing clustering processing on the feature vector corresponding to the first log instance sample to obtain multiple cluster feature vectors.

[0154] S8, determining the similarity between each feature vector in each cluster feature vector, and performing deduplication processing on feature vectors whose similarities all meet a preset similarity condition, to obtain processed multi-cluster feature vectors.

[0155] S9, generating a new first log instance sample according to the processed multi-cluster feature vector; and labeling the new first log instance sample to obtain a labeled first log instance sample.

[0156] S10, inputting the annotated first log instance sample into the second reference model for analysis to obtain a third output value; inputting the new first log instance sample into the second initial model for analysis to obtain a fourth output value.

[0157] S11, determining a second loss value according to the third output value and the fourth output value; training a second initial model according to the second loss value to obtain a target analysis model.

[0158] S12, obtaining a log analysis target corresponding to the log instance to be analyzed; the format of the log instance to be analyzed is a structure of a machine language.

[0159] S13, selecting a target analysis model corresponding to the log analysis target from multiple candidate analysis models.

[0160] S14, inputting the log instance to be analyzed into the target analysis model for analysis, and obtaining the analysis result of the log instance to be analyzed.

[0161] In the above log analysis method, the log analysis target corresponding to the log instance to be analyzed is obtained; the format of the log instance to be analyzed is a structure of machine language; the target analysis model corresponding to the log analysis target is screened out from multiple candidate analysis models; the log instance to be analyzed is input into the target analysis model for analysis, and the analysis result of the log instance to be analyzed is obtained. According to the log analysis target of the instance to be analyzed, the target analysis model is matched for the log to be analyzed, which improves the accuracy of the analysis of the log to be analyzed, and the target analysis model is a model that can process machine language, which further improves the accuracy of the analysis of the log to be analyzed.

[0162] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0163] Based on the same inventive concept, the embodiment of the present application also provides a log analysis device for implementing the log analysis method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more log analysis device embodiments provided below can refer to the limitations on the log analysis method above, and will not be repeated here.

[0164] In one embodiment, Fig. 9 As shown, a log analysis device is provided, including: an acquisition module 10, a screening module 11 and an analysis module 12, wherein:

[0165] The acquisition module 10 is used to acquire the log analysis target corresponding to the log instance to be analyzed; the format of the log instance to be analyzed is a structure of machine language.

[0166] The screening module 11 is used to screen out a target analysis model corresponding to the log analysis target from a plurality of candidate analysis models.

[0167] The analysis module 12 is used to input the log instance to be analyzed into the target analysis model for analysis to obtain the analysis result of the log instance to be analyzed.

[0168] In one embodiment, the method further comprises: a second acquisition module, an enhancement module and a training module, wherein:

[0169] The second acquisition module is used to acquire the log template sample and the first log instance sample.

[0170] The enhancement module is used to perform data enhancement processing on the log template samples to obtain enhanced log template samples, and to label the enhanced log template samples to obtain labeled log template samples.

[0171] The training module is used to train the second initial model according to the enhanced log template sample, the labeled log template sample and the first log instance sample to obtain a target analysis model.

[0172] In one embodiment, the training module includes: a first analysis unit, a second analysis unit, a first training unit, a third analysis unit and a second training unit, wherein:

[0173] The first analysis unit is used to input the labeled log template sample into the first reference model for analysis to obtain a first output value.

[0174] The second analysis unit is used to input the enhanced log template sample into the first initial model for analysis to obtain a second output value.

[0175] The first training unit is used to determine a first loss value according to the first reference model and the first initial model, and train the first initial model according to the first loss value to obtain a trained first initial model.

[0176] a third analysis unit, configured to input the first log instance sample into the trained first initial model for analysis to obtain a new first log instance sample, and to annotate the new first log instance sample to obtain an annotated first log instance sample;

[0177] The second training unit is used to train the second initial model according to the new first log instance sample and the labeled first log instance sample to obtain a target analysis model.

[0178] In one embodiment, the above-mentioned second training unit is specifically used to input the labeled log template sample into the first reference model for analysis to obtain a first output value; input the enhanced log template sample into the first initial model for analysis to obtain a second output value; determine the first loss value according to the first reference model and the first initial model, and train the first initial model according to the first loss value to obtain the trained first initial model; input the first log instance sample into the trained first initial model for analysis to obtain a new first log instance sample, and label the new first log instance sample to obtain a labeled first log instance sample; train the second initial model according to the new first log instance sample and the labeled first log instance sample to obtain a target analysis model.

[0179] In one embodiment, the above-mentioned second training unit is specifically used to input the labeled first log instance sample into the second reference model for analysis to obtain a third output value; input the new first log instance sample into the second initial model for analysis to obtain a fourth output value; determine the second loss value based on the third output value and the fourth output value; train the second initial model based on the second loss value to obtain a target analysis model.

[0180] In one embodiment, the above-mentioned second training unit is specifically used to train the second initial model according to the second loss value to obtain an intermediate analysis model; determine a preset log analysis target, and obtain a second log instance sample corresponding to the preset log analysis target; customize the intermediate analysis model according to the second log instance sample to obtain a target analysis model.

[0181] In one embodiment, the second training unit is specifically used to input the first log instance sample into the trained first initial model for analysis to obtain a feature vector corresponding to the first log instance sample; cluster the feature vector corresponding to the first log instance sample to obtain multiple cluster feature vectors; determine the similarity between each feature vector in each cluster feature vector, and deduplicate the feature vectors whose similarities meet preset similarity conditions to obtain processed multiple cluster feature vectors; generate a new first log instance sample based on the processed multiple cluster feature vectors.

[0182] Each module in the above-mentioned log analysis device can be implemented in whole or in part by software, hardware and their combination. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0183] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0184] Obtain the log analysis target corresponding to the log instance to be analyzed; the format of the log instance to be analyzed is a structure of machine language;

[0185] Select a target analysis model corresponding to the log analysis target from multiple candidate analysis models;

[0186] The log instance to be analyzed is input into the target analysis model for analysis to obtain the analysis result of the log instance to be analyzed.

[0187] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0188] Obtain a log template sample and a first log instance sample;

[0189] Performing data enhancement processing on the log template sample to obtain an enhanced log template sample, and annotating the enhanced log template sample to obtain an annotated log template sample;

[0190] The second initial model is trained according to the enhanced log template sample, the labeled log template sample and the first log instance sample to obtain a target analysis model.

[0191] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0192] Inputting the labeled log template sample into the first reference model for analysis to obtain a first output value;

[0193] Inputting the enhanced log template sample into the first initial model for analysis to obtain a second output value;

[0194] Determine a first loss value according to the first reference model and the first initial model, and train the first initial model according to the first loss value to obtain a trained first initial model;

[0195] Inputting the first log instance sample into the trained first initial model for analysis to obtain a new first log instance sample, and labeling the new first log instance sample to obtain a labeled first log instance sample;

[0196] The second initial model is trained according to the new first log instance sample and the labeled first log instance sample to obtain a target analysis model.

[0197] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0198] Inputting the labeled first log instance sample into the second reference model for analysis to obtain a third output value;

[0199] Inputting the new first log instance sample into the second initial model for analysis to obtain a fourth output value;

[0200] Determine a second loss value according to the third output value and the fourth output value;

[0201] The second initial model is trained according to the second loss value to obtain a target analysis model.

[0202] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0203] Training the second initial model according to the second loss value to obtain an intermediate analysis model;

[0204] Determine a preset log analysis target, and obtain a second log instance sample corresponding to the preset log analysis target;

[0205] The intermediate analysis model is customized and trained according to the second log instance sample to obtain a target analysis model.

[0206] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0207] Inputting the first log instance sample into the trained first initial model for analysis to obtain a feature vector corresponding to the first log instance sample;

[0208] Performing clustering processing on the feature vector corresponding to the first log instance sample to obtain multiple cluster feature vectors;

[0209] Determine the similarity between each feature vector in each cluster feature vector, and perform deduplication processing on feature vectors whose similarities all meet the preset similarity condition to obtain processed multi-cluster feature vectors;

[0210] Generate a new first log instance sample according to the processed multi-cluster feature vector.

[0211] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0212] Obtain the log analysis target corresponding to the log instance to be analyzed; the format of the log instance to be analyzed is a structure of machine language;

[0213] Select a target analysis model corresponding to the log analysis target from multiple candidate analysis models;

[0214] The log instance to be analyzed is input into the target analysis model for analysis to obtain the analysis result of the log instance to be analyzed.

[0215] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0216] Obtain a log template sample and a first log instance sample;

[0217] Performing data enhancement processing on the log template sample to obtain an enhanced log template sample, and annotating the enhanced log template sample to obtain an annotated log template sample;

[0218] The second initial model is trained according to the enhanced log template sample, the labeled log template sample and the first log instance sample to obtain a target analysis model.

[0219] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0220] Inputting the labeled log template sample into the first reference model for analysis to obtain a first output value;

[0221] Inputting the enhanced log template sample into the first initial model for analysis to obtain a second output value;

[0222] Determine a first loss value according to the first reference model and the first initial model, and train the first initial model according to the first loss value to obtain a trained first initial model;

[0223] Inputting the first log instance sample into the trained first initial model for analysis to obtain a new first log instance sample, and labeling the new first log instance sample to obtain a labeled first log instance sample;

[0224] The second initial model is trained according to the new first log instance sample and the labeled first log instance sample to obtain a target analysis model.

[0225] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0226] Inputting the labeled first log instance sample into the second reference model for analysis to obtain a third output value;

[0227] Inputting the new first log instance sample into the second initial model for analysis to obtain a fourth output value;

[0228] Determine a second loss value according to the third output value and the fourth output value;

[0229] The second initial model is trained according to the second loss value to obtain a target analysis model.

[0230] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0231] Training the second initial model according to the second loss value to obtain an intermediate analysis model;

[0232] Determine a preset log analysis target, and obtain a second log instance sample corresponding to the preset log analysis target;

[0233] The intermediate analysis model is customized and trained according to the second log instance sample to obtain a target analysis model.

[0234] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0235] Inputting the first log instance sample into the trained first initial model for analysis to obtain a feature vector corresponding to the first log instance sample;

[0236] Performing clustering processing on the feature vector corresponding to the first log instance sample to obtain multiple cluster feature vectors;

[0237] Determine the similarity between each feature vector in each cluster feature vector, and perform deduplication processing on feature vectors whose similarities all meet the preset similarity condition to obtain processed multi-cluster feature vectors;

[0238] Generate a new first log instance sample according to the processed multi-cluster feature vector.

[0239] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0240] Obtain the log analysis target corresponding to the log instance to be analyzed; the format of the log instance to be analyzed is a structure of machine language;

[0241] Select a target analysis model corresponding to the log analysis target from multiple candidate analysis models;

[0242] The log instance to be analyzed is input into the target analysis model for analysis to obtain the analysis result of the log instance to be analyzed.

[0243] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0244] Obtain a log template sample and a first log instance sample;

[0245] Performing data enhancement processing on the log template sample to obtain an enhanced log template sample, and annotating the enhanced log template sample to obtain an annotated log template sample;

[0246] The second initial model is trained according to the enhanced log template sample, the labeled log template sample and the first log instance sample to obtain a target analysis model.

[0247] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0248] Inputting the labeled log template sample into the first reference model for analysis to obtain a first output value;

[0249] Inputting the enhanced log template sample into the first initial model for analysis to obtain a second output value;

[0250] Determine a first loss value according to the first reference model and the first initial model, and train the first initial model according to the first loss value to obtain a trained first initial model;

[0251] Inputting the first log instance sample into the trained first initial model for analysis to obtain a new first log instance sample, and labeling the new first log instance sample to obtain a labeled first log instance sample;

[0252] The second initial model is trained according to the new first log instance sample and the labeled first log instance sample to obtain a target analysis model.

[0253] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0254] Inputting the labeled first log instance sample into the second reference model for analysis to obtain a third output value;

[0255] Inputting the new first log instance sample into the second initial model for analysis to obtain a fourth output value;

[0256] Determine a second loss value according to the third output value and the fourth output value;

[0257] The second initial model is trained according to the second loss value to obtain the target analysis model. In one embodiment, when the computer program is executed by the processor, the following steps are also implemented:

[0258] Training the second initial model according to the second loss value to obtain an intermediate analysis model;

[0259] Determine a preset log analysis target, and obtain a second log instance sample corresponding to the preset log analysis target;

[0260] The intermediate analysis model is customized and trained according to the second log instance sample to obtain a target analysis model.

[0261] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0262] Inputting the first log instance sample into the trained first initial model for analysis to obtain a feature vector corresponding to the first log instance sample;

[0263] Performing clustering processing on the feature vector corresponding to the first log instance sample to obtain multiple cluster feature vectors;

[0264] Determine the similarity between each feature vector in each cluster feature vector, and perform deduplication processing on feature vectors whose similarities all meet the preset similarity condition to obtain processed multi-cluster feature vectors;

[0265] Generate a new first log instance sample according to the processed multi-cluster feature vector.

[0266] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0267] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0268] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A log analysis method, characterized in that: The method comprises: Obtaining a log analysis target corresponding to a log instance to be analyzed; the format of the log instance to be analyzed is a structure of a machine language; Filtering out a target analysis model corresponding to the log analysis target from a plurality of candidate analysis models; The log instance to be analyzed is input into the target analysis model for analysis to obtain an analysis result of the log instance to be analyzed.

2. The method according to claim 1, characterized in that The training process of the target analysis model includes: Obtain a log template sample and a first log instance sample; Performing data enhancement processing on the log template sample to obtain an enhanced log template sample, and annotating the enhanced log template sample to obtain an annotated log template sample; The second initial model is trained according to the enhanced log template sample, the annotated log template sample and the first log instance sample to obtain the target analysis model.

3. The method according to claim 2, characterized in that The step of training the first initial model according to the enhanced log template sample, the annotated log template sample and the first log instance sample to obtain the target analysis model includes: Inputting the annotated log template sample into a first reference model for analysis to obtain a first output value; Inputting the enhanced log template sample into the first initial model for analysis to obtain a second output value; Determine a first loss value according to the first reference model and the first initial model, and train the first initial model according to the first loss value to obtain a trained first initial model; Inputting the first log instance sample into the trained first initial model for analysis to obtain a new first log instance sample, and labeling the new first log instance sample to obtain a labeled first log instance sample; The second initial model is trained according to the new first log instance sample and the labeled first log instance sample to obtain a target analysis model.

4. The method according to claim 3, characterized in that The step of training the second initial model according to the new first log instance sample and the labeled first log instance sample to obtain a target analysis model includes: Inputting the labeled first log instance sample into the second reference model for analysis to obtain a third output value; Inputting the new first log instance sample into the second initial model for analysis to obtain a fourth output value; Determining a second loss value according to the third output value and the fourth output value; The second initial model is trained according to the second loss value to obtain the target analysis model.

5. The method according to claim 4, characterized in that The step of training the second initial model according to the second loss value to obtain the target analysis model includes: Training the second initial model according to the second loss value to obtain an intermediate analysis model; Determine a preset log analysis target, and obtain a second log instance sample corresponding to the preset log analysis target; The intermediate analysis model is customized and trained according to the second log instance sample to obtain the target analysis model.

6. The method according to claim 3, characterized in that The step of inputting the first log instance sample into the trained first initial model for analysis to obtain a new first log instance sample includes: Inputting the first log instance sample into the trained first initial model for analysis to obtain a feature vector corresponding to the first log instance sample; Performing clustering processing on the feature vectors corresponding to the first log instance samples to obtain multiple cluster feature vectors; Determine the similarity between each feature vector in each cluster feature vector, and perform deduplication processing on feature vectors whose similarities all meet the preset similarity condition to obtain processed multi-cluster feature vectors; The new first log instance sample is generated according to the processed multi-cluster feature vectors.

7. A log analysis device, characterized in that: The device comprises: A first acquisition module is used to acquire a log analysis target corresponding to a log instance to be analyzed; the format of the log instance to be analyzed is a structure of a machine language; A screening module, used to screen out a target analysis model corresponding to the log analysis target from a plurality of candidate analysis models; The analysis module is used to input the log instance to be analyzed into the target analysis model for analysis to obtain the analysis result of the log instance to be analyzed.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images