Application log analysis method and device

Abstract

The embodiment of the present invention discloses an analysis method, device, electronic device and storage medium for application logs, which relates to the field of electronic information, including: performing similarity calculation on multiple collected application logs, and filtering similar application logs according to the similarity calculation results; training each application log after filtering by a first algorithm and a second algorithm to obtain multiple prediction models; inputting the application logs to be classified into multiple prediction models, and determining the classification results of the application logs to be classified according to the output results of the multiple prediction models; analyzing the application logs to be classified according to the classification results of the application logs to be classified. This method can filter similar application logs, so that the prediction model obtained by training is more accurate, and then accurately determine the log classification situation based on the prediction model, so as to perform log analysis according to the log classification situation. This method improves the efficiency and accuracy of log analysis.

Description

Technical Field

[0001] The present invention relates to the field of electronic information, and in particular to an analysis method and device for application logs. Background Art

[0002] At present, with the rapid development of Internet technology, artificial intelligence technology has gradually spread to various fields of computers, such as intelligent recommendation, intelligent face recognition, intelligent voice, etc. At the same time, as computer systems become increasingly complex and network data grows explosively, network operation and maintenance are also becoming increasingly complex and difficult. Therefore, in the field of operation and maintenance, it is hoped that artificial intelligence technology can be introduced to achieve intelligent operation and maintenance and improve efficiency. To this end, the efficiency of daily operation and maintenance work can be improved through intelligent analysis of application logs.

[0003] In the prior art, when analyzing application logs, it is not possible to quickly and efficiently classify the logs, resulting in different types of logs being mixed together, low analysis efficiency, and low analysis accuracy. Therefore, a more efficient and accurate log analysis method is urgently needed. Summary of the invention

[0004] In view of the above problems, the present invention is proposed to provide an application log analysis method and device that overcomes the above problems or at least partially solves the above problems.

[0005] According to one aspect of the present invention, there is provided a method for analyzing an application log, comprising:

[0006] Calculate the similarity of multiple collected application logs, and filter similar application logs according to the similarity calculation results;

[0007] The first algorithm and the second algorithm are used to train each application log after filtering to obtain multiple prediction models;

[0008] Inputting the application logs to be classified into the multiple prediction models, and determining the classification results of the application logs to be classified according to the output results of the multiple prediction models;

[0009] The application log to be classified is analyzed according to the classification result of the application log to be classified.

[0010] Optionally, the performing similarity calculation on the collected multiple application logs includes:

[0011] For each application log, a preset number of log classes contained in the application log are extracted, and log tags are added to the application log according to the extracted log classes;

[0012] Word segmentation is performed on multiple application logs under the same log tag, the log word vector of each application log is determined according to the word segmentation result, and the similarity score between each application log is calculated according to the log word vector of each application log.

[0013] Optionally, the first algorithm and the second algorithm are used to train each application log after filtering to obtain multiple prediction models, including:

[0014] For each log tag, at least one prediction model corresponding to the log tag is trained.

[0015] Optionally, calculating the similarity scores between the application logs according to the log word vectors of the application logs includes:

[0016] The similarity between the log word vectors of each application log is calculated by the smoothed inverse frequency similarity comparison method, and the similarity score between each application log is determined according to the similarity calculation result.

[0017] Optionally, the training of each filtered application log by using the first algorithm and the second algorithm includes:

[0018] Determining that the application log corresponds to a first calculation result of the first algorithm and that the application log corresponds to a second calculation result of the second algorithm;

[0019] Calculate the average similarity value of the application log according to the smoothed inverse frequency similarity comparison method, and set a first weight corresponding to the first calculation result and a second weight corresponding to the second calculation result according to the average similarity value;

[0020] According to the first weight and the second weight, a weighted operation is performed on the first calculation result and the second calculation result, a prediction result of the application log is determined according to the operation result, and training is performed based on the prediction result.

[0021] Optionally, the first algorithm is used to obtain log vectors of application logs and perform multi-classification processing based on the log vectors, and the second algorithm is used to extract statement features in the application logs through convolution to obtain multiple feature sets, and obtain log classification output based on the multiple feature sets.

[0022] Optionally, the preset number of log classes contained in the application log include: three JAVA classes located at the header of the application log.

[0023] According to another aspect of the present invention, there is provided an analysis device for application logs, comprising:

[0024] A filtering module, adapted to perform similarity calculation on a plurality of collected application logs, and perform filtering processing on similar application logs according to the similarity calculation result;

[0025] A training module, adapted to train each application log after filtering by using the first algorithm and the second algorithm to obtain multiple prediction models;

[0026] A classification module, adapted to input the application logs to be classified into the multiple prediction models, and determine the classification results of the application logs to be classified according to the output results of the multiple prediction models;

[0027] The analysis module is adapted to analyze the application log to be classified according to the classification result of the application log to be classified.

[0028] Optionally, the filtering module is specifically suitable for:

[0029] For each application log, a preset number of log classes contained in the application log are extracted, and log tags are added to the application log according to the extracted log classes;

[0030] Word segmentation is performed on multiple application logs under the same log tag, the log word vector of each application log is determined according to the word segmentation result, and the similarity score between each application log is calculated according to the log word vector of each application log.

[0031] Optionally, the training module is specifically suitable for:

[0032] For each log tag, at least one prediction model corresponding to the log tag is trained.

[0033] Optionally, the filtering module is specifically suitable for:

[0034] The similarity between the log word vectors of each application log is calculated by the smoothed inverse frequency similarity comparison method, and the similarity score between each application log is determined according to the similarity calculation result.

[0035] Optionally, the training module is specifically suitable for:

[0036] Determining that the application log corresponds to a first calculation result of the first algorithm and that the application log corresponds to a second calculation result of the second algorithm;

[0037] Calculate the average similarity value of the application log according to the smoothed inverse frequency similarity comparison method, and set a first weight corresponding to the first calculation result and a second weight corresponding to the second calculation result according to the average similarity value;

[0038] According to the first weight and the second weight, a weighted operation is performed on the first calculation result and the second calculation result, a prediction result of the application log is determined according to the operation result, and training is performed based on the prediction result.

[0039] Optionally, the first algorithm is used to obtain log vectors of application logs and perform multi-classification processing based on the log vectors, and the second algorithm is used to extract statement features in the application logs through convolution to obtain multiple feature sets, and obtain log classification output based on the multiple feature sets.

[0040] Optionally, the preset number of log classes contained in the application log include: three JAVA classes located at the header of the application log.

[0041] According to another aspect of the present invention, there is provided an electronic device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0042] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the above-mentioned application log analysis method.

[0043] According to another aspect of the present invention, a computer storage medium is provided, wherein the storage medium stores at least one executable instruction, and the executable instruction enables a processor to execute operations corresponding to the above-mentioned application log analysis method.

[0044] In the application log analysis method and device provided by the present invention, it is possible to perform similarity calculation on multiple collected application logs, and perform filtering processing on similar application logs based on the similarity calculation results; train each application log after filtering processing through the first algorithm and the second algorithm to obtain multiple prediction models, thereby determining the classification results of the application logs to be classified based on the output results of the multiple prediction models, and analyzing the application logs to be classified based on the classification results. It can be seen that this method can filter similar application logs, so that the prediction model obtained by training is more accurate, and then accurately determine the log classification situation based on the prediction model, so as to perform log analysis based on the log classification situation. This method improves the efficiency and accuracy of log analysis.

[0045] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0047] Figure 1 A flowchart of an application log analysis method provided by Embodiment 1 of the present invention is shown;

[0048] Figure 2 A flow chart of an application log analysis method provided by Embodiment 2 of the present invention is shown;

[0049] Figure 3 A structural diagram of an application log analysis device provided in Embodiment 3 of the present invention is shown;

[0050] Figure 4 A schematic diagram of the structure of an electronic device provided by Embodiment 5 of the present invention is shown;

[0051] Figure 5 A schematic diagram showing various modules involved in the overall process of intelligent analysis of application logs;

[0052] Figure 6 A schematic diagram of the process flow of the fasttext algorithm is shown. DETAILED DESCRIPTION

[0053] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0054] Embodiment 1

[0055] Figure 1 A flow chart of an application log analysis method provided in the first embodiment of the present invention is shown.

[0056] like Figure 1 As shown, the method includes:

[0057] Step S110: performing similarity calculation on the collected multiple application logs, and performing filtering processing on similar application logs according to the similarity calculation result.

[0058] Among them, the collected multiple application logs are used as log samples to train the prediction model. In order to ensure the purity of the samples, similar application logs need to be filtered to improve the efficiency of model training. Specifically, the similarity calculation method can be flexibly set by those skilled in the art. For example, it can be calculated by a similarity algorithm, or several log classes contained in the application log can be extracted and determined based on the extracted log classes.

[0059] Step S120: training each application log after filtering by using the first algorithm and the second algorithm to obtain multiple prediction models.

[0060] The first algorithm is used to obtain the log vector of the application log and perform multi-classification processing based on the log vector. The second algorithm is used to extract the sentence features in the application log through convolution to obtain multiple feature sets, and obtain the log classification output according to the multiple feature sets. After being processed by the first algorithm and the second algorithm, each application log is trained to obtain multiple prediction models. Each prediction model can correspond to different log types, so as to flexibly realize the prediction of various types of logs.

[0061] Step S130: inputting the application logs to be classified into a plurality of prediction models, and determining the classification results of the application logs to be classified according to the output results of the plurality of prediction models.

[0062] Specifically, the application logs to be classified are input into a plurality of prediction models, and the classification results of the application logs to be classified can be determined quickly and accurately according to the output results of the plurality of prediction models, thereby determining the specific types of the application logs.

[0063] Step S140: analyzing the application log to be classified according to the classification result of the application log to be classified.

[0064] The classification result of the application log to be classified is used to indicate the specific type of the application log. Accordingly, according to the log type of the application log, the characteristics of the type of log can be determined, and then the analysis can be performed based on the characteristics of the type of log to improve the efficiency and accuracy of the analysis.

[0065] It can be seen that in the application log analysis method provided by the present invention, it is possible to perform similarity calculation on multiple collected application logs, and perform filtering processing on similar application logs based on the similarity calculation results; the first algorithm and the second algorithm are used to train each application log after filtering processing to obtain multiple prediction models, thereby determining the classification results of the application logs to be classified based on the output results of the multiple prediction models, and analyzing the application logs to be classified based on the classification results. It can be seen that this method can filter similar application logs, so that the prediction model obtained by training is more accurate, and then accurately determine the log classification situation based on the prediction model, so as to perform log analysis based on the log classification situation. This method improves the efficiency and accuracy of log analysis.

[0066] Embodiment 2

[0067] Figure 2 A flow chart of an application log analysis method provided by the second embodiment of the present invention is shown.

[0068] like Figure 2As shown, the method includes:

[0069] Step S210: performing similarity calculation on the collected multiple application logs.

[0070] Specifically, for each application log, a preset number of log classes contained in the application log are extracted, and a log tag is added to the application log according to the extracted log class; word segmentation is performed on multiple application logs under the same log tag, and the log word vector of each application log is determined according to the word segmentation result, and the similarity score between each application log is calculated according to the log word vector of each application log. Among them, the preset number of log classes contained in the application log include: three JAVA classes located at the head of the application log.

[0071] When calculating the similarity scores between application logs based on the log word vectors of each application log, the similarity between the log word vectors of each application log is calculated by a smoothed inverse frequency similarity comparison method, and the similarity scores between the application logs are determined based on the similarity calculation results.

[0072] Step S220: Filter similar application logs according to the similarity calculation result.

[0073] Specifically, when performing the filtering process, multiple application logs that are similar to each other are filtered, so that multiple application logs with high similarity are not retained at the same time, thereby improving the purity of the sample.

[0074] Step S230: training each application log after filtering by using the first algorithm and the second algorithm to obtain multiple prediction models.

[0075] In specific implementation, for each log tag, at least one prediction model corresponding to the log tag is trained. The log tag corresponds to the log type, and accordingly, the method of training the corresponding prediction model for each log tag can ensure that there is a corresponding prediction model for each type of log.

[0076] Among them, when training each application log after filtering processing through the first algorithm and the second algorithm, first, determine the first calculation result of the application log corresponding to the first algorithm and the second calculation result of the application log corresponding to the second algorithm; then, calculate the similarity average value of the application log according to the smoothed inverse frequency similarity comparison method, and set the first weight corresponding to the first calculation result and the second weight corresponding to the second calculation result according to the similarity average value; finally, according to the first weight and the second weight, perform weighted operation on the first calculation result and the second calculation result, determine the prediction result of the application log according to the operation result, and perform training based on the prediction result.

[0077] Among them, the first algorithm is used to obtain the log vector of the application log and perform multi-classification processing based on the log vector, and the second algorithm is used to extract the statement features in the application log through convolution to obtain multiple feature sets, and obtain the log classification output according to the multiple feature sets.

[0078] Step S240: inputting the application logs to be classified into a plurality of prediction models, and determining the classification results of the application logs to be classified according to the output results of the plurality of prediction models.

[0079] Specifically, the application logs to be classified are input into a plurality of prediction models, and the classification results of the application logs to be classified can be determined quickly and accurately according to the output results of the plurality of prediction models, thereby determining the specific types of the application logs.

[0080] Step S250: analyzing the application log to be classified according to the classification result of the application log to be classified.

[0081] The classification result of the application log to be classified is used to indicate the specific type of the application log. Accordingly, according to the log type of the application log, the characteristics of the type of log can be determined, and then the analysis can be performed based on the characteristics of the type of log to improve the efficiency and accuracy of the analysis.

[0082] In summary, in the application log analysis method provided in the embodiment of the present invention, similar application logs can be filtered, so that the trained prediction model is more accurate, and then the log classification is accurately determined based on the prediction model, so that log analysis can be performed according to the log classification. This method improves the efficiency and accuracy of log analysis.

[0083] For ease of understanding, the following takes a specific example as an example to describe in detail the specific implementation details of the application log analysis method provided in this embodiment:

[0084] In traditional log analysis methods, it is impossible to predict unclassified log data based on the segmented log data. It can only be detected based on business common sense rules. If new application log data appears, it is impossible to accurately determine the type of application log. In this example, the application log analysis process is implemented based on the data analysis flow of ElasticSearch. In the log parsing process, it was previously analyzed by professional business personnel based on the type and rules of log errors and marked with type labels. This example predicts application logs by similarity filtering the application logs during the application log parsing process and adding AI model algorithms combined with similarity coefficient weighting during training. This example mainly extracts the first three Java classes of each log data from the acquired log data, compares the similarity based on the first three Java classes of the application log, and selects the count. Then, the log is labeled and classified based on the first three Java classes, and the log data is segmented. The segmented data is then passed into the neural network training model to obtain multiple prediction models. The model output results are weighted according to the similarity coefficient to achieve the purpose of filtering application log types and statistical analysis. This example proposes a new method for data processing and screening of application logs - the smoothed inverse frequency similarity algorithm, which can provide a basis for model training. In addition, the algorithm is combined with the similarity coefficient to perform reasonable model weighting to achieve the purpose of predicting new log categories; this method saves some manpower time for tagging using scripts and improves the accuracy of tagging. In this example, the accuracy of log classification using the prediction model tagging can reach 90%, while the accuracy of script tagging cannot be evaluated.

[0085] Figure 5 The schematic diagram of each module involved in the overall process of application log intelligent analysis is shown in FIG. Figure 5 As described above, the process mainly involves application log data collection module, application log parsing module, application log archiving module, application log aggregation module and application log analysis module, etc. Among them, the application log parsing module can perform data screening, introduce the smooth inverse frequency similarity algorithm, filter similar logs, and then add intelligent AI neural network learning model training to achieve the effect of online prediction and classification of offline training model.

[0086] The following is a detailed description of the log analysis method:

[0087] Log data collection process:

[0088] For more than 60 topics and more than 370 log modules corresponding to more than 60 business systems, more than 6T of log data needs to be collected every day. Accordingly, 42 topic logs are screened out. The log data types are shown in Table 1:

[0089] Table 1

[0090]

[0091] Data filtering and annotation process:

[0092] After the log data is collected, the similarity comparison is performed based on the first three Java classes in the application log and the numbers are counted. The similarity comparison method is as follows: for the application logs containing three Java classes, auxiliary tags are added based on business common sense; then the application logs are segmented and the similarity between the application logs is analyzed.

[0093] Among them, it is necessary to use the smoothed inverse frequency similarity algorithm to analyze the similarity. The following is a detailed description of the algorithm:

[0094] First, assume that the word vectors contained in the vocabulary are {υ w :w∈V}, the application log is recorded as S, and the word vector {υ s :s∈S}

[0095] Then, based on the word segmentation, calculate the weighted coefficient of each word:

[0096]

[0097] where a is a smoothing parameter and {p(w):w∈V} is the (estimated) frequency of occurrence of each word.

[0098] Next, update the log word vectors of all logs

[0099] Finally, the calculated log word vectors are compared for similarity

[0100]

[0101] For the similarity q i The application logs corresponding to the log word vectors that reach a preset percentage (such as 95%) are filtered, because similar application logs have little effect on the learning of model training. i Application log counts below 95% are incorporated into the tag library.

[0102] Log tag library:

[0103] For log data that has been manually labeled, it is stored in the database. If an existing new log is not labeled, first perform a smoothed inverse frequency similarity comparison based on its first three Java classes. If the similarity is very high and reaches above 95%, the log is directly filtered out. If the similarity is lower than 95% and there is a corresponding type in the database, the log is labeled with the corresponding log tag. If there is no corresponding type in the database, the type of the log is predicted using the trained prediction model, and a log tag is added to the log based on the prediction result. Then the log is added to the tag library to facilitate the update of the next model training.

[0104] Model offline training:

[0105] The overall process of model building is as follows: first, data preprocessing is performed, then data sampling is performed, and then the model is evaluated through the first algorithm and the second algorithm, so as to predict the type of application log according to the model.

[0106] First, the preprocessing process of application log data is introduced: according to the screening data and annotation during data filtering, the application log data is taken out from the tag library and the log data is obtained by line, the message field is analyzed, the word segmentation is adopted, stop words are removed, stemming is performed, and tokens are obtained.

[0107] Then, perform application log data sampling: analyze the proportion of type labels in application log data, and separate the sampling ratio training set and validation set of the corresponding labels. The ratio between the sampling ratio training set and the validation set is 8:2.

[0108] Next, the model is built using the fasttext algorithm and the TextCNN algorithm.

[0109] The fasttext algorithm (i.e., the first algorithm) superimposes and averages the words and n-gram vectors of the entire log to obtain the log vector, and then uses the log vector for softmax multi-classification. The TextCNN algorithm (i.e., the second algorithm) extracts sentence features through convolution, extracts the features with the highest activation degree for each feature set (maximum pooling), and obtains the log classification output after splicing. This example aims to merge the two algorithms and perform weighted optimization on the output results of the algorithms. For different application logs, due to their different log features, the weight emphasis will be different.

[0110] The output result definition formula is as follows:

[0111] f i =q i f a (i)+(1-q i )T(i)

[0112] where fa (i) is the result of the FastText algorithm outputting i logs, T(i) is the result of the TextCNN algorithm outputting i logs, q i is the average of the smoothed inverse frequency similarities between log i and other logs, f i is the prediction result of log i. Figure 6 Figure 2 shows a schematic diagram of the fasttext algorithm. a (i) is the first calculation result of the application log corresponding to the first algorithm, q i That is, the first weight corresponding to the first calculation result set according to the similarity average mentioned above; T(i) is the second calculation result of the application log corresponding to the second algorithm, 1-q i It is the second weight corresponding to the second calculation result set according to the similarity average mentioned above. It can be seen that the present application combines the similarity calculation result with the model training process, so as to make the weight setting more reasonable.

[0113] Finally, the model is evaluated: the prediction model is obtained by training and learning with the training samples, and then the validation set is used for testing and verification, and the parameters are tuned. The accuracy, recall rate, and F1-Score are used to measure the model prediction results.

[0114] Online prediction:

[0115] The prediction model generated by training on the deep learning platform is automatically deployed on the classification detection service cluster by regularly selecting and rolling updating modules according to the test results. At the same time, the original logstash source code is transformed and the corresponding interface call plug-in is developed to achieve a smooth transition of the original data flow.

[0116] The specific instructions are as follows:

[0117] Step 1: Collect log data from Kafka in real time;

[0118] Step 2: Analyze the collected logs and use the algorithm model training method to perform label analysis on the collected logs;

[0119] Step 3: Archive the marked logs, analyze the alarm information and analyze the log anomaly types through the log output information;

[0120] Step 4: Cluster the archived logs and summarize and classify the logs belonging to the same abnormal category;

[0121] Step 5: Perform visual analysis on the classified logs, analyze the logs with specific anomalies, and filter out useless logs;

[0122] Step 6: After filtering the archived clustered logs, add them to the label library to prepare for updating the label library and training model;

[0123] At this point, the method for intelligent analysis of application logs ends.

[0124] In summary, this example uses the smoothed inverse frequency similarity comparison method to filter the application logs when the application log data collection module performs data filtering, thereby improving the filtering effect. In addition, the model pool includes multiple prediction models trained for different application log attributes (types), which are combined with the model results according to the similarity coefficient. It can be seen that this example makes up for the lack of classification prediction models in existing log parsing technology, and achieves the effect of updating the log label library and updating the training model in real time. Since log errors also have real-time characteristics, the label library is updated in time, and the model of new log data is trained to more effectively predict new log data. At the same time, the present invention combines multiple models trained for different application log attributes with model results according to the similarity coefficient, thereby increasing the robustness of the training effect.

[0125] Embodiment 3

[0126] Figure 3 A schematic diagram of the structure of an application log analysis device provided by Embodiment 3 of the present invention is shown, which specifically includes:

[0127] The filtering module 31 is adapted to perform similarity calculation on the collected multiple application logs, and perform filtering processing on similar application logs according to the similarity calculation result;

[0128] The training module 32 is adapted to train each application log after filtering by using the first algorithm and the second algorithm to obtain multiple prediction models;

[0129] A classification module 33, adapted to input the application logs to be classified into the multiple prediction models, and determine the classification results of the application logs to be classified according to the output results of the multiple prediction models;

[0130] The analysis module 34 is adapted to analyze the application log to be classified according to the classification result of the application log to be classified.

[0131] Optionally, the filtering module is specifically suitable for:

[0132] For each application log, a preset number of log classes contained in the application log are extracted, and log tags are added to the application log according to the extracted log classes;

[0133] Word segmentation is performed on multiple application logs under the same log tag, the log word vector of each application log is determined according to the word segmentation result, and the similarity score between each application log is calculated according to the log word vector of each application log.

[0134] Optionally, the training module is specifically suitable for:

[0135] For each log tag, at least one prediction model corresponding to the log tag is trained.

[0136] Optionally, the filtering module is specifically suitable for:

[0137] The similarity between the log word vectors of each application log is calculated by the smoothed inverse frequency similarity comparison method, and the similarity score between each application log is determined according to the similarity calculation result.

[0138] Optionally, the training module is specifically suitable for:

[0139] Determining that the application log corresponds to a first calculation result of the first algorithm and that the application log corresponds to a second calculation result of the second algorithm;

[0140] Calculate the average similarity value of the application log according to the smoothed inverse frequency similarity comparison method, and set a first weight corresponding to the first calculation result and a second weight corresponding to the second calculation result according to the average similarity value;

[0141] According to the first weight and the second weight, a weighted operation is performed on the first calculation result and the second calculation result, a prediction result of the application log is determined according to the operation result, and training is performed based on the prediction result.

[0142] Optionally, the first algorithm is used to obtain log vectors of application logs and perform multi-classification processing based on the log vectors, and the second algorithm is used to extract statement features in the application logs through convolution to obtain multiple feature sets, and obtain log classification output based on the multiple feature sets.

[0143] Optionally, the preset number of log classes contained in the application log include: three JAVA classes located at the header of the application log.

[0144] Embodiment 4

[0145] Embodiment 4 of the present application provides a non-volatile computer storage medium, wherein the computer storage medium stores at least one executable instruction, and the computer executable instruction can execute the analysis method of the application log in any of the above method embodiments. The executable instruction can be specifically used to enable the processor to execute the corresponding operations in the above method embodiments.

[0146] Embodiment 5

[0147] Figure 4 A schematic structural diagram of an electronic device according to Embodiment 5 of the present invention is shown. The specific embodiment of the present invention does not limit the specific implementation of the electronic device.

[0148] like Figure 4 As shown, the electronic device may include: a processor (processor) 402 , a communication interface (Communications Interface) 406 , a memory (memory) 404 , and a communication bus 408 .

[0149] in:

[0150] The processor 402 , the communication interface 406 , and the memory 404 communicate with each other via a communication bus 408 .

[0151] The communication interface 406 is used to communicate with other devices such as clients or other servers.

[0152] The processor 402 is used to execute the program 410, and specifically can execute the relevant steps in the above-mentioned application log analysis method embodiment.

[0153] Specifically, the program 410 may include program codes, which include computer operation instructions.

[0154] The processor 402 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the electronic device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0155] The memory 404 is used to store the program 410. The memory 404 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0156] The program 510 may be specifically used to enable the processor 502 to execute the corresponding operations in the above method embodiment.

[0157] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the description of the above specific languages ​​is for disclosing the best mode of the present invention.

[0158] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0159] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the intention that the claimed invention requires more features than those explicitly recited in each claim. Rather, as reflected in the claims below, inventive aspects lie in all of the features of the individual embodiments disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present invention.

[0160] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0161] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0162] The various component embodiments of the present invention can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functions of some or all of the components in the embodiments of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., computer program and computer program product) for executing part or all of the methods described herein. Such a program implementing the present invention can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0163] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

Claims

1. A method for analyzing application logs, comprising: Calculate the similarity of multiple collected application logs, and filter similar application logs according to the similarity calculation results; Training each application log after filtering by the first algorithm and the second algorithm to obtain multiple prediction models; wherein, determining the first calculation result of the application log corresponding to the first algorithm and the second calculation result of the application log corresponding to the second algorithm; calculating the average similarity value of the application log according to the smoothed inverse frequency similarity comparison method, setting a first weight corresponding to the first calculation result and a second weight corresponding to the second calculation result according to the similarity average value; performing a weighted operation on the first calculation result and the second calculation result according to the first weight and the second weight, determining the prediction result of the application log according to the operation result, and performing training based on the prediction result; Inputting the application logs to be classified into the multiple prediction models, and determining the classification results of the application logs to be classified according to the output results of the multiple prediction models; The application log to be classified is analyzed according to the classification result of the application log to be classified.

2. The method according to claim 1, wherein: The similarity calculation for the collected multiple application logs includes: For each application log, a preset number of log classes contained in the application log are extracted, and log tags are added to the application log according to the extracted log classes; Word segmentation is performed on multiple application logs under the same log tag, the log word vector of each application log is determined according to the word segmentation result, and the similarity score between each application log is calculated according to the log word vector of each application log.

3. The method according to claim 2, wherein: The first algorithm and the second algorithm are used to train each application log after filtering to obtain multiple prediction models including: For each log tag, at least one prediction model corresponding to the log tag is trained.

4. The method according to claim 2, wherein: Calculating the similarity scores between the application logs according to the log word vectors of the application logs includes: The similarity between the log word vectors of each application log is calculated by the smoothed inverse frequency similarity comparison method, and the similarity score between each application log is determined according to the similarity calculation result.

5. The method according to claim 1, wherein: The first algorithm is used to obtain the log vector of the application log and perform multi-classification processing based on the log vector, and the second algorithm is used to extract the statement features in the application log through convolution to obtain multiple feature sets, and obtain the log classification output according to the multiple feature sets.

6. The method according to claim 2, wherein: The preset number of log classes contained in the application log include: three JAVA classes located at the head of the application log.

7. An analysis device for application logs, comprising: A filtering module, adapted to perform similarity calculation on a plurality of collected application logs, and perform filtering processing on similar application logs according to the similarity calculation result; A training module, adapted to train each application log after filtering by a first algorithm and a second algorithm to obtain a plurality of prediction models; wherein, determining a first calculation result of the application log corresponding to the first algorithm and a second calculation result of the application log corresponding to the second algorithm; calculating the average similarity value of the application log according to a smoothed inverse frequency similarity comparison method, setting a first weight corresponding to the first calculation result and a second weight corresponding to the second calculation result according to the similarity average value; performing a weighted operation on the first calculation result and the second calculation result according to the first weight and the second weight, determining a prediction result of the application log according to the operation result, and performing training based on the prediction result; A classification module, adapted to input the application logs to be classified into the multiple prediction models, and determine the classification results of the application logs to be classified according to the output results of the multiple prediction models; The analysis module is adapted to analyze the application log to be classified according to the classification result of the application log to be classified.

8. An electronic device, comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the application log analysis method according to any one of claims 1-6.

9. A computer storage medium, wherein at least one executable instruction is stored in the storage medium, and the executable instruction enables a processor to execute an operation corresponding to the application log analysis method according to any one of claims 1 to 6.

Images