Log processing method and device, computer device and storage medium

Abstract

The application relates to a log processing method and device, computer equipment and a storage medium, and applies to the technical field of artificial intelligence. The method comprises the following steps: performing information extraction processing on a to-be-processed log to obtain keyword information in the to-be-processed log; performing re-extraction processing on the keyword information according to a middleware category corresponding to the to-be-processed log to obtain target keyword information matched with the middleware category; the middleware category is the category of a server used for deploying an application program corresponding to the to-be-processed log; determining fault information corresponding to the to-be-processed log based on the target keyword information; and finding a pre-constructed decision tree according to the fault information to obtain a decision result corresponding to the to-be-processed log. The method can shorten the operation and maintenance emergency time length and reduce the risk of production accidents.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a log processing method and device, computer equipment, storage medium and computer program product. BACKGROUND

[0002] The running and maintenance of software is the last stage of the software life cycle and also the longest duration, which directly faces the end user, therefore, it is of great significance to do a good job in software operation and maintenance, and the production accident in the software operation and maintenance process is often caused by the application itself, which is reflected in the log of non-system side monitoring. Therefore, the analysis and judgment of the log become an important part of the operation and maintenance work.

[0003] However, the current judgment of log output is mostly obtained by the developer himself, which needs to consume a lot of time, thereby affecting the emergency timeliness. SUMMARY

[0004] Therefore, it is necessary to provide a log processing method, device, computer equipment, computer readable storage medium and computer program product to solve the technical problems that the personalized log cannot be analyzed by the conventional monitoring tool and the timely response.

[0005] In a first aspect, the present application provides a log processing method. The method comprises:

[0006] information extraction processing is performed on the to-be-processed log to obtain keyword information in the to-be-processed log;

[0007] According to the middleware category corresponding to the to-be-processed log, the keyword information is extracted again to obtain target keyword information matched with the middleware category; the middleware category is the category of the server used to deploy the application program corresponding to the to-be-processed log;

[0008] Based on the target keyword information, the fault information corresponding to the to-be-processed log is determined;

[0009] According to the fault information, a decision tree constructed in advance is searched to obtain a decision result corresponding to the to-be-processed log.

[0010] In one of the embodiments, the information extraction processing on the to-be-processed log to obtain the keyword information in the to-be-processed log comprises:

[0011] Based on the weight of the sample error keyword in the pre-constructed keyword information library, a plurality of target sample error keywords are extracted from the keyword information library; the keyword information library stores a plurality of sample error keywords and the weight of the plurality of sample error keywords;

[0012] According to each target sample error keyword, information extraction processing is performed on the to-be-processed log to obtain keyword information in the to-be-processed log.

[0013] In one of the embodiments, the weight of the sample error keyword is determined by the following manner:

[0014] A plurality of sample error keywords in a sample log are obtained.

[0015] The occurrence times of each sample error keyword in the sample log are counted respectively, and the cumulative sum of the occurrence times of each sample error keyword is counted.

[0016] For any sample error keyword, the ratio of the occurrence time of the sample error keyword to the cumulative sum of the occurrence times is obtained as the weight of the sample error keyword.

[0017] In one of the embodiments, the determination of the fault information corresponding to the to-be-processed log based on the target keyword information comprises:

[0018] The target keyword information is processed by the trained classification model to obtain the fault information of the to-be-processed log; the classification model is trained by sample data with the target keyword information as input and the fault information as output.

[0019] In a second aspect, the present application further provides a log processing device. The device comprises:

[0020] A first extraction module is configured to perform information extraction processing on a to-be-processed log to obtain keyword information in the to-be-processed log.

[0021] A second extraction module is configured to determine a middleware category corresponding to the to-be-processed log, and perform re-extraction processing on the keyword information according to the middleware category to obtain target keyword information matched with the middleware category; the middleware category is a category of a server used to deploy an application program corresponding to the to-be-processed log.

[0022] A determination module is configured to determine fault information corresponding to the to-be-processed log based on the target keyword information.

[0023] A search module is configured to search a pre-constructed decision tree according to the fault information to obtain a decision result corresponding to the to-be-processed log.

[0024] In a third aspect, the present application further provides a computer device. The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0025] performing information extraction processing on the to-be-processed log to obtain keyword information in the to-be-processed log;

[0026] performing re-extraction processing on the keyword information according to a middleware category corresponding to the to-be-processed log to obtain target keyword information matched with the middleware category; the middleware category is a category of a server used to deploy an application program corresponding to the to-be-processed log;

[0027] determining fault information corresponding to the to-be-processed log based on the target keyword information;

[0028] finding a pre-constructed decision tree according to the fault information to obtain a decision result corresponding to the to-be-processed log.

[0029] In a fourth aspect, the present application further provides a computer readable storage medium. The computer readable storage medium has a computer program stored thereon, and the computer program is executed by a processor to implement the following steps:

[0030] performing information extraction processing on the to-be-processed log to obtain keyword information in the to-be-processed log;

[0031] performing re-extraction processing on the keyword information according to a middleware category corresponding to the to-be-processed log to obtain target keyword information matched with the middleware category; the middleware category is a category of a server used to deploy an application program corresponding to the to-be-processed log;

[0032] determining fault information corresponding to the to-be-processed log based on the target keyword information;

[0033] finding a pre-constructed decision tree according to the fault information to obtain a decision result corresponding to the to-be-processed log.

[0034] In a fifth aspect, the present application further provides a computer program product. The computer program product comprises a computer program, and the computer program is executed by a processor to implement the following steps:

[0035] performing information extraction processing on the to-be-processed log to obtain keyword information in the to-be-processed log;

[0036] performing re-extraction processing on the keyword information according to a middleware category corresponding to the to-be-processed log to obtain target keyword information matched with the middleware category; the middleware category is a category of a server used to deploy an application program corresponding to the to-be-processed log;

[0037] determining fault information corresponding to the to-be-processed log based on the target keyword information;

[0038] According to the fault information, a decision tree previously constructed is searched to obtain a decision result corresponding to the log to be processed.

[0039] The log processing method, device, computer device, storage medium and computer program product, first extract information from the log to be processed to obtain keyword information in the log to be processed, then perform re-extraction processing on the keyword information according to the middleware category corresponding to the log to be processed to obtain target keyword information matched with the middleware category, determine the fault information corresponding to the log to be processed based on the target keyword information, and finally search a decision tree previously constructed according to the fault information to obtain a decision result corresponding to the log to be processed. The method establishes a model through the decision tree in the terminal to realize artificial intelligence scheduling of the log to be processed, without the need for developers to analyze and judge by themselves, thereby shortening the operation and maintenance emergency time and improving the operation and maintenance efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 An application environment diagram of the log processing method in an embodiment;

[0041] Figure 2 A flowchart of the log processing method in an embodiment;

[0042] Figure 3 A knowledge graph based on middleware in an embodiment;

[0043] Figure 4 A diagram of the decision tree in the log processing method in an embodiment;

[0044] Figure 5 A flowchart of the log processing method in another embodiment;

[0045] Figure 6 A flowchart of the log processing method in an application example;

[0046] Figure 7 A structural block diagram of the log processing device in an embodiment;

[0047] Figure 8 An internal structure diagram of the computer device in an embodiment. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application is further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0049] Operation and maintenance is the last stage of software life cycle and also the longest one, which directly faces the end user. Therefore, its importance is self-evident, and good operation and maintenance work also has very far-reaching significance. For system operation and maintenance, the common method at present is to deploy monitoring program in advance through technical means, to ensure the availability of services by actively sending agreed messages to the monitoring system during production operation, or to check whether the system can be normally accessed by network detection, so as to isolate the fault server after discovering the abnormality.

[0050] With the development of computer technology, the configuration of software and hardware is gradually upgraded, and enterprises are also constantly promoting the transformation of architecture. However, after the change of the basic platform, the application side and the monitoring end need to re-agree a new set of rules to adapt to the new platform. In order to achieve a new balance, the initial intention of relatively uncertain adaptation and operation and maintenance-stability is different. At the same time, the monitoring of system and network is only a part of application operation and maintenance, and more production accidents often occur due to the application itself, which is reflected in the logs of non-system side monitoring. However, the judgment of non-standard log output at present is mostly based on the log analysis and judgment of the developer himself, lacking of intelligent judgment of automatic tools, thereby affecting the emergency timeliness, and finally affecting the satisfaction and trust of the end customer to the enterprise.

[0051] Therefore, in order to solve the problem that the personalized log cannot be analyzed through the conventional monitoring tool and the problem of timely response, the present application provides an intelligent operation and maintenance scheme based on decision tree algorithm automatic evaluation and rapid feedback decision-making-log processing method.

[0052] The log processing method provided by the embodiments of the present application can be applied in an application environment as shown in Figure 1 The terminal 102 communicates with the server 104 through the network. The data storage system can store the data required to be processed by the server 104. The data storage system can be integrated on the server 104, or can be placed on the cloud or other network servers. In the application scenario of the present application, the terminal 102 obtains the to-be-processed log from the server 104, performs information extraction processing on the to-be-processed log, obtains the keyword information in the to-be-processed log, and performs re-extraction processing on the keyword information according to the middleware category corresponding to the to-be-processed log, to obtain target keyword information matched with the middleware category; based on the target keyword information, the fault information corresponding to the to-be-processed log is determined; according to the fault information, the decision tree constructed in advance is searched to obtain the decision result corresponding to the to-be-processed log.

[0053] In the present application, the terminal 102 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0054] In one embodiment, as shown in Figure 2 , a log processing method is provided. Taking the terminal 102 in Figure 1 as an example, the method includes the following steps:

[0055] In step S210, information extraction processing is performed on the to-be-processed log to obtain keyword information in the to-be-processed log.

[0056] The to-be-processed log can represent a log generated when an application runs abnormally.

[0057] The keyword information can be keyword information related to the abnormal running of the application.

[0058] During the daily operation of the application system, logs are continuously generated on the server. A log collection program is triggered by deploying a timing task, and the application-side logs are downloaded to a special analysis platform within a certain interval. Specifically, the required log files on the server can be downloaded by using the existing jar (a computer file format) package ganymed-ssh2-build.jar in Java (a computer programming language). The ganymed-ssh2 is a package for implementing the SSH-2 protocol (Secure Shell) using Java, which can be used to connect to an SSH server and call the sz instruction (a command for sending files from the server to the client) of Linux (an operating system) to send the log files from the server 104 to the terminal 102.

[0059] After receiving the log file sent by server 104, terminal 102 treats the log file as a log to be processed. Using lexical and syntactic analysis of Natural Language Processing (NLP), it splits the text in the log into independent words and extracts feature words related to the error keywords using the Term Frequency-Inverse Document Frequency (TF-IDF) algorithm, which serve as keyword information. More specifically, to extract keyword information from the log to be processed using NLP, multiple target sample error keywords can be obtained by searching a pre-built keyword information database. Information extraction is then performed on the log according to each target sample error keyword to obtain the keyword information from the log to be processed.

[0060] Step S220: Based on the middleware category corresponding to the log to be processed, the keyword information is extracted again to obtain the target keyword information that matches the middleware category; the middleware category is the category of the server used to deploy the application corresponding to the log to be processed.

[0061] Middleware refers to the type of server used to deploy the application corresponding to the logs to be processed. Middleware types may include Was, Liberty, Tomcat, TongWeb, etc.

[0062] The target keyword information may include error messages and potential fault information.

[0063] In practice, different middleware may provide different error messages, such as... Figure 3 As shown, the keyword information corresponding to the middleware Was is "com.ibm.ws.rsadapter.DSConfigHelper$1.run(DSConfigHelper.Java:1273)," while the keyword information corresponding to the middleware Liberty is "com.ibm.ejs.j2c.MCWrapper.cleanup(MCWrapper.java:1680)." Therefore, in order to more accurately determine the fault information corresponding to the log to be processed, after extracting the keyword information from the log to be processed through natural language processing, the category of the server that deploys the application corresponding to the log to be processed can be further determined as the middleware category. Using knowledge graph technology, the keyword information is further extracted and processed according to the middleware category to obtain the error information and potential fault information that match the middleware category corresponding to the log to be processed, which are used as target keyword information. Furthermore, the middleware category, error information, and potential fault information can be used to form a structured keyword text, thereby realizing the transformation of unstructured information into a structured data system.

[0064] Wherein, the knowledge graph is a structured semantic knowledge base, used to quickly describe the concepts in the physical world and their mutual relations, which is through effective processing, processing, integration of complex data documents, into simple, clear "entity, relationship, entity" triple, finally aggregate a large amount of knowledge, so as to realize the quick response and reasoning of knowledge, such as Figure 3 As shown in a middleware log based knowledge graph. Two nodes with existence relationship are connected together through an undirected edge, the node is called entity, and the edge between them is called relationship.

[0065] Step S230, based on the target keyword information, determine the fault information corresponding to the log to be processed.

[0066] Wherein, the fault information can include information of multiple fault indicators, for example, the fault information can include information of fault level and production accident probability.

[0067] In a specific implementation, the error information and fault hidden danger information in the target keyword information can be input, and the fault level and production accident probability can be output, and the classification model can be pre-trained, and the target keyword information and the influence range information of the fault can be processed by the classification model to obtain the fault level and the production accident probability of the log to be processed as the fault information of the log to be processed.

[0068] Step S240, according to the fault information, find the decision tree constructed in advance, obtain the decision result corresponding to the log to be processed.

[0069] Wherein, the decision tree represents a tree structure similar to flowchart, each node inside the tree represents a test on a feature, the branches of the tree represent each test result of the feature, and each leaf node of the tree represents a classification, the highest layer of the tree is the root node, such as Figure 4 As shown in a schematic diagram of a decision tree, the internal node is represented by a rectangle, and the leaf node is represented by an ellipse.

[0070] Wherein, the decision result can include application program needs to restart and application program does not need to restart.

[0071] In a specific implementation, logs of different decision results can be obtained in a certain proportion to form training samples, each sample containing fault information of multiple fault indicators, for example, each sample containing fault levels and production accident occurrence probabilities, a decision tree is constructed based on a decision tree construction algorithm ID3 of information gain, and the fault levels of each fault indicator in the fault information are determined. After obtaining the fault information of the to-be-processed log, the decision tree constructed is searched layer by layer in the order of the fault levels of each fault indicator in the fault information from high to low, the corresponding leaf node of the fault information in the decision tree is obtained, and the decision result corresponding to the leaf node is taken as the decision result corresponding to the to-be-processed log.

[0072] For example, as shown in the decision tree, Figure 4 For example, as shown in the decision tree, when searching the decision tree, first compare the fault level indicator of the root node, if the fault level of the to-be-processed log is greater than A, enter the leaf node of the left branch to obtain the decision result: restart is needed. If the fault level of the to-be-processed log is not greater than A, enter the node influence range node of the right branch to determine whether the influence range of the to-be-processed log is greater than B, if yes, enter the leaf node of the left branch to obtain the decision result: restart is needed. If not, continue to enter the production accident probability node of the right branch to determine, if the production accident probability of the to-be-processed log is greater than C, enter the leaf node of the left branch to obtain the decision result: restart is needed. Otherwise, enter the right branch to continue to determine whether the production accident probability of the to-be-processed log is less than D, if yes, obtain the decision result: no restart is needed; if not, obtain the decision result: restart is needed.

[0073] In the above log processing method, first, the keyword information in the to-be-processed log is obtained through information extraction processing; then, the keyword information is extracted again according to the middleware category corresponding to the to-be-processed log to obtain target keyword information matched with the middleware category, and based on the target keyword information, the fault information corresponding to the to-be-processed log is determined, and finally, according to the fault information, the decision tree constructed in advance is searched to obtain the decision result corresponding to the to-be-processed log. This method establishes a model through a decision tree on a terminal to realize artificial intelligence scheduling of to-be-processed logs, without the need for developers to analyze and judge themselves, thereby shortening the operation and maintenance emergency time and improving operation and maintenance efficiency.

[0074] In an example embodiment, the information extraction processing on the to-be-processed log in step S210 to obtain the keyword information in the to-be-processed log comprises: extracting a plurality of target sample error keywords from the keyword information library based on the weights of the sample error keywords in the pre-constructed keyword information library; the keyword information library stores a plurality of sample error keywords and weights of the plurality of sample error keywords; and performing information extraction processing on the to-be-processed log according to each target sample error keyword to obtain the keyword information in the to-be-processed log.

[0075] The weight can represent the frequency of the error keyword in the fault log.

[0076] In a specific implementation, the sample error keywords in the sample log can be extracted in advance, the frequency of each sample error keyword can be calculated as the weight of each sample error keyword, and an association between each sample error keyword and the corresponding weight of each sample error keyword can be established and stored in the keyword information library. After obtaining the to-be-processed log, each sample error keyword can be sorted in descending order of the weight value to obtain a sample error keyword sequence, a plurality of target sample error keywords can be extracted from the keyword information library in descending order of the weight value, and information extraction can be performed on the target sample error keywords in the to-be-processed log to obtain the keyword information in the to-be-processed log.

[0077] Further, in an example embodiment, the weight of the sample error keyword is determined by: obtaining a plurality of sample error keywords in the sample log; respectively counting the number of occurrences of each sample error keyword in the sample log and the cumulative sum of the number of occurrences of each sample error keyword; and obtaining the ratio of the number of occurrences of the sample error keyword to the cumulative sum of the number of occurrences of the sample error keyword as the weight of the sample error keyword.

[0078] In a specific implementation, the TF-IDF algorithm (Term Frequency-Inverse Document Frequency) can be used to extract feature words related to the error keywords in the sample log to calculate the weight of the sample error keyword. More specifically, first, keyword extraction is performed on the sample log to obtain a plurality of sample error keywords in the sample log, the number of occurrences of each sample error keyword in the sample log and the cumulative sum of the number of occurrences of each sample error keyword are counted, and the ratio of the number of occurrences of the sample error keyword to the cumulative sum of the number of occurrences of the sample error keyword is calculated as the weight of the sample error keyword.

[0079] In this embodiment, by acquiring sample error keywords and calculating their weights in the sample logs, a keyword information database is pre-built, which can better locate keyword information in the logs to be processed and improve the efficiency of keyword information acquisition.

[0080] In an exemplary embodiment, the step S230 above, which determines the fault information corresponding to the log to be processed based on the target keyword information, can be implemented in the following way: the target keyword information is processed by a trained classification model to obtain the fault information of the log to be processed; the classification model is trained by sample data with the target keyword information as input and the fault information as output.

[0081] In practical implementation, the Naive Bayes method can be used to classify target keyword information. First, sample data is acquired and divided into training and testing data. The target keyword information is used as input, and fault information is used as output. The Bayesian model is trained using the training data and tested using the testing data. A Bayesian model with accuracy meeting preset conditions is obtained and serves as the trained classification model. After obtaining the target keyword information of the log to be processed, the Bayesian model is used to classify the target keyword information to obtain the fault information of the log.

[0082] The Naive Bayes algorithm is a classification method based on Bayes' theorem and the assumption of conditional independence. The idea is that when the exact nature of a thing cannot be known, the probability of its essential attribute can be judged by the frequency of events related to that thing's specific nature. For a given training set, the joint probability distribution of input and output is first learned based on conditional independence. Then, on this model, for a given input x, Bayes' theorem is used to find the output y with the highest posterior probability. The basic method of the algorithm is as follows:

[0083] Let the input space be Given a set of n-dimensional vectors, the output space is a set of class labels Y = {c1, c2, ..., ck}. The input is a feature vector, and the output is the class label. The training set is:

[0084] T={(x1,y1),(x2,y2),…,(xN,yN)}

[0085] The joint probability distribution P(X,Y) of an instance is learned through a training set. The joint probability P(X,Y) of an instance is calculated as follows:

[0086] P(X,Y)=P(X|Y)·P(Y)=P(Y|X)·P(X)

[0087] Based on the above equation, we can obtain the general form of Bayesian theory:

[0088]

[0089] Naive Bayes can also be represented as:

[0090]

[0091] After removing the common denominators, the formula simplifies to:

[0092]

[0093] In this embodiment, by constructing an automatic classification model with learning capabilities, the correlation between target keyword information and fault information is established, thereby improving the prediction efficiency of log faults to be processed.

[0094] In an exemplary embodiment, step S230 above, which determines the fault information corresponding to the log to be processed based on the target keyword information, further includes: determining the influence range information of the log to be processed; and determining the fault information corresponding to the log to be processed based on the influence range information and the target keyword information.

[0095] In practical implementation, when extracting keyword information again using knowledge graph technology, the scope of the fault's impact can also be identified simultaneously. Based on the scope of impact information and the target keyword information, the fault level and the probability of a production accident corresponding to the log to be processed are determined. More specifically, error information, potential fault information, and the scope of the fault's impact information from the target keyword information can be used as inputs, with the fault level and the probability of a production accident as outputs. A classification model is pre-trained, and the target keyword information and the scope of the fault's impact information are processed by the classification model to obtain the fault level and the probability of a production accident for the log to be processed, which serve as the fault information for the log to be processed.

[0096] In this embodiment, by combining the impact range information of the log to be processed with the target keyword information, the fault information corresponding to the log to be processed can be determined, which can further improve the accuracy of the determined fault information.

[0097] In an exemplary embodiment, the step S140 above, which searches a pre-constructed decision tree based on the fault information to obtain the decision result corresponding to the log to be processed, can be achieved in the following way: sorting each fault indicator in the fault information according to a predetermined fault level; the fault information includes multiple fault indicators and the indicator values ​​of multiple fault indicators; searching the decision tree layer by layer according to the fault level of each fault indicator from high to low to obtain the leaf node corresponding to the fault information in the decision tree; determining the decision result corresponding to the leaf node as the decision result corresponding to the log to be processed.

[0098] In the specific implementation, the sorting structure of the fault levels of each fault indicator in the fault information is the same as the sorting result of the fault levels of each fault indicator when constructing the decision tree. After obtaining the fault information of the log to be processed, the decision tree can be searched layer by layer according to the fault level of each fault indicator in the fault information from high to low, and compared with the conditions of each decision node, until the leaf node is reached. The decision result corresponding to the leaf node is taken as the decision result corresponding to the log to be processed.

[0099] The decision tree can be constructed based on the ID3 decision tree construction algorithm with information gain. The following example, using the application error log information of the Was middleware, illustrates the construction of the decision tree:

[0100] Suppose the obtained samples are the five samples shown in Table 1 below. Each sample has three attributes: fault level, scope of impact, and probability of production accident. Finally, it is necessary to determine whether automatic restart is required. Various decision trees can be trained using these samples, such as... Figure 5 The image shows one type of decision tree.

[0101] Table 1 Sample application error logs for Was middleware

[0102]

[0103] By studying the data in Table 1, we can set up settings such as... Figure 5 The fault indicators A, B, C, and D shown represent decision thresholds for fault level, scope of impact, and probability of production accidents, and these decision thresholds should minimize the classification error rate.

[0104] The construction of a decision tree mainly involves two steps: node splitting and threshold determination. Node splitting means that when the attribute represented by a node cannot be determined, it is generally split into two child nodes (or n child nodes in the case of a non-binary tree). Determining the threshold requires choosing an appropriate threshold to ensure high classification accuracy. The algorithm for generating a decision tree is a classification method that approximates discrete function values. First, the data is processed, and an inductive algorithm is used to generate readable rules and a decision tree. Then, the decision tree is used to analyze new data. Decision tree algorithms construct decision trees to discover the classification rules inherent in the data. Taking the information gain-based decision tree construction algorithm ID3 as an example, the entropy principle determines which node should be the parent node and which node needs to be split. For a set of data, the lower the entropy, the better the classification result.

[0105] The definition of entropy is as follows:

[0106] Where p(x) i ) is x i The probability of occurrence.

[0107] If it's a binary classification case, with class A and class B each accounting for 50%,

[0108] Entropy=-(0.5*log2(0.5)+0.5*log2(0.5))=1

[0109] When there is only category A or only category B

[0110] Entropy = -(1*log2(1)+0) = 0

[0111] Therefore, when Entropy is at its maximum of 1, it represents the worst classification performance, and when it is at its minimum of 0, it represents perfect classification. Entropy equal to 0 is the ideal state. In reality, entropy generally lies between 0 and 1. Continuously minimizing entropy is the process of improving classification accuracy.

[0112] For example, the three attributes in Table 1 can be categorized individually using the following statement:

[0113] 1) Low fault level [no restart required]: 1 incorrectly assigned;

[0114] 2) Wide-ranging impact [requires restart]: 1 misclassified instance;

[0115] 3) Production accident probability greater than 50% [restart required]: No errors detected;

[0116] Finally, it was found that the probability of a production accident was greater than 50% [requiring a restart], resulting in the fewest errors, i.e., the lowest entropy. Therefore, this should be chosen as the parent node for tree generation. The same logic applies when splitting the parent node: compared to the classification error rate before splitting, the choice that improves accuracy is retained.

[0117] After constructing the decision tree, each leaf node has a corresponding execution plan. For common system maintenance tasks such as application server restarts and directory cleanup, a standard of no human intervention can be achieved by directly scheduling deployed automated tools. For the parts of the leaf nodes that require human intervention and analysis, manual intervention will be assigned in the short term, while in the medium to long term, the maintenance tools and decision tree modules for these parts will be continuously improved to achieve a higher level of intelligent operation and maintenance.

[0118] In this embodiment, a model is built using a decision tree to achieve artificial intelligence scheduling of automated tools, which greatly shortens the operation and maintenance emergency response time and reduces the risk of production accidents.

[0119] In one embodiment, such as Figure 5 As shown, a log processing method is provided. In this embodiment, the method includes the following steps:

[0120] Step S510: Based on the weights of sample error keywords in the pre-built keyword information database, extract multiple target sample error keywords from the keyword information database; the keyword information database stores multiple sample error keywords and their weights.

[0121] Step S520: Extract information from the log to be processed according to the error keywords of each target sample to obtain the keyword information in the log to be processed.

[0122] Step S530: Determine the impact scope information and middleware category of the logs to be processed;

[0123] Step S540: Based on the middleware category corresponding to the log to be processed, the keyword information is extracted and processed again to obtain the target keyword information that matches the middleware category;

[0124] Step S550: The target keyword information and the scope of influence information are processed through the trained classification model to obtain the fault information of the log to be processed; the classification model is trained using sample data with target keyword information as input and fault information as output.

[0125] Step S560: Sort the various fault indicators in the fault information according to the predetermined fault level; the fault information includes multiple fault indicators and the indicator values ​​of the multiple fault indicators.

[0126] Step S570: According to the fault level of each fault indicator from high to low, search the decision tree layer by layer to obtain the leaf node corresponding to the fault information in the decision tree.

[0127] Step S580: Determine the decision result corresponding to the leaf node, and use it as the decision result corresponding to the log to be processed.

[0128] Understandably, operations and maintenance (O&M) work typically encompasses both applications and systems. The system side is maintained and supported by specialized system and network technical departments, including monitoring CPU utilization, disk capacity, tablespace utilization, and database session counts. The application side relies more on its own set metrics, comparing them with pre-defined thresholds to determine system risks. This application, based on application-layer logs, utilizes data mining techniques to analyze and match output information, and then intelligently schedules system-layer emergency measures using a decision tree algorithm, thereby improving O&M efficiency and flexibility.

[0129] In one embodiment, to facilitate understanding of the embodiments of this application by those skilled in the art, specific examples will be described below in conjunction with the accompanying drawings. References Figure 6 The diagram illustrates a log processing method, including the following steps:

[0130] (1) Obtain the file content of the error log, extract information from the file content of the error log through natural language processing (NLP), and obtain the keyword information of the error log;

[0131] Natural Language Processing (NLP) refers to the research on processing and manipulating language text (sentences, texts, or discourses, etc.) using computer technology. Research content includes various processing methods and implementation technologies for identifying, classifying, extracting, transforming, and generating lexical, syntactic, semantic, and pragmatic information. In this invention, lexical and syntactic analysis from NLP is used to split the text in the application log into independent word segments, and feature words related to error keywords are extracted using the TF-IDF algorithm.

[0132] (2) Use knowledge graph technology to determine the middleware category and scope of impact of the error log, and extract the keyword information again based on the middleware category to obtain the target keyword information;

[0133] (3) Use the Bayesian algorithm in machine learning to organize and summarize the middleware category, scope of influence and target keyword information to form structured data;

[0134] (4) The Bayesian algorithm is used again to calculate the fault level and the probability of production accident in the error log based on the middleware category, the scope of impact information and the target keyword information, as the fault information;

[0135] (5) Search based on fault level and probability of production accident occurrence Figure 4 The decision tree shown yields a decision on whether the application needs to be restarted.

[0136] TF-IDF (Term Frequency-Inverse Document Frequency) is a weighted technique used in information retrieval and text mining to evaluate the importance of a word to a document within a document set or corpus. A word's importance increases proportionally to its frequency of occurrence in a document, but decreases inversely proportionally to its frequency of occurrence in the corpus. The main idea of ​​TF-IDF is that if a word or phrase has a high TF frequency in one document but rarely appears in other documents, it is considered to have good class-discriminating ability and is suitable for classification.

[0137] Term frequency (TF) represents the frequency with which a term (keyword) appears in a text.

[0138] formula: Where n ijThe denominator is the number of times the word appears in file j, and the sum of the total number of occurrences of all words in file j is:

[0139] Inverse Document Frequency (IDF) is the number of documents in a document set in which a word appears. The fewer documents containing a term t, the larger the IDF, indicating that the term has a good ability to distinguish categories.

[0140] Where |D| is the total number of documents in the corpus. |{j:t i ∈d j}| indicates that the word t is included. i Number of files (i.e., n) i,j (Number of files ≠ 0). If the word is not in the corpus, the denominator will be 0. Therefore, in general, 1+|{j:ti∈dj}| is used, that is: The reason for adding 1 to the denominator is to avoid the denominator being 0.

[0141] A high word frequency within a specific file and a low document frequency for that word across the entire document set can result in a high-weighted TF-IDF. Therefore, TF-IDF tends to filter out common words and retain important ones, leading to the final formula: TF-IDF = TF * IDF.

[0142] Since this application is related to application logs, data mining work needs to be done in advance. Documents that record error information are integrated into a document set, and commonly used feature words are assigned weights to better locate keywords and phrases.

[0143] Furthermore, this application supports programs running on different middleware, such as traditional and cloud nodes. Based on the differences in log data structures, matching relationships are searched from the database using a knowledge graph and then categorized.

[0144] This application, starting from intelligent operation and maintenance, can connect and interoperate application-specific logs, universal error reports, and corresponding solutions, and has the following advantages:

[0145] 1. Fill the gap in application-side log monitoring. A unified intelligent operations and maintenance solution addresses the pain point of applications running on different platforms operating independently, providing more reliable support for application transformation.

[0146] 2. Fully consider the shortcomings in current operation and maintenance work, and strengthen the professionalization of operation and maintenance. By automating log extraction and analysis, reduce the occurrence of various problems caused by unclear division of responsibilities or human error, and promote the development of production management towards a more refined direction.

[0147] 3. Combining DevOps (a combination of Development and Operations, a set of processes, methods and systems used to promote communication, collaboration and integration between development (application / software engineering), technical operations and quality assurance (QA) departments), a decision tree model is built to achieve artificial intelligence scheduling of automation tools, which greatly shortens the operation and maintenance emergency response time and reduces the risk of production accidents.

[0148] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0149] Based on the same inventive concept, this application also provides a log processing apparatus for implementing the log processing method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more log processing apparatus embodiments provided below can be found in the limitations of the log processing method described above, and will not be repeated here.

[0150] In one embodiment, such as Figure 7 As shown, a log processing apparatus is provided, including: a first extraction module 710, a second extraction module 720, a determination module 730, and a search module 740, wherein:

[0151] The first extraction module 710 is used to extract information from the log to be processed and obtain keyword information from the log to be processed.

[0152] The second extraction module 720 is used to determine the middleware category corresponding to the log to be processed, and to extract the keyword information again according to the middleware category to obtain the target keyword information that matches the middleware category; the middleware category is the category of the server used to deploy the application corresponding to the log to be processed.

[0153] The determination module 730 is used to determine the fault information corresponding to the log to be processed based on the target keyword information;

[0154] The lookup module 740 is used to search a pre-built decision tree based on the fault information to obtain the decision result corresponding to the log to be processed.

[0155] In one embodiment, the first extraction module 710 is specifically used to extract multiple target sample error keywords from the keyword information library based on the weights of sample error keywords in the pre-built keyword information library; the keyword information library stores multiple sample error keywords and the weights of multiple sample error keywords; according to each target sample error keyword, information extraction processing is performed on the log to be processed to obtain keyword information in the log to be processed.

[0156] In one embodiment, the above-mentioned apparatus further includes a weight determination module, used to obtain multiple sample error keywords in the sample log; count the occurrence times of each sample error keyword in the sample log, and the sum of the occurrence times of each sample error keyword; for any sample error keyword, obtain the ratio of the occurrence times of the sample error keyword to the sum of the occurrence times, as the weight of the sample error keyword.

[0157] In one embodiment, the determining module 730 is specifically used to process the target keyword information through a trained classification model to obtain the fault information of the log to be processed; the classification model is trained using sample data with the target keyword information as input and the fault information as output.

[0158] In one embodiment, the determining module 730 is further configured to determine the scope of influence information of the log to be processed; and based on the scope of influence information and the target keyword information, determine the fault information corresponding to the log to be processed.

[0159] In one embodiment, the search module 740 is specifically used to sort the various fault indicators in the fault information according to a predetermined fault level; the fault information includes multiple fault indicators and the indicator values ​​of the multiple fault indicators; according to the fault level of each fault indicator from high to low, the decision tree is searched layer by layer to obtain the leaf node corresponding to the fault information in the decision tree; the decision result corresponding to the leaf node is determined as the decision result corresponding to the log to be processed.

[0160] Each module in the aforementioned log processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can invoke and execute the operations corresponding to each module.

[0161] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 8As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a log processing method. The display screen can be an LCD screen or an e-ink display screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0162] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0163] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0164] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0165] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0166] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0167] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0168] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0169] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A log processing method, characterized in that, The method includes: The log to be processed is processed to extract keyword information from the log to be processed. Based on the middleware category corresponding to the log to be processed, the keyword information is extracted again to obtain target keyword information that matches the middleware category; the middleware category is the category of the server used to deploy the application corresponding to the log to be processed. Based on the target keyword information, the fault information corresponding to the log to be processed is determined; Based on the fault information, a pre-built decision tree is searched to obtain the decision result corresponding to the log to be processed; Based on the middleware category corresponding to the log to be processed, the keyword information is extracted and processed again to obtain target keyword information that matches the middleware category. This includes: using knowledge graph technology, extracting and processing the keyword information again based on the middleware category to obtain error information and potential fault information that match the middleware category corresponding to the log to be processed, which are used as the target keyword information. The step of extracting information from the log to be processed to obtain keyword information includes: extracting multiple target sample error keywords from the keyword information library based on the weights of sample error keywords in the pre-built keyword information library; the keyword information library stores multiple sample error keywords and the weights of the multiple sample error keywords; and performing information extraction processing on the log to be processed according to each of the target sample error keywords to obtain keyword information in the log to be processed. The method further includes: inputting the target keyword information and fault impact range information into the trained classification model, and processing the classification model to obtain the fault level and production accident probability of the log to be processed, which are used as the fault information of the log to be processed; The decision result includes whether the application needs to be restarted or not.

2. The method according to claim 1, characterized in that, The weights of the sample error keywords are determined in the following way: Retrieve multiple sample error keywords from the sample logs; The occurrence count of each sample error keyword in the sample log is counted separately, and the sum of the occurrence counts of each sample error keyword is calculated. For any of the sample error keywords, the ratio of the number of times the sample error keyword appears to the sum of the number of times it appears is used as the weight of the sample error keyword.

3. The method according to claim 1, characterized in that, The step of determining the fault information corresponding to the log to be processed based on the target keyword information includes: The target keyword information is processed by the trained classification model to obtain the fault information of the log to be processed; the classification model is trained by sample data with target keyword information as input and fault information as output.

4. The method according to claim 1, characterized in that, The step of determining the fault information corresponding to the log to be processed based on the target keyword information further includes: Determine the scope of impact information of the logs to be processed; Based on the impact range information and the target keyword information, the fault information corresponding to the log to be processed is determined.

5. The method according to claim 1, characterized in that, The fault information includes information on multiple fault indicators; the step of searching a pre-constructed decision tree based on the fault information to obtain the decision result corresponding to the log to be processed includes: The fault indicators in the fault information are sorted according to a predetermined fault level; the fault information includes multiple fault indicators and the indicator values ​​of the multiple fault indicators. According to the fault level of each fault indicator in descending order, the decision tree is searched layer by layer to obtain the leaf node corresponding to the fault information in the decision tree. The decision result corresponding to the leaf node is determined and used as the decision result corresponding to the log to be processed.

6. A log processing device, characterized in that, The device includes: The first extraction module is used to extract information from the log to be processed, and obtain the keyword information in the log to be processed. The second extraction module is used to determine the middleware category corresponding to the log to be processed, and to extract the keyword information again according to the middleware category to obtain target keyword information that matches the middleware category; the middleware category is the category of the server used to deploy the application corresponding to the log to be processed. The determination module is used to determine the fault information corresponding to the log to be processed based on the target keyword information; The search module is used to search a pre-built decision tree based on the fault information to obtain the decision result corresponding to the log to be processed; Based on the middleware category corresponding to the log to be processed, the keyword information is extracted and processed again to obtain target keyword information that matches the middleware category. This includes: using knowledge graph technology, extracting and processing the keyword information again based on the middleware category to obtain error information and potential fault information that match the middleware category corresponding to the log to be processed, which are used as the target keyword information. The step of extracting information from the log to be processed to obtain keyword information includes: extracting multiple target sample error keywords from the keyword information library based on the weights of sample error keywords in the pre-built keyword information library; the keyword information library stores multiple sample error keywords and the weights of the multiple sample error keywords; and performing information extraction processing on the log to be processed according to each of the target sample error keywords to obtain keyword information in the log to be processed. The device is further configured to input the target keyword information and the fault impact range information into the trained classification model, and process the classification model to obtain the fault level and the probability of production accident occurrence of the log to be processed, which are used as the fault information of the log to be processed. The decision result includes whether the application needs to be restarted or not.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images