A problem log processing method, device and product

Abstract

The application provides a problem log processing method, device and product, relates to the technical field of data processing, and comprises the following steps: obtaining target log data; performing link tracking on the target log data to obtain context information of the target log data; inputting the target log data and the context information of the target log data into a pre-trained problem log processing model to obtain a characteristic value of the target log data; and determining a solution mapped by the characteristic value of the target log data by using a pre-trained solution analysis adapter; wherein the problem log processing model and the solution analysis adapter are obtained by training with sample problem log data as a training sample, and with a sample characteristic value and a sample solution corresponding to the sample problem log data as labels.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a problem log processing method, device and product. BACKGROUND

[0002] At present, in a distributed system, due to factors such as large system scale and complex service complexity, the number of system logs is large and the structure is complex. After an abnormality is captured by a log collection tool, manual intervention is required regardless of the size of the log problem, and the problem needs to be manually analyzed and located to solve the abnormality and ensure the normal operation of the system, which requires a large amount of time and labor cost. Therefore, it is urgent to provide a problem log processing method, device and product to improve the processing efficiency of the problem log. SUMMARY

[0003] In view of the above problems, the embodiments of the present application provide a problem log processing method, device and product to improve the processing efficiency of the problem log.

[0004] In a first aspect, the embodiments of the present application provide a problem log processing method, comprising:

[0005] obtaining target log data;

[0006] performing link tracking on the target log data to obtain context information of the target log data;

[0007] inputting the target log data and the context information of the target log data into a pre-trained problem log processing model to obtain a feature value of the target log data;

[0008] determining a solution mapped by the feature value of the target log data by using a pre-trained solution analysis adapter;

[0009] The problem log processing model and the solution analysis adapter are trained by taking sample problem log data as a training sample, and taking a sample feature value and a sample solution corresponding to the sample problem log data as a label.

[0010] In a possible implementation, the performing link tracking on the target log data to obtain the context information of the target log data comprises:

[0011] capturing a plurality of associated log data related to the target log data in reverse according to a Trace ID of the target log data;

[0012] According to a predefined regular matching rule, context information of the target log data is matched from the plurality of associated log data, and the context information of the target log data at least includes a problem log protocol, an interface path, and an input parameter.

[0013] In a possible implementation, the solution is a first solution, or a second solution; and the method further includes:

[0014] In a case where the solution of the feature value mapping of the target log data is the first solution, the first solution is pushed to a user, or the first solution is executed at a timing and a result of the execution is pushed to the user; and the first solution at least includes a function solution procedure.

[0015] In a case where the solution of the feature value mapping of the target log data is the second solution, the second solution is pushed to the user, and the second solution at least includes suspected feature value information and context information of the target log data.

[0016] In a possible implementation, in a case where the solution is the second solution, the method further includes:

[0017] A processing result of the user is acquired.

[0018] According to the processing result, a corrected feature value of the target log data and a corrected solution are determined.

[0019] The target log data is taken as a new training sample, and the corrected feature value and the corrected solution are taken as labels, and the problem log processing model and the solution analysis adapter are trained again.

[0020] In a possible implementation, the problem log processing model and the solution analysis adapter are trained through the following steps:

[0021] The sample problem log data is collected.

[0022] Link tracking is performed on the sample problem log data to obtain context information of the sample problem log data.

[0023] According to the sample problem log data and the context information of the sample problem log data, the sample problem log data is classified to determine sample feature values and sample solutions of the sample problem log data.

[0024] Feature extraction is performed on the sample problem log data and the context information of the sample problem log data to obtain error data features.

[0025] inputting the error data features into a machine model to be trained to obtain predicted feature values;

[0026] inputting the predicted feature values into an adapter to be trained to obtain a predicted solution;

[0027] calculating a loss function value according to the predicted feature values, the predicted solution, the sample feature values and the sample solution;

[0028] repeating the above steps until a preset training number or the loss function value converges, ending the training, and obtaining the trained problem log processing model and the solution analysis adapter.

[0029] In a possible implementation, before the feature extraction of the sample problem log data and the context information of the sample problem log data, the method further includes:

[0030] performing data cleaning on the sample problem log data and the context information of the sample problem log data, the data cleaning including one or more of the following: missing value filling, outlier detection, data deduplication, data normalization;

[0031] performing data normalization on the sample problem log data and the context information of the sample problem log data by a Z-Score standardization method;

[0032] the feature extraction of the sample problem log data and the context information of the sample problem log data includes:

[0033] performing feature extraction on the sample problem log data and the context information of the sample problem log data by a statistical feature extraction and a text feature extraction to obtain the error data features.

[0034] In a possible implementation, after the training ends, the method further includes:

[0035] performing one or more of the following monitoring on the trained problem log processing model and the solution analysis adapter: model data monitoring, parameter monitoring, log monitoring, data distribution monitoring, error monitoring.

[0036] The second aspect of the embodiments of the present application provides a problem log processing device, applied to the problem log processing method proposed in the first aspect of the embodiments of the present application, and the device includes:

[0037] a data acquisition module, configured to acquire target log data;

[0038] a link tracking module configured to perform link tracking on the target log data to obtain context information of the target log data;

[0039] a model processing module configured to input the target log data and the context information of the target log data into a pre-trained problem log processing model to obtain a feature value of the target log data;

[0040] a solution determination module configured to determine a solution mapped by the feature value of the target log data by using a pre-trained solution analysis adapter; wherein the problem log processing model and the solution analysis adapter are obtained by training with sample problem log data as a training sample, and with a sample feature value and a sample solution corresponding to the sample problem log data as labels.

[0041] The third aspect of the embodiments of the present application provides an electronic device, which includes a memory and a processor, the memory stores a computer program capable of running on the processor, and the processor implements the steps of the problem log processing method of the first aspect of the embodiments of the present application when executing the computer program.

[0042] The fifth aspect of the embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the problem log processing method proposed in the first aspect of the embodiments of the present application.

[0043] The sixth aspect of the embodiments of the present application provides a computer program product, which includes a computer program, and the computer program is executed by a processor to implement the steps of the problem log processing method proposed in the first aspect of the embodiments of the present application.

[0044] The embodiments of the present application provide a problem log processing method, device and product, the method including: obtaining target log data; performing link tracking on the target log data to obtain context information of the target log data; inputting the target log data and the context information of the target log data into a pre-trained problem log processing model to obtain a feature value of the target log data; determining a solution mapped by the feature value of the target log data by using a pre-trained solution analysis adapter; wherein the problem log processing model and the solution analysis adapter are obtained by training with sample problem log data as a training sample, and with a sample feature value and a sample solution corresponding to the sample problem log data as labels.

[0045] The specific beneficial effects are as follows:

[0046] The embodiment of the application adopts link tracking technology to monitor the system, obtains extended information (context information of target log data), analyzes the log by using a machine learning algorithm (pre-trained problem log processing model and solution analysis adapter), can quickly and accurately identify and locate the problem, and provides an effective solution. Through an automated manner, log analysis and troubleshooting are realized, which can reduce labor cost, improve problem log processing efficiency, and thus improve the overall reliability and stability of the system. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings needed to be used in the description of the embodiments of the application will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0048] Figure 1 is a step flow chart of a problem log processing method provided by the embodiment of the application;

[0049] Figure 2 is a problem log processing flowchart provided by the embodiment of the application;

[0050] Figure 3 is a training flowchart of a problem log processing model and a solution analysis adapter provided by the embodiment of the application;

[0051] Figure 4 is a structural schematic diagram of a problem log processing device provided by the embodiment of the application;

[0052] Figure 5 is a structural schematic diagram of an electronic device provided by the embodiment of the application. DETAILED DESCRIPTION

[0053] The exemplary embodiments of the application will be described in more detail below with reference to the accompanying drawings of the embodiments of the application. Although the exemplary embodiments of the application are shown in the drawings, it should be understood that the application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the application and to fully convey the scope of the application to those skilled in the art.

[0054] Distributed system technology: with the continuous expansion of computer application scale, the traditional single machine system has been unable to meet the demand of large-scale data processing and analysis. Therefore, distributed system emerges as the times require. Distributed system has the advantages of high reliability, high concurrency, scalability, etc., and is widely used in log collection, storage and analysis.

[0055] Big data analysis technology: With the advent of the era of big data, how to quickly and effectively process massive amounts of data has become a hot topic. Big data analysis technology includes data mining, machine learning, deep learning and other algorithms, which can extract valuable information from large amounts of data to support business decision-making.

[0056] Artificial intelligence technology: Artificial intelligence technology has developed rapidly in recent years, among which machine learning is the most representative technology. Machine learning algorithms can automatically generate models through learning from large amounts of data to achieve classification, prediction and other tasks on data. Machine learning technology is a key technology of artificial intelligence, which enables computer systems to analyze and learn from large amounts of data and automatically make predictions and decisions. The core of machine learning technology is to use algorithms and statistical methods to enable computers to "learn" patterns from data without human intervention, and design models to make autonomous decisions.

[0057] Link tracking technology: Link tracking is a technology used to monitor distributed systems, such as Zipkin, Jaeger, etc. to monitor distributed log systems, which can track the transmission process of requests in the system and record key information such as processing time and error information of each component to help troubleshoot.

[0058] Log collection technology: By deploying log collection agents on each node of the system, the logs generated by the nodes are sent to the log center or storage system in real time for subsequent offline analysis and processing.

[0059] Currently, in a distributed system, due to the large size of the system, the complexity of the service, and other factors, the number of system logs is large, and the structure is complex. When the log collection tool captures an exception, manual intervention is required regardless of the size of the log problem to manually analyze and locate the problem to solve the abnormal situation and ensure the normal operation of the system, which requires a lot of time and labor costs. Moreover, the existing link tracking system can only collect, analyze and count historical data, and cannot automatically predict risks based on input or alarms.

[0060] In view of the above problems, the embodiments of the present application provide a problem log processing method, device and product to improve the problem log processing efficiency. In the following, some embodiments and their application scenarios will be described in detail.

[0061] The first aspect of the embodiments of the present application provides a problem log processing method, and the following sections 1.1, 1.2, 1.3 and 1.4 will introduce the problem log processing method of the first aspect.

[0062] 1.1 Brief overview of the problem log processing method:

[0063] Referring to Figure 1 , Figure 1 A step flowchart of a problem log processing method is shown in Figure 1 , which includes:

[0064] Step S101, obtaining target log data.

[0065] Specifically, log data of each microservice or each module in the current system (which can be a distributed system) is collected as target log data to be detected and processed for abnormalities. In this embodiment, one or more log collection tools such as ELK (Elasticsearch, Logstash, Kibana), Splunk, Graylog, etc. can be used for log data collection.

[0066] The log data collection can be performed once every certain time interval, and the newly generated log data collected is taken as the target log data. Alternatively, for each microservice or module, each time a new log data is detected, it is taken as the target log data and is subjected to abnormality detection and processing according to subsequent steps S102-S104.

[0067] Step S102, link tracking of the target log data to obtain context information of the target log data.

[0068] Specifically, when the target log data is obtained, the Trace ID according to the target log data of the target log data can also be obtained, which can be a GUID or other unique ID. Then, using the link tracking technology, all microservices or modules related information (i.e. context information) related to the target log data is captured in reverse according to the Trace ID. In this embodiment, open source tools such as Jaeger and Zipkin can be selected to perform link tracking. The link tracking technology can track the transmission of requests between different services and provide detailed information about the requests, such as response time, error rate, delay, etc. It can track the entire request lifecycle, so that users can better understand the interaction between services and capture requests and exceptions. Traditional link tracking technology requires installing an agent on each node and modifying the original code, which is more troublesome to deploy and maintain. The scheme based on machine learning algorithm in this embodiment can analyze the collected link information (target log data and context information) offline without installing an agent on each node, so it is more scalable.

[0069] The log data in the related microservices associated with the Trace ID in series through link tracking is captured by the embodiments of the present application. For example, the log data 1-1 of the microservice A is normal log data, the log data 1-2 of the microservice B is abnormal log data, and the log data 1-3 of the microservice C is normal log data, and the three log data belong to the same link, that is, are associated with the same Trace ID. If the log data 1-3 of the microservice C is detected for abnormality alone, it is difficult to find the abnormality. Therefore, the embodiments of the present application propose to capture the context information in the whole link in reverse based on the Trace ID through link tracking, for example, for the log data 1-2 of the microservice B, the log data 1-1 of the microservice A and the log data 1-3 of the microservice C are captured as context information through link tracking, and by increasing the extension information (that is, the context information), the log data is input into the problem log processing model for analysis, which is beneficial to subsequent problem positioning analysis in abnormality detection and improves the accuracy of abnormality detection of the log data.

[0070] In step S103, the target log data and the context information of the target log data are input into a pre-trained problem log processing model to obtain a feature value of the target log data.

[0071] Specifically, the problem log processing model can be a logistic regression algorithm model or a random forest algorithm model, and in addition, an ensemble learning algorithm Bagging can be used to improve the generalization ability and accuracy of the model. For example, the problem log processing model can use a partial random forest algorithm, constructs a decision tree, and uses the constructed decision tree to perform a classification task on the input log data (the target log data and the context information of the target log data) to obtain a classification result y. The classification result y can be as shown in the following formula:

[0072] ;

[0073] wherein T represents the number of decision trees in the model, y i is the classification result predicted by the i-th decision tree. The type represented by the classification result y is different, and the corresponding solution is different, that is, the function solution procedure is different. The result output by the problem log processing model can be in the form of a feature value or a combination of multiple feature values, and different feature values correspond to different processing methods, that is, solutions. For example, when the feature value is 1, it corresponds to solution A, and when the feature value is 2, it corresponds to solution B.

[0074] Step S104, using a pre-trained solution analysis adapter, determine the solution mapped by the feature value of the target log data. Wherein, the problem log processing model and the solution analysis adapter are trained with sample problem log data as training samples, and the sample feature values and sample solutions corresponding to the sample problem log data as labels.

[0075] Specifically, according to the feature value output by the model, through the self-defined solution analysis adapter, it is mapped to an executable solution (i.e. corresponding function solution procedure). Wherein, the solution analysis adapter includes: a plurality of different solutions, and the mapping relationship between different feature values and solutions. The solution can be a series of function solution procedures summarized in the model training process.

[0076] In one possible implementation, the solution is a first solution, or a second solution; the method further comprises:

[0077] In the case where the solution mapped by the feature value of the target log data is the first solution, the first solution is pushed to the user, or the first solution is executed at a timing and the execution result is pushed to the user; the first solution at least includes: function solution procedure;

[0078] In the case where the solution mapped by the feature value of the target log data is the second solution, the second solution is pushed to the user, and the second solution at least includes: suspected feature value information and context information of the target log data.

[0079] Reference Figure 2 , Figure 2 A problem log processing flow diagram is shown, as shown in Figure 2 The final output of the model solution exists in the following two cases:

[0080] One case is that through the problem log processing model and the solution analysis adapter, the target log data can be identified (the specific type to which the target log data belongs can be determined) and processed, a first solution (which contains a corresponding function solution procedure) is generated, and the first solution can be directly pushed to the user. The user determines whether to execute the function solution procedure in the first solution, or executes the first solution at a timing, automatically executes the function solution procedure (such as restarting) in the first solution within a first time period, and then pushes the execution result to the user.

[0081] In another case, if the target log data cannot be identified (it cannot be determined to which specific type the target log data belongs) and processed by the problem log processing model and the solution analysis adapter, a second solution is generated, which contains suspected feature value information (i.e., the possible type to which the target log data belongs predicted by the problem log processing model, i.e., the feature value) and context information of the target log data, so as to facilitate the user to locate the problem and handle the exception according to the information in the second solution. For example, when the feature value output by the problem log processing model is 0, it means that the model cannot identify the target log data, and the corresponding second solution can be found according to the solution analysis adapter; when the feature value output by the problem log processing model is 1, it means that the model can identify the target log data, and the corresponding first solution A (i.e., function solution procedure a) can be found according to the solution analysis adapter; when the feature value output by the problem log processing model is 2, it means that the model can identify the target log data, and the corresponding first solution B (i.e., function solution procedure b) can be found according to the solution analysis adapter.

[0082] The embodiments of the present application monitor the system by using link tracking technology to obtain extended information (context information of the target log data), analyze the log by using machine learning algorithm (pre-trained problem log processing model and solution analysis adapter), and can quickly and accurately identify and locate the problem, and provide effective solutions. Through the automatic way of realizing log analysis and troubleshooting, the human cost can be reduced, the problem log processing efficiency can be improved, and thus the overall reliability and stability of the system can be improved.

[0083] The traditional link tracking technology can only provide the transmission process of the request, and cannot directly identify the specific fault point. However, the solution based on the machine learning algorithm of the embodiments of the present application can learn a large amount of data (training samples), use the trained problem log processing model and solution analysis adapter to deeply mine the hidden rules and abnormalities in the link information, and thus more accurately locate the fault point. Moreover, the common fault types are summarized as automatic processing solutions for automatic processing, and the problem log processing efficiency is improved.

[0084] 1.2 Further optimization of link tracking:

[0085] Considering that link tracking will obtain all related log data of the entire link (i.e., all log data associated with the Trace ID), part of the log data is useless for abnormality checking and processing, and if it is input into the model together, it will affect the analysis performance of the model. In the embodiments, by predefining a regular matching rule, useful key information (i.e., context information) is extracted from the information obtained by link tracking, and useless information is reduced, so as to improve the processing efficiency of the problem log.

[0086] In a possible implementation, the step S102 of performing link tracking on the target log data to obtain context information of the target log data comprises:

[0087] The step S1021 comprises: capturing, according to a Trace ID of the target log data, a plurality of associated log data related to the target log data.

[0088] The step S1022 comprises: matching, according to a predefined regular matching rule, the context information of the target log data from the plurality of associated log data, wherein the context information of the target log data at least comprises a problem log protocol, an interface path and an input parameter.

[0089] Specifically, link tracking generally concatenates the relevant log data of all microservices, and normal log data is not helpful for the model to perform problem positioning processing, and the increased data volume will only cause unnecessary interference to the model calculation. Therefore, the embodiment of the present application defines a regular matching rule (regular matching and enumeration of suspected behavior attributes and feature attributes) in advance, so as to extract useful context information (problem log protocol, path and input parameter) from the plurality of associated log data obtained by link tracking according to the rule. Specifically, the problem log protocol is an interface calling protocol, for example: POST, PUT, DELETE, GET, etc.; the interface path is, for example: openapi / v1 / vpc, etc.; and the input parameter is, for example: name=test Resource, ip Version=4, etc. The specific regular matching rule (and the key value of the regular matching rule) can be selected according to the actual business requirements of the system, and is not limited in the embodiment.

[0090] 1.3 A specific training method of the problem log processing model and the scheme analysis adapter:

[0091] The embodiment of the present application adopts a supervised learning algorithm, and trains a model (problem log processing model and scheme analysis adapter) based on known training samples (sample problem log data) and labels (sample feature values and sample solutions corresponding to the sample problem log data), so that the model learns to classify based on input data to infer the corresponding solution, that is, a function solving procedure (the function solving procedure is not universal and needs to be used in combination with a specific business scenario and a feature value set).

[0092] 1.3.1 Training process of the problem log processing model and the scheme analysis adapter:

[0093] In a possible implementation, the problem log processing model and the scheme analysis adapter are trained by the following steps:

[0094] Step S201: Collect the sample problem log data.

[0095] Reference Figure 3 , Figure 3 A schematic diagram of the training process for a problem log processing model and a solution parsing adapter is shown, such as... Figure 3 As shown, the first step is to collect sample problem log data, i.e., obtain training samples. Specifically, problem log data can be collected from various microservices or modules in the distributed system, including at least event, error, fault, and exception types of problem log data, as sample problem log data.

[0096] Furthermore, for the collected sample problem log data, the Trace IDs generated at the beginning of each microservice are classified and identified, and the sample problem log data is initially classified. Specifically, based on the log keyword attributes, the obtained sample problem log data can be divided into several different types, such as creation exception type, update exception type, and null pointer exception type, which facilitates subsequent behavior tracking and feature data classification (helping to classify the sample problem log data in step S203 and determine the sample feature values ​​corresponding to the sample problem logs).

[0097] Furthermore, when classifying sample problem log data, it can be initially divided into two categories: If the sample problem log data belongs to one of the predefined log data types, it is assigned to a designated first data warehouse; if it does not belong to any of the log data types, it is assigned to a designated second data warehouse, where all log data is of unknown type. For the log data in the second data warehouse, further cleaning and refinement can be performed later based on the trained model, extracting and mapping processing methods.

[0098] Step S202: Perform link tracing on the sample problem log data to obtain the context information of the sample problem log data.

[0099] Specifically, according to the Trace ID of the sample problem log data determined in step S201, the method same as that in step S102 is used to reversely capture a plurality of associated sample log data related to the sample problem log data. Then, according to a predefined regular matching rule, context information of the sample problem log data is matched from the plurality of associated sample log data, and the context information of the sample problem log data at least includes a problem log protocol, an interface path and an input parameter. The embodiment of the present application uses the predefined regular matching rule to extract useful context information (problem log protocol, interface path and input parameter) from the plurality of associated sample log data obtained by link tracking, so as to reduce the influence of useless data on model training and improve the performance of the model.

[0100] In step S203, the sample problem log data is classified according to the sample problem log data and the context information of the sample problem log data, and sample feature values and sample solutions of the sample problem log are determined.

[0101] Specifically, the classification of the sample problem log data is further refined in this step (i.e., more refined classification can be performed on the basis of the first data warehouse and the second data warehouse after the initial classification in step S201), and suspected behavior attributes, feature attributes and other attribute labels are added to the sample problem log data, so as to facilitate the preprocessing of the data in the later stage. The source of the attributes and the specific processing of the problems also come from the training and customization of the model, and the subsequent induction is used to assist the tracking in the early stage, so as to improve the efficiency. Each type corresponds to a sample feature value and a sample solution. Therefore, for each sample problem log data, the corresponding label (i.e., the corresponding sample feature value and sample solution) is determined according to the corresponding classification, and the training sample and the label of the training sample are obtained.

[0102] In addition, before the machine model is trained, the exploration of classification variables, the exploration of error keywords, and the correlation analysis and clustering analysis are performed respectively, so as to deeply understand the characteristics of the problem log data, such as error types, levels, abnormal points and trends, and provide a reference for subsequent model training, anomaly detection and problem positioning. Different exploration methods and different processing schemes can be used for different system scenarios.

[0103] In step S204, feature extraction is performed on the sample problem log data and the context information of the sample problem log data, and error data features are obtained. In this step, effective extraction of data features can provide strong support for subsequent data analysis and modeling.

[0104] Step S205, input the error data features into the machine model to be trained to obtain predicted feature values. In this embodiment, the machine model to be trained can be a random forest model or a logistic regression model.

[0105] Step S206, input the predicted feature values into the adapter to be trained to obtain a predicted solution.

[0106] Step S207, calculate a loss function value according to the predicted feature values, the predicted solution, the sample feature values and the sample solution.

[0107] Specifically, the closer the predicted feature values output by the model to the sample feature values in the label, and the closer the predicted solution output by the adapter to the sample solution in the label, the closer the target is. The loss function value is calculated, and the gradient back propagation is performed according to the calculated loss function value to update the parameters of the machine model and the adapter. Wherein, updating the parameters of the adapter includes: updating the related parameters of the mapping relationship between the feature values and the solutions in the adapter, and updating the related parameters of the function solution procedure in the solution.

[0108] Step S208, repeat the above steps until a preset training number or a loss function value is converged, end the training, and obtain the trained problem log processing model and the solution analysis adapter.

[0109] After each training is completed, the above steps S201-S207 are repeated to reacquire new training samples (sample problem log data) and labels (corresponding sample feature values and sample solutions), and the next round of training is performed on the model until the training number reaches the pre-set training number, or the calculated loss function value gradually decreases until it is stable within a certain threshold range, the training is stopped, and the trained problem log processing model and the solution analysis adapter are obtained. Therefore, by using the supervised learning method, the model is trained by using the collected problem log data, the model learns to make prediction analysis based on the input data, the potential problems that may exist are quickly identified, and the corresponding warning information (i.e. solution) is given, so that corresponding measures are taken in time before the fault occurs, and efficient problem log processing is realized.

[0110] 1.3.2 Preprocessing and normalization of the collected sample problem log data:

[0111] In the model training process, before step S204 is performed, the collected sample problem log data and context information are first preprocessed and normalized to ensure the accuracy and consistency of the required data in the subsequent process and improve the model training efficiency.

[0112] In a possible implementation, before feature extraction is performed on the sample problem log data and the context information of the sample problem log data, the method further includes:

[0113] Data cleaning is performed on the sample problem log data and the context information of the sample problem log data, and the data cleaning includes one or more of the following: missing value filling, outlier detection, data deduplication, and data normalization. Specifically, by using a set logical order rule of missing value filling, outlier detection, data deduplication, data normalization, matching, and the like, errors, missing values, repeated data, abnormal data range, and inconsistent unit data in the collected log data are detected and corrected, so that key information of the sample problem log data and the context information is extracted, and the sample problem log data and the context information are in a standardized and normalized format and enter a data warehouse, thereby facilitating deduplication of the data processed according to the logic based on a hash algorithm, to ensure accuracy and consistency of data required for subsequent model training.

[0114] Data normalization is performed on the sample problem log data and the context information of the sample problem log data by using a Z-Score normalization method. Specifically, the Z-Score normalization method is used to convert original data into a distribution with zero mean and unit variance, and the formula is: (x - μ) / σ. Wherein, x represents original data (i.e., the sample problem log data and the context information of the sample problem log data), μ and σ are the mean and standard deviation of the sample, respectively. Standardization effectively reduces the difficulty of comparison between different feature values, reduces errors caused by feature values, and improves the accuracy of the algorithm. Different normalization methods, such as minimum-maximum and decimal scaling normalization, can be used according to different data type scenarios and specific conditions, and can be autonomously selected according to different system business scenarios.

[0115] In the step S204, feature extraction is performed on the sample problem log data and the context information of the sample problem log data, including:

[0116] Feature extraction is performed on the sample problem log data and the context information of the sample problem log data by using statistical feature extraction and text feature extraction, to obtain the error data features. Specifically, different environment systems can use different feature extraction methods and suspected attribute settings according to specific application conditions. Effective extraction of data features can provide strong support for subsequent data analysis and modeling.

[0117] In the embodiment, the basic information, feature distribution, and abnormal value condition of the collected sample problem log data are understood through feature engineering techniques, i.e., through feature extraction, transformation, selection, and intersection, and the feature values are classified and summarized in detail according to the integrated information, so as to constantly improve the precision of the model.

[0118] 1.3.3 After the training is completed, the model quality is monitored:

[0119] In a possible implementation, after the training is completed, the method further includes:

[0120] The trained problem log processing model and the solution analysis adapter are monitored in one or more of the following aspects: model data monitoring, parameter monitoring, log monitoring, data distribution monitoring, and error monitoring.

[0121] In the embodiment, after the trained model (the problem log processing model and the solution analysis adapter) obtained through steps S201-S208 is obtained, one or more of the following model quality monitoring methods can be performed in the actual application process, i.e., the process of performing steps S101-S104 multiple times, to maintain the reliability and stability of the long-term operation of the model. In addition, according to actual needs, specific feature values can be alarmed according to the monitoring information.

[0122] 1.3.4 Hyperparameter tuning of the model:

[0123] In the embodiment, the Bayesian theory can be used to adaptively adjust the hyperparameter search direction of the model (the problem log processing model and the solution analysis adapter) according to the existing information, so as to efficiently find the optimal solution, i.e., the optimal hyperparameter combination, to improve the model performance of the problem log processing model. The Bayesian theory is P(A|B) = P(B|A) * P(A) / P(B), where P(A|B) represents the probability of event A occurring under the condition that event B occurs; P(B|A) represents the probability of event B occurring under the condition that event A occurs; and P(A) and P(B) represent the prior probabilities of events A and B, respectively.

[0124] 1.4 In the subsequent model application process, the feature values are further subdivided to further optimize the model:

[0125] In the embodiment, in the process of applying the trained models (the problem log processing model and the solution analysis adapter) to realize the problem log processing according to the method of steps S101-S104, the characteristic values can be further subdivided (more fine division is performed on the types of the problem log data) according to the processing result and other information, so as to further optimize the models. Based on the self-learning ability of the machine learning algorithm, the models can be continuously learned from the collected log data and automatically updated, so as to better adapt to the changing business scenarios and environments, and make the models realize more accurate identification and processing of the problem log.

[0126] In a possible implementation, in the case where the solution is the second solution, the method further includes:

[0127] obtaining a processing result of the user;

[0128] determining a corrected characteristic value of the target log data and a corrected solution according to the processing result;

[0129] training the problem log processing model and the solution analysis adapter again by taking the target log data as a new training sample and taking the corrected characteristic value and the corrected solution as labels.

[0130] In the embodiment, when the solution is the second solution, it indicates that the current model cannot recognize the target log data and cannot find the corresponding function solution program to realize automatic processing, so the related information (i.e., the second solution) of the target log data is pushed to the user, and the user manually completes the problem log positioning, abnormality detection and problem log processing according to the information. In this case, the processing result can be obtained, which can include the function solution program (corrected solution) used by the user, and the attribute information, the type or the corresponding characteristic value (i.e., the corrected characteristic value) of the target log data determined by the user.

[0131] According to the processing result, the feature value can be further subdivided, that is, the type of log data is subdivided. For example, after the model training process (steps S201-S208) described in 1.3 is completed, there are 10 types of problem log data, and the corresponding feature value output by the model will have 10: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, and the remaining types of log data cannot be identified by the model, and the model will correspondingly obtain a second solution; according to the method described in section 1.4, the model is retrained, so that the model learns to identify and process new types of log data, and further subdivides the feature value. For example, for the type of log data with a feature value of 1, further subdivision is performed to obtain three types of log data with feature values of 1-1, 1-2, and 1-3. By further subdividing the feature value, the classification and recognition ability of the model for log data is improved to obtain more accurate and reliable solutions.

[0132] Based on the processing result, new training samples (target log data) and corresponding labels (corrected feature values and corrected solutions) are generated. Using the newly generated training samples and labels, the model (problem log processing model and solution analysis adapter) is trained again according to the model training method (steps S201-S208) described in section 1.3. Specifically: feature extraction is performed on the target log data and the context information of the target log data to obtain error data features; the error data features are input into the problem log processing model to obtain predicted feature values; the predicted feature values are input into the solution analysis adapter to obtain predicted solutions; the loss function value is calculated according to the predicted feature values, the predicted solutions, the corrected feature values, and the corrected solutions; and the parameters of the problem log processing model and the solution analysis adapter are updated according to the loss function value.

[0133] The embodiments of the present application use link tracking technology to monitor the distributed system log, optimize the link tracking technology, gradually refine the classification of feature values through model training, automatically analyze log data using machine learning algorithms, and quickly and accurately identify and locate problems, and provide effective solutions. Through automatic log analysis and troubleshooting, human cost is reduced, efficiency is improved, and the reliability and stability of the entire system are improved.

[0134] The second part of the embodiments of the present application provides a problem log processing device applied to the problem log processing method proposed in the first aspect of the embodiments of the present application, which is described with reference to Figure 4 , Figure 4 A structural schematic diagram of a problem log processing device is shown, as shown in Figure 4 The device includes:

[0135] a data acquisition module configured to acquire target log data;

[0136] a link tracking module configured to perform link tracking on the target log data to obtain context information of the target log data;

[0137] a model processing module configured to input the target log data and the context information of the target log data into a pre-trained problem log processing model to obtain a feature value of the target log data;

[0138] a solution determination module configured to determine a solution to which the feature value of the target log data is mapped by using a pre-trained solution analysis adapter; wherein the problem log processing model and the solution analysis adapter are trained by taking sample problem log data as a training sample, and taking a sample feature value and a sample solution corresponding to the sample problem log data as labels.

[0139] In a possible implementation, the link tracking module comprises:

[0140] a first link tracking submodule configured to reversely capture a plurality of associated log data related to the target log data according to a Trace ID of the target log data;

[0141] a second link tracking submodule configured to match the context information of the target log data from the plurality of associated log data according to a pre-defined regular matching rule, wherein the context information of the target log data at least includes a problem log protocol, an interface path, and an input parameter.

[0142] In a possible implementation, the solution is a first solution, or a second solution; and the apparatus further comprises:

[0143] a first solution determination submodule configured to, in a case where the solution to which the feature value of the target log data is mapped is the first solution, push the first solution to a user, or execute the first solution at a timing and push an execution result to the user; wherein the first solution at least includes a function solution procedure.

[0144] a second solution determination submodule configured to, in a case where the solution to which the feature value of the target log data is mapped is the second solution, push the second solution to the user; wherein the second solution at least includes suspected feature value information and context information of the target log data.

[0145] In a possible implementation, in the case where the solution is the second solution, the apparatus further comprises:

[0146] a processing result obtaining module configured to obtain a processing result of the user;

[0147] a label determining module configured to determine, according to the processing result, a corrected feature value and a corrected solution of the target log data;

[0148] a retraining module configured to take the target log data as a new training sample, and take the corrected feature value and the corrected solution as labels, and retrain the problem log processing model and the solution analysis adapter.

[0149] In a possible implementation, the apparatus comprises a model training module configured to train the problem log processing model and the solution analysis adapter by the following steps:

[0150] collecting the sample problem log data;

[0151] performing link tracking on the sample problem log data to obtain context information of the sample problem log data;

[0152] classifying the sample problem log data according to the sample problem log data and the context information of the sample problem log data, to determine a sample feature value and a sample solution of the sample problem log data;

[0153] extracting features from the sample problem log data and the context information of the sample problem log data to obtain error data features;

[0154] inputting the error data features into a machine model to be trained to obtain a predicted feature value;

[0155] inputting the predicted feature value into an adapter to be trained to obtain a predicted solution;

[0156] calculating a loss function value according to the predicted feature value, the predicted solution, the sample feature value and the sample solution;

[0157] repeating the above steps until a preset training number or the loss function value converges, ending the training, and obtaining the trained problem log processing model and the solution analysis adapter.

[0158] In a possible implementation, before the features are extracted from the sample problem log data and the context information of the sample problem log data, the model training module is further configured to:

[0159] The sample problem log data and context information of the sample problem log data are subjected to data cleaning, which includes one or more of the following: missing value filling, outlier detection, data deduplication, data normalization;

[0160] The sample problem log data and context information of the sample problem log data are subjected to data normalization by a Z-Score standardization method.

[0161] The feature extraction on the sample problem log data and context information of the sample problem log data includes:

[0162] The sample problem log data and context information of the sample problem log data are subjected to feature extraction by statistical feature extraction and text feature extraction, to obtain the error data features.

[0163] In a possible implementation, the apparatus further includes:

[0164] The model monitoring module is configured to, after the training is completed, perform one or more of the following monitoring on the trained problem log processing model and the solution analysis adapter: model data monitoring, parameter monitoring, log monitoring, data distribution monitoring, and error monitoring.

[0165] The embodiments of the present application further provide an electronic device, which refers to Figure 5 , Figure 5 is a structural schematic diagram of the electronic device provided by the embodiments of the present application. As shown in Figure 5 , the electronic device 100 includes a memory 110 and a processor 120, the memory 110 and the processor 120 are communicatively connected through a bus, the memory 110 stores a computer program, the computer program can run on the processor 120, and then the steps of the problem log processing method in the first aspect of the embodiments of the present application are implemented.

[0166] The embodiments of the present application further provide a computer readable storage medium, which stores a computer program / instruction, and the computer program / instruction is executed by a processor to implement the steps of the problem log processing method in the first aspect of the embodiments of the present application.

[0167] The embodiments of the present application further provide a computer program product, which includes a computer program, and the computer program is executed by a processor to implement the steps of the problem log processing method in the first aspect of the embodiments of the present application.

[0168] Each of the embodiments in the present specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same and similar parts of each embodiment can be referred to each other.

[0169] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, apparatuses, electronic device and computer program products according to the embodiments of the present application. It is understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing terminal devices to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal devices, create means for implementing the functions specified in the flowcharts and / or block diagrams block or blocks. Figure 1 one or more flows and / or blocks Figure 1 an apparatus with a function specified in one or more flows and / or blocks.

[0170] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal devices to work in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction apparatus that implements the functions specified in the flowcharts and / or block diagrams block or blocks. Figure 1 one or more flows and / or blocks Figure 1 an apparatus with a function specified in one or more flows and / or blocks.

[0171] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, so that a series of operation steps are performed on the computer or other programmable terminal device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide a process for implementing the functions specified in the flowcharts and / or block diagrams block or blocks. Figure 1 one or more flows and / or blocks Figure 1 an apparatus with a function specified in one or more flows and / or blocks.

[0172] Although the preferred embodiments of the present application have been described, those skilled in the art can make additional changes and modifications to the embodiments once they know the basic inventive concept. Therefore, the appended claims are intended to be interpreted as including all the preferred embodiments and all the changes and modifications falling within the scope of the embodiments of the present application.

[0173] Finally, it is to be understood that the phraseology or terminology such as "first" and "second" etc. used herein is merely intended to differentiate one entity or operation from another entity or operation, without necessarily requiring or implying any actual such relationship or order between such entities or operations. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0174] The above provides a detailed description of the problem log processing method, device and product provided by the present application. The principles and implementation modes of the present application are described by applying specific examples. The above example is only used to help understand the method and core idea of the present application. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A problem log processing method characterized by comprising: The method comprises: acquiring target log data; link tracking the target log data to obtain context information of the target log data; inputting the target log data and the context information of the target log data into a pre-trained problem log processing model to obtain a feature value of the target log data; determining a solution mapped by the feature value of the target log data by using a pre-trained solution analysis adapter; wherein the problem log processing model and the solution analysis adapter are obtained by training with sample problem log data as a training sample, and sample feature values and sample solutions corresponding to the sample problem log data as labels; the link tracking the target log data to obtain the context information of the target log data comprises: capturing a plurality of associated log data related to the target log data in reverse according to a Trace ID of the target log data; matching the context information of the target log data from the plurality of associated log data according to a pre-defined regular matching rule, wherein the context information of the target log data at least comprises a problem log protocol, an interface path and an input parameter; the problem log processing model and the solution analysis adapter are obtained by training through the following steps: collecting the sample problem log data; link tracking the sample problem log data to obtain context information of the sample problem log data; classifying the sample problem log data according to the sample problem log data and the context information of the sample problem log data to determine sample feature values and sample solutions of the sample problem log data; extracting features from the sample problem log data and the context information of the sample problem log data to obtain error data features; inputting the error data features into a machine model to be trained to obtain predicted feature values; inputting the predicted feature values into an adapter to be trained to obtain predicted solutions; calculating a loss function value according to the predicted feature values, the predicted solutions, the sample feature values and the sample solutions; repeating the above steps until a preset training number or a loss function value converges, ending the training to obtain the trained problem log processing model and the solution analysis adapter.

2. The problem log processing method according to claim 1, characterized by, The solution is a first solution, or a second solution; the method further comprises: in the case that the solution mapped by the feature value of the target log data is the first solution, pushing the first solution to the user, or executing the first solution at a timing and pushing the execution result to the user; the first solution at least comprises a function solution procedure; in the case that the solution mapped by the feature value of the target log data is the second solution, pushing the second solution to the user, wherein the second solution at least comprises suspected feature value information and context information of the target log data.

3. The problem log processing method according to claim 2, characterized by, in the case that the solution is the second solution, the method further comprises: acquiring a processing result of a user; According to the processing result, a corrected feature value of the target log data is determined, and a corrected solution is determined; The target log data is taken as a new training sample, and the corrected feature value and the corrected solution are taken as labels to retrain the problem log processing model and the solution analysis adapter.

4. The problem log processing method according to claim 1, characterized by, Before the feature extraction of the sample problem log data and the context information of the sample problem log data, the method further comprises: The sample problem log data and the context information of the sample problem log data are subjected to data cleaning, and the data cleaning comprises one or more of the following: missing value filling, outlier detection, data deduplication, and data normalization; The sample problem log data and the context information of the sample problem log data are subjected to data normalization by a Z-Score standardization method. The feature extraction of the sample problem log data and the context information of the sample problem log data comprises: The sample problem log data and the context information of the sample problem log data are subjected to feature extraction by statistical feature extraction and text feature extraction to obtain error data features.

5. The problem log processing method according to claim 1, characterized by, After the training is completed, the method further comprises: The trained problem log processing model and the solution analysis adapter are subjected to one or more of the following monitoring: model data monitoring, parameter monitoring, log monitoring, data distribution monitoring, and error monitoring.

6. A problem log processing apparatus characterized by comprising: The device is applied to the problem log processing method according to any one of claims 1-5, and the device comprises: a data acquisition module configured to acquire target log data; a link tracking module configured to perform link tracking on the target log data to obtain context information of the target log data; a model processing module configured to input the target log data and the context information of the target log data into a pre-trained problem log processing model to obtain a feature value of the target log data; a solution determination module configured to determine a solution mapped by the feature value of the target log data by using a pre-trained solution analysis adapter; wherein the problem log processing model and the solution analysis adapter are obtained by training with sample problem log data as training samples and sample feature values and sample solutions corresponding to the sample problem log data as labels.

7. An electronic device comprising a memory, a processor, the memory having stored thereon a computer program executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the problem log processing method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer program is stored on the computer readable medium and is executed by the processor to implement the steps of the problem log processing method according to any one of claims 1-5.

Images