Problem log processing method and device and product

Abstract

The invention provides a problem log processing method and device and a product, and relates to the technical field of data processing, and the method comprises the steps: obtaining target log data; performing link tracking on the target log data to obtain context information of the target log data; inputting the target log data and the context information of the target log data into a pre-trained problem log processing model to obtain a feature value of the target log data; analyzing an adapter by utilizing a pre-trained scheme, and determining a solution mapped by the characteristic value of the target log data; wherein the problem log processing model and the scheme analysis adapter are obtained through training by taking sample problem log data as a training sample and taking a sample feature value and a sample solution corresponding to the sample problem log data as labels.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a problem log processing method, device and product. Background Art

[0002] At present, in distributed systems, due to the large scale of the system and the high complexity of services, the number of system logs is huge and the structure is complex. When the log collection tool captures an exception, no matter how big or small the log problem is, manual intervention is required to manually analyze and locate the problem to solve the abnormal situation and ensure the normal operation of the system, which requires a lot of time and manpower costs. Therefore, it is urgent to provide a problem log processing method, device and product to improve the efficiency of problem log processing. Summary of the invention

[0003] In view of the above problems, the embodiments of the present application provide a problem log processing method, device and product to improve the processing efficiency of the problem log.

[0004] A first aspect of an embodiment of the present application provides a problem log processing method, including: Get target log data; Performing link tracing on the target log data to obtain context information of the target log data; Inputting the target log data and the context information of the target log data into a pre-trained problem log processing model to obtain a feature value of the target log data; Determine, using a pre-trained solution parsing adapter, a solution mapped to the feature value of the target log data; The problem log processing model and the solution parsing adapter are obtained through training using sample problem log data as training samples and using sample feature values ​​and sample solutions corresponding to the sample problem log data as labels.

[0005] In a possible implementation, the link tracing of the target log data to obtain context information of the target log data includes: According to the Trace ID of the target log data, reversely capture a plurality of associated log data related to the target log data; According to a predefined regular matching rule, context information of the target log data is obtained by matching from the multiple associated log data, and the context information of the target log data at least includes: a problem log protocol, an interface path, and an input parameter.

[0006] In a possible implementation manner, the solution is the first solution, or the second solution; the method further includes: In the case where the solution to the feature value mapping of the target log data is the first solution, the first solution is pushed to the user, or the first solution is executed regularly and the execution result is pushed to the user; the first solution at least includes: a function solution program; When the solution of the feature value mapping of the target log data is the second solution, the second solution is pushed to the user, and the second solution at least includes: suspected feature value information and context information of the target log data.

[0007] In a possible implementation manner, when the solution is the second solution, the method further includes: Get the user's processing results; Determine, according to the processing result, a corrected characteristic value of the target log data and a corrected solution; The target log data is used as a new training sample, and the corrected feature value and the corrected solution are used as labels to train the problem log processing model and the solution parsing adapter again.

[0008] In a possible implementation manner, the problem log processing model and the solution parsing adapter are trained through the following steps: Collecting the sample problem log data; Performing link tracing on the sample problem log data to obtain context information of the sample problem log data; Classifying the sample problem log data according to the sample problem log data and context information of the sample problem log data, and determining sample feature values ​​and sample solutions of the sample problem log; Extracting features from the sample problem log data and context information of the sample problem log data to obtain error report data features; Inputting the error data features into the machine model to be trained to obtain a predicted feature value; Inputting the predicted feature value into the adapter to be trained to obtain a prediction solution; Calculating a loss function value according to the predicted feature value, the predicted solution, the sample feature value, and the sample solution; Repeat the above steps until the preset number of training times is reached or the loss function value converges, and then end the training to obtain the trained problem log processing model and the solution parsing adapter.

[0009] In a possible implementation, before extracting features from the sample problem log data and context information of the sample problem log data, the method further includes: Performing data cleaning on the sample problem log data and context information of the sample problem log data, wherein the data cleaning includes one or more of the following: missing value filling, outlier detection, data deduplication, and data normalization; Normalizing the sample problem log data and the context information of the sample problem log data by using a Z-Score normalization method; The extracting features of the sample problem log data and context information of the sample problem log data includes: By means of statistical feature extraction and text feature extraction, feature extraction is performed on the sample problem log data and context information of the sample problem log data to obtain the error data feature.

[0010] In a possible implementation, after the training is completed, the method further includes: For the trained problem log processing model and the solution parsing adapter, perform one or more of the following monitoring: model data monitoring, parameter monitoring, log monitoring, data distribution monitoring, and error monitoring.

[0011] A second aspect of the embodiment of the present application provides a problem log processing device, which is applied to the problem log processing method proposed in the first aspect of the embodiment of the present application, and the device includes: A data acquisition module is used to acquire target log data; A link tracking module, used to perform link tracking on the target log data to obtain context information of the target log data; A model processing module, used for inputting the target log data and the context information of the target log data into a pre-trained problem log processing model to obtain a feature value of the target log data; A solution determination module is used to use a pre-trained solution parsing adapter to determine the solution mapped by the characteristic value of the target log data; wherein the problem log processing model and the solution parsing adapter are trained using sample problem log data as training samples and using sample characteristic values ​​and sample solutions corresponding to the sample problem log data as labels.

[0012] A third aspect of an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the steps of the problem log processing method described in the first aspect of the embodiment of the present application are implemented.

[0013] A fifth aspect of an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the problem log processing method proposed in the first aspect of the embodiment of the present application are implemented.

[0014] A sixth aspect of an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the problem log processing method proposed in the first aspect of the embodiment of the present application.

[0015] An embodiment of the present application provides a problem log processing method, device and product, the method comprising: acquiring target log data; performing link tracing on the target log data to obtain context information of the target log data; inputting the target log data and the context information of the target log data into a pre-trained problem log processing model to obtain feature values ​​of the target log data; using a pre-trained solution parsing adapter to determine a solution mapped to the feature value of the target log data; wherein the problem log processing model and the solution parsing adapter are obtained by training using sample problem log data as training samples and using sample feature values ​​and sample solutions corresponding to the sample problem log data as labels.

[0016] The specific beneficial effects are: The embodiment of the present application uses link tracking technology to monitor the system, obtain extended information (context information of the target log data), and analyze the log using a machine learning algorithm (pre-trained problem log processing model and solution parsing adapter), which can quickly and accurately identify and locate the problem and provide an effective solution. Log analysis and troubleshooting are achieved in an automated manner, which can reduce labor costs, improve problem log processing efficiency, and thus improve the overall reliability and stability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0018] Figure 1 It is a flowchart of a problem log processing method provided by an embodiment of the present application; Figure 2 This is a schematic diagram of a problem log processing flow provided by an embodiment of the present application; Figure 3It is a schematic diagram of a training process for a problem log processing model and a solution parsing adapter provided in an embodiment of the present application; Figure 4 It is a structural schematic diagram of a problem log processing device provided in an embodiment of the present application; Figure 5 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] The exemplary embodiments of the present application will be described in more detail below in conjunction with the accompanying drawings in the embodiments of the present application. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to enable the scope of the present application to be fully communicated to those skilled in the art.

[0020] Distributed system technology: With the continuous expansion of computer application scale, traditional single-machine systems can no longer meet the needs of large-scale data processing and analysis. Therefore, distributed systems came into being. Distributed systems have the advantages of high reliability, high concurrency, and scalability, and are widely used in log collection, storage, and analysis.

[0021] Big data analysis technology: With the advent of the big data era, how to quickly and effectively process massive amounts of data has become a hot topic. Big data analysis technology includes a variety of algorithms such as data mining, machine learning, and deep learning, which can extract valuable information from large amounts of data to provide support for business decisions.

[0022] Artificial intelligence technology: Artificial intelligence technology has developed rapidly in recent years, among which machine learning is the most representative technology. Machine learning algorithms can automatically generate models by learning from large amounts of data to achieve tasks such as data classification and prediction. Machine learning technology is a key technology of artificial intelligence. It enables computer systems to analyze and learn from large amounts of data to automatically make predictions and decisions. The core of machine learning technology is to use algorithms and statistical methods to allow computers to "learn" patterns from data without direct human intervention, and to design models based on this to make autonomous decisions.

[0023] Tracking technology: Tracking is a technology used to monitor distributed systems. For example, Zipkin and Jaeger monitor distributed log systems. It can track the transmission process of requests in the system and record key information such as the processing time and error information of each component to help troubleshoot.

[0024] Log collection technology: By deploying log collection agents on each node of the system, the logs generated by the nodes are sent to the log center or storage system in real time for subsequent offline analysis and processing.

[0025] Currently, in distributed systems, due to the large scale of the system and the high complexity of services, the number of system logs is huge and the structure is complex. When the log collection tool captures an anomaly, no matter how big or small the log problem is, manual intervention is required to manually analyze and locate the problem to solve the anomaly and ensure the normal operation of the system, which requires a lot of time and manpower costs. In addition, the existing link tracking system can only collect, analyze and count historical data, and cannot automatically predict risks based on input or alarms.

[0026] In view of the above problems, the embodiments of the present application propose a problem log processing method, device and product to improve the problem log processing efficiency. The following is a detailed description of a problem log processing method, device and product provided by the embodiments of the present application through some embodiments and their application scenarios in conjunction with the accompanying drawings.

[0027] A first aspect of an embodiment of the present application provides a problem log processing method. The problem log processing method of the first aspect is introduced below through Sections 1.1, 1.2, 1.3 and 1.4 respectively.

[0028] 1.1 A brief overview of the problem log processing method: Reference Figure 1 , Figure 1 A flowchart of a problem log processing method is shown. Figure 1 As shown, the method includes: Step S101, obtaining target log data.

[0029] Specifically, log data of each microservice or each module in the current system (which may be a distributed system) is collected as target log data to be detected and processed. In this embodiment, one or more log collection tools such as ELK (Elasticsearch, Logstash, Kibana), Splunk, Graylog, etc. can be used to collect log data.

[0030] Log data collection can be performed once every certain period of time, and the newly generated log data collected can be used as the target log data. Alternatively, for each microservice or module, each time a new log data is detected and generated, it can be used as the target log data, and abnormality detection and processing can be performed on it according to the subsequent steps S102-S104.

[0031] Step S102: perform link tracing on the target log data to obtain context information of the target log data.

[0032] Specifically, when obtaining the target log data, the Trace ID of the target log data can also be obtained, and the trace ID can be a GUID or other unique ID. Then, using the link tracking technology, according to the Trace ID, the relevant information (i.e., context information) of all microservices or modules related to the target log data is reversely captured. In this embodiment, open source tools such as Jaeger and Zipkin can be selected to perform link tracking. Link tracking technology can track the transfer of requests between different services and provide detailed information about the request, such as response time, error rate, delay, etc., and can achieve distributed tracking of the entire request life cycle, so that users can better understand the interaction between services and capture requests and exceptions. Traditional link tracking technology requires the installation of an agent on each node, and the original code needs to be modified, and deployment and maintenance are more troublesome. The solution based on the machine learning algorithm in the embodiment of the present application can be analyzed offline by collecting link information (target log data and context information), without installing an agent on each node, so it is more scalable.

[0033] The embodiment of the present application connects the relevant log data of all microservices associated with Trace ID in series through link tracking. Exemplarily, the log data 1-1 of microservice A is a normal log, the log data 1-2 of microservice B is an abnormal log, and the log data 1-3 of microservice C is a normal log, and the three log data belong to the same link, that is, they are associated with the same Trace ID. If the log data 1-3 of microservice C is detected for abnormality alone, it is difficult to find abnormal situations. Therefore, the embodiment of the present application proposes to reversely capture the context information in the entire link based on Trace ID through link tracking. For example, for the log data 1-2 of microservice B, the log data 1-1 of microservice A and the log data 1-3 of microservice C are captured as context information through link tracking. By adding extended information (i.e., context information), they are input into the problem log processing model for analysis, which is conducive to subsequent problem location analysis in abnormal detection and improves the accuracy of abnormal detection of log data.

[0034] Step S103: input the target log data and the context information of the target log data into a pre-trained problem log processing model to obtain a feature value of the target log data.

[0035] Specifically, the problem log processing model can be a logistic regression algorithm model or a random forest algorithm model. In addition, the integrated learning algorithm Bagging can be used to improve the generalization ability and accuracy of the model. Exemplarily, the problem log processing model can use part of the random forest algorithm to construct a decision tree, and use the constructed decision tree to perform a classification task on the input log data (target log data and context information of the target log data) to obtain a classification result y. The classification result y can be shown in the following formula: ; Where T represents the number of decision trees in the model, y i is the classification result predicted by the i-th decision tree. Different types of classification results y represent different corresponding solutions, that is, different function solvers. The output of the problem log processing model can be in the form of a feature value or a combination of multiple feature values. Different feature values ​​correspond to different processing methods, that is, solutions. For example, when the feature value is 1, it corresponds to solution A, and when the feature value is 2, it corresponds to solution B.

[0036] Step S104, using a pre-trained solution parsing adapter to determine the solution mapped to the feature value of the target log data. The problem log processing model and the solution parsing adapter are trained using sample problem log data as training samples and sample feature values ​​and sample solutions corresponding to the sample problem log data as labels.

[0037] Specifically, according to the feature values ​​output by the model, a custom solution parsing adapter is used to map to an executable solution (i.e., the corresponding function solver). The solution parsing adapter includes: a variety of different solutions, and the mapping relationship between different feature values ​​and solutions. The solution can be a series of function solvers summarized during the model training process.

[0038] In a possible implementation manner, the solution is the first solution, or the second solution; the method further includes: In the case where the solution to the feature value mapping of the target log data is the first solution, the first solution is pushed to the user, or the first solution is executed regularly and the execution result is pushed to the user; the first solution at least includes: a function solution program; When the solution of the feature value mapping of the target log data is the second solution, the second solution is pushed to the user, and the second solution at least includes: suspected feature value information and context information of the target log data.

[0039] Reference Figure 2, Figure 2 A schematic diagram of a problem log processing flow is shown, such as Figure 2 As shown in the figure, the final output of the model has the following two solutions: In one case, the target log data can be identified (the specific type of the target log data can be determined) and processed through the problem log processing model and the solution parsing adapter, and a first solution (which includes a corresponding function solution program) can be generated. The first solution can be pushed directly to the user, and the user can determine whether to execute the function solution program in the first solution. Alternatively, the first solution can be executed at a scheduled time, and the function solution program in the first solution can be automatically executed within a first time period (for example, restarting), and the execution result can then be pushed to the user.

[0040] Another situation is that the target log data cannot be identified (the specific type of the target log data cannot be determined) and processed through the problem log processing model and the solution parsing adapter, then a second solution is generated, and the second solution contains suspected feature value information (i.e., the possible type of the target log data predicted by the problem log processing model, i.e., the feature value), and the context information of the target log data, so that the user can locate the problem and handle the exception according to the information in the second solution. Exemplarily, when the feature value output by the problem log processing model is 0, it means that the model cannot identify the target log data, and the corresponding second solution can be found according to the solution parsing adapter; when the feature value output by the problem log processing model is 1, it means that the model can identify the target log data, and the corresponding first solution A (i.e., function solution program a) can be found according to the solution parsing adapter; when the feature value output by the problem log processing model is 2, it means that the model can identify the target log data, and the corresponding first solution B (i.e., function solution program b) can be found according to the solution parsing adapter.

[0041] The embodiment of the present application uses link tracking technology to monitor the system, obtain extended information (context information of the target log data), and analyze the log using a machine learning algorithm (pre-trained problem log processing model and solution parsing adapter), which can quickly and accurately identify and locate the problem and provide an effective solution. Log analysis and troubleshooting are achieved in an automated manner, which can reduce labor costs, improve problem log processing efficiency, and thus improve the overall reliability and stability of the system.

[0042] Traditional link tracking technology can only provide the request transmission process, and cannot directly identify the specific fault point. However, the solution based on the machine learning algorithm in the embodiment of the present application can learn a large amount of data (training samples), and use the trained problem log processing model and solution analysis adapter to deeply explore the hidden rules and anomalies in the link information, so as to locate the fault point more accurately. In addition, common fault types are summarized into automated processing solutions for automated processing, which improves the efficiency of problem log processing.

[0043] 1.2 Further optimization of link tracking: Considering that link tracing will obtain all relevant log data of the entire link (i.e., all log data associated with the Trace ID), some of the log data is useless for abnormality inspection and processing. If it is input into the model together, it will affect the analysis performance of the model. In this embodiment, by pre-defining regular matching rules, useful key information (i.e., context information) is extracted from the information obtained through link tracing, and useless information is reduced, thereby improving the processing efficiency of problem logs.

[0044] In a possible implementation, the step S102 of performing link tracing on the target log data to obtain context information of the target log data includes: Step S1021: reversely capture a plurality of associated log data related to the target log data according to the Trace ID of the target log data.

[0045] Step S1022: According to a predefined regular matching rule, context information of the target log data is obtained by matching from the multiple associated log data, where the context information of the target log data at least includes: a problem log protocol, an interface path, and input parameters.

[0046] Specifically, link tracking generally connects the relevant logs of all microservices in series, and normal log data is not helpful for model problem location processing, and the increased amount of data will only cause unnecessary interference to the model calculation. Therefore, the embodiment of the present application predefines regular matching rules (regular matching and enumeration of suspected behavior attributes and feature attributes) to extract useful contextual information (problem log protocol, path and input parameters) from multiple related log data obtained by link tracking according to the rules. Specifically, the problem log protocol is an interface call protocol, such as: POST, PUT, DELETE, GET, etc.; interface path, such as: openapi / v1 / vpc, etc.; input parameters such as: name=test Resource, ip Version=4, etc. The specific regular matching rules (and the key values ​​of the regular matching rules) can be selected according to the actual business needs of the system, and are not limited in this embodiment.

[0047] 1.3 Specific training methods for problem log processing model and solution parsing adapter: The embodiment of the present application adopts a supervised learning algorithm to train a model (problem log processing model and solution parsing adapter) based on known training samples (sample problem log data) and labels (sample feature values ​​and sample solutions corresponding to the sample problem log data), so that the model learns to classify based on the input data to infer the corresponding solution, that is, the function solver (the function solver is not universal and needs to be used in combination with specific business scenarios and feature value sets).

[0048] 1.3.1 Training process of problem log processing model and solution parsing adapter: In a possible implementation manner, the problem log processing model and the solution parsing adapter are trained through the following steps: Step S201, collecting the sample problem log data.

[0049] Reference Figure 3 , Figure 3 A schematic diagram of the training process of the problem log processing model and the solution parsing adapter is shown, Figure 3 As shown, first, it is necessary to collect sample problem log data, that is, obtain training samples. Specifically, the problem log data of each microservice or each module in the distributed system can be collected, including at least event, error, failure, and exception type problem log data as sample problem log data.

[0050] In addition, for the collected sample problem log data, the Trace ID generated by each microservice is classified and identified, and the sample problem log data is initially classified. Specifically, the obtained sample problem log data can be divided into multiple different types according to the log keyword attribute: creation exception type, update exception type, null pointer exception type, etc., to facilitate subsequent behavior tracking and feature data classification (helping to classify the sample problem log data in step S203 and determine the sample feature value corresponding to the sample problem log).

[0051] In addition, when classifying sample problem log data, it can be initially divided into two categories: according to the pre-defined multiple log data types, if it is determined that the sample problem log data belongs to one of the multiple log data types, the sample problem log data is classified into the designated first data warehouse; if it is determined that the sample problem log data does not belong to any of the multiple log data types, the sample problem log data is classified into the designated second data warehouse, and the log data in the second data warehouse are all of unknown type. For the log data in the second data warehouse, it can be further cleaned and refined according to the trained model, and the processing method can be extracted and mapped.

[0052] Step S202: perform link tracing on the sample problem log data to obtain context information of the sample problem log data.

[0053] Specifically, the method is the same as the above-mentioned step S102, and according to the Trace ID of the sample problem log data determined in step S201, multiple associated sample log data related to the sample problem log data are reversely captured. Then, according to the predefined regular matching rules, the context information of the sample problem log data is matched from the multiple associated sample log data, and the context information of the sample problem log data includes at least: problem log protocol, interface path and input parameters. The embodiment of the present application utilizes predefined regular matching rules to extract useful context information (problem log protocol, interface path and input parameters) from multiple associated sample log data obtained by link tracking, so as to reduce the impact of useless data on model training and improve model performance.

[0054] Step S203: classify the sample problem log data according to the sample problem log data and context information of the sample problem log data, and determine sample feature values ​​and sample solutions of the sample problem logs.

[0055] Specifically, in this step, the classification of the sample problem log data is further refined (that is, a more refined classification can be performed on the basis of the first data warehouse and the second data warehouse after the initial classification in step S201), and attribute labels such as suspected behavior attributes and characteristic attributes are added to the sample problem log data. The purpose is to facilitate the subsequent preprocessing of the data. The source of attributes and the specific processing of problems also come from the training and customization of the model. A subsequent induction is used to assist the early tracking, thereby improving efficiency. Each type corresponds to a sample characteristic value and a sample solution. Therefore, for each sample problem log data, the corresponding label is determined according to its corresponding classification (that is, the corresponding sample characteristic value and sample solution are determined), and the training sample and the label of the training sample are obtained.

[0056] In addition, before machine model training, we conduct exploration of classification variables, exploration of error keywords, correlation analysis, and cluster analysis to gain a deep understanding of the characteristics of problem log data, such as error type, level, anomaly points, trend, etc., to provide reference for subsequent model training, anomaly detection, and problem location. Different system scenarios can use different exploration methods and set different processing solutions.

[0057] Step S204: extract features from the sample problem log data and the context information of the sample problem log data to obtain error data features. In this step, effective extraction of data features can provide strong support for subsequent data analysis and modeling.

[0058] Step S205, input the error data features into the machine model to be trained to obtain a predicted feature value. In this embodiment, the machine model to be trained can be a random forest model or a logistic regression model.

[0059] Step S206: input the predicted feature value into the adapter to be trained to obtain a prediction solution.

[0060] Step S207, calculating a loss function value according to the predicted feature value, the predicted solution, the sample feature value and the sample solution.

[0061] Specifically, with the goal of making the predicted feature values ​​output by the model closer to the sample feature values ​​in the label, and the predicted solution output by the adapter closer to the sample solution in the label, the loss function value is calculated, and the gradient is returned based on the calculated loss function value to update the parameters of the machine model and the adapter. The updating of the adapter parameters includes: updating the parameters related to the mapping relationship between the feature values ​​in the adapter and the solution, and updating the parameters related to the function solver in the solution.

[0062] Step S208, repeat the above steps until the preset number of training times is reached or the loss function value converges, and then the training is terminated to obtain the trained problem log processing model and the solution analysis adapter.

[0063] After each training is completed, the above steps S201-S207 are repeated to obtain new training samples (sample problem log data) and labels (corresponding sample feature values ​​and sample solutions) and to perform the next round of training on the model until the number of trainings reaches the preset number of trainings, or the calculated loss function value gradually decreases until it stabilizes within a certain threshold range, and then the training is stopped to obtain a trained problem log processing model and solution parsing adapter. Therefore, the embodiment of the present application uses a supervised learning method to perform model training using the collected problem log data, so that the model learning performs predictive analysis based on the input data, quickly identifies potential problems that may exist, and gives corresponding warning information (i.e., solutions), so that corresponding measures can be taken in time before a failure occurs, thereby achieving efficient problem log processing.

[0064] 1.3.2 Preprocess and normalize the collected sample problem log data: During the model training process, before executing step S204, the collected sample problem log data and context information are first preprocessed and normalized to ensure the accuracy and consistency of the data required subsequently and improve the model training efficiency.

[0065] In a possible implementation, before extracting features from the sample problem log data and context information of the sample problem log data, the method further includes: The sample problem log data and the context information of the sample problem log data are cleaned, and the data cleaning includes one or more of the following: missing value filling, outlier detection, data deduplication, and data normalization. Specifically, by using the set logical sequence rules such as missing value filling, outlier detection, data deduplication, data normalization, and matching, errors, missing values, duplicate data removal, detection of the range of abnormal data, and inconsistent unit data in the collected log data are detected and corrected, so as to extract key information from the sample problem log data and context information, and enter the data warehouse in a standardized and normalized format, so as to facilitate the use of a hash algorithm to deduplicate the data after logical processing, so as to ensure the accuracy and consistency of the data required for subsequent model training.

[0066] The sample problem log data and the context information of the sample problem log data are normalized by the Z-Score standardization method. Specifically, the original data is converted into a distribution with zero mean and unit variance by the Z-Score standardization method, and the formula is: (x - μ) / σ. Wherein, x represents the original data (i.e., the sample problem log data and the context information of the sample problem log data), and μ and σ are the mean and standard deviation of the sample, respectively. Standardization effectively reduces the difficulty of comparison between different eigenvalues, reduces the errors caused by eigenvalues, and improves the accuracy of the algorithm. Different normalization methods can be used for different data type scenarios and specific situations, such as minimum-maximum, decimal calibration standardization, etc., which can be selected independently according to different system business scenarios.

[0067] The step S204, extracting features from the sample problem log data and context information of the sample problem log data, includes: By means of statistical feature extraction and text feature extraction, feature extraction is performed on the sample problem log data and the context information of the sample problem log data to obtain the error data features. Specifically, different environmental systems can adopt different feature extraction methods and settings of suspected attributes according to specific application situations. Effective extraction of data features can help provide strong support for subsequent data analysis and modeling.

[0068] In this embodiment, feature engineering technology is used, that is, feature extraction, transformation, selection and crossover are used to understand the basic information, feature distribution and outlier conditions of the collected sample problem log data, and the feature values ​​are refined and classified based on the integrated information to continuously improve the accuracy of the model.

[0069] 1.3.3 After training is completed, monitor the model quality: In a possible implementation, after the training is completed, the method further includes: For the trained problem log processing model and the solution parsing adapter, perform one or more of the following monitoring: model data monitoring, parameter monitoring, log monitoring, data distribution monitoring, and error monitoring.

[0070] In the embodiment of the present application, after obtaining the trained model (problem log processing model and solution parsing adapter) through steps S201-208, in the actual application process, that is, in the process of executing steps S101-S104 multiple times, one or more model quality monitoring methods of model data monitoring, parameter monitoring, log monitoring, data distribution, and error monitoring can be adopted for the quality of the model to maintain the reliability and stability of the long-term operation of the model. In addition, according to actual needs, an alarm can be issued for a specific feature value based on the monitoring information.

[0071] 1.3.4 Tune the hyperparameters of the model: In the embodiment of the present application, Bayesian theory can also be used to adaptively adjust the hyperparameter search direction of the model (problem log processing model and solution parsing adapter) according to the existing information, so as to efficiently find the optimal solution, that is, the optimal hyperparameter combination, to improve the model performance of the problem log processing model. Among them, Bayesian theory refers to: P(A|B) = P(B|A) * P(A) / P(B), where P(A|B) represents the probability of event A occurring when event B is known to occur; P(B|A) represents the probability of event B occurring when event A occurs; P(A) and P(B) represent the prior probabilities of event A and event B, respectively.

[0072] 1.4 In the subsequent model application process, the eigenvalues ​​are further subdivided to further optimize the model: In this embodiment, in the process of applying the trained model (problem log processing model and solution parsing adapter) according to the method of steps S101-S104 to realize problem log processing, the feature value can be further subdivided (the type of problem log data can be more finely divided) according to information such as processing results to further optimize the model. Based on the self-learning ability of the machine learning algorithm, it can continuously learn from the collected log data and automatically update the model to better adapt to the changing business scenarios and environments, so that the model can realize more accurate identification and processing of problem logs.

[0073] In a possible implementation manner, when the solution is the second solution, the method further includes: Get the user's processing results; Determine, according to the processing result, a corrected characteristic value of the target log data and a corrected solution; The target log data is used as a new training sample, and the corrected feature value and the corrected solution are used as labels to train the problem log processing model and the solution parsing adapter again.

[0074] In this embodiment, when the solution is the second solution, it means that the current model cannot identify the target log data and cannot find the corresponding function solution program to realize automatic processing, so the relevant information of the target log data (i.e., the second solution) is pushed to the user, and the user locates the problem log according to the information therein, and manually completes the anomaly detection and problem log processing. In this case, the processing result can be obtained, and the processing result may include: the function solution program (corrected solution) used by the user, and the attribute information, type, or corresponding feature value (i.e., corrected feature value) of the target log data determined by the user.

[0075] According to the processing results, the characteristic values ​​can be further subdivided, that is, the types of log data can be subdivided. Exemplarily, after completing the training according to the model training process described in 1.3 (steps S201-S208), there are 10 types of problem log data, and the corresponding model output characteristic values ​​will have 10: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9. The remaining types of log data cannot be recognized by the model, and the model will obtain the second solution accordingly; according to the method described in Section 1.4, the model is retrained so that the model learns to recognize and process the new type of log data, and the characteristic values ​​are further subdivided. Exemplarily, the type of log data with a characteristic value of 1 is further subdivided to obtain three types of log data with characteristic values ​​of 1-1, 1-2, and 1-3 respectively. By further subdividing the characteristic values, the model's ability to classify and recognize log data is improved to obtain a more accurate and reliable solution.

[0076] Based on the processing results, new training samples (target log data) and corresponding labels (corrected feature values ​​and corrected solutions) are generated. Using the newly generated training samples and labels, a new round of training is performed on the model (problem log processing model and solution parsing adapter) according to the model training method described in Section 1.3 (steps S201-S208). Specifically: feature extraction is performed on the target log data and the context information of the target log data to obtain error data features; the error data features are input into the problem log processing model to obtain predicted feature values; the predicted feature values ​​are input into the solution parsing adapter to obtain predicted solutions; the loss function value is calculated based on the predicted feature values, the predicted solutions, the corrected feature values ​​and the corrected solutions; and the parameters of the problem log processing model and the solution parsing adapter are updated based on the loss function value.

[0077] The embodiment of the present application uses link tracking technology to monitor the logs of the distributed system, optimizes the link tracking technology, classifies the feature values ​​in a step-by-step manner through model training, and uses machine learning algorithms to automatically analyze the log data, which can quickly and accurately identify and locate the problem and provide an effective solution. Log analysis and troubleshooting are realized in an automated manner, which reduces labor costs and improves efficiency, thereby improving the reliability and stability of the entire system.

[0078] The second part of the embodiment of the present application provides a problem log processing device, which is applied to the problem log processing method proposed in the first aspect of the embodiment of the present application, referring to Figure 4 , Figure 4 A schematic diagram of the structure of a problem log processing device is shown. Figure 4 As shown, the device comprises: A data acquisition module is used to acquire target log data; A link tracking module, used to perform link tracking on the target log data to obtain context information of the target log data; A model processing module, used for inputting the target log data and the context information of the target log data into a pre-trained problem log processing model to obtain a feature value of the target log data; A solution determination module is used to use a pre-trained solution parsing adapter to determine the solution mapped by the characteristic value of the target log data; wherein the problem log processing model and the solution parsing adapter are trained using sample problem log data as training samples and using sample characteristic values ​​and sample solutions corresponding to the sample problem log data as labels.

[0079] In a possible implementation, the link tracking module includes: A first link tracing submodule, configured to reversely capture a plurality of associated log data related to the target log data according to the Trace ID of the target log data; The second link tracking submodule is used to match the context information of the target log data from the multiple associated log data according to a predefined regular matching rule, and the context information of the target log data at least includes: problem log protocol, interface path and input parameter. In a possible implementation manner, the solution is the first solution, or the second solution; the device further includes: A first solution determination submodule is used for, when the solution of the feature value mapping of the target log data is the first solution, pushing the first solution to the user, or executing the first solution at a fixed time and pushing the execution result to the user; the first solution at least includes: a function solution program; The second solution determination submodule is used to push the second solution to the user when the solution of the feature value mapping of the target log data is the second solution, and the second solution at least includes: suspected feature value information and context information of the target log data.

[0080] In a possible implementation manner, when the solution is the second solution, the device further includes: The processing result acquisition module is used to obtain the user's processing results; A label determination module, used to determine the corrected feature value and the corrected solution of the target log data according to the processing result; The retraining module is used to use the target log data as new training samples, the corrected feature values ​​and the corrected solutions as labels, and train the problem log processing model and the solution parsing adapter again.

[0081] In a possible implementation, the device includes: a model training module, configured to obtain the problem log processing model and the solution parsing adapter through training through the following steps: Collecting the sample problem log data; Performing link tracing on the sample problem log data to obtain context information of the sample problem log data; Classifying the sample problem log data according to the sample problem log data and context information of the sample problem log data, and determining sample feature values ​​and sample solutions of the sample problem log; Extracting features from the sample problem log data and context information of the sample problem log data to obtain error report data features; Inputting the error data features into the machine model to be trained to obtain a predicted feature value; Inputting the predicted feature value into the adapter to be trained to obtain a prediction solution; Calculating a loss function value according to the predicted feature value, the predicted solution, the sample feature value, and the sample solution; Repeat the above steps until the preset number of training times is reached or the loss function value converges, and then end the training to obtain the trained problem log processing model and the solution parsing adapter.

[0082] In a possible implementation, before extracting features from the sample question log data and context information of the sample question log data, the model training module is further used to: Performing data cleaning on the sample problem log data and context information of the sample problem log data, wherein the data cleaning includes one or more of the following: missing value filling, outlier detection, data deduplication, and data normalization; Normalizing the sample problem log data and the context information of the sample problem log data by using a Z-Score normalization method; The extracting features of the sample problem log data and context information of the sample problem log data includes: By means of statistical feature extraction and text feature extraction, feature extraction is performed on the sample problem log data and context information of the sample problem log data to obtain the error data feature.

[0083] In a possible implementation, the device further includes: The model monitoring module is used to perform one or more of the following monitoring on the problem log processing model and the solution parsing adapter after the training is completed: model data monitoring, parameter monitoring, log monitoring, data distribution monitoring, and error monitoring.

[0084] The present application also provides an electronic device, referring to Figure 5 , Figure 5 Schematic diagram of the structure of the electronic device proposed in the embodiment of the present application. Figure 5 As shown, the electronic device 100 includes: a memory 110 and a processor 120. The memory 110 and the processor 120 are connected via a bus communication. A computer program is stored in the memory 110. The computer program can be run on the processor 120, thereby implementing the steps of the problem log processing method described in the first aspect of the embodiment of the present application.

[0085] The embodiment of the present application also provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of the problem log processing method described in the first aspect of the embodiment of the present application are implemented.

[0086] The embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the problem log processing method as described in the first aspect of the embodiment of the present application.

[0087] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0088] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices, electronic devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0089] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0090] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0091] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0092] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0093] The above is a detailed introduction to a problem log processing method, device and product provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A problem log processing method, characterized in that: The method comprises: Get target log data; Performing link tracing on the target log data to obtain context information of the target log data; Inputting the target log data and the context information of the target log data into a pre-trained problem log processing model to obtain a feature value of the target log data; Determine, using a pre-trained solution parsing adapter, a solution mapped to the feature value of the target log data; The problem log processing model and the solution parsing adapter are obtained through training using sample problem log data as training samples and using sample feature values ​​and sample solutions corresponding to the sample problem log data as labels.

2. The problem log processing method according to claim 1, characterized in that: The link tracking of the target log data to obtain context information of the target log data includes: According to the Trace ID of the target log data, reversely capture a plurality of associated log data related to the target log data; According to a predefined regular matching rule, context information of the target log data is obtained by matching from the multiple associated log data, and the context information of the target log data at least includes: a problem log protocol, an interface path, and an input parameter.

3. The problem log processing method according to claim 1, characterized in that: The solution is the first solution, or the second solution; the method further includes: In the case where the solution to the feature value mapping of the target log data is the first solution, the first solution is pushed to the user, or the first solution is executed regularly and the execution result is pushed to the user; the first solution at least includes: a function solution program; When the solution of the feature value mapping of the target log data is the second solution, the second solution is pushed to the user, and the second solution at least includes: suspected feature value information and context information of the target log data.

4. The problem log processing method according to claim 3 is characterized in that: In the case where the solution is the second solution, the method further includes: Get the user's processing results; Determine, according to the processing result, a corrected characteristic value of the target log data and a corrected solution; The target log data is used as a new training sample, and the corrected feature value and the corrected solution are used as labels to train the problem log processing model and the solution parsing adapter again.

5. The problem log processing method according to claim 1, characterized in that: The problem log processing model and the solution parsing adapter are trained through the following steps: Collecting the sample problem log data; Performing link tracing on the sample problem log data to obtain context information of the sample problem log data; Classifying the sample problem log data according to the sample problem log data and context information of the sample problem log data, and determining sample feature values ​​and sample solutions of the sample problem log; Extracting features from the sample problem log data and context information of the sample problem log data to obtain error report data features; Inputting the error data features into the machine model to be trained to obtain a predicted feature value; Inputting the predicted feature value into the adapter to be trained to obtain a prediction solution; Calculating a loss function value according to the predicted feature value, the predicted solution, the sample feature value, and the sample solution; Repeat the above steps until the preset number of training times is reached or the loss function value converges, and then end the training to obtain the trained problem log processing model and the solution parsing adapter.

6. The problem log processing method according to claim 5, characterized in that: Before extracting features from the sample problem log data and context information of the sample problem log data, the method further includes: Performing data cleaning on the sample problem log data and context information of the sample problem log data, wherein the data cleaning includes one or more of the following: missing value filling, outlier detection, data deduplication, and data normalization; Normalizing the sample problem log data and the context information of the sample problem log data by using a Z-Score normalization method; The extracting features of the sample problem log data and context information of the sample problem log data includes: By means of statistical feature extraction and text feature extraction, feature extraction is performed on the sample problem log data and context information of the sample problem log data to obtain the error data feature.

7. The problem log processing method according to claim 5, characterized in that: After the training is completed, the method further includes: For the trained problem log processing model and the solution parsing adapter, perform one or more of the following monitoring: model data monitoring, parameter monitoring, log monitoring, data distribution monitoring, and error monitoring.

8. A problem log processing device, characterized in that: The problem log processing method applied to any one of claims 1 to 7, wherein the device comprises: A data acquisition module is used to acquire target log data; A link tracking module, used to perform link tracking on the target log data to obtain context information of the target log data; A model processing module, used for inputting the target log data and the context information of the target log data into a pre-trained problem log processing model to obtain a feature value of the target log data; A solution determination module is used to use a pre-trained solution parsing adapter to determine the solution mapped by the characteristic value of the target log data; wherein the problem log processing model and the solution parsing adapter are trained using sample problem log data as training samples and using sample characteristic values ​​and sample solutions corresponding to the sample problem log data as labels.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, wherein: When the processor executes the computer program, the steps of the problem log processing method as proposed in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the steps of the problem log processing method proposed in any one of claims 1 to 7 are implemented.

Images