Data exception processing method and device, electronic equipment and storage medium

Abstract

The invention provides a data exception processing method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a data tracking identifier of a to-be-detected business system, carrying out the query of the data tracking identifier to obtain exception log data carrying the data tracking identifier, carrying out the keyword extraction of the exception log data, and obtaining a keyword extraction result; determining an error type and a call chain corresponding to the abnormal log data, dividing the abnormal log data in a preset time window to obtain a target log data set, performing anomaly recognition on the target log data set by adopting a pre-trained inference engine to obtain an anomaly confidence coefficient of the target log data set and a processing strategy corresponding to the anomaly confidence coefficient, and executing the processing strategy on the service system. According to the method, the log data are automatically associated through the data tracking identifier, the tedious process of manually associating the log data is avoided, and the inference engine deployed privately is adopted to accurately identify the abnormity of the log data, so that the operation and maintenance efficiency of a service system is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data exception processing method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of information technology, the scale and complexity of business systems continue to increase, and the challenges faced by operations and maintenance personnel are becoming increasingly severe. Especially in fields such as finance and government affairs that have extremely high requirements for data security and privacy, the rapid location and resolution of system anomalies have become the key to ensuring business continuity.

[0003] Currently, when handling exceptions in business systems, operations and maintenance personnel usually need to manually correlate log data and then use a common open source model to analyze the manually correlated log data. Open source models have response delays and are prone to data leakage risks. Therefore, this processing method results in time-consuming exception handling in business systems and low accuracy in exception location, further affecting the operation and maintenance efficiency of business systems. Summary of the Invention

[0004] In view of this, the present invention aims to propose a data anomaly processing method, device, electronic device and storage medium to solve the problem that manual log association is used to handle business system anomalies, which is time-consuming, has a large response delay, has a low accuracy rate in anomaly location, and affects the operation and maintenance efficiency of the business system.

[0005] According to a first aspect of the present invention, a method for handling data anomalies is provided, the method comprising:

[0006] Obtaining a data tracking identifier of the business system to be detected, and using the data tracking identifier to query and obtain abnormal log data carrying the data tracking identifier;

[0007] Perform keyword extraction on the abnormal log data to determine the error type and call chain corresponding to the abnormal log data;

[0008] Dividing the abnormal log data in a preset time window according to the error type and the call chain to obtain a target log data set;

[0009] Using a pre-trained inference engine to perform anomaly identification on the target log data set, and obtain an anomaly confidence level of the target log data set and a processing strategy corresponding to the anomaly confidence level;

[0010] According to the anomaly confidence of the target log data set and the processing strategy corresponding to the anomaly confidence, the processing strategy is executed on the business system.

[0011] Optionally, the step of obtaining a data tracking identifier of the business system to be detected and using the data tracking identifier to query and obtain abnormal log data carrying the data tracking identifier includes:

[0012] Obtaining the data tracking identifier of the business system to be tested input by the user;

[0013] Inputting the data tracking identifier into the log collection layer of the business system to track log data;

[0014] The data tracking identifier is used in the log collection layer to query the log data of the business system to obtain abnormal log data carrying the data tracking identifier.

[0015] Optionally, performing keyword extraction on the abnormal log data to determine the error type and call chain corresponding to the abnormal log data includes:

[0016] Associating the abnormal log data carrying the data tracking identifier to obtain the abnormal log data across services;

[0017] Perform keyword extraction on the cross-service abnormal log data to extract abnormal data and service nodes in the abnormal log data;

[0018] An error type corresponding to the abnormal log data is determined according to the abnormal data, and a call chain of the abnormal log data is determined according to the service node.

[0019] Optionally, dividing the abnormal log data in a preset time window according to the error type and the call chain to obtain a target log data set includes:

[0020] Determine the time information of the abnormal log data, match the time information of the abnormal log data with a preset time window, and determine the abnormal log data in the preset time window;

[0021] Dividing the abnormal log data in the preset time window according to the error type to obtain abnormal log data corresponding to each error type;

[0022] The abnormal log data corresponding to each error type is aggregated according to the call chain to obtain a target log data set.

[0023] Optionally, the using a pre-trained inference engine to perform anomaly identification on the target log data set to obtain an anomaly confidence level of the target log data set and a processing strategy corresponding to the anomaly confidence level includes:

[0024] Pre-generating inference prompt words for anomaly identification of the log data set using a preset template engine; wherein the inference prompt words include high-frequency word extraction and context division;

[0025] Inputting the target log data set and the reasoning prompt words into a pre-trained reasoning engine, using the reasoning engine to perform anomaly identification on the target log data set based on the reasoning prompt words, and obtaining anomaly confidence of the target log data set;

[0026] According to the corresponding relationship between the abnormality confidence and the processing strategy, the processing strategy corresponding to the abnormality confidence is determined.

[0027] Optionally, inputting the target log data set and the inference prompt words into a pre-trained inference engine, using the inference engine to perform anomaly identification on the target log data set based on the inference prompt words, and obtaining anomaly confidence of the target log data set includes:

[0028] Inputting the target log data set and the inference prompt words into a pre-trained inference engine, using the inference engine to extract high-frequency words and perform context division on the target log data set, and identifying abnormal problems in the abnormal log data;

[0029] Matching the abnormal problems in the abnormal log data with known abnormal problems in the inference engine to generate a first confidence level;

[0030] Performing semantic recognition on context information of the abnormal problem in the abnormal log data to generate a second confidence level;

[0031] The first confidence level and the second confidence level are combined to obtain an anomaly confidence level of the target log data set.

[0032] Optionally, executing the processing strategy on the business system according to the anomaly confidence of the target log data set and the processing strategy corresponding to the anomaly confidence includes:

[0033] If the anomaly confidence level of the target log data set is greater than or equal to a first threshold, repairing the business system using a preset repair script;

[0034] If the exception confidence of the target log data set is greater than or equal to the second threshold and less than the first threshold, generating an exception handling form for the business system, and sending the exception handling form to the user for confirmation;

[0035] If the anomaly confidence of the target log data set is less than the second threshold, a prompt message for manual processing is sent to the user.

[0036] According to a second aspect of the present invention, a data anomaly processing device is provided, the device comprising:

[0037] A log query module is used to obtain a data tracking identifier of the business system to be detected, and use the data tracking identifier to query and obtain abnormal log data carrying the data tracking identifier;

[0038] An information extraction module is used to extract keywords from the abnormal log data and determine the error type and call chain corresponding to the abnormal log data;

[0039] A log partitioning module, configured to partition the abnormal log data in a preset time window according to the error type and the call chain to obtain a target log data set;

[0040] An anomaly identification module is used to use a pre-trained inference engine to identify anomalies of the target log data set, obtain an anomaly confidence level of the target log data set and a processing strategy corresponding to the anomaly confidence level;

[0041] The exception handling module is configured to execute the processing strategy on the business system according to the exception confidence of the target log data set and the processing strategy corresponding to the exception confidence.

[0042] According to another aspect of the present invention, there is provided an electronic device, comprising:

[0043] processor;

[0044] a memory for storing instructions executable by the processor;

[0045] The processor is configured to execute the instructions to implement the data exception processing method as described above.

[0046] According to another aspect of the present invention, a readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data exception processing method described above are implemented.

[0047] The data anomaly processing method provided by the embodiment of the present invention obtains the data tracking identifier of the business system to be detected, uses the data tracking identifier to query to obtain abnormal log data carrying the data tracking identifier, performs keyword extraction on the abnormal log data, determines the error type and call chain corresponding to the abnormal log data, divides the abnormal log data in a preset time window according to the error type and call chain, obtains a target log data set, uses a pre-trained reasoning engine to identify anomalies on the target log data set, obtains anomaly confidence of the target log data set and a processing strategy corresponding to the anomaly confidence, and executes the processing strategy on the business system based on the anomaly confidence of the target log data set and the processing strategy corresponding to the anomaly confidence. The embodiment of the present invention automatically associates log data from different services of the business system through the data tracking identifier to form a complete call chain, avoiding the tedious process of manually associating log data, and uses a privately deployed reasoning engine to accurately identify anomalies in the log data, generate a quantitative confidence and a reasonable processing strategy, and automatically execute anomaly processing operations based on the anomaly confidence, ensuring the accuracy and reliability of exception processing, quickly responding to and identifying anomalies, greatly improving the operation and maintenance efficiency of the business system, minimizing the impact of abnormal problems on the business system, and ensuring the stability of the business system.

[0048] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present invention. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0050] Figure 1 This is a flowchart of a method for handling data anomalies provided by an embodiment of the present invention;

[0051] Figure 2 yes Figure 1 Flowchart of step 101 in the data anomaly processing method provided by an embodiment of the present invention;

[0052] Figure 3 yes Figure 1 Flowchart of step 102 in the data anomaly processing method provided by an embodiment of the present invention;

[0053] Figure 4 yes Figure 1Flowchart of step 103 in the data anomaly processing method provided by an embodiment of the present invention;

[0054] Figure 5 yes Figure 1 Flowchart of step 104 in the data anomaly processing method provided by an embodiment of the present invention;

[0055] Figure 6 yes Figure 1 Flowchart of step 105 in the data anomaly processing method provided by an embodiment of the present invention;

[0056] Figure 7 This is an interactive diagram of a data anomaly processing method provided by an embodiment of the present invention;

[0057] Figure 8 1 is a structural diagram of a data anomaly processing device provided by an embodiment of the present invention;

[0058] Figure 9 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, each embodiment of the present invention will be described in detail below with reference to the accompanying drawings. However, it will be understood by those skilled in the art that in each embodiment of the present invention, many technical details are provided to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in this application can be implemented. The division of the following embodiments is for convenience of description and should not constitute any limitation on the specific implementation of the present invention. The various embodiments can be combined with each other and referenced to each other under the premise that there is no contradiction.

[0060] Reference Figure 1 , which shows a flowchart of the steps of a data anomaly processing method provided by an embodiment of the present invention, the method may include:

[0061] Step 101: obtain a data tracking identifier of a business system to be detected, and use the data tracking identifier to query and obtain abnormal log data carrying the data tracking identifier.

[0062] In an embodiment of the present invention, in order to solve the problem of operation and maintenance of the current business system, it is necessary to manually associate log data and then use a general open source model to analyze the manually associated log data, which leads to a long time-consuming exception handling of the business system and low accuracy of exception positioning, further affecting the operation and maintenance efficiency of the business system. This embodiment automatically queries and associates abnormal log data through data tracking identification, combines the deployed private reasoning engine to accurately identify anomalies in log data, generates quantitative confidence and reasonable processing strategies, automatically executes exception handling operations, avoids the tedious process of manually associating log data, and improves the operation and maintenance efficiency of the business system.

[0063] It should be noted that the executor of the embodiment of the present invention is a server that detects and analyzes the business system. The server adopts a dual-driven analysis framework of a data tracking identifier combined with an inference engine deployed in a private cloud to perform anomaly analysis and anomaly location processing on the log data of business systems in various fields. Among them, the data tracking identifier (traceId) is a core identifier used to uniquely identify the flow path of a request in the entire call chain. A data tracking identifier represents a complete business request life cycle and can be transmitted across services, that is, it is transmitted between services in the system with requests or logs to form a call chain. The data tracking identifier is usually represented as a 16 or 32-bit hexadecimal string. For example, in the anomaly analysis of a payment business system of an e-commerce company, TraceId: 5a472e0a-ddee-487c-8a5f-12dafe79f29d. This embodiment does not make specific limitations on this.

[0064] In this embodiment, when the current business system is operated and maintained, logs, metrics, and traces are the three core observability data used to monitor system status and troubleshoot. In this embodiment, a call chain tracing tool can be used to generate and record a unique data tracing identifier for each request during the operation of the business system. For example, when a user initiates an application log or HTTP request, the business system will generate a traceId for the request and embed it into the context of the request. The business system will also record the traceId in the log for subsequent analysis.

[0065] Specifically, the server first obtains the data tracking identifier of the business system to be detected, wherein the data tracking identifier is usually input by the operation and maintenance personnel based on the abnormality analysis requirements of the business system. The operation and maintenance personnel can determine it based on the data tracking identifier of the stored records in the business system. The server queries the original log stream in the business data based on the data tracking identifier, that is, the abnormal log data carrying the data tracking identifier is obtained by querying the data tracking identifier, wherein the abnormal log data is the log data generated in the time window when the business system has an abnormality, so as to further analyze the system abnormality based on the abnormal log data and locate and handle the abnormality. This embodiment can automatically associate the log data generated by different modules and different services through the data tracking identifier to form a complete call chain view. When an abnormality occurs in the business system, the operation and maintenance personnel can quickly locate all log data related to the abnormality through the data tracking identifier, avoiding the tedious process of manually associating log data.

[0066] Step 102: extract keywords from the abnormal log data to determine the error type and call chain corresponding to the abnormal log data.

[0067] In an embodiment of the present invention, after obtaining the abnormal log data carrying the data tracking identifier in each service module in the business system, the server performs keyword extraction on the abnormal log data to determine the error type and call chain corresponding to the abnormal log data. Specifically, the keywords include abnormal data and service nodes. Based on the extracted abnormal data, the error type of the abnormal log data can be determined. The service node can determine the different modules and different services included in the log data of the same data tracking identifier. Among them, the error type includes two levels: error and fatal error. The abnormal data at the error level is a common error that affects the system function, which may include service connection errors, response timeouts, etc. The abnormal data at the fatal error level is an error that causes the system service to be unavailable, such as memory overflow, missing key configuration files, service downtime, etc., which are not specifically limited here.

[0068] Step 103 : Divide the abnormal log data in the preset time window according to the error type and the call chain to obtain a target log data set.

[0069] In an embodiment of the present invention, after determining the abnormal log data, in order to facilitate the inference engine to identify and locate the abnormality, the abnormal log data in the preset time window is divided according to the error type and call chain to obtain a target log data set. The target log data set includes the call chain of abnormal log data of different error types in various time windows, as well as the abnormal log data itself. The preset time window is the time period when the abnormality occurs in the business system.

[0070] Specifically, the time information or timestamp contained in the exception log data can be obtained to determine the time information of the exception log data, and the time window of the business system exception can be matched with the time information of the exception log data to determine the exception log data that matches the preset time window. The exception log data is filtered and divided. First, the exception log data in the preset time window is divided according to the error type to obtain the exception log data corresponding to each error type. Secondly, the exception log data corresponding to each error type is aggregated according to the call chain to obtain the target log data set, that is, the exception log data is associated according to the call chain of the exception log data of different error types in different time windows to obtain the target log data set.

[0071] It should be noted that after obtaining the target log data set, in order to facilitate the inference engine to identify and locate anomalies in the log data, this embodiment encodes the discrete abnormal log data, continuous indicators and call chain information in the log data set, and uniformly encodes them into a matrix format that can be understood by the model, for example, the format of 1641020400, payment service, ERROR, DB_conn_failed@0.87, so that the inference engine can directly process the encoded log data.

[0072] In step 104 , a pre-trained inference engine is used to perform anomaly identification on the target log data set to obtain anomaly confidence of the target log data set and a processing strategy corresponding to the anomaly confidence.

[0073] In an embodiment of the present invention, a pre-trained inference engine is used to analyze the target log data set. Since the server first encodes the log data and converts it into a matrix format that can be understood by the model, the inference engine can directly identify anomalies through a deep learning algorithm and output the anomaly confidence level and corresponding processing strategy.

[0074] Specifically, a preset template engine is used to generate inference prompts for identifying anomalies in a log dataset. The target log dataset and the inference prompts are then fed into the pre-trained inference engine. The inference engine then uses the inference prompts to identify anomalies in the target log dataset, obtaining an anomaly confidence level for the target log dataset and a corresponding processing strategy for the anomaly confidence level. In this embodiment, the anomaly confidence level of the target log dataset reflects the type of error in the anomaly log data and the credibility of the anomaly data. The processing strategies include automatically executing repair operations on the task executor and prompting operations and maintenance personnel for manual intervention, which are not detailed here.

[0075] It should be noted that the pre-trained reasoning engine in this embodiment can be a DeepSeek model. The DeepSeek model is a private deployment container that can ensure the security of data processing and has better Chinese comprehension ability. It is used to perform anomaly analysis, confidence assessment, and processing strategy generation on log data. Specifically, the inference engine, namely the DeepSeek model, is an intelligent analysis tool based on deep learning. Its basic architecture includes an input layer for receiving log data, an encoding layer for using a custom encoder to convert matrix-formatted log data into a high-dimensional feature vector, a feature extraction layer for extracting keyword features in log data and capturing abnormal patterns in logs, an inference layer for inferring keyword features in log data, identifying anomalies and calculating anomaly confidence, and an output layer for outputting anomaly confidence and corresponding processing strategies.

[0076] Step 105 : executing a processing strategy on the business system according to the anomaly confidence level of the target log data set and the processing strategy corresponding to the anomaly confidence level.

[0077] In an embodiment of the present invention, a processing strategy is executed on a business system based on the anomaly confidence of a target log data set and the processing strategy corresponding to the anomaly confidence, that is, different anomaly confidences correspond to different processing strategies, and the anomaly confidence is used to reflect the error type in the anomaly log data and the degree of credibility of the anomaly data. The higher the anomaly confidence, the higher the credibility of the error type of the identified anomaly log data and the specific anomaly problem, and the anomaly repair process can be automatically performed according to the repair strategy of the anomaly problem. On the contrary, the lower the anomaly confidence, the lower the credibility of the error type of the identified anomaly log data and the specific anomaly problem, and manual intervention and control are required.

[0078] The data anomaly processing method provided by the embodiment of the present invention obtains the data tracking identifier of the business system to be detected, uses the data tracking identifier to query to obtain abnormal log data carrying the data tracking identifier, performs keyword extraction on the abnormal log data, determines the error type and call chain corresponding to the abnormal log data, divides the abnormal log data in a preset time window according to the error type and call chain, obtains a target log data set, uses a pre-trained reasoning engine to identify anomalies on the target log data set, obtains anomaly confidence of the target log data set and a processing strategy corresponding to the anomaly confidence, and executes the processing strategy on the business system based on the anomaly confidence of the target log data set and the processing strategy corresponding to the anomaly confidence. The embodiment of the present invention automatically associates log data from different services of the business system through the data tracking identifier to form a complete call chain, avoiding the tedious process of manually associating log data, and uses a privately deployed reasoning engine to accurately identify anomalies in the log data, generate a quantitative confidence and a reasonable processing strategy, and automatically execute anomaly processing operations based on the anomaly confidence, ensuring the accuracy and reliability of exception processing, quickly responding to and identifying anomalies, greatly improving the operation and maintenance efficiency of the business system, minimizing the impact of abnormal problems on the business system, and ensuring the stability of the business system.

[0079] Further, refer to Figure 2 , showing Figure 1 A flowchart of step 101 of a data anomaly processing method is provided. This method is substantially the same as the data anomaly processing method provided in the first embodiment of the present invention. Step 101 may include:

[0080] Step 1011: Obtain the data tracking identifier of the business system to be detected input by the user;

[0081] Step 1012: input the data tracking identifier into the log collection layer of the business system to perform log data tracking;

[0082] Step 1013: Use the data tracking identifier to query the log data of the business system at the log collection layer to obtain abnormal log data carrying the data tracking identifier.

[0083] It should be noted that, in an embodiment of the present invention, the server can obtain the data tracking identifier of the business system to be detected input by the user, wherein the user can input the data tracking identifier of the business system to be detected through the console interface or command line tool. After the business system receives the data tracking identifier input by the user, the data tracking identifier is passed to the log collection layer of the business system. After the log collection layer receives the data tracking identifier, it starts the log tracking function and begins to collect log data related to the data tracking identifier, and finally tracks and queries all log data carrying the data tracking identifier. The user can select different data tracking identifiers for analysis according to specific needs, avoiding errors that may be caused by automatic selection by the system, and can automatically query the log data related to the data tracking identifier.

[0084] For example, in the scenario of abnormal diagnosis of an e-commerce payment system, the user enters the data trace identifier TraceId: 5a472e0a-ddee-487c-8a5f-12dafe79f29d. The log collection layer queries the relevant log: 5a472e0a-ddee-487c-8a5f-12dafe79f29d, payment service, payment service JDBC connection error, service response timeout.

[0085] The embodiment of the present invention can automatically and quickly filter out log data related to the current exception by querying logs through data tracking identifiers, eliminating the need for manual association of log data and improving log data processing efficiency.

[0086] Further, refer to Figure 3 , showing Figure 1 A flowchart of step 102 of a data anomaly processing method is provided. This method is substantially the same as the data anomaly processing method provided in the first embodiment of the present invention. Step 102 may include:

[0087] Step 1021: Correlate the abnormal log data carrying the data tracking identifier to obtain cross-service abnormal log data;

[0088] Step 1022: perform keyword extraction on the cross-service abnormal log data to extract abnormal data and service nodes in the abnormal log data;

[0089] Step 1023: Determine the error type corresponding to the abnormal log data according to the abnormal data, and determine the call chain of the abnormal log data according to the service node.

[0090] It should be noted that in the embodiments of the present invention, since there are multiple service calls in the business system, when an exception occurs in the business system, multiple service nodes may be involved. Therefore, the exception log data carrying the data tracking identifier may be distributed on multiple service nodes. The server associates these exception log data using the same data tracking identifier. For example, assuming that a user request involves service A, service B, and service C, the system will collect exception log data across services A, B, and C, and associate these exception log data to form a cross-service exception log data set. By associating the log data distributed on different service nodes through the data tracking identifier, a complete call chain view is formed, which allows us to understand the context of the exception from a global perspective and quickly locate the exception problem.

[0091] Specifically, the server can use natural language processing (NLP) technology to extract keywords from cross-service abnormal log data. The keywords include abnormal data and service nodes. Based on the extracted abnormal data, the error type of the abnormal log data can be determined. The service node can determine the different services involved in the log data of the same data tracking identifier. Among them, the error type includes two levels: error and fatal error. Error-level abnormal data includes service connection errors, response timeouts, etc., and fatal error-level abnormal data includes memory overflows, missing key configuration files, service downtime, etc. Through keyword extraction, the error type and call chain of the abnormal log data can be determined based on the extracted abnormal data and the corresponding service nodes, which can quickly focus on the core abnormal information in the log, avoid interference from irrelevant information, and at the same time clarify the service node corresponding to the abnormal log data.

[0092] The embodiment of the present invention automatically associates log data in different services of a business system through data tracking identifiers to form a complete call chain, avoiding the tedious process of manually associating log data and improving the operation and maintenance efficiency of the business system.

[0093] Further, refer to Figure 4 , showing Figure 1 A flowchart of step 103 of a data anomaly processing method is provided. This method is substantially the same as the data anomaly processing method provided in the first embodiment of the present invention. Step 103 may include:

[0094] Step 1031, determining the time information of the abnormal log data, matching the time information of the abnormal log data with a preset time window, and determining the abnormal log data in the preset time window;

[0095] Step 1032: Divide the abnormal log data in the preset time window according to the error type to obtain the abnormal log data corresponding to each error type;

[0096] Step 1033 : Aggregate the abnormal log data corresponding to each error type according to the call chain to obtain a target log data set.

[0097] It should be noted that, in the embodiment of the present invention, time information is first extracted from the abnormal log data, and a preset time window is used to match the time information. Assuming that the preset time window for the business abnormality is 2025-03-01-12:05:27 to 2025-03-01-12:08:02, the abnormal log data generated in the time window is matched. Through the preset time window, the abnormal log data within a specific time range can be locked at any time, reducing unnecessary data processing.

[0098] Specifically, according to the error type of the determined abnormal log data, the abnormal log data in the preset time window is divided. Through the error type division, the abnormal log data is classified by error type to facilitate subsequent targeted analysis. According to the call chain of the determined abnormal log data, the abnormal log data corresponding to each error type is aggregated to form a target log data set, which is conducive to intuitively understanding the context of the abnormality and quickly locating the abnormal problem.

[0099] The embodiment of the present invention dynamically matches abnormal log data through a preset time window to avoid interference from irrelevant logs, classifies abnormal log data by error type and call chain, simplifies the target log data set, and makes subsequent abnormality analysis more efficient.

[0100] Further, refer to Figure 5 , showing Figure 1 A flowchart of step 104 of a data anomaly processing method is provided. This method is substantially the same as the data anomaly processing method provided in the first embodiment of the present invention. Step 104 may include:

[0101] Step 1041: Pre-generate inference prompt words for anomaly identification of the log data set using a preset template engine; wherein the inference prompt words include high-frequency word extraction and context division;

[0102] Step 1042: Input the target log data set and the inference prompt words into a pre-trained inference engine, and use the inference engine to perform anomaly identification on the target log data set based on the inference prompt words to obtain anomaly confidence of the target log data set.

[0103] Step 1043: Determine the processing strategy corresponding to the abnormality confidence level according to the corresponding relationship between the abnormality confidence level and the processing strategy.

[0104] It should be noted that, in an embodiment of the present invention, the target log data set and reasoning prompt words are input into a pre-trained reasoning engine, and the reasoning prompt words are inference prompt words generated in advance using a preset template engine to identify anomalies in the log data set; wherein, the reasoning prompt words include high-frequency word extraction and context division, for example: "If you are an operation and maintenance expert, please perform high-frequency word extraction and context division based on the log data set, identify anomalies in the target log data set, calculate the confidence of the anomaly problem, and give processing suggestions", so that the reasoning engine can identify anomalies in the target log data set based on the reasoning prompt words.

[0105] Specifically, the target log dataset and inference prompts are fed into a pre-trained inference engine. The inference engine then uses the inference prompts to identify anomalies in the target log dataset, obtaining the anomaly confidence level for the target log dataset. Based on the correspondence between anomaly confidence and processing strategies, the corresponding processing strategy is determined. Dynamically matching processing strategies based on anomaly confidence levels enables automated exception handling. Strategies like automatic repair can quickly respond to high-confidence anomalies, reducing manual intervention and improving operational efficiency.

[0106] The embodiment of the present invention can provide accurate prompts for the inference engine through high-frequency word extraction and context division, thereby improving the accuracy of anomaly recognition. Through the guidance of inference prompt words, the inference engine can more accurately identify abnormal patterns in logs, dynamically match processing strategies according to the anomaly confidence level, realize the automation of exception handling, and improve operation and maintenance efficiency.

[0107] Specifically, step 1042 inputs the target log data set and the inference prompt words into a pre-trained inference engine, and uses the inference engine to identify anomalies in the target log data set based on the inference prompt words to obtain anomaly confidence of the target log data set, which may specifically include:

[0108] Sub-step 01: Input the target log data set and inference prompt words into the pre-trained inference engine. Use the inference engine to extract high-frequency words and perform context division on the target log data set to identify abnormal problems in the abnormal log data.

[0109] Sub-step 02, matching the abnormal problems in the abnormal log data with the known abnormal problems in the inference engine to generate a first confidence level;

[0110] Sub-step 03: performing semantic recognition on the context information of the abnormal problem in the abnormal log data to generate a second confidence level;

[0111] Sub-step 04: fusing the first confidence level and the second confidence level to obtain the anomaly confidence level of the target log data set.

[0112] It should be noted that in the above steps of the embodiment of the present invention, the target log data set and the reasoning prompt words are input into the pre-trained reasoning engine. The reasoning engine first extracts high-frequency words from the target log data set and identifies high-frequency words such as "Database", "connection", "failed", and "Query". The reasoning engine divides the context of the log according to the context information (call chain) and identifies abnormal problems. The reasoning engine matches the identified abnormal problems with the known abnormal problem library and generates a first confidence level based on the matching results. The reasoning engine performs semantic recognition on the context information of the abnormal problem. According to the context semantics, the reasoning engine generates a second confidence level. The second confidence level indicates that the context semantics supports the identification of abnormal problems. The reasoning engine uses the weighted average method to fuse the first confidence level and the second confidence level to calculate the abnormal confidence level of the target log data set.

[0113] Further, refer to Figure 6 , showing Figure 1 A flowchart of step 105 of a data anomaly processing method is provided. This method is substantially the same as the data anomaly processing method provided in the first embodiment of the present invention. Step 105 may include:

[0114] Step 1051: If the anomaly confidence level of the target log data set is greater than or equal to a first threshold, a preset repair script is used to repair the business system;

[0115] Step 1052: If the anomaly confidence level of the target log data set is greater than or equal to the second threshold and less than the first threshold, an exception handling form for the business system is generated and sent to the user for confirmation.

[0116] Step 1053: If the anomaly confidence level of the target log data set is less than the second threshold, a prompt message for manual processing is sent to the user.

[0117] It should be noted that, in an embodiment of the present invention, if the abnormality confidence of the target log data set is greater than or equal to the first threshold, a preset repair script is used to repair the business system. If the abnormality confidence of the target log data set is greater than or equal to the second threshold and less than the first threshold, an abnormality handling form for the business system is generated and the abnormality handling form is sent to the user for confirmation. If the abnormality confidence of the target log data set is less than the second threshold, a prompt message for manual processing is sent to the user, wherein the first threshold is greater than the second threshold, and the first threshold and the second threshold are set according to the actual analysis situation and are not specifically limited here. For example, if the abnormality confidence is greater than or equal to 90%, it is automatically repaired; if the abnormality confidence is greater than or equal to 70% and less than 90%, manual review is required; if the abnormality confidence is less than 70%, an expert consultation is required.

[0118] Specifically, if the anomaly confidence of the target log data set is greater than or equal to the first threshold, it indicates that the error type of the identified abnormal log data and the credibility of the specific abnormal problem are high, and the abnormal repair process can be automatically performed according to the repair strategy of the abnormal problem, wherein a preset repair script is used to repair the business system according to the repair strategy. The repair strategy is the operation corresponding to the common abnormal problem, and this implementation does not make specific restrictions on this; if the anomaly confidence of the target log data set is less than the first threshold, it indicates that the error type of the identified abnormal log data and the credibility of the specific abnormal problem are low, and manual intervention control is required. Specifically, if the anomaly confidence of the target log data set is greater than or equal to the second threshold and less than the first threshold, an abnormality handling form for the business system is generated and sent to the user for confirmation. If the anomaly confidence of the target log data set is less than the second threshold, a manual processing prompt message is sent to the user. The abnormality handling form includes information such as abnormal log data, abnormality confidence, and repair operation, so that the user can confirm whether the repair operation is feasible. The manual processing prompt message indicates that the credibility of the anomaly is very low, and an expert consultation is required to confirm the abnormal situation.

[0119] The embodiment of the present invention quickly responds to high-confidence abnormal problems through preset repair scripts, reduces manual intervention, improves repair efficiency, and promptly restores the normal operation of the business system. By generating exception handling forms and manual processing prompt information, users are allowed to participate in the decision-making process of exception handling, thereby improving the rationality of processing strategies, flexibly responding to abnormal problems of different degrees, effectively reducing the risk of misjudgment due to low confidence, and improving the accuracy of exception handling.

[0120] In order to enable those skilled in the art to more clearly understand the above-mentioned data anomaly processing method, refer to Figure 7 , showing an interactive schematic diagram of a data anomaly processing method provided by an embodiment of the present invention, specifically: the user inputs the data tracking identifier of the business system to be detected, inputs the data tracking identifier into the log collection layer of the business system for log data tracking, obtains abnormal log data carrying the data tracking identifier, associates the abnormal log data carrying the data tracking identifier, obtains cross-service abnormal log data, uses an inference engine to identify anomalies in the abnormal log data, obtains anomaly confidence, dynamically matches processing strategies based on different anomaly confidence levels, determines whether to use the preset script of the task executor to automatically repair the anomaly, or sends the anomaly confidence result and identification status to the console for manual intervention.

[0121] Reference Figure 8 , shows a schematic structural diagram of a data anomaly processing device provided by an embodiment of the present invention, the device comprising:

[0122] The log query module 201 is used to obtain a data tracking identifier of the business system to be detected, and use the data tracking identifier to query and obtain abnormal log data carrying the data tracking identifier;

[0123] An information extraction module 202 is configured to extract keywords from the abnormal log data and determine the error type and call chain corresponding to the abnormal log data;

[0124] The log partitioning module 203 is configured to partition the abnormal log data in a preset time window according to the error type and the call chain to obtain a target log data set;

[0125] Anomaly identification module 204, configured to use a pre-trained inference engine to perform anomaly identification on the target log data set, and obtain anomaly confidence of the target log data set and a processing strategy corresponding to the anomaly confidence;

[0126] The exception handling module 205 is configured to execute the processing strategy on the business system according to the exception confidence of the target log data set and the processing strategy corresponding to the exception confidence.

[0127] Optionally, the log query module 201 includes:

[0128] The acquisition submodule is used to obtain the data tracking identifier of the business system to be detected input by the user;

[0129] A tracking submodule, configured to input the data tracking identifier into the log collection layer of the business system for log data tracking;

[0130] The query submodule is used to use the data tracking identifier to query the log data of the business system at the log collection layer to obtain abnormal log data carrying the data tracking identifier.

[0131] Optionally, the information extraction module 202 includes:

[0132] An association submodule, configured to associate the abnormal log data carrying the data tracking identifier to obtain the abnormal log data across services;

[0133] An extraction submodule, configured to perform keyword extraction on the cross-service abnormal log data, and extract abnormal data and service nodes from the abnormal log data;

[0134] The first determining submodule is configured to determine an error type corresponding to the abnormal log data according to the abnormal data, and to determine a call chain of the abnormal log data according to the service node.

[0135] Optionally, the log partitioning module 203 includes:

[0136] A matching submodule, configured to determine the time information of the abnormal log data, match the time information of the abnormal log data with a preset time window, and determine the abnormal log data in the preset time window;

[0137] A division submodule is used to divide the abnormal log data in the preset time window according to the error type to obtain abnormal log data corresponding to each error type;

[0138] The aggregation submodule is used to aggregate the abnormal log data corresponding to each error type according to the call chain to obtain a target log data set.

[0139] Optionally, the abnormality identification module 204 includes:

[0140] A generation submodule is used to pre-generate inference prompt words for anomaly identification of the log data set using a preset template engine; wherein the inference prompt words include high-frequency word extraction and context division;

[0141] an identification submodule, configured to input the target log data set and the inference prompt words into a pre-trained inference engine, employ the inference engine to perform anomaly identification on the target log data set based on the inference prompt words, and obtain an anomaly confidence level of the target log data set;

[0142] The second determining submodule is configured to determine a processing strategy corresponding to the anomaly confidence level according to a corresponding relationship between the anomaly confidence level and the processing strategy.

[0143] Optionally, the identification submodule includes:

[0144] an identification unit, configured to input the target log data set and the inference prompt words into a pre-trained inference engine, use the inference engine to extract high-frequency words and perform context division on the target log data set, and identify abnormal problems in the abnormal log data;

[0145] A first generating unit is configured to match the abnormal problems in the abnormal log data with known abnormal problems in the inference engine to generate a first confidence level;

[0146] A second generating unit is configured to perform semantic recognition on context information of the abnormal problem in the abnormal log data to generate a second confidence level;

[0147] A fusion unit is used to fuse the first confidence level and the second confidence level to obtain an anomaly confidence level of the target log data set.

[0148] Optionally, the exception handling module 205 includes:

[0149] A first processing submodule, configured to use a preset repair script to repair the business system if the anomaly confidence level of the target log data set is greater than or equal to a first threshold;

[0150] a second processing submodule, configured to generate an exception handling form for the business system if the exception confidence level of the target log data set is greater than or equal to a second threshold and less than the first threshold, and send the exception handling form to a user for confirmation;

[0151] The third processing submodule is configured to send a manual processing prompt to the user if the anomaly confidence level of the target log data set is less than the second threshold.

[0152] The data anomaly processing device provided by the embodiment of the present invention obtains the data tracking identifier of the business system to be detected, uses the data tracking identifier to query to obtain abnormal log data carrying the data tracking identifier, performs keyword extraction on the abnormal log data, determines the error type and call chain corresponding to the abnormal log data, divides the abnormal log data in a preset time window according to the error type and call chain, obtains a target log data set, uses a pre-trained reasoning engine to identify anomalies on the target log data set, obtains anomaly confidence of the target log data set and a processing strategy corresponding to the anomaly confidence, and executes the processing strategy on the business system based on the anomaly confidence of the target log data set and the processing strategy corresponding to the anomaly confidence. The embodiment of the present invention automatically associates log data from different services of the business system through the data tracking identifier to form a complete call chain, avoiding the tedious process of manually associating log data, and uses a privately deployed reasoning engine to accurately identify anomalies in the log data, generate a quantitative confidence and a reasonable processing strategy, and automatically execute anomaly handling operations based on the anomaly confidence, ensuring the accuracy and reliability of exception handling, quickly responding to and identifying anomalies, greatly improving the operation and maintenance efficiency of the business system, minimizing the impact of abnormal problems on the business system, and ensuring the stability of the business system.

[0153] Reference Figure 9 , an embodiment of the present invention further provides an electronic device, such as Figure 9 As shown, it includes a processor 301, a communication interface 302, a memory 303 and a communication bus 304, wherein the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304.

[0154] Processor 301, memory 303 for storing processor-executable instructions;

[0155] The processor 301 is configured to execute the instructions to implement the data exception processing method as described above:

[0156] Obtaining a data tracking identifier of the business system to be detected, and using the data tracking identifier to query and obtain abnormal log data carrying the data tracking identifier;

[0157] Perform keyword extraction on the abnormal log data to determine the error type and call chain corresponding to the abnormal log data;

[0158] Dividing the abnormal log data in a preset time window according to the error type and the call chain to obtain a target log data set;

[0159] Using a pre-trained inference engine to perform anomaly identification on the target log data set, and obtain an anomaly confidence level of the target log data set and a processing strategy corresponding to the anomaly confidence level;

[0160] According to the anomaly confidence of the target log data set and the processing strategy corresponding to the anomaly confidence, the processing strategy is executed on the business system.

[0161] The communication bus mentioned in the terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0162] The communication interface is used for communication between the above terminal and other devices.

[0163] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0164] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0165] In another embodiment of the present invention, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the data exception processing method described in any one of the above embodiments is implemented.

[0166] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0167] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0168] Each embodiment in this specification is described in a related manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiment is generally similar to the method embodiment, so the description is relatively simple. For related parts, refer to the description of the method embodiment.

[0169] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included in the scope of protection of the present invention.

Claims

1. A data anomaly processing method, characterized in that: The method comprises: Obtaining a data tracking identifier of the business system to be detected, and using the data tracking identifier to query and obtain abnormal log data carrying the data tracking identifier; Perform keyword extraction on the abnormal log data to determine the error type and call chain corresponding to the abnormal log data; Dividing the abnormal log data in a preset time window according to the error type and the call chain to obtain a target log data set; Using a pre-trained inference engine to perform anomaly identification on the target log data set, and obtain an anomaly confidence level of the target log data set and a processing strategy corresponding to the anomaly confidence level; According to the anomaly confidence of the target log data set and the processing strategy corresponding to the anomaly confidence, the processing strategy is executed on the business system.

2. The method according to claim 1, characterized in that The step of obtaining a data tracking identifier of the business system to be detected and using the data tracking identifier to query and obtain abnormal log data carrying the data tracking identifier includes: Obtaining the data tracking identifier of the business system to be tested input by the user; Inputting the data tracking identifier into the log collection layer of the business system to track log data; The data tracking identifier is used in the log collection layer to query the log data of the business system to obtain abnormal log data carrying the data tracking identifier.

3. The method according to claim 1, characterized in that The performing keyword extraction on the abnormal log data to determine the error type and call chain corresponding to the abnormal log data includes: Associating the abnormal log data carrying the data tracking identifier to obtain the abnormal log data across services; Perform keyword extraction on the cross-service abnormal log data to extract abnormal data and service nodes in the abnormal log data; An error type corresponding to the abnormal log data is determined according to the abnormal data, and a call chain of the abnormal log data is determined according to the service node.

4. The method according to claim 1, wherein The abnormal log data in the preset time window is divided according to the error type and the call chain to obtain a target log data set, including: Determine the time information of the abnormal log data, match the time information of the abnormal log data with a preset time window, and determine the abnormal log data in the preset time window; Dividing the abnormal log data in the preset time window according to the error type to obtain abnormal log data corresponding to each error type; The abnormal log data corresponding to each error type is aggregated according to the call chain to obtain a target log data set.

5. The method according to claim 1, wherein The method of using a pre-trained inference engine to perform anomaly identification on the target log data set to obtain an anomaly confidence level of the target log data set and a processing strategy corresponding to the anomaly confidence level includes: Pre-generating inference prompt words for anomaly identification of the log data set using a preset template engine; wherein the inference prompt words include high-frequency word extraction and context division; Inputting the target log data set and the reasoning prompt words into a pre-trained reasoning engine, using the reasoning engine to perform anomaly identification on the target log data set based on the reasoning prompt words, and obtaining anomaly confidence of the target log data set; According to the corresponding relationship between the abnormality confidence and the processing strategy, the processing strategy corresponding to the abnormality confidence is determined.

6. The method according to claim 5, characterized in that Inputting the target log data set and the inference prompt words into a pre-trained inference engine, using the inference engine to perform anomaly identification on the target log data set based on the inference prompt words, and obtaining anomaly confidence of the target log data set includes: Inputting the target log data set and the inference prompt words into a pre-trained inference engine, using the inference engine to extract high-frequency words and perform context division on the target log data set, and identifying abnormal problems in the abnormal log data; Matching the abnormal problems in the abnormal log data with known abnormal problems in the inference engine to generate a first confidence level; Performing semantic recognition on context information of the abnormal problem in the abnormal log data to generate a second confidence level; The first confidence level and the second confidence level are combined to obtain an anomaly confidence level of the target log data set.

7. The method according to claim 1, characterized in that The executing the processing strategy on the business system according to the anomaly confidence of the target log data set and the processing strategy corresponding to the anomaly confidence includes: If the anomaly confidence level of the target log data set is greater than or equal to a first threshold, repairing the business system using a preset repair script; If the exception confidence of the target log data set is greater than or equal to the second threshold and less than the first threshold, generating an exception handling form for the business system, and sending the exception handling form to the user for confirmation; If the anomaly confidence of the target log data set is less than the second threshold, a prompt message for manual processing is sent to the user.

8. A data anomaly processing device, characterized in that: The device comprises: A log query module is used to obtain a data tracking identifier of the business system to be detected, and use the data tracking identifier to query and obtain abnormal log data carrying the data tracking identifier; An information extraction module is used to extract keywords from the abnormal log data and determine the error type and call chain corresponding to the abnormal log data; A log partitioning module, configured to partition the abnormal log data in a preset time window according to the error type and the call chain to obtain a target log data set; An anomaly identification module is used to use a pre-trained inference engine to identify anomalies of the target log data set, obtain an anomaly confidence level of the target log data set and a processing strategy corresponding to the anomaly confidence level; The exception handling module is configured to execute the processing strategy on the business system according to the exception confidence of the target log data set and the processing strategy corresponding to the exception confidence.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute the instruction to implement the data exception processing method according to any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the data exception processing method according to any one of claims 1 to 7 is implemented.

Images