Log data processing method, device, electronic device and readable storage medium

Abstract

The present application provides a log data processing method, device, electronic device and readable storage medium, which obtain multiple logs to be processed and extract key field information from each log to be processed; for any log to be processed, based on the key field information and a preset black, white and gray list rule library, determine the processing priority of the log to be processed; based on the key field information and processing priority of the log to be processed, determine the associated log with an association relationship with the log to be processed, and merge the associated log and the log to be processed to obtain a joint log; based on the abnormal information extracted from the joint log, determine the abnormal classification result of the joint log; for the first priority log to be processed, based on the key field information and abnormal classification result of the corresponding joint log, generate a target work order. In this way, through the black, white and gray list priority classification combined with the log association mechanism, the abnormal classification result can be accurately determined and a work order can be generated, thereby improving the accuracy of log abnormality determination.

Description

Technical Field

[0001] The present application relates to the technical field of log analysis and anomaly monitoring, and in particular to a log data processing method, device, electronic device, and readable storage medium. Background Art

[0002] In international business systems, especially cross-border transaction processing scenarios, log data of various transaction types and multiple languages ​​is involved. The log data formats are diverse and highly mobile across systems, which makes the detection and capture of log anomalies more difficult.

[0003] Traditional log analysis and anomaly monitoring usually rely on manual inspection and processing of log data one by one to determine and report anomalies. Because manual processing is time-consuming and there are problems such as missed anomalies and false alarms, the accuracy of log anomaly determination is low. Summary of the Invention

[0004] In view of this, the embodiments of the present application at least provide a log data processing method, device, electronic device and readable storage medium, which improve the accuracy of log anomaly determination.

[0005] This application mainly includes the following aspects:

[0006] In a first aspect, an embodiment of the present application provides a log data processing method, the method comprising:

[0007] Obtain multiple logs to be processed and extract key field information from each log to be processed;

[0008] For any of the logs to be processed, determining the processing priority of the log to be processed based on the key field information of the log to be processed and the preset black, white and gray list rule library;

[0009] Determine, based on key field information and processing priority of the log to be processed, an associated log associated with the log to be processed, and merge the associated log and the log to be processed to obtain a joint log;

[0010] Determining an anomaly classification result of the joint log based on the anomaly information extracted from the joint log;

[0011] For a log to be processed with a processing priority of the first priority, a target work order corresponding to the joint log is generated based on key field information and an exception classification result of the joint log corresponding to the log to be processed.

[0012] In a second aspect, an embodiment of the present application further provides a log data processing device, the log data processing device comprising:

[0013] The collection and analysis module is used to obtain multiple logs to be processed and extract key field information from each log to be processed;

[0014] A log screening module is used to determine the processing priority of any of the logs to be processed based on the key field information of the logs to be processed and a preset black, white and gray list rule library;

[0015] A log association module is configured to determine, based on key field information and processing priority of the log to be processed, an associated log associated with the log to be processed, and merge the associated log and the log to be processed to obtain a combined log;

[0016] an anomaly extraction module, configured to determine an anomaly classification result of the joint log based on the anomaly information extracted from the joint log;

[0017] The work order generation module is used to generate a target work order corresponding to the joint log based on the key field information and the abnormality classification result of the joint log corresponding to the unprocessed log with a processing priority of the first priority.

[0018] In a third aspect, an embodiment of the present application further provides an electronic device comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the log data processing method described above.

[0019] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the log data processing method described above are executed.

[0020] The embodiment of the present application provides a log data processing method, device, electronic device and readable storage medium, which obtains multiple logs to be processed and extracts key field information from each log to be processed; for any log to be processed, based on the key field information of the log to be processed and a preset black-white-gray list rule library, determines the processing priority of the log to be processed; based on the key field information and processing priority of the log to be processed, determines the associated log with an association relationship with the log to be processed, and merges the associated log and the log to be processed to obtain a joint log; based on the abnormal information extracted from the joint log, determines the abnormal classification result of the joint log; for the log to be processed with the first priority, based on the key field information and abnormal classification result of the joint log corresponding to the log to be processed, generates a target work order corresponding to the joint log. In this way, through the black-white-gray list priority classification combined with the log association mechanism, the abnormal classification result can be accurately determined and a work order can be generated, thereby improving the accuracy of log abnormality determination.

[0021] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 A flow chart of a log data processing method provided by an embodiment of the present application is shown;

[0024] Figure 2 One of the functional module diagrams of a log data processing device provided in an embodiment of the present application is shown;

[0025] Figure 3 The second functional module diagram of a log data processing device provided by an embodiment of the present application is shown;

[0026] Figure 4 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0028] In order to enable those skilled in the art to use the contents of this application, the following implementation method is provided in combination with the specific application scenario of "log analysis and anomaly monitoring of international business systems". For those skilled in the art, the general principles defined herein can be applied to other embodiments and application scenarios without departing from the spirit and scope of this application.

[0029] To facilitate understanding of the present application, the technical solutions provided in the present application are described in detail below in conjunction with specific embodiments.

[0030] See also Figure 1 , Figure 1 This is a flow chart of a log data processing method provided by an embodiment of the present application. Figure 1 As shown, the log data processing method provided in the embodiment of the present application includes the following steps:

[0031] S101: Acquire multiple logs to be processed, and extract key field information from each log to be processed.

[0032] Here, first, the international business system needs to collect raw log data recording anomalies in real time from a variety of different log sources, that is, obtain multiple unprocessed logs. Then, after log collection is completed, log parsing is performed on each unprocessed log, and key field information is extracted from each unprocessed log. Among these various log sources, various types of logs include API logs and SWIFT messages. SWIFT messages are a standardized message format used by international banks to transmit financial transaction information and other services. Key field information is the field information that must be included in log data processing. For example, API logs must include the field information of the "transaction_id" field; SWIFT messages must include the field information of the "reference_id" field, and so on.

[0033] In an embodiment of the present application, firstly, a streaming log collection tool such as the distributed message queue Kafka and the log collection tool Fluentd is used to access multi-source log streams. According to different log sources and types, the logs are routed to independent message classification units (Topics) to ensure efficient and independent processing of logs. Specifically, a different Topic is configured for each log source in Kafka; for example: "topic_api_logs" is used to store API logs; "topic_swift_logs" is used to store SWIFT message logs; Fluentd is configured to collect logs, and the log labels and routing rules are defined to ensure that logs from different sources can be accurately diverted. For example: API logs are pushed to Kafka's "topic_api_logs" through Fluentd; SWIFT messages are collected to Kafka's "topic_swift_logs" after being recorded through the system log.

[0034] After accessing the log stream, the system consumes the log stream from Kafka, determines the log format (e.g., JSON, SWIFT message, plain text, etc.) based on its characteristics, and routes it to the appropriate processing logic. Specifically, it attempts to parse the log data into JSON format to confirm whether it is valid JSON. It also determines whether the log contains specific SWIFT message fields (e.g., :20: or 32A:) to confirm that it is a SWIFT message. If the log is neither in JSON format nor contains the characteristic SWIFT message fields, it is classified as a plain text log.

[0035] Next, we use the corresponding parsing method based on the log type to extract key field information. Specifically, for JSON-formatted logs, we use a JSON parser to extract key field information. For SWIFT message-formatted logs, we parse the log content based on SWIFT message field rules (such as using the delimiter ":" to extract fields). For plain text logs, we use regular expressions to match key field information, such as timestamps, transaction IDs, and error messages.

[0036] After extracting the key field information, the extracted key field information can also be preliminarily verified to ensure that the key field information obtained is complete and conforms to the expected format. For logs where parsing fails or fields are missing, error information is recorded and the log is sent to the exception queue. Specifically, the verification rules include mandatory field checks and field format checks, where mandatory field checks are used to ensure that key fields exist. For example, API logs must include the "transaction_id" field; SWIFT logs must include the "reference_id" field. Field format verification is used to ensure that the format of each key field is correct, such as the timestamp field format is valid, the amount field format is a valid number, etc.

[0037] Finally, the extracted key field information is combined with the original log content of the log to be processed and stored, providing a unified input format for subsequent screening, correlation, and anomaly analysis. Specifically, source information can be added to each log record, such as source="API" or source="SWIFT", while the original log content is retained to ensure traceability during subsequent processing or troubleshooting. Furthermore, retaining the original log data can be used for auditing and secondary analysis.

[0038] This step collects logs from various sources (such as API logs and SWIFT messages) in real time, automatically identifies their formats, and extracts key field information. By cleaning and structuring the data, it provides the foundational data for subsequent log data processing. Efficient log stream access, format recognition, dynamic field parsing, and exception tolerance ensure efficient, accurate, and scalable log collection and parsing.

[0039] S102: For any of the logs to be processed, based on key field information of the log to be processed and a preset black, white and grey list rule library, determining a processing priority of the log to be processed.

[0040] Here, after extracting the key field information from each log to be processed, for any log to be processed, the system will match the rules in the preset black, white and gray list library based on the key field information of the log to be processed, and prioritize the log to be processed according to the rules in the preset black, white and gray list rule library to determine the processing priority of the log to be processed. Among them, the rules in the preset black, white and gray list library are stored in the remote dictionary service (RemoteDictionary Server, Redis) in the form of key-value pairs. Each rule includes four core fields for defining the rule, including the rule unique identifier "rule_id", the matching log keyword "keyword", the processing priority "priority" and the rule exception trigger frequency "frequency". The processing priority is divided into first priority, second priority and third priority in order from high to low. The rules in the preset black, white and gray list library are divided into three categories according to the processing priority. The rules with the first priority "high" are blacklist rules, the rules with the second priority "medium" are graylist rules, and the rules with the third priority "low" are whitelist rules.

[0041] Pending logs are sorted into different priority categories based on the processing priority determined by the matching rules, and then processed. This prioritized processing order of logs can help reduce invalid alerts and improve system response efficiency. Furthermore, prioritizing the sorting of pending logs ensures more accurate and efficient log correlation and anomaly identification.

[0042] S103: Based on the key field information and processing priority of the log to be processed, determine an associated log associated with the log to be processed, and merge the associated log and the log to be processed to obtain a combined log.

[0043] Here, the system will associate the pending log with related logs from different sources that have a correlation with the pending log based on the key fields and processing priority of the pending log. The associated logs and the pending log are then merged into a joint log to generate a complete transaction chain. The associated logs are all the other logs in the pending logs, excluding the pending log. This log association process provides sufficient context for subsequent anomaly extraction, anomaly classification, and work order generation, ensuring more accurate anomaly identification.

[0044] S104: Determine an abnormality classification result of the joint log based on the abnormality information extracted from the joint log.

[0045] Here, after the logs to be processed are parsed, classified, and associated, a joint log is obtained. The system extracts exception information from the joint log and determines the exception classification result of the joint log based on the extracted exception information, providing accurate guidance for subsequent work order generation and exception handling based on the generated work order.

[0046] S105 , for a log to be processed with a first priority, generating a target work order corresponding to the joint log based on key field information and an exception classification result of the joint log corresponding to the log to be processed.

[0047] Here, for the pending log with the first priority, a target work order corresponding to the joint log is generated based on the key field information and exception classification results of the joint log corresponding to the pending log. This step enables the system to automatically generate and distribute work orders corresponding to exceptions in the joint log corresponding to the first priority pending log. This ensures that exceptions corresponding to the first priority, that is, those corresponding to the blacklist rules, can be quickly located and processed, thereby improving the accuracy and efficiency of the system's exception handling.

[0048] Furthermore, for any of the logs to be processed, determining the processing priority of the log to be processed based on the key field information of the log to be processed and a preset black, white and gray list rule library includes:

[0049] Step a1: query at least one first rule that matches the key field information of the log to be processed from the preset black, white and gray list rule library.

[0050] Here, at least one first rule that matches the key field information of the log to be processed is queried from the preset black, white and gray list rule library. For example, assuming that the key field information of the log to be processed is {"transaction_id":"TX123","status":"failed","reason":"payment failure"}, and the rule in the preset black, white and gray list rule library is {"rule_id":"R001","keyword":"payment failure","priority":"high"}, and "payment failure" in the rule matches "payment failure" in the key log information, then the rule is determined as the first rule.

[0051] Step a2: determining the highest processing priority among the processing priorities corresponding to the at least one first rule as the processing priority of the log to be processed; wherein, the preset black, white and gray list rule library is provided with a plurality of rules, and any of the rules is configured with a processing priority.

[0052] Here, multiple key field data of the same log may match multiple processing priority rules at the same time. At this time, the highest processing priority among the processing priorities corresponding to at least one first rule is determined as the processing priority of the log to be processed; wherein, multiple rules are set in the preset black, white and gray list rule library, and any rule is configured with a processing priority.

[0053] Furthermore, the determining, based on the key field information and processing priority of the log to be processed, an associated log having an associated relationship with the log to be processed includes:

[0054] Step b1: Determine whether there is a second log in at least one first log that has the same key field information as the log to be processed; the first log is another log in the multiple logs to be processed that has the same processing priority as the log to be processed.

[0055] Here, it is determined whether there is a second log in at least one first log having the same key field information as the log to be processed; wherein the first log is another log in the plurality of logs to be processed having the same processing priority as the log to be processed.

[0056] Step b2: If the second log exists, determine the second log as an associated log associated with the log to be processed.

[0057] Here, if it exists, the second log is determined as an associated log with an association relationship with the log to be processed. In an embodiment of the present application, the first-level initial association result is generated by directly matching the key field information. The matching priority is centered on field matching to ensure high precision. Specifically, the "transaction_id" of the API log is matched with the "reference_id" of the SWIFT message. If the field match is successful, the record is stored in the initial association intermediate table to generate an associated record. Among them, strict field matching is used first to avoid redundant matching.

[0058] Step b3: If no such log exists, then based on the dynamic time window and fuzzy matching algorithm, determine an associated log that has an associated relationship with the log to be processed.

[0059] Here, for the logs to be processed whose key field information cannot be directly matched, a supplementary match is performed on the initial associated intermediate table through a dynamic time window and fuzzy matching algorithm to ensure the coverage of cross-source associations.

[0060] Specifically, the dynamic time window dynamically adjusts the time window threshold based on the business scenario. If the timestamp of a log differs from the timestamp of the log to be processed within the time window threshold, it is considered that they may belong to the same transaction link and the log is identified as an associated log with the log to be processed. The fuzzy matching algorithm uses the similarity of the prefix and suffix of key field information for fuzzy matching, such as matching "TX123" and "TX12345", and is combined with string similarity algorithms, such as the edit distance algorithm (Levenshtein), to improve the accuracy of fuzzy matching.

[0061] Furthermore, to meet dynamic adjustment needs, the system supports real-time updates of associated intermediate tables, automatically correcting incorrect associations or adding new fields. Specifically, it regularly scans the associated intermediate tables to supplement and correct records with missing fields or matching conflicts. When new log fields (such as API response time) are added, the intermediate table structure is dynamically updated to ensure the integrity of the associated logs. Verification rules are used to detect redundant or incorrectly associated records and clean them up.

[0062] Furthermore, all associated logs can be stored in a unified intermediate table, providing a flexible query interface to support log chain analysis. Specifically, the associated logs use a unified storage structure, so that each joint log records complete information including API and SWIFT messages. The associated log fields include: transaction ID "transaction_id", SWIFT message reference number "swift_reference", timestamp "timestamp", status "status", and other extended fields such as response time and error code.

[0063] Furthermore, the associated log also provides a query interface based on "transaction_id" or "swift_reference" to support link-level transaction diagnosis.

[0064] Furthermore, determining an anomaly classification result of the joint log based on the anomaly information extracted from the joint log includes:

[0065] Step c1: performing semantic extraction on the joint log according to the trained natural language processing model to obtain abnormal information of the joint log.

[0066] Here, natural language processing (NLP) technology is used to perform semantic extraction on the joint log according to the trained natural language processing model to obtain the abnormal information of the joint log. In an embodiment of the present application, the natural language processing model adopts the BERT model, which supports deep semantic understanding of the context, is suitable for parsing long texts and complex structures, can perform semantic analysis on the abnormal messages recorded in the joint log and the contextual information of the abnormal messages, and can accurately extract the abnormal information of the joint log. In addition, the BERT model supports multi-language and multi-field adaptation, and can be trained on the BERT model through diversified business log samples to ensure that it can adapt to different types of abnormal messages, and fine-tune for multiple languages ​​(such as English, French, etc.) and specific field terms (such as key fields in SWIFT messages).

[0067] Step c2: generating an anomaly classification label for the joint log based on the anomaly information, and determining the confidence of the anomaly classification label.

[0068] Here, we use a trained natural language processing model to extract semantic information from the joint logs and convert the extracted semantic information into anomaly classification labels, such as network timeout and payment rejection. We also attach a confidence level to the anomaly classification label, which ranges from 0 to 1.

[0069] Step c3: determining an anomaly classification result of the joint log according to the confidence of the anomaly classification label.

[0070] Here, the anomaly classification result of the joint log is determined according to the confidence of the anomaly classification label.

[0071] Furthermore, the generated anomaly classification labels and corresponding confidence levels can be stored in a database, such as an Elasticsearch database, to provide real-time data query support services.

[0072] Furthermore, for anomalies with frequently appearing abnormal classification labels, the black, white, and gray list rule base can be automatically optimized to increase the ability to identify such anomalies. Specifically, the system will automatically generate new rules based on the abnormal classification label data in the historical logs.

[0073] Furthermore, determining the anomaly classification result of the joint log according to the confidence of the anomaly classification label includes:

[0074] Step d1: If the confidence of the abnormal classification label is less than a first threshold, the manual classification result of the abnormal information is determined as the abnormal classification result of the joint log.

[0075] Here, if the confidence level of the anomaly classification label is less than the first threshold, it indicates that the confidence level of the anomaly classification label is low, the anomaly in the joint log is an unknown anomaly, and needs to be sent for manual review for anomaly classification. The classification result of the anomaly information fed back by the manual review is determined as the anomaly classification result of the joint log. In this embodiment of the present application, the first threshold is 60%.

[0076] Step d2: If the confidence of the anomaly classification label is within the target threshold interval, the anomaly classification result of the joint log is determined based on the number of times the confidence of the anomaly classification label in the historical log is within the target threshold interval; the target threshold interval is an interval greater than or equal to the first threshold and less than or equal to the second threshold.

[0077] Here, if the confidence of the abnormal classification label is within the target threshold interval, the abnormal classification result of the joint log is determined based on the number of times the confidence of the abnormal classification label in the historical log is within the target threshold interval; wherein, the target threshold interval is an interval greater than or equal to the first threshold and less than or equal to the second threshold. In the embodiment of the present application, the second threshold is 90%, that is, the target threshold interval is an interval of [60%, 90%]. Specifically, if the number of times the confidence of the abnormal classification label in the historical log is within the target threshold interval is less than the preset number threshold, the abnormal classification label is determined as the abnormal classification result of the joint log; if the number of times the confidence of the abnormal classification label in the historical log is within the target threshold interval reaches the preset number threshold, it is necessary to push manual review for abnormal classification, and the classification result of the abnormal information fed back by the manual review is determined as the abnormal classification result of the joint log.

[0078] Step d3: If the confidence of the abnormal classification label is greater than a second threshold, the abnormal classification label is determined as the abnormal classification result of the joint log.

[0079] Here, if the confidence of the abnormal classification label is greater than the second threshold, that is, the confidence is greater than 90%, it means that the confidence of the abnormal classification label is very high, and the abnormal classification label can be directly determined as the abnormal classification result of the joint log.

[0080] In this application, the embodiment uses natural language processing (NLP) technology to perform semantic analysis on the abnormal information in the log, extract key abnormal information, and generate accurate classification labels for each log to determine the abnormal classification results. This can ensure the accuracy of abnormal classification and further optimize the log abnormality processing process.

[0081] Furthermore, after generating a target work order corresponding to the joint log based on key field information and anomaly classification results of the joint log corresponding to the log to be processed, the method further includes:

[0082] Step e1: Based on the feedback information of the target work order and / or the rule exception triggering frequency, the preset black, white and gray list rule base is adjusted to obtain an adjusted preset black, white and gray list rule base.

[0083] Here, according to the feedback information of the target work order and / or the rule exception triggering frequency, the preset black, white and gray list rule base is dynamically adjusted to obtain an adjusted preset black, white and gray list rule base.

[0084] Step e2, determining the processing priority of the log to be processed based on the key field information of the log to be processed and the preset black, white and gray list rule library, includes:

[0085] Based on the key field information of the log to be processed and the adjusted preset black, white and gray list rule library, the processing priority of the log to be processed is determined.

[0086] Here, adjusted rules are updated to Redis in real time to ensure the system always uses the latest rules. Specifically, the processing priority of pending logs is determined based on the key fields of the logs and the adjusted preset black, white, and gray list rule base. This further improves the system's exception handling efficiency and reduces duplication and invalid warnings in work orders.

[0087] Furthermore, the preset black, white and gray list rule base is adjusted based on the feedback information of the target work order and / or the rule abnormality triggering frequency to obtain the adjusted preset black, white and gray list rule base, including:

[0088] Step f1: If the feedback information of the target work order indicates that the exception corresponding to any second rule is resolved, the second rule is adjusted from a rule corresponding to a blacklist to a rule corresponding to a whitelist.

[0089] Here, the rules in the preset black, white and gray list rule base are adjusted according to the feedback of the work order status, and the rules corresponding to the blacklist whose corresponding exceptions have been fixed are downgraded to the rules corresponding to the whitelist. Specifically, if the feedback information of the target work order indicates that the exception corresponding to any second rule has been resolved, the second rule is adjusted from the rule corresponding to the blacklist to the rule corresponding to the whitelist; wherein the second rule is the rule corresponding to the blacklist in the preset black, white and gray list rule base. For example, taking the rule {"rule_id":"R001","keyword":"payment failure","priority":"high"} corresponding to the blacklist as an example, if the feedback information of the target work order indicates that the exception corresponding to the rule has been resolved, the rule is adjusted to {"rule_id":"R001","keyword":"payment failure","priority":"low"}, that is, the rule corresponding to the whitelist.

[0090] Step f2: If the rule abnormal triggering frequency of any third rule reaches a preset triggering frequency threshold, the third rule is adjusted from a gray list rule to a black list rule.

[0091] Here, by monitoring the logs to be processed, the frequency of rule anomaly triggering is counted, and the rules in the preset black, white and gray list rule base are adjusted according to the frequency of rule anomaly triggering. Specifically, if the rule anomaly triggering frequency of any third rule reaches the preset trigger frequency threshold, the third rule is adjusted from a gray list rule to a black list rule; wherein the third rule is the rule corresponding to the gray list in the preset black, white and gray list rule base. For example, taking the rule {"rule_id":"R003","keyword":"timeout","priority":"medium","frequency":10} corresponding to the gray list as an example, if the rule anomaly triggering frequency of this rule reaches the preset trigger frequency threshold of 15, then the rule is adjusted to {"rule_id":"R003","keyword":"timeout","priority":"high"}, that is, the rule corresponding to the black list.

[0092] Furthermore, all rule adjustment operations can be recorded in a log system for auditing and backtracking. Taking the example in step f2 as an example, the rule adjustment operation can be recorded as:

[0093] "[2025-01-08 12:00:00]Rule R003 priority updated from medium to highdue to 15triggers in 1hour".

[0094] Furthermore, the target work order includes the fields required for the target work order and the exception resolution suggestions dynamically generated based on the exception classification results and the preset exception resolution suggestion rule base. The required fields for the target work order include the transaction ID, which is used to identify the abnormal business transaction; the exception classification result, which is used to indicate the exception classification result; the time when the exception occurred, which is used to locate the problem; and the context log information, which is used to provide more background support for the responsible team. The rules in the preset exception resolution suggestion rule base can be dynamically and sustainably optimized. Examples of exception resolution suggestions include: payment failure suggestion: contact the payment gateway team to check service availability; database timeout suggestion: check for slow queries or database lock issues.

[0095] Furthermore, the target work order can be formatted into JSON format for easy storage and distribution.

[0096] Furthermore, the method further comprises:

[0097] For pending logs with the first and second priority levels, no work order is generated, and only the rule exception triggering frequency is recorded.

[0098] Specifically, for the second priority pending logs, record the rule exception triggering frequency and observe whether the priority needs to be adjusted; for the third priority pending logs, only record the triggering frequency to reduce resource usage.

[0099] Furthermore, after generating the target work order, the method further includes:

[0100] Based on the results of exception classification, dynamic distribution algorithms and load balancing mechanisms are used to push work orders to the queues of the corresponding responsible teams to avoid overloading a single team. Specifically, a mapping relationship between exception types and responsible teams is established. Use polling or priority-based load balancing algorithms to dynamically adjust the priority of the distribution queue. For high-priority exceptions, priority is given to queues with lower current loads. Use message queue tools (such as RabbitMQ and Kafka) to push work orders to team queues in real time. Each responsible team maintains an independent queue, and queue messages support confirmation and retry mechanisms. After the responsible team receives the work order, it confirms the receipt status through the callback interface, and the system marks the work order as "distributed". A REST API interface is provided for the responsible team to update the work order status and processing feedback in real time. A status field is designed for each work order, including: Unprocessed: the work order has been generated but not yet distributed; In Process: the responsible team has received the work order and is processing it; Resolved: the responsible team has completed processing and the problem has been resolved; Unresolved: escalation is required.

[0101] The embodiment of the present application provides a log data processing method, including: obtaining multiple logs to be processed and extracting key field information from each log to be processed; for any log to be processed, determining the processing priority of the log to be processed based on the key field information and a preset black, white, and gray list rule library; based on the key field information and processing priority of the log to be processed, determining an associated log that has an association relationship with the log to be processed, and merging the associated log and the log to be processed to obtain a joint log; based on the abnormal information extracted from the joint log, determining the abnormal classification result of the joint log; for the first priority log to be processed, generating a target work order based on the key field information and abnormal classification result of the corresponding joint log. In this way, through the black, white, and gray list priority classification combined with the log association mechanism, the abnormal classification result can be accurately determined and a work order can be generated, thereby improving the accuracy of log abnormality determination.

[0102] Based on the same application concept, the embodiments of the present application also provide a log data processing device corresponding to the log data processing method provided in the above embodiments. Since the principle of solving the problem by the device in the embodiments of the present application is similar to the log data processing method in the above embodiments of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0103] See also Figure 2 , Figure 2 This is one of the functional module diagrams of a log data processing device provided in an embodiment of the present application. Figure 2 As shown, the log data processing device 200 includes:

[0104] The collection and analysis module 210 is used to obtain multiple logs to be processed and extract key field information from each log to be processed.

[0105] The log screening module 220 is configured to determine the processing priority of any of the logs to be processed based on key field information of the log and a preset black, white and grey list rule library.

[0106] The log association module 230 is configured to determine an associated log associated with the log to be processed based on key field information and processing priority of the log to be processed, and merge the associated log and the log to be processed to obtain a combined log.

[0107] The anomaly extraction module 240 is configured to determine an anomaly classification result of the joint log based on the anomaly information extracted from the joint log.

[0108] The work order generating module 250 is configured to generate a target work order corresponding to a first priority unprocessed log based on key field information and anomaly classification results of a joint log corresponding to the unprocessed log.

[0109] Furthermore, when the log screening module 220 is used to determine the processing priority of any of the logs to be processed based on the key field information of the logs to be processed and the preset black, white and gray list rule base, the log screening module 220 is specifically used to:

[0110] Querying at least one first rule that matches the key field information of the log to be processed from the preset black, white and gray list rule library;

[0111] Determining the highest processing priority among the processing priorities corresponding to the at least one first rule as the processing priority of the log to be processed;

[0112] There are multiple rules set in the preset black, white and grey list rule library, and any of the rules is configured with a processing priority.

[0113] Furthermore, when the log association module 230 is used to determine the associated log associated with the log to be processed based on the key field information and processing priority of the log to be processed, the log association module 230 is specifically used to:

[0114] Determine whether there is a second log in at least one first log that has the same key field information as the log to be processed; the first log is another log in the multiple logs to be processed that has the same processing priority as the log to be processed;

[0115] If so, determining the second log as an associated log associated with the log to be processed;

[0116] If not, then based on the dynamic time window and fuzzy matching algorithm, determine the associated log that has an associated relationship with the log to be processed.

[0117] Furthermore, when the anomaly extraction module 240 is used to determine an anomaly classification result of the joint log based on the anomaly information extracted from the joint log, the anomaly extraction module 240 is specifically used to:

[0118] Performing semantic extraction on the joint log according to the trained natural language processing model to obtain abnormal information of the joint log;

[0119] generating an anomaly classification label for the joint log based on the anomaly information, and determining a confidence level of the anomaly classification label;

[0120] An anomaly classification result of the joint log is determined according to the confidence of the anomaly classification label.

[0121] Furthermore, when the anomaly extraction module 240 is used to determine the anomaly classification result of the joint log according to the confidence of the anomaly classification label, the anomaly extraction module 240 is specifically used to:

[0122] If the confidence level of the abnormal classification label is less than a first threshold, determining the manual classification result of the abnormal information as the abnormal classification result of the joint log;

[0123] If the confidence level of the abnormal classification label is within a target threshold interval, the abnormal classification result of the joint log is determined based on the number of times the confidence level of the abnormal classification label in the historical log is within the target threshold interval; the target threshold interval is an interval greater than or equal to the first threshold and less than or equal to the second threshold;

[0124] If the confidence of the anomaly classification label is greater than a second threshold, the anomaly classification label is determined as the anomaly classification result of the joint log.

[0125] Further, see Figure 3 , Figure 3 This is a second functional module diagram of a log data processing device provided in an embodiment of the present application. Figure 3 As shown, the log data processing device 200 further includes:

[0126] The feedback optimization module 260 is configured to adjust the preset black, white and gray list rule base based on the feedback information of the target work order and / or the rule exception triggering frequency to obtain an adjusted preset black, white and gray list rule base.

[0127] The log screening module 220 is further configured to:

[0128] Based on the key field information of the log to be processed and the adjusted preset black, white and gray list rule library, the processing priority of the log to be processed is determined.

[0129] Furthermore, when the feedback optimization module 260 is used to adjust the preset black, white, and gray list rule base based on the feedback information of the target work order and / or the rule exception triggering frequency to obtain the adjusted preset black, white, and gray list rule base, the feedback optimization module 260 is specifically used to:

[0130] If the feedback information of the target work order indicates that the exception corresponding to any second rule is resolved, adjust the second rule from a rule corresponding to the blacklist to a rule corresponding to the whitelist;

[0131] If the rule abnormal triggering frequency of any third rule reaches a preset triggering frequency threshold, the third rule is adjusted from a gray list rule to a black list rule.

[0132] The embodiment of the present application provides a log data processing device, including a collection and analysis module for obtaining multiple logs to be processed and extracting key field information from each log to be processed; a log screening module for determining the processing priority of any log to be processed based on the key field information of the log to be processed and a preset black, white and gray list rule library; a log association module for determining the associated logs that have an associated relationship with the log to be processed based on the key field information and processing priority of the log to be processed, and merging the associated logs and the log to be processed to obtain a joint log; an anomaly extraction module for determining the anomaly classification result of the joint log based on the anomaly information extracted from the joint log; a work order generation module for generating a target work order corresponding to the joint log based on the key field information and anomaly classification result of the joint log corresponding to the log to be processed with a processing priority of the first priority. In this way, through the black, white and gray list priority classification combined with the log association mechanism, the anomaly classification result can be accurately determined and a work order can be generated, thereby improving the accuracy of log anomaly determination.

[0133] Based on the same application idea, please refer to Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 4 As shown, the electronic device 400 includes a processor 410 , a memory 420 and a bus 430 .

[0134] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 through the bus 430. The machine-readable instructions are executed by the processor 410 to execute the steps of the log data processing method provided in the above embodiment. The specific implementation method can be found in the method embodiment and will not be repeated here.

[0135] Based on the same application concept, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is run by a processor, the steps of the log data processing method provided in the above embodiment are executed. The specific implementation method can be found in the method embodiment and will not be repeated here.

[0136] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0137] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interface, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0138] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0139] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0140] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0141] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0142] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The scope of protection of the present application is not limited thereto. Although the present application has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed in the present application, or make equivalent replacements for some of the technical features thereof. However, these modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A log data processing method, characterized in that: The method comprises: Obtain multiple logs to be processed and extract key field information from each log to be processed; For any of the logs to be processed, determining the processing priority of the log to be processed based on the key field information of the log to be processed and the preset black, white and gray list rule library; Determine, based on key field information and processing priority of the log to be processed, an associated log associated with the log to be processed, and merge the associated log and the log to be processed to obtain a joint log; Determining an anomaly classification result of the joint log based on the anomaly information extracted from the joint log; For a pending log with a processing priority of the first priority, generating a target work order corresponding to the joint log based on key field information and anomaly classification results of the joint log corresponding to the pending log; The determining of the associated logs associated with the log to be processed based on the key field information and the processing priority of the log to be processed includes: Determine whether there is a second log in at least one first log that has the same key field information as the log to be processed; the first log is another log in the multiple logs to be processed that has the same processing priority as the log to be processed; If so, determining the second log as an associated log associated with the log to be processed; If not, then based on the dynamic time window and fuzzy matching algorithm, determine the associated log that has an associated relationship with the log to be processed.

2. The log data processing method according to claim 1, characterized in that: For any of the logs to be processed, determining the processing priority of the log to be processed based on the key field information of the log to be processed and a preset black, white and gray list rule library includes: Querying at least one first rule that matches the key field information of the log to be processed from the preset black, white and gray list rule library; Determining the highest processing priority among the processing priorities corresponding to the at least one first rule as the processing priority of the log to be processed; There are multiple rules set in the preset black, white and grey list rule library, and any of the rules is configured with a processing priority.

3. The log data processing method according to claim 1, wherein: The determining, based on the abnormal information extracted from the combined log, an abnormal classification result of the combined log includes: Performing semantic extraction on the joint log according to the trained natural language processing model to obtain abnormal information of the joint log; generating an anomaly classification label for the joint log based on the anomaly information, and determining a confidence level of the anomaly classification label; An anomaly classification result of the joint log is determined according to the confidence of the anomaly classification label.

4. The log data processing method according to claim 3, characterized in that: Determining the anomaly classification result of the joint log according to the confidence of the anomaly classification label includes: If the confidence level of the abnormal classification label is less than a first threshold, determining the manual classification result of the abnormal information as the abnormal classification result of the joint log; If the confidence level of the abnormal classification label is within a target threshold interval, the abnormal classification result of the joint log is determined based on the number of times the confidence level of the abnormal classification label in the historical log is within the target threshold interval; the target threshold interval is an interval greater than or equal to the first threshold and less than or equal to the second threshold; If the confidence of the anomaly classification label is greater than a second threshold, the anomaly classification label is determined as the anomaly classification result of the joint log.

5. The log data processing method according to claim 1, characterized in that: After generating a target work order corresponding to the joint log based on key field information and an exception classification result of the joint log corresponding to the log to be processed, the method further includes: Based on the feedback information of the target work order and / or the rule exception triggering frequency, adjusting the preset black, white and gray list rule library to obtain an adjusted preset black, white and gray list rule library; Determining the processing priority of the log to be processed based on the key field information of the log to be processed and the preset black, white and gray list rule library includes: Based on the key field information of the log to be processed and the adjusted preset black, white and gray list rule library, the processing priority of the log to be processed is determined.

6. The log data processing method according to claim 5, characterized in that: The step of adjusting the preset black, white, and gray list rule base based on the feedback information of the target work order and / or the rule exception triggering frequency to obtain the adjusted preset black, white, and gray list rule base includes: If the feedback information of the target work order indicates that the exception corresponding to any second rule is resolved, adjust the second rule from a rule corresponding to the blacklist to a rule corresponding to the whitelist; If the rule abnormal triggering frequency of any third rule reaches a preset triggering frequency threshold, the third rule is adjusted from a gray list rule to a black list rule.

7. A log data processing device, characterized in that: The log data processing device includes: The collection and analysis module is used to obtain multiple logs to be processed and extract key field information from each log to be processed; A log screening module is used to determine the processing priority of any of the logs to be processed based on the key field information of the logs to be processed and a preset black, white and gray list rule library; A log association module is configured to determine, based on key field information and processing priority of the log to be processed, an associated log associated with the log to be processed, and merge the associated log and the log to be processed to obtain a combined log; an anomaly extraction module, configured to determine an anomaly classification result of the joint log based on the anomaly information extracted from the joint log; A work order generation module is configured to generate, for a to-be-processed log having a processing priority of the first priority, a target work order corresponding to the joint log based on key field information and an exception classification result of the joint log corresponding to the to-be-processed log; When the log association module is used to determine the associated log associated with the log to be processed based on the key field information and processing priority of the log to be processed, the log association module is specifically used to: Determine whether a second log having the same key field information as the log to be processed exists in at least one first log; the first log is another log among the multiple logs to be processed that has the same processing priority as the log to be processed; if so, determine the second log as an associated log having an associated relationship with the log to be processed; if not, determine an associated log having an associated relationship with the log to be processed based on a dynamic time window and a fuzzy matching algorithm.

8. An electronic device, characterized in that: include: A processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the log data processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the log data processing method according to any one of claims 1 to 6 are executed.

Images