A log processing method, device, computer equipment and storage medium

Abstract

The present application relates to a log processing method, device, computer equipment and storage medium. The method includes: obtaining at least one log information containing link tracking information; cleaning the log information to obtain valid log information; configuring abnormal log feature information; diverting the log information according to the link tracking information in the valid log information to obtain complete link log information; based on the complete link log information, combined with the abnormal log feature information, making a log abnormality judgment to obtain a judgment result; according to the judgment result, obtaining a corresponding log processing strategy and executing the log processing strategy. The method can quickly classify the logs, generate processing strategies and process the logs.

Description

Technical Field

[0001] The present application relates to the technical field of big data processing, and in particular to a log processing method, apparatus, computer equipment and storage medium. Background Art

[0002] As the business functions of application systems continue to expand, more and more functional modules are involved in the system, and the traditional vertical expansion model can no longer meet the needs. Therefore, many systems choose to adopt distributed and microservice architectures to split different functional modules and have multiple services work together to complete the functions of the overall system. However, the transmission of a single business data in a distributed system usually involves interactions between multiple services, which brings huge challenges to the daily operation and maintenance of the system.

[0003] When a system fails or needs to be troubleshooted, it is difficult to track the complete path of the entire data transmission in the traditional single service logging method because each service generates its own logs and these logs are scattered on different physical nodes. In this case, it is often necessary to start from the source of the business logic and troubleshoot one by one. This method cannot meet the needs of timely fault response and problem location. In order to effectively solve this problem, modern systems usually use link tracing technology to track system calls. Link tracing technology helps to achieve complete tracking of data transmission by creating a tracing context for data transmission in the application system and propagating it on the call link. The sampler records the input, output, and execution time of each service call and stores this information as a separate span object. Subsequently, the collector aggregates all generated span objects and writes them to the analysis system. The analysis system can reconstruct the entire call chain, and users can query the input, output parameters, execution time and other information of each call in different ways.

[0004] However, link tracking technology is mainly used to collect system performance indicators, such as CPU usage, memory consumption, process information, etc., as well as the input and output parameters and time of specific calls. It mainly focuses on monitoring and performance analysis, but cannot effectively process system log information. Its core function is to monitor system performance, and it cannot perform in-depth log analysis. Since link tracking needs to record the details of the entire call chain, this means that a large amount of irrelevant data may be generated, making data storage and retrieval a challenge. At the same time, due to the large amount of data generated by link tracking, some data is usually discarded through the sampling rate algorithm. As a result, some abnormal event data may be missed, affecting the accuracy of troubleshooting. Summary of the invention

[0005] Based on this, it is necessary to provide a log processing method, device, computer equipment and storage medium that can quickly classify logs, generate processing strategies and process logs in response to the above technical problems.

[0006] On the one hand, a log processing method is provided, the method comprising:

[0007] Obtain at least one log information containing link tracking information, where the link tracking information at least includes a link identifier;

[0008] Configure abnormal log feature information;

[0009] Grouping the log information according to the link identifier to obtain multiple log information groups of the same link;

[0010] Based on the same link log information group, combined with abnormal log feature information, log abnormality judgment is performed to obtain a judgment result;

[0011] According to the judgment result, the corresponding log processing strategy is obtained and executed.

[0012] In one embodiment, grouping the log information according to the link identifier to obtain multiple log information groups of the same link includes:

[0013] Extract key fields from log information, filter out duplicate information from log information, and convert log information format into standard format;

[0014] Verify the log information after conversion;

[0015] In response to the log information after the conversion not conforming to the standard format, the log information is discarded.

[0016] In one embodiment, the log information is grouped according to the link identifier to obtain multiple log information groups related to the same link, including:

[0017] The link tracking information also includes a node identifier;

[0018] According to the link identifier in the log information, extract all node identifiers in the same link log;

[0019] All node identifiers under the same data transmission link identifier are connected in series to obtain multiple log information groups of the same link.

[0020] In one embodiment, based on the same link log information group and in combination with abnormal log feature information, log abnormality judgment is performed to obtain a judgment result, including:

[0021] Parsing the same link log information to obtain log key fields, link identifiers, and node identifiers, wherein the log key fields include: one or more of an operation type, a log message, and a timestamp;

[0022] Determine whether the key fields of the log and the characteristic information of the abnormal log match, and obtain the determination result;

[0023] In response to at least one item of information in the key field of the log being the same as the characteristic information of the abnormal log, it is an abnormal log, and the link where the log is located is the abnormal log link;

[0024] In response to the information in the key field of the log being different from the abnormal characteristic information, it is a normal log, and the link where the log is located is a normal log link.

[0025] In one embodiment, according to the judgment result, a corresponding log processing strategy is obtained, and the log processing strategy is executed, including:

[0026] In response to the judgment result being a normal log link, the normal log link is discarded;

[0027] In response to the judgment result being an abnormal log link, the abnormal log link is sent to the retrieval system.

[0028] In one embodiment, in response to the determination result that the abnormal log link is an abnormal log link, the abnormal log link is sent to a retrieval system, including:

[0029] Extract all logs under the abnormal log link according to the link identifier and node identifier in the abnormal log link;

[0030] Sort the logs according to the node identifier to obtain a complete abnormal log link in the correct order;

[0031] Send the complete exception log chain in the correct order to the retrieval system.

[0032] In one embodiment, before obtaining at least one log information containing link tracking information, the process includes:

[0033] Initialize link tracking information;

[0034] propagating link tracking information to at least one processing node;

[0035] In response to the data transmission jumping from the first processing node to the second processing node, the second processing node determines whether the data transmission has a link identifier and a node identifier;

[0036] In response to the existence of the link identifier and the node identifier in the second processing node, determining whether the link identifier and the node identifier conform to a preset format;

[0037] In response to the link identifier and the node identifier conforming to a preset format, generating a child node identifier based on the existing node identifier, appending it to the data transmission and generating a log;

[0038] In response to the link identifier and the node identifier not conforming to a preset format, the data transmission of the current processing node is marked as abnormal, the link identifier of the first processing node is inherited, and the abnormal mark is attached to the data transmission, and a log is generated;

[0039] In response to the data transmission within the second processing node not having the link identifier and the node identifier, a new link identifier and a new node identifier are generated, attached to the data transmission and a log is generated.

[0040] On the other hand, a log processing device is provided, the device comprising:

[0041] A log acquisition module, used to acquire at least one log information containing link tracking information, where the link tracking information at least includes a link identifier;

[0042] The exception log configuration module is used to configure the exception log feature information;

[0043] A log grouping module, used to group log information according to link identifiers to obtain log information of the same link;

[0044] The abnormal log judgment module is used to judge the abnormality of the log based on the same link log information and the abnormal log feature information to obtain the judgment result;

[0045] The execution module is used to obtain the corresponding log processing strategy according to the judgment result and execute the log processing strategy.

[0046] In another aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented:

[0047] Obtain at least one log information containing link tracking information, where the link tracking information at least includes a link identifier;

[0048] Configure abnormal log feature information;

[0049] Grouping the log information according to the link identifier to obtain multiple log information groups of the same link;

[0050] Based on the same link log information group, combined with abnormal log feature information, log abnormality judgment is performed to obtain a judgment result;

[0051] According to the judgment result, the corresponding log processing strategy is obtained and executed.

[0052] In another aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0053] Obtain at least one log information containing link tracking information, where the link tracking information at least includes a link identifier;

[0054] Configure abnormal log feature information;

[0055] Grouping the log information according to the link identifier to obtain multiple log information groups of the same link;

[0056] Based on the same link log information group, combined with abnormal log feature information, log abnormality judgment is performed to obtain a judgment result;

[0057] According to the judgment result, the corresponding log processing strategy is obtained and executed.

[0058] The log processing method, device, computer equipment and storage medium described above, by acquiring and cleaning logs containing link tracking information, remove redundant, erroneous or irrelevant logs, and retain only valid log information. By link diversion and induction of valid logs, the log structure is further simplified, making the entire log processing process clearer and easier to manage. Through the summarized complete link logs, the system can perform automatic abnormality judgment based on pre-configured abnormal features. This process effectively improves the processing speed and accuracy of the system when facing a large amount of log data, avoiding the tediousness and inefficiency of manually reviewing all logs. By combining abnormal judgment with log processing strategies, the system can automatically generate corresponding processing strategies according to the specific abnormal type of the log. It can effectively reduce the interference of irrelevant logs and reduce the false alarm rate. At the same time, through efficient log processing strategies, the possibility of missed reports is reduced, ensuring that the health status of the system is monitored in real time. This can not only ensure a rapid response to problems in the system, but also reduce human intervention and improve the overall system operation and maintenance automation level. In addition, the whole process can improve the operability and traceability of log data. Each log link information is clearly associated with the abnormal mark. Operation and maintenance personnel can quickly trace the specific link where the abnormality occurred through the link information, thereby greatly speeding up troubleshooting. The summarized link log structure makes the log more compact and centralized, which is convenient for subsequent retrieval and analysis. The cleaned log not only reduces storage pressure, but also improves the system's retrieval efficiency for logs, further improving system performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 An application environment diagram of a log processing method in an embodiment;

[0060] Figure 2A flow chart of a log processing method in one embodiment;

[0061] Figure 3 is a structural block diagram of a log processing device in an embodiment;

[0062] Figure 4 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0064] The log processing method provided in this application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 through a network. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices, and the server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0065] In one embodiment, Figure 2 As shown, a log processing method is provided, which is applied to Figure 1 The terminal in is used as an example to illustrate, including the following steps:

[0066] Step 202, obtaining at least one log information containing link tracking information, where the link tracking information at least includes a link identifier;

[0067] Link tracing is a technology used to monitor and analyze the flow of data transmission in distributed systems. By marking the execution steps of each data transmission in the system and assigning unique identifiers to these steps, a clear data transmission link is constructed. Link tracing can effectively monitor the flow of data transmission in distributed systems, locate performance bottlenecks, discover anomalies, and improve the observability of the system.

[0068] Step 204, configure abnormal log feature information;

[0069] Specifically, a set of abnormal features are preset and defined, and the abnormal log features provide a pre-set rule set that can automatically filter out potential anomalies, which can greatly shorten the time required for anomaly identification and improve the response speed of the system.

[0070] Step 206, grouping the log information according to the link identifier to obtain multiple log information groups of the same link;

[0071] Specifically, by diverting logs according to link tracking information, we can ensure that all related logs are classified into the same link. Diverting and classifying logs helps to organize scattered and fragmented log data into a more structured form. By associating link tracking information, the system can clearly integrate cross-service and cross-module log information into a complete and easy-to-analyze link view. This structured log organization method makes the log content more concise and clear.

[0072] Step 208, based on the same link log information group and in combination with the abnormal log feature information, log abnormality judgment is performed to obtain a judgment result;

[0073] Specifically, by combining the complete link log information and the abnormal log feature information, the system can accurately determine whether the log is abnormal based on the preset abnormal feature rules and the context of the log link. The link log information provides the complete path of data transmission from the beginning to the end, which can clearly show the location and context of the abnormality, while the abnormal feature information ensures that the system can accurately identify the real abnormal situation in these logs. This combination makes anomaly detection more accurate and reduces false positives and false negatives. At the same time, it reduces misjudgments and erroneous analysis.

[0074] Step 210: Obtain a corresponding log processing strategy based on the judgment result, and execute the log processing strategy.

[0075] Specifically, by formulating and executing corresponding log processing strategies based on the abnormal log judgment results, the system can achieve automatic response. This means that after detecting abnormal logs, the system can automatically take actions according to the pre-set processing strategies, such as triggering alarms, starting automatic recovery, restarting services, or performing load balancing. This automation reduces the need for manual intervention, improves processing efficiency, and speeds up response time.

[0076] By obtaining and cleaning logs containing link tracking information, redundant, erroneous or irrelevant logs are eliminated, and only valid log information is retained. By link-dividing and summarizing valid logs, the log structure is further simplified, making the entire log processing process clearer and easier to manage. Through the summarized complete link logs, the system can automatically judge anomalies based on pre-configured anomaly features. This process effectively improves the processing speed and accuracy of the system when facing a large amount of log data, avoiding the tediousness and inefficiency of manually reviewing all logs. By combining anomaly judgment with log processing strategies, the system can automatically generate corresponding processing strategies based on the specific anomaly type of the log. It can effectively reduce the interference of irrelevant logs and reduce the false alarm rate. This can not only ensure a quick response to problems in the system, but also reduce human intervention and improve the overall system operation and maintenance automation level. In addition, the entire process can improve the operability and traceability of log data. Each log link information is clearly associated with the anomaly mark, and the operation and maintenance personnel can quickly trace the specific link where the anomaly occurs through the link information, thereby greatly speeding up the troubleshooting.

[0077] In one embodiment,

[0078] Extract key fields from log information, filter out duplicate information from log information, and convert log information format into standard format;

[0079] Verify the log information after conversion;

[0080] In response to the log information after the conversion not conforming to the standard format, the log information is discarded.

[0081] Specifically, by extracting key fields and filtering out duplicate information, redundant and invalid information is removed. This not only simplifies the log content, but also improves the accuracy and simplicity of the log. It reduces the interference of useless data, making the log information more practical. It avoids the storage of invalid data, optimizes the storage resources of logs, reduces the cost of log management, and also reduces the amount of calculation in the system and improves the calculation speed. It can ensure that all logs are presented in a consistent structure, which is convenient for the automated system to quickly process and analyze. The unified standard format simplifies the log parsing process and avoids the additional processing burden caused by inconsistent formats, thereby improving the efficiency of log processing and analysis.

[0082] In one embodiment,

[0083] The link tracking information also includes a node identifier;

[0084] According to the link identifier in the log information, extract all node identifiers in the same link log;

[0085] All node identifiers under the same data transmission link identifier are connected in series to obtain multiple log information groups of the same link.

[0086] Specifically, the valid log information contains multiple data transmissions, each of which contains a link identifier and at least one node identifier. The data transmission is divided according to the link identifier, and the logs of the same data transmission are summarized into a set to obtain the complete log information of the data transmission link. Concatenating all node identifiers into complete log information helps the system to centrally manage logs. Aggregating related logs through link identifiers can avoid logs being scattered in multiple locations or different links, and improve the uniformity of log management.

[0087] In one embodiment,

[0088] Parsing the same link log information to obtain log key fields, link identifiers, and node identifiers, wherein the log key fields include: one or more of an operation type, a log message, and a timestamp;

[0089] Determine whether the key fields of the log and the characteristic information of the abnormal log match, and obtain the determination result;

[0090] In response to at least one item of information in the key field of the log being the same as the characteristic information of the abnormal log, it is an abnormal log, and the link where the log is located is the abnormal log link;

[0091] In response to the information in the key field of the log being different from the abnormal characteristic information, it is a normal log, and the link where the log is located is a normal log link.

[0092] Specifically, the system records logs of multiple applications. Each log contains the following fields: log level, application name, process information, method information, and log content.

[0093] If there are two logs as follows:

[0094] Log 1: Log level: ERROR, Application name: PaymentService, Process information: PID-1234, Method information: processPayment(), Log content: Payment failed due to network timeout;

[0095] Log 2: Log level: INFO, Application name: OrderService, Process information: PID-5678, Method information: placeOrder(), Log content: Order placed successfully;

[0096] Assume that the abnormal log characteristic information is: log level: ERROR, application name: PaymentService, method information: processPayment(), log content: Payment failed due to network timeout;

[0097] Match the specific fields of each log with the abnormal log feature information.

[0098] Result: Log 1 completely matches the abnormal log feature information and is determined to be an abnormal log. The link LNK-123456 is the abnormal log link.

[0099] Result: Log 2 does not match the abnormal log feature information and is determined to be a normal log. The link LNK-123456 where it is located is a normal log link.

[0100] By matching the key fields of the logs with the features of the abnormal logs, you can quickly identify which logs are abnormal logs. The clear distinction between abnormal log links and normal log links helps to manage the system in layers. Storing and analyzing abnormal logs and normal logs separately helps to improve the processing efficiency of log data. By focusing only on the abnormal log links, operation and maintenance personnel can focus on troubleshooting and anomaly detection, avoiding too many normal logs interfering with analysis. It not only improves the timeliness and accuracy of abnormal logs, but also helps to quickly locate problems, optimize log storage and analysis, and improve the operation and maintenance efficiency and reliability of the overall system.

[0101] In one embodiment,

[0102] In response to the judgment result being a normal log link, the normal log link is discarded;

[0103] In response to the judgment result being an abnormal log link, the abnormal log link is sent to the retrieval system.

[0104] Specifically, normal log links are quickly screened out, which can reduce the resource usage and analysis burden on the system. Normal log links usually do not involve system errors or potential problems. Discarding these logs can effectively save storage space, improve processing speed, and avoid wasting energy on a large amount of irrelevant data. Abnormal log links are identified and sent to the retrieval system. This ensures that abnormal situations can be detected and processed centrally in a timely manner. By centrally managing these abnormal log links, the retrieval system can conduct in-depth analysis, root cause investigation, or trigger automated alarms and response mechanisms, thereby improving the response speed and accuracy of problem solving.

[0105] In one embodiment,

[0106] According to the link identifier and the node identifier in the abnormal log link, all logs under the same data transmission link in the abnormal log are extracted;

[0107] Sort the logs according to the node identifier to obtain the complete exception log chain in the correct order.

[0108] Send the complete exception log chain in the correct order to the retrieval system.

[0109] Specifically, by extracting the link identifier and node identifier in the abnormal log link, the system can accurately identify all logs belonging to the same data transmission link. This ensures that the abnormal logs can be aggregated by data transmission link without missing any relevant log information, sorts the logs by node identifiers, ensures the correct order of the logs, and sends the complete abnormal log link in the correct order to the retrieval system, ensuring that the retrieval system can perform in-depth analysis based on the complete and correctly ordered log link, thereby quickly identifying the cause of the system failure.

[0110] In one embodiment,

[0111] Initialize link tracking information;

[0112] propagating link tracking information to at least one processing node;

[0113] In response to the data transmission jumping from the first processing node to the second processing node, the second processing node determines whether the data transmission has a link identifier and a node identifier;

[0114] In response to the existence of the link identifier and the node identifier in the second processing node, determining whether the link identifier and the node identifier conform to a preset format;

[0115] In response to the link identifier and the node identifier conforming to a preset format, generating a child node identifier based on the existing node identifier, appending it to the data transmission and generating a log;

[0116] In response to the link identifier and the node identifier not conforming to a preset format, the data transmission of the current processing node is marked as abnormal, the link identifier of the first processing node is inherited, and the abnormal mark is attached to the data transmission, and a log is generated;

[0117] In response to the data transmission within the second processing node not having the link identifier and the node identifier, a new link identifier and a new node identifier are generated, attached to the data transmission and a log is generated.

[0118] Specifically, by including link tracking information in data transmission, it is possible to ensure that each link can inherit and propagate link identifiers and node identifiers, so that data transmission flows between different services in the system can be tracked uniformly. No matter how many services and modules the data transmission spans, it can be visualized and tracked in a unified link view, improving the transparency and manageability of the system. This process improves the traceability of data transmission, enhances the relevance of log data, and supports the dynamic generation of new link information, ensuring that link tracking of complex data transmission is not lost.

[0119] In one embodiment,

[0120] In response to at least one item of information in the key field of the log being identical to the abnormal log characteristic information, marking the log in the field identical to the abnormal log characteristic information as abnormal;

[0121] Update the abnormal link log according to the abnormal mark, and send the abnormal link log to the retrieval system;

[0122] By inputting the link identifier and the abnormal mark of the abnormal link in the retrieval system, the abnormal field in the abnormal link is highlighted.

[0123] Among them, the exception mark is a unified identifier, such as error code 001 or error code 002. When the system detects that the key fields of the log match the defined abnormal characteristic information such as error information, these abnormal fields will be marked.

[0124] Specifically, the abnormal link log is updated according to the abnormal mark and sent to the retrieval system, ensuring the centralized management and immediate response of the abnormal log in the system. By associating the abnormal mark information with the abnormal link log, the retrieval system can quickly locate the complete log link containing the abnormal field, thereby avoiding the inefficient operation of searching for relevant information one by one in multiple log entries. By entering the link identifier and abnormal mark, the operation and maintenance personnel can quickly identify the specific abnormal location, avoiding the tedious process of manually searching for problems through lengthy log contents. Highlighting not only improves the efficiency of log analysis, but also improves the efficiency of abnormal log processing and the speed of fault response.

[0125] By automatically marking, updating, and highlighting abnormal fields, the system can accurately identify and track abnormal problems in a short period of time, and operation and maintenance personnel can quickly locate the specific location of the problem in the abnormal log link, thereby solving the problem efficiently. In addition, the integration of the retrieval system further improves the operability and queryability of log data, providing a strong guarantee for the stability and maintainability of the system. This efficient and accurate abnormal log management method has significantly improved the overall operation and maintenance efficiency of the system.

[0126] It should be understood that although Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0127] In one embodiment, Figure 3 As shown, a log processing device is provided, including: a log acquisition module 302, an abnormal log configuration module 304, a log grouping module 306, an abnormal log judgment module 308 and an execution module 310, wherein:

[0128] The log acquisition module 302 acquires at least one log information containing link tracking information, where the link tracking information at least includes a link identifier;

[0129] The abnormal log configuration module 304 is used to configure abnormal log feature information;

[0130] The log grouping module 306 is used to group the log information according to the link identifier to obtain multiple log information groups of the same link;

[0131] The abnormal log judgment module 308 judges the abnormality of the log based on the same link log information group and the abnormal log feature information to obtain a judgment result;

[0132] The execution module 310 obtains a corresponding log processing strategy according to the judgment result and executes the log processing strategy.

[0133] In one embodiment, the log grouping module 306 includes:

[0134] The link tracking information also includes a node identifier;

[0135] Used to extract all node identifiers in the same link log according to the link identifier in the log information;

[0136] It is used to connect all node identifiers under the same data transmission link identifier in series to obtain multiple log information groups of the same link.

[0137] In one embodiment, the abnormal log determination module 308 includes:

[0138] Used to initialize link tracking information;

[0139] for propagating link tracking information to at least one processing node;

[0140] for, in response to a data transmission jumping from a first processing node to a second processing node, the second processing node determining whether a link identifier and a node identifier exist in the data transmission;

[0141] In response to the existence of the link identifier and the node identifier in the second processing node, determining whether the link identifier and the node identifier conform to a preset format;

[0142] In response to the link identifier and the node identifier conforming to a preset format, generating a child node identifier according to the existing node identifier, attaching it to the data transmission and generating a log;

[0143] In response to the link identifier and the node identifier not conforming to a preset format, the data transmission of the current processing node is marked as abnormal, the link identifier of the first processing node is inherited, and the abnormal mark is attached to the data transmission, and a log is generated;

[0144] In response to the absence of a link identifier and a node identifier in the data transmission within the second processing node, a new link identifier and a new node identifier are generated, attached to the data transmission, and a log is generated.

[0145] For the specific definition of the log processing device, please refer to the definition of the log processing method above, which will not be repeated here. Each module in the above-mentioned log processing device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0146] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 4As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a log processing method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0147] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0148] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0149] Obtain at least one log message containing link tracking information;

[0150] Clean the log information to obtain valid log information; configure abnormal log feature information; divert the log information according to the link tracking information in the valid log information to obtain complete link log information; based on the complete link log information and combined with the abnormal log feature information, make a log abnormality judgment to obtain a judgment result; according to the judgment result, obtain the corresponding log processing strategy and execute the log processing strategy.

[0151] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0152] Extract key fields from the log information, filter out duplicate information from the log information, and convert the log information format into a standard format; verify the valid log information after the conversion; and discard the valid log information in response to the valid log information after the conversion format not conforming to the standard format.

[0153] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0154] According to the link identifier in the valid log information, all node identifiers under the same data transmission link identifier in the valid log are extracted; all node identifiers under the same data transmission link identifier are connected in series to obtain the complete log information of the same data transmission link.

[0155] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0156] Initialize link tracking information;

[0157] propagating link tracking information to at least one processing node;

[0158] In response to the data transmission jumping from the first processing node to the second processing node, the second processing node determines whether the data transmission has a link identifier and a node identifier;

[0159] In response to the existence of the link identifier and the node identifier in the second processing node, determining whether the link identifier and the node identifier conform to a preset format;

[0160] In response to the link identifier and the node identifier conforming to a preset format, generating a child node identifier based on the existing node identifier, appending it to the data transmission and generating a log;

[0161] In response to the link identifier and the node identifier not conforming to a preset format, the data transmission of the current processing node is marked as abnormal, the link identifier of the first processing node is inherited, and the abnormal mark is attached to the data transmission, and a log is generated;

[0162] In response to the data transmission within the second processing node not having a link identifier and a node identifier, a new link identifier and a new node identifier are generated, attached to the data transmission and a log is generated. In one embodiment, the computer program further implements the following steps when executed by the processor:

[0163] Parsing the same link log information to obtain log key fields, link identifiers, and node identifiers, wherein the log key fields include: one or more of an operation type, a log message, and a timestamp;

[0164] Determine whether the key fields of the log and the characteristic information of the abnormal log match, and obtain the determination result;

[0165] In response to at least one item of information in the key field of the log being the same as the characteristic information of the abnormal log, it is an abnormal log, and the link where the log is located is the abnormal log link;

[0166] In response to the information in the key field of the log being different from the abnormal characteristic information, it is a normal log, and the link where the log is located is a normal log link.

[0167] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0168] The abnormal log link includes at least one link identifier and a node identifier; according to the link identifier and the node identifier in the abnormal log link, all logs under the same data transmission link in the abnormal log are extracted; the logs are sorted according to the node identifier to obtain a complete abnormal log link in the correct order, and the complete abnormal log link in the correct order is sent to the retrieval system.

[0169] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0170] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0171] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A log processing method, characterized in that: include: Obtain at least one log information containing link tracking information, wherein the link tracking information at least includes a link identifier; Configure abnormal log feature information; Grouping the log information according to the link identifier to obtain a plurality of log information groups of the same link; Based on the same link log information group and in combination with the abnormal log feature information, a log abnormality judgment is performed to obtain a judgment result; According to the judgment result, a corresponding log processing strategy is obtained and the log processing strategy is executed.

2. A log processing method according to claim 1, characterized in that: The log information is grouped according to the link identifier to obtain a plurality of log information groups of the same link, including: Extract key fields from the log information, filter out duplicate information from the log information, and convert the log information format into a standard format; Verifying the log information after the conversion; In response to the log information after the format conversion not conforming to the standard format, the log information is discarded.

3. A log processing method according to claim 2, characterized in that: The log information is grouped according to the link identifier to obtain multiple log information groups related to the same link, including: The link tracking information also includes a node identifier; Extracting all node identifiers in the same link log according to the link identifier in the log information; All node identifiers under the same data transmission link identifier are connected in series to obtain multiple log information groups of the same link.

4. A log processing method according to claim 3, characterized in that: Based on the same link log information group and in combination with the abnormal log feature information, a log abnormality judgment is performed to obtain a judgment result, including: Parsing the same link log information to obtain log key fields, link identifiers, and node identifiers, wherein the log key fields include: one or more of an operation type, a log message, and a timestamp; Determine whether the log key field and the abnormal log characteristic information match, and obtain a determination result; In response to at least one item of information in the key field of the log being the same as the characteristic information of the abnormal log, it is an abnormal log, and the link where the log is located is an abnormal log link; In response to the information in the key field of the log being different from the abnormal characteristic information, it is a normal log, and the link where the log is located is a normal log link.

5. A log processing method according to claim 4, characterized in that: According to the judgment result, a corresponding log processing strategy is obtained, and the log processing strategy is executed, including: In response to the judgment result being a normal log link, discarding the normal log link; In response to the determination result being an abnormal log link, the abnormal log link is sent to a retrieval system.

6. A log processing method according to claim 5, characterized in that: In response to the judgment result being an abnormal log link, sending the abnormal log link to a retrieval system, including: Extracting all logs under the abnormal log link according to the link identifier and the node identifier in the abnormal log link; Sort the logs according to the node identifiers to obtain a complete abnormal log link in the correct order; The complete exception log link in the correct order is sent to the retrieval system.

7. A log processing method according to claim 1, characterized in that: Before obtaining at least one log message containing link tracking information, include: Initializing the link tracking information; propagating the link tracking information to at least one processing node; In response to the data transmission jumping from the first processing node to the second processing node, the second processing node determines whether the data transmission has the link identifier and the node identifier; In response to the presence of the link identifier and the node identifier in the second processing node, determining whether the link identifier and the node identifier conform to a preset format; In response to the link identifier and the node identifier conforming to a preset format, generating a child node identifier according to the existing node identifier, appending the child node identifier to the data transmission and generating a log; In response to the link identifier and the node identifier not conforming to a preset format, marking the data transmission of the current processing node as abnormal, inheriting the link identifier of the first processing node, and attaching an abnormal mark to the data transmission, and generating a log; In response to the data transmission within the second processing node not having the link identifier and the node identifier, a new link identifier and a new node identifier are generated, attached to the data transmission, and a log is generated.

8. A log processing device, characterized in that: include: A log acquisition module, used to acquire at least one log information containing link tracking information, wherein the link tracking information at least includes a link identifier; The exception log configuration module is used to configure the exception log feature information; A log grouping module, used for grouping the log information according to the link identifier to obtain log information of the same link; An abnormal log judgment module is used to judge the abnormality of the log based on the same link log information and the abnormal log feature information to obtain a judgment result; The execution module is used to obtain a corresponding log processing strategy according to the judgment result and execute the log processing strategy.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images