A method and system for analyzing a multi-system call process based on a log chain

Abstract

This invention provides a method for analyzing multi-system call processes based on log chains, characterized by steps including: defining a custom log protocol; generating application logs; extracting logs from services; and analyzing the service log chain. Another aspect of this invention discloses a system for analyzing multi-system call processes based on log chains, characterized by its function of implementing the aforementioned method. This invention employs a log chain approach, allowing users to extract and arrange the logs of completed business processes in a pre-organized manner, eliminating the need to view logs individually in each application—simple and efficient. The log component outputs logs in a unified and concise format, and the logging methods of each application are controllable, masking application differences and significantly improving log readability.

Description

Technical Field

[0001] This invention relates to a method and system for unified integration and analysis of log chains from multiple systems to reconstruct the call process and quickly locate anomalies. Background Technology

[0002] The stability and real-time operational status of a program are primary concerns for most systems after deployment. Given that debugging with breakpoints is not easily performed during program execution, logging systems have become a widely used method for reflecting operational status and reporting anomalies. Currently, commonly used logging components and libraries such as log4j, logback, and SLF4J offer good support for flexible configuration of print formats, output targets, and output filtering, allowing for customization of log formats based on program characteristics and needs. Developers and operations personnel can understand the program's operational status and monitor for anomalies by reading program logs.

[0003] With the development of technologies such as internet microservices and distributed systems, the functions of a system are no longer undertaken by a single program. The business of a system tends to be decoupled and functionally singular. Processing a business process often requires the cooperation of multiple services and programs. Therefore, it is particularly important to manage the calling process of each program in a single business process and to locate problems when they occur.

[0004] Traditional program log printing only focuses on the current program itself. However, a system's business logic is often handled by multiple applications, each with its own single responsibility, to achieve the complete business function. In this case, traditional program log printing and analysis will have the following problems:

[0005] 1. The log viewing process is cumbersome:

[0006] When reviewing logs, it is necessary to identify each application involved in the business and check the log directory of each application.

[0007] 2. Low log readability:

[0008] Log components or log libraries support custom log formats and output configurations. Different programs handle logs differently, so you need to check the program status according to the different program log formats when reading them.

[0009] 3. Difficulty in pinpointing the problem:

[0010] When an anomaly occurs during system business processing, it is difficult to identify the application causing the problem. It is necessary to check the logs of all involved programs one by one to identify the problematic program.

[0011] 4. The overall operating status of the system is difficult to ascertain:

[0012] Although logs can provide feedback on the current status of one's own application, they cannot provide information on the overall business status and metrics of the entire system, such as overall business performance, anomaly distribution, and trends.

[0013] 5. Timeliness issues:

[0014] Traditional logging methods often only involve checking logs to investigate anomalies when problems occur online and are reflected in system business anomalies, resulting in a lack of timeliness in exposing problems.

[0015] 6. High labor maintenance costs:

[0016] All of the above-mentioned operations, maintenance, troubleshooting, and procedures require a significant investment of human resources in development and maintenance; this high investment does not yield any real or effective returns. Summary of the Invention

[0017] The purpose of this invention is to utilize a unified log format, associate all application logs involved according to business processes, assemble them into a log chain, thereby reconstructing the overall business processing process, marking the status, time consumption, error points, etc. of each program, and realizing the overall management and monitoring system.

[0018] To achieve the above objectives, the technical solution of this invention discloses a method for analyzing multiple system call processes based on log chains, characterized by comprising:

[0019] The steps for customizing the log protocol include log header information, business parameters, and execution information. The log header information includes the processing chain ID, and the execution information includes the time taken for the program to execute business processing, status information, and error information.

[0020] The steps involved in generating application logs include:

[0021] Step 101: The application calls the logging component to enable logging;

[0022] Step 102: The logging component creates a thread-local variable to record and save all log information of the current business thread in subsequent steps;

[0023] Step 103: The logging component obtains the log chain ID from the upstream. If it cannot obtain it, the current application is the starting point of the business processing. A new log chain ID is generated and encapsulated as log header information and recorded in the log information.

[0024] Step 104: The application executes all business steps. When executing each business step, the logging component is triggered to record the business step information, and the status, time consumption, and error information of the business step are recorded in the log information.

[0025] Step 105: After all business steps of the application have been executed, the logging component sums up the total time of all business steps to calculate the total time of the program's business.

[0026] Step 106: The log component parses each business step, determines whether it contains error information, and if so, generates a step error code and records the error information.

[0027] Step 107: After the application's business logic is completed, call the logging component to disable logging.

[0028] Step 108: The logging component obtains thread-local variables, retrieves log information, and assembles logs according to the logging protocol;

[0029] Step 109: The logging component will assemble the logs and print them to a log file in a unified directory.

[0030] The steps for extracting service logs include:

[0031] Step 201: Obtain configuration information;

[0032] Step 202: Start the log fetching thread;

[0033] Step 203: Obtain the current application identifier, instance identifier, and log storage directory based on the configuration information;

[0034] Step 204: Obtain the file stream and read the log file according to the log storage directory;

[0035] Step 205: Wait for the application to write to the log, and read the latest log line by line;

[0036] Step 206: Obtain log information, assemble log messages, and supplement the log header with application information and current instance information;

[0037] Step 207: Call the analysis service receiving interface to send log messages to the analysis service;

[0038] Step 208: Determine if the current thread has stopped. If it has not stopped, continue to wait for the log to be written, read the next log entry, and repeat steps 205, 206, and 207 to pull and send the log.

[0039] Step 209: If a stop command is received, close the current read thread and stop the log fetching operation;

[0040] The steps involved in analyzing service log chains include:

[0041] Step 301: Perform initialization tasks;

[0042] Step 302: Start receiving thread T1 to receive log messages sent by the extraction service;

[0043] Step 303: Start the log chain assembly thread T2 to assemble the received logs into a log chain;

[0044] Step 304: Create a log assembly queue to temporarily store log messages received by receiving thread T1;

[0045] Step 305: Create a log chain cache to store the log chains being assembled.

[0046] Preferably, the log header information includes the processing chain ID and instance information, wherein:

[0047] For the processing chain ID, all programs in a single business process have the same processing chain ID, which is used to connect the log chains.

[0048] Instance information is used to record the currently running instance of the application.

[0049] Preferably, the service parameters are used to identify custom information recorded by each program.

[0050] Preferably, the execution information includes key information recorded in the log, information on the time consumed by the program's business processing, and information on sub-steps.

[0051] Preferably, the key information recorded in the log includes the status and error codes used to mark errors;

[0052] The sub-step information refers to the status and time consumption records of each operation involved in the processing within the application.

[0053] Preferably, in step 302, the receiving thread T1 includes the following steps:

[0054] Step 3021: Receive log information, verify the log header, and determine whether the application is legitimate;

[0055] Step 3022: Call the log assembly queue and add the logs to the queue;

[0056] Step 3023: If a termination command is received, terminate reception and stop; otherwise, continue receiving log information and repeat steps 3021 and 3022.

[0057] Preferably, in step 303, the log chain assembly thread T2 includes the following steps:

[0058] Step 3031: Call the log assembly queue and read a log message A;

[0059] Step 3032: Obtain the log chain configuration associated with the business based on the application information in the log header of log information A;

[0060] Step 3033: Obtain the link ID of log information A and match it in the log chain cache to see if a log chain with the same ID exists in the cache. If it does not exist, add log information A to the log chain cache and start from step 3031 to retrieve a new log information from the queue. If it exists, retrieve the matching corresponding log chain B from the cache.

[0061] Step 3034: Based on the log chain configuration obtained in step 3032, arrange the current log information A into log chain B according to the business link rules to generate log chain C;

[0062] Step 3035: Calculate and update the total time taken for log chain C;

[0063] Step 3036: Determine whether log chain C contains error code information. If it does, integrate the error codes and update the error identifier.

[0064] Step 3037: Determine whether the business executed by log chain C has ended based on the business configuration: If it has not ended, add it back to the log chain cache and wait for subsequent logs to be assembled completely; otherwise, put the completed log chain C into the database for subsequent platform queries.

[0065] Another aspect of the present invention discloses a multi-system call process analysis system based on log chains, characterized in that, for implementing the above-mentioned multi-system call process analysis method based on log chains, it includes:

[0066] A custom logging protocol module is used to implement the steps of the custom logging protocol described above;

[0067] The application log generation module is used to implement the above steps for generating application logs.

[0068] The service log extraction module is used to implement the above-mentioned steps of extracting service logs.

[0069] The service log chain analysis module is used to implement the steps of the service log chain analysis described above.

[0070] Compared with existing technical solutions, the beneficial effects of the present invention are as follows:

[0071] 1. By using a log chain approach, users can extract the logs of the business process and display them in an organized manner, without having to view the logs in each application individually, which is simple and efficient;

[0072] 2. The logging component outputs logs in a unified and concise format, and the logging methods of each application are controllable, which hides the differences between applications and greatly improves the readability of logs.

[0073] 3. Thanks to the design of error codes, the logging component packages program error information into error codes when recording logs. The analysis service connects the logs of various applications according to the business chain to display the processing process and error codes. When an exception occurs, the problem steps and corresponding applications can be quickly located by relying on the chain and error codes.

[0074] 4. The analysis service uniformly processes the log chain, abstracting the business process into log behavior. Developers can statistically analyze the abnormal situations in each link of the business based on the key information of the assembled log chain (step time, overall time, error code distribution, etc.), which is conducive to obtaining the overall working status and timely discovering potential problems. Attached Figure Description

[0075] Figure 1 This illustrates a method for analyzing multiple system call processes based on log chains and a system log protocol;

[0076] Figure 2 This diagram illustrates a method for analyzing multiple system call processes based on log chains and the overall system structure.

[0077] Figure 3 Generate flowcharts for application logs;

[0078] Figure 4 Flowchart for extracting logs from the service;

[0079] Figure 5 Flowchart for analyzing service log chain analysis. Detailed Implementation

[0080] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0081] The technical solution disclosed in this invention defines a custom logging protocol for marking key log information, unifying log output across all applications, and assembling log chains according to the protocol, etc. Figure 1 As shown. The log protocol disclosed in this embodiment of the invention includes the following information:

[0082] 1) Log header information: contains processing chain ID and instance information. Among them: the processing chain ID is the same for all programs in a single business, and is used to connect the log chains; the instance information is used to record the running instance of the current application.

[0083] 2) Business parameters: Each program records custom business information.

[0084] 3) Execution information, including:

[0085] a) Key log entries include status and error codes used to mark errors;

[0086] b) Time consumed during program execution of business processes;

[0087] c) Sub-step information, which records the status and time consumption of each operation involved in the processing within the application.

[0088] The overall structure and process of the solution disclosed in the embodiments of the present invention are as follows: Figure 2 As shown, it mainly includes three modules and corresponding processes: application log generation process, service log extraction process, and service log chain analysis process.

[0089] 1) The main process of application log generation is as follows: Figure 3 As shown, the specific steps include:

[0090] Step 1: The application calls the logging component to enable logging;

[0091] Step 2: The logging component creates a thread-local variable to record and save all log information of the current business thread in subsequent steps;

[0092] Step 3: The logging component obtains the log chain ID from the upstream. If it cannot obtain it, the program becomes the starting point of business processing, generates a new log chain ID, encapsulates it as log header information, and records it in the log information.

[0093] Step 4: The application executes the step, triggering the logging component to record step information, and records the step's status, time consumption, error information, etc., in the log information;

[0094] The application continues to execute the next step, and so on, until all steps have been completed;

[0095] Step 5: After the business steps are completed, the logging component will add up the total time of each step to calculate the total time of the program's business.

[0096] Step 6: The logging component parses each step, determines whether it contains error information, and if so, generates a step error code and records the error information.

[0097] Step 7: After the application's business logic is completed, call the logging component to disable logging.

[0098] Step 8: The logging component obtains thread-local variables, retrieves log information, and assembles the log according to the logging protocol;

[0099] Step 9: The logging component will assemble the logs and print them to a log file in a unified directory.

[0100] 2) The main process of extracting service logs is as follows: Figure 4 As shown, the specific steps include:

[0101] Step 1: Obtain configuration information, including the address information of the applications involved in the business and the target address of the analysis service;

[0102] Step 2: Start the log fetching thread;

[0103] Step 3: Obtain the current application identifier, instance identifier, and log storage directory based on the configuration information;

[0104] Step 4: Obtain the file stream and read the log file based on the log storage directory;

[0105] Step 5: Wait for the application to write to the log, and read the latest log line by line;

[0106] Step 6: Obtain log information, assemble log messages, and supplement the log header with application information and current instance information;

[0107] Step 7: Call the analysis service receiving interface to send log messages to the analysis service;

[0108] Step 8: Determine if the current thread has stopped. If it has not stopped, continue to wait for the log to be written, read the next log entry, and repeat steps 5, 6, and 7 to pull and send the log.

[0109] Step 9: If a stop command is received, close the current reading thread and stop the log fetching operation.

[0110] 3) The main process of analyzing service log chains is as follows: Figure 5 As shown, the specific work and steps include the following:

[0111] a) Initialization work:

[0112] Step 1: Perform initialization tasks;

[0113] Step 2: Start the receiving thread T1 to receive log messages sent by the extraction service;

[0114] Step 3: Start the log chain assembly thread T2, which is used to assemble the received logs into a log chain;

[0115] Step 4: Create a log assembly queue to temporarily store log messages received by receiving thread T1;

[0116] Step 5: Create a log chain cache to store the log chains being assembled.

[0117] b) Main process steps of receiving thread T1:

[0118] Step 1: Receive log information, verify the log header, and determine whether the application is legitimate;

[0119] Step 2: Call the log assembly queue and add the logs to the queue;

[0120] Step 3: If a termination command is received, stop receiving the log information; otherwise, continue receiving log information and repeat steps 1 and 2.

[0121] c) Main process steps of log chain assembly thread T2:

[0122] Step 1: Call the log assembly queue and read a log message A;

[0123] Step 2: Obtain the log chain configuration associated with the business based on the application information in the log header of log information A;

[0124] Step 3: Obtain the link ID of log information A and match it in the log chain cache to see if a log chain with the same ID exists in the cache. If it does not exist, add log information A to the log chain cache and start from step 1 to retrieve a new log information from the queue. If it exists, retrieve the corresponding matching log chain B from the cache.

[0125] Step 4: Based on the log chain configuration obtained in Step 2, arrange the current log information A into log chain B according to the business link rules to generate log chain C;

[0126] Step 5: Calculate and update the total time taken for log chain C;

[0127] Step 6: Determine whether log chain C contains error code information. If it does, integrate the error codes and update the error identifier.

[0128] Step 7: Determine whether the business executed by log chain C has ended based on the business configuration. If it has not ended, add it back to the log chain cache and wait for subsequent logs to be assembled. Otherwise, put the completed log chain C into the database for subsequent platform queries.

Claims

1. A method for analyzing multi-system call processes based on log chains, characterized in that, include: The steps for customizing the log protocol include log header information, business parameters, and execution information. The log header information includes the processing chain ID, and the execution information includes the time taken for the program to execute business processing, status information, and error information. The steps involved in generating application logs include: Step 101: The application calls the logging component to enable logging; Step 102: The logging component creates a thread-local variable to record and save all log information of the current business thread in subsequent steps; Step 103: The logging component obtains the log chain ID from the upstream. If it cannot obtain it, the current application is the starting point of the business processing. A new log chain ID is generated and encapsulated as log header information and recorded in the log information. Step 104: The application executes all business steps. When executing each business step, the logging component is triggered to record the business step information, and the status, time consumption, and error information of the business step are recorded in the log information. Step 105: After all business steps of the application have been executed, the logging component sums up the total time of all business steps to calculate the total time of the program's business. Step 106: The log component parses each business step, determines whether it contains error information, and if so, generates a step error code and records the error information. Step 107: After the application's business logic is completed, call the logging component to disable logging. Step 108: The logging component obtains thread-local variables, retrieves log information, and assembles logs according to the logging protocol; Step 109: The logging component will assemble the logs and print them to log files in a unified directory; The steps for extracting service logs include: Step 201: Obtain configuration information; Step 202: Start the log fetching thread; Step 203: Obtain the current application identifier, instance identifier, and log storage directory based on the configuration information; Step 204: Obtain the file stream and read the log file according to the log storage directory; Step 205: Wait for the application to write to the log, and read the latest log line by line; Step 206: Obtain log information, assemble log messages, and supplement the log header with application information and current instance information; Step 207: Call the analysis service receiving interface to send log messages to the analysis service; Step 208: Determine if the current thread has stopped. If it has not stopped, continue to wait for the log to be written, read the next log entry, and repeat steps 205, 206, and 207 to pull and send the log. Step 209: If a stop command is received, close the current read thread and stop the log fetching operation; The steps involved in analyzing service log chains include: Step 301: Perform initialization tasks; Step 302: Start receiving thread T1 to receive log messages sent by the extraction service; Step 303: Start the log chain assembly thread T2 to assemble the received logs into a log chain; Step 304: Create a log assembly queue to temporarily store log messages received by receiving thread T1; Step 305: Create a log chain cache to store the log chains being assembled.

2. The method for analyzing multi-system call processes based on log chains as described in claim 1, characterized in that, The log header information includes the processing chain ID and instance information, wherein: For the processing chain ID, all programs in a single business process have the same processing chain ID, which is used to connect the log chains. Instance information is used to record the currently running instance of the application.

3. The method for analyzing multiple system call processes based on log chains as described in claim 1, characterized in that, The business parameters are used to identify the custom information recorded by each program.

4. The method for analyzing multi-system call processes based on log chains as described in claim 1, characterized in that, The execution information includes key information recorded in the logs, information on the time consumed by the program's business processing, and information on sub-steps.

5. The method for analyzing multi-system call processes based on log chains as described in claim 4, characterized in that, The key information recorded in the log includes the status and error codes used to mark errors; The sub-step information refers to the status and time consumption records of each operation involved in the processing within the application.

6. The method for analyzing multi-system call processes based on log chains as described in claim 1, characterized in that, In step 302, the receiving thread T1 includes the following steps: Step 3021: Receive log information, verify the log header, and determine whether the application is legitimate; Step 3022: Call the log assembly queue and add the logs to the queue; Step 3023: If a termination command is received, terminate reception and stop; otherwise, continue receiving log information and repeat steps 3021 and 3022.

7. The method for analyzing multi-system call processes based on log chains as described in claim 1, characterized in that, In step 303, the log chain assembly thread T2 includes the following steps: Step 3031: Call the log assembly queue and read a log message A; Step 3032: Obtain the log chain configuration associated with the business based on the application information in the log header of log information A; Step 3033: Obtain the link ID of log information A and match it in the log chain cache to see if a log chain with the same ID exists in the cache. If it does not exist, add log information A to the log chain cache and start from step 3031 to retrieve a new log information from the queue. If it exists, retrieve the matching corresponding log chain B from the cache. Step 3034: Based on the log chain configuration obtained in step 3032, arrange the current log information A into log chain B according to the business link rules to generate log chain C; Step 3035: Calculate and update the total time taken for log chain C; Step 3036: Determine whether log chain C contains error code information. If it does, integrate the error codes and update the error identifier. Step 3037: Determine whether the business executed by log chain C has ended based on the business configuration: If it has not ended, add it back to the log chain cache and wait for subsequent logs to be assembled completely; otherwise, put the completed log chain C into the database for subsequent platform queries.

8. A multi-system call process analysis system based on log chains, characterized in that, To implement the log chain-based multi-system call process analysis method as described in claim 1, the method includes: A custom log protocol module is used to implement the steps of the custom log protocol as described in claim 1; An application log generation module is used to implement the application log generation steps described in claim 1; The service extraction log module is used to implement the steps of extracting service logs as described in claim 1. The service log chain analysis module is used to implement the service log chain analysis steps described in claim 1.

Images