Multi-system calling process analysis method and system based on log chain

Abstract

The invention provides a multi-system calling process analysis method based on a log chain, which is characterized by comprising the following steps of: customizing a log protocol; an application program log is generated; a step of extracting the log by the extraction service; and analyzing the service log chain. On the other hand, the invention discloses a multi-system calling process analysis system based on the log chain, which is characterized by being used for realizing the multi-system calling process analysis method based on the log chain. According to the method, a log chain mode is adopted, a user can take out a link log after service completion and display the link log in a well arranged mode, the log does not need to be checked independently in each application program, and the method is simple and efficient; the log component outputs the logs in a unified and simple log format, the log printing mode of each application is controllable, the difference of the applications is shielded, and the log readability is greatly improved.

Description

Technical Field

[0001] The present invention relates to a method and system for unified integration and analysis of log chains based on multiple systems to restore the call process and quickly locate anomalies. Background Art

[0002] The stability of the program and its running state at all times are the primary concerns after the vast majority of systems are launched. Given that it is not easy to perform operations such as breakpoint debugging during program operation, the log system is currently widely used as a means to reflect the running state and report anomalies. Currently, log components and libraries such as log4j, logback, and SLF4J, which are widely used in the industry, have good support for flexible configuration of printing formats, output targets, filtering outputs, etc., and can customize the form of logs according to program characteristics and needs. Developers or operators can understand the program running state and monitor anomalies by reading the program logs.

[0003] With the development of technologies such as Internet microservices and distributed systems, the functions of a system are no longer borne by a single program. The business of a set of systems is more inclined to decoupling and functional singleness. Processing a business process often requires the cooperation of multiple services and programs. Therefore, how to manage the call process of each program during a single business process and locate problems when they occur becomes particularly important.

[0004] Currently, the printing of traditional program logs only focuses on the current program itself. However, the business of a system is often borne by multiple program applications, each responsible for a single duty to achieve a complete business. In this case, there will be the following problems in the printing and analysis of traditional program logs:

[0005] 1. The log viewing process is cumbersome:

[0006] When viewing logs, it is necessary to confirm each application program involved in the business and view them in the log directory of each program.

[0007] 2. The log readability is low:

[0008] Log components or log libraries support customizing log formats and output configurations. The log processing methods among different programs are not the same. When reading, it is necessary to confirm the program state according to different program log formats.

[0009] 3. It is difficult to locate problems when they occur:

[0010] When an anomaly occurs during the system business processing process, it is difficult to confirm the application program where the problem occurs. It is necessary to check the logs of all involved programs one by one to confirm the problem program.

[0011] 4. It is difficult to obtain the overall operating conditions of the system:

[0012] Although the current status of the application itself can be reflected based on logs, the operating conditions and metrics of the entire system's business cannot be obtained, such as information on overall business performance, exception distribution, trends, etc.

[0013] 5. Timeliness issue:

[0014] For the traditional log recording mode, problems often occur online and are reflected in system business exceptions before logs are checked to troubleshoot exceptions, lacking timeliness in problem exposure.

[0015] 6. High manual maintenance cost:

[0016] All of the above operations, maintenance, troubleshooting, and operations require a large amount of human resources such as development and operation and maintenance. High investment cannot obtain actual effective returns. Summary of the Invention

[0017] The object of the present invention is to use a unified log format, associate all application program logs involved according to the business process, assemble them into a log chain, thereby restoring the overall business processing process, marking the status, time consumption, error points, etc. of each program, and realizing the overall management and monitoring system.

[0018] To achieve the above object, the technical solution of the present invention discloses a method for analyzing the multi-system call process based on a log chain, which is characterized by including:

[0019] The step of customizing the log protocol, wherein the log protocol includes log header information, business parameters, and execution information. Among them, the log header information includes the processing chain id, and the execution information includes the time consumption information, status information, and error information of the program executing the business process.

[0020] The step of generating application program logs, including:

[0021] Step 101: The application program calls the log component to start log recording.

[0022] Step 102: The log component creates thread-local variables for subsequent steps to record and save all log information of the current business thread.

[0023] Step 103: The log component obtains the log chain id from the upstream. If it cannot be obtained, the current application program is the starting point of the business process, and a new log chain id is generated and encapsulated as log header information and recorded in the log information.

[0024] Step 104: The application program executes all business steps. When each business step is executed, it triggers the log component to record the business step information event, and records the status, time consumption, and error information of the business step in the log information.

[0025] Step 105: After all business steps of the application are completed, the logging component accumulates the total time taken for all business steps and calculates the total business time of the program.

[0026] Step 106: The logging component parses each business step to determine if it contains error information. If it does, it generates a step error code and records the error information.

[0027] Step 107: After the application's business execution is completed, the logging component is called to close the logging.

[0028] Step 108: The logging component retrieves the thread-local variable, extracts the log information, and assembles the log according to the logging protocol.

[0029] Step 109: The logging component prints the assembled log to a log file in a unified directory.

[0030] Steps for the extraction service to extract logs include:

[0031] Step 201: Obtain configuration information.

[0032] Step 202: Start the log pulling thread.

[0033] Step 203: Obtain the current application identifier, instance identifier, and log storage directory according to the configuration information.

[0034] Step 204: Obtain a file stream based on the log storage directory and read the log file.

[0035] Step 205: Wait for the application to write logs and read the latest logs line by line.

[0036] Step 206: Obtain the log information, assemble the log message, and supplement the application information and current instance information in the log header.

[0037] Step 207: Call the analysis service receiving interface to send the log message to the analysis service.

[0038] Step 208: Determine if the current thread has stopped. If not, continue to wait for log writing, read the next log, and repeat steps 205, 206, and 207 to pull and send logs.

[0039] Step 209: If a stop command is received, close the current reading thread and stop the log pulling work.

[0040] Steps for the analysis service to analyze the log chain include:

[0041] Step 301: Perform initialization work.

[0042] Step 302: Start the receiving thread T1 to receive the log messages sent by the extraction service;

[0043] Step 303: Start the log chain assembly thread T2 to assemble the received logs into a log chain;

[0044] Step 304: Create a log assembly queue to temporarily store the log messages received by the receiving thread T1;

[0045] Step 305: Create a log chain cache to store the log chain being assembled.

[0046] Preferably, the log header information includes a processing chain id and instance information, where:

[0047] For the processing chain id, the processing chain ids of all programs in a single business operation are the same and are used to concatenate the log chain;

[0048] The instance information is used to record the running instance of the current application program.

[0049] Preferably, the service parameters are used to identify the business custom information recorded by each program.

[0050] Preferably, the execution information includes the key information of the log record, the time-consuming information of the program executing the business process, and the sub-step information.

[0051] Preferably, the key information of the log record includes the status and an error code used to mark errors;

[0052] The sub-step information is the status and time-consuming record of each operation involved in the internal processing of the application program.

[0053] Preferably, in step 302, the receiving thread T1 includes the following steps:

[0054] Step 3021: Receive the log information, verify the log header, and determine whether the application program is legal;

[0055] Step 3022: Call the log assembly queue and add the log to the queue;

[0056] Step 3023: If an end command is received, end the reception and stop; otherwise, continue to receive the log information and repeat the operations of steps 3021 and 3022.

[0057] Preferably, in step 303, the log chain assembly thread T2 includes the following steps:

[0058] Step 3031: Call the log assembly queue and read a log message A;

[0059] Step 3032: Obtain the log chain configuration associated with the service according to the application information in the log header of the log message A;

[0060] Step 3033: Obtain the link ID of log information A and check if there is an identical ID log chain in the log chain cache. If not, add log information A to the log chain cache and start over from step 3031 to obtain a new piece of log information from the queue. If so, retrieve the corresponding log chain B that matches from the cache.

[0061] Step 3034: According to the log chain configuration obtained in step 3032, arrange the current log information A into log chain B according to the business link rules to generate log chain C.

[0062] Step 3035: Calculate and update the overall elapsed time of log chain C.

[0063] Step 3036: Determine whether log chain C contains error code information. If it does, integrate the error codes and update the error flag.

[0064] Step 3037: Determine whether the business executed by log chain C has ended according to the business configuration. If not, re-add it to the log chain cache and wait for subsequent logs to be assembled completely. Otherwise, store the completed log chain C in the database for subsequent platform queries.

[0065] Another aspect of the present invention discloses a multi-system call process analysis system based on log chains, which is characterized in that it is used to implement the above-mentioned multi-system call process analysis method based on log chains, including:

[0066] A custom log protocol module for implementing the steps of the above-mentioned custom log protocol.

[0067] An application log generation module for implementing the steps of the above-mentioned application log generation.

[0068] An extraction service log extraction module for implementing the steps of the above-mentioned extraction service log extraction

[0069] An analysis service log chain analysis module for implementing the steps of the above-mentioned analysis service log chain analysis.

[0070] Compared with the prior art solutions, the beneficial effects of the present invention are as follows:

[0071] 1. By adopting the log chain method, users can retrieve the link logs of completed operations and display them in an arranged manner, eliminating the need to view logs separately in each application, which is simple and efficient.

[0072] 2. The log component outputs logs in a unified and concise log format, and the log printing methods of each application are controllable, masking the differences between applications and greatly improving the readability of logs.

[0073] 3. Thanks to the design of error codes, when the log component records logs, it will wrap the error information of the program into error codes. The analysis service concatenates the logs of each application program according to the business link to display the processing process and error codes. When an exception occurs, the problem steps and the corresponding application programs can be quickly located based on the link and error codes;

[0074] 4. The analysis service uniformly processes the log chain, abstracts the business process into log behaviors. Developers can statistically analyze the abnormal situations of each link of the business based on the key information of the assembled log chain (step time-consuming, overall time-consuming, error code distribution, etc.), which is beneficial to obtaining the overall working condition and timely discovering potential problems. Brief Description of the Drawings

[0075] Figure 1 Illustrates a method for analyzing the multi-system call process based on a log chain and a system log protocol;

[0076] Figure 2 Illustrates a method for analyzing the multi-system call process based on a log chain and the overall structure of the system;

[0077] Figure 3 Is a flowchart for generating application program logs;

[0078] Figure 4 Is a flowchart for the extraction service to extract logs;

[0079] Figure 5 Is a flowchart for the analysis service to analyze the log chain. Detailed Embodiments

[0080] The following further elaborates the present invention in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and not to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope defined by the appended claims of this application.

[0081] The technical solution disclosed in the embodiment of the present invention customizes a set of log recording protocols for marking key log information, unifying the log output of all application programs, assembling log chains according to the protocol, etc., as Figure 1 shown. The log protocol disclosed in the embodiment of the present invention includes the following information:

[0082] 1) Log header information: includes the processing chain id and instance information. Among them: for the processing chain id, the processing chain ids of all programs in a single business are the same and are used to concatenate the log chain; the instance information is used to record the running instance of the current application program;

[0083] 2) Business parameters: each program records business custom information

[0084] 3) Execution information, including:

[0085] a) Keys for logging, including status and error codes for marking errors;

[0086] b) Time taken for the program to execute business processing;

[0087] c) Sub-step information, which is the status and time consumption records of each operation involved in the processing within the application program.

[0088] The overall structure and process of the solution disclosed in the embodiments of the present invention are as Figure 2 shown, mainly including three modules and corresponding processes - the application program log generation process, the extraction service log extraction process, and the analysis service log chain analysis process.

[0089] 1) The main process of application program log generation is as Figure 3 shown, specifically including the following steps:

[0090] Step 1: The application program calls the log component to start logging;

[0091] Step 2: The log component creates thread-local variables for subsequent steps to record and save all the log information of the current business thread;

[0092] Step 3: The log component obtains the log chain id from the upstream. If it cannot be obtained, the program is the starting point of business processing, and a new log chain id is generated and encapsulated as log header information and recorded in the log information;

[0093] Step 4: The application program executes the step, triggers the log component to record the step information event, and records the status, time consumption, error information, etc. of the step in the log information;

[0094] The application program continues to execute the next step, and so on until all steps are executed;

[0095] Step 5: After the business step is executed, the log component accumulates the overall time consumption of the step and calculates the total business time consumption of the program;

[0096] Step 6: The log component parses each step to determine whether it contains error information. If it does, a step error code is generated and the error information is recorded;

[0097] Step 7: After the application program business is executed, it calls the log component to close logging;

[0098] Step 8: The log component obtains the thread-local variables, extracts the log information, and assembles the log according to the log recording protocol;

[0099] Step 9: The log component prints the assembled log to the log file in a unified directory.

[0100] 2) The main process of extracting service extraction logs is as follows Figure 4 shown, specifically including the following steps:

[0101] Step 1: Obtain configuration information, including the address information of the application involved in the business, the target address of the analysis service, etc.;

[0102] Step 2: Start the log pulling thread;

[0103] Step 3: Obtain the current application identifier, instance identifier, and log storage directory according to the configuration information;

[0104] Step 4: Obtain the file stream according to the log storage directory and read the log file;

[0105] Step 5: Wait for the application to write logs, and read the latest logs line by line;

[0106] Step 6: Obtain the log information, assemble the log message, and supplement the application information and current instance information in the log header;

[0107] Step 7: Call the analysis service receiving interface and send the log message to the analysis service;

[0108] Step 8: Judge whether the current thread has stopped. If it has not stopped, continue to wait for log writing, read the next log, and repeat steps 5, 6, and 7 to pull and send logs;

[0109] Step 9: If a stop command is received, close the current reading thread and stop the log pulling work.

[0110] 3) The main process of analyzing the service log chain is as follows Figure 5 shown, specifically including the following work and steps:

[0111] a) Initialization work:

[0112] Step 1: Execute the initialization work;

[0113] Step 2: Start the receiving thread T1 to receive the log messages sent by the extraction service;

[0114] Step 3: Start the log chain assembly thread T2 to assemble the received logs into a log chain;

[0115] Step 4: Create a log assembly queue to temporarily store the log messages received by the receiving thread T1;

[0116] Step 5: Create a log chain cache to store the log chain being assembled.

[0117] b) The main process steps of the receiving thread T1:

[0118] Step 1: Receive log information, verify the log header, and determine whether the application is legal;

[0119] Step 2: Call the log assembly queue and add the log to the queue;

[0120] Step 3: If an end command is received, end the reception and stop. Otherwise, continue to receive log information and repeat the operations in Steps 1 and 2.

[0121] c) Main process steps of the log chain assembly thread T2:

[0122] Step 1: Call the log assembly queue and read a piece of log information A;

[0123] Step 2: Obtain the log chain configuration associated with the service according to the application information in the log header of the log information A;

[0124] Step 3: Check whether there is a log chain with the same id in the log chain cache for the link id of the log information A. If not, add the log information A to the log chain cache and start over to obtain a new piece of log information from the queue in Step 1. If it exists, take out the corresponding log chain B that matches in the cache;

[0125] Step 4: According to the log chain configuration obtained in Step 2, arrange the current log information A into the log chain B according to the service link rule to generate the log chain C;

[0126] Step 5: Calculate and update the overall time consumption of the log chain C;

[0127] Step 6: Determine whether the log chain C contains error code information. If it does, integrate the error codes and update the error flag;

[0128] Step 7: Determine whether the service executed by the log chain C has ended according to the service configuration. If not, add it back to the log chain cache and wait for the subsequent logs to be assembled completely. Otherwise, store the completed log chain C in the database for subsequent platform queries.

Claims

1. A multi-system call process analysis method based on log chain, characterized in that: include: The step of customizing the log protocol, wherein the log protocol includes log header information, business parameters and execution information, wherein the log header information includes the processing chain ID, and the execution information includes the time-consuming information, status information and error information of the program execution business processing; The steps to generate application logs include: Step 101, the application calls the log component to start logging; Step 102: The log component creates a thread local variable, which is used in subsequent steps to record and save all log information of the current business thread; Step 103: The log component obtains the log chain ID from the upstream. If it cannot be obtained, the current application is used as the starting point of the business processing, and a new log chain ID is generated, which is encapsulated as log header information and recorded in the log information. Step 104: The application executes all business steps. When executing each business step, the log component is triggered to record the business step information event, and the state, time consumption, and error information of the business step are recorded in the log information; Step 105: After all business steps of the application are executed, the log component accumulates the overall time consumption of all business steps to calculate the total time consumption of the program business; Step 106: The log component parses each business step to determine whether it contains error information. If so, it generates a step error code and records the error information. Step 107: After the application service is executed, the log component is called to close the log record; Step 108, the log component obtains thread local variables, takes out log information, and assembles logs according to the log recording protocol; Step 109, the log component prints the assembly log to a log file in a unified directory; The steps to extract service logs include: Step 201: Obtain configuration information; Step 202: Start the log pulling thread; Step 203: Obtain the current application ID, instance ID, and log storage directory according to the configuration information; Step 204: Obtain the file stream and read the log file according to the log storage directory; Step 205, wait for the application to write the log, and read the latest log line by line; Step 206: Obtain log information, assemble log messages, and add application information and current instance information to the log header; Step 207: Call the analysis service receiving interface and send the log message to the analysis service; Step 208: Determine whether the current thread is stopped. If not, continue to wait for log writing, read the next log, and repeat steps 205, 206, and 207 to pull and send the log. Step 209: If a stop command is received, the current reading thread is closed and the log pulling work is stopped; The steps of analyzing the service log chain include: Step 301, perform initialization work; Step 302: Start the receiving thread T1 to receive the log message sent by the extraction service; Step 303: Start the log chain assembly thread T2 to assemble the received logs into a log chain; Step 304: Create a log assembly queue to temporarily store the log messages received by the receiving thread T1; Step 305: Create a log chain cache to store the log chain being assembled.

2. A multi-system call process analysis method based on log chain as claimed in claim 1, characterized in that: The log header information includes the processing chain ID and instance information, where: For the processing chain id, the processing chain id of all programs of a single business is the same, which is used to connect the log chain in series; Instance information is used to record the running instance of the current application.

3. A multi-system call process analysis method based on log chain as claimed in claim 1, characterized in that: The service parameters are used to identify the service custom information recorded by each program.

4. A multi-system call process analysis method based on log chain as claimed in claim 1, characterized in that: The execution information includes key information recorded in the log, time-consuming information of program execution business processing, and sub-step information.

5. A multi-system call process analysis method based on log chain as described in claim 4, characterized in that: The key information recorded in the log includes the status and the error code used to mark the error; The sub-step information is the status and time-consuming record of each operation involved in the processing process within the application.

6. A method for analyzing multiple system call processes based on log chains as claimed in claim 1, characterized in that: In step 302, the receiving thread T1 includes the following steps: Step 3021: Receive log information, check the log header, and determine whether the application is legal; Step 3022: Call the log assembly queue and add the log to the queue; Step 3023: If an end command is received, end the reception and stop; otherwise, continue to receive log information and repeat steps 3021 and 3022.

7. A multi-system call process analysis method based on log chain as claimed in claim 1, characterized in that: In step 303, the log chain assembly thread T2 includes the following steps: Step 3031, call the log assembly queue and read a log information A; Step 3032: Obtain the log chain configuration associated with the business according to the log header application information of the log information A; Step 3033: Get the link ID of log information A and check whether there is a log chain with the same ID in the log chain cache: if not, add log information A to the log chain cache, and start from step 3031 to get a new log information in the queue; if so, take out the matching corresponding log chain B in the cache; Step 3034: According to the log chain configuration obtained in step 3032, the current log information A is arranged into the log chain B according to the business chain rule to generate the log chain C; Step 3035: Calculate and update the overall time consumption of log chain C; Step 3036: determine whether the log chain C contains error code information. If so, integrate the error code and update the error flag. Step 3037: Determine whether the business executed by log chain C has ended according to the business configuration: if not, re-add it to the log chain cache and wait for subsequent log assembly to be complete; otherwise, store the completed log chain C in the warehouse and provide subsequent platform query.

8. A multi-system call process analysis system based on log chain, characterized in that: The method for implementing the multi-system call process analysis method based on log chain as claimed in claim 1 comprises: A custom log protocol module, used to implement the steps of the custom log protocol described in claim 1; An application log generation module, used to implement the step of generating the application log according to claim 1; Extraction service extraction log module, used to implement the step of extracting service extraction log as claimed in claim 1 The analysis service log chain analysis module is used to implement the analysis service log chain analysis steps described in claim 1.

Images