Log burying point-based service cluster link tracking method and system

Abstract

The invention provides a link tracking method and system for a service cluster based on log burying points, and belongs to the technical field of distributed monitoring. According to the method, a link tracking ID is generated through a snowflake algorithm at a starting point of service calling, and service IDs are distributed to micro-service nodes participating in service calling; setting log burying points at the entrance, the exit and the key logic processing points of each service according to preset burying point logic; asynchronously recording link tracking information contained in the service node through the log burying point, and uploading the information to a storage system for storage; according to the link tracking ID and the service ID in the link tracking information, the link tracking information is connected in series, a complete calling link called by the service is restored, and meanwhile a service calling topological graph is generated. According to the method, real link information can be quickly and accurately tracked, and quick retrieval and analysis of a large amount of real-time link data can be supported.

Description

Technical Field

[0001] The present invention relates to the field of distributed monitoring technology, and in particular to a link tracking method and system for a service cluster based on log embedding. Background Art

[0002] In a microservice architecture, user requests often need to go through multiple service nodes and complete a series of calls before they can be responded to. This process involves many services, each of which is deployed on different servers or even across different data centers. Such complexity can lead to the inability to quickly locate problems when requests are delayed or fail. Traditional monitoring methods can often only provide the health status of a single service, and are unable to cope with link problems caused by cross-service calls.

[0003] The emergence of link tracking technology is precisely to deal with the problems of complex service call links and difficult fault location under the microservice architecture. It records, collects and analyzes the call relationships between services in the entire distributed system, providing developers with a global perspective to observe and understand the complete process of each request. However, when tracking links, existing link tracking technologies require manual intervention in the tracking process to ensure that the entire call link can be accurately tracked. However, this method is difficult to support the rapid retrieval and analysis of a large amount of existing real-time link data. Therefore, link tracking technology needs to be further improved. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a link tracking method and system for a service cluster based on log tracking points, so as to solve the problem that the existing link tracking technology is difficult to realize automatic and accurate data collection, and is difficult to support the rapid retrieval and analysis of large amounts of real-time link data.

[0005] The technical solution adopted by the present invention is: In a first aspect, the present invention provides a link tracking method for a service cluster based on log embedding, comprising: At the starting point of the service call, a link tracking ID is generated through the snowflake algorithm, and a corresponding service ID is assigned to each microservice node participating in the service call in the service cluster; According to the preset tracking logic, set log tracking points at the entry, exit and key logic processing points of each service; During the service call process, the link tracking information including the link tracking ID and the service ID of each microservice node is asynchronously recorded through the log embedding point, and the link tracking information is uploaded to the tracking data storage system through the message queue for storage; According to the link tracking ID and service ID in the link tracking information, the link tracking information of each microservice node is concatenated and restored to restore the complete call link of the service call, and a service call topology diagram is generated at the same time.

[0006] Furthermore, the preset tracking logic is specifically as follows: The link tracking ID is carried in the request header of the service call request. When passing from the current service to the next service, the service ID of the current service is attached to the request header, and the service ID of the next service to be called is recorded; the services include service trace processing service, election trace processing service, log synchronization trace service, log cleaning service, sqlite storage service and MySQL storage service.

[0007] Furthermore, according to the link tracking ID and service ID in the link tracking information, the link tracking information of each microservice node is connected in series and a complete call link of the service call is restored, and a service call topology diagram is generated, including: Parse the link tracking information of each microservice node separately to obtain the link tracking ID and service ID; the link tracking information also includes the service call timestamp, response time, call result, passed parameter summary, return result summary, and called interface name and operation type; Based on the link tracking ID and service ID, the link tracking information of each microservice node is concatenated in the order of service calls to form the complete link tracking information in the service call process, and the corresponding complete call link is obtained; Use the stream computing engine to build a distributed consistency component, and use the distributed consistency component to perform data consistency verification on the complete link tracking information; After the data consistency check is correct, the complete link tracking information is log cleaned and the business logic is processed. The dependency, call sequence and time distribution between services are analyzed based on the complete call link, and the corresponding service call topology diagram is constructed through a graphical interface.

[0008] Furthermore, the method also includes: performing statistical analysis on the collected link tracking information to obtain the operating indicators of the system service cluster, and evaluating whether the operating indicators have reached the system bottleneck based on preset indicator thresholds. If the system bottleneck is reached, the service cluster is optimized for performance based on preset load balancing strategies and caching strategies.

[0009] In a second aspect, the present invention further provides a link tracking system for a service cluster based on log embedding points, comprising: The tracking configuration module is used to generate a link tracking ID through the snowflake algorithm at the starting point of the service call, and assign a corresponding service ID to each microservice node participating in the service call in the service cluster; The tracking point configuration module is used to set log tracking points at the entry, exit and key logic processing points of each service according to the preset tracking point logic; A link tracking module is used to asynchronously record the link tracking information containing the link tracking ID and the service ID of each microservice node through the log embedding point during the service call process, and upload the link tracking information to the tracking data storage system through the message queue for storage; The link restoration module is used to concatenate the link tracking information of each microservice node according to the link tracking ID and service ID in the link tracking information and restore the complete call link of the service call, and generate a service call topology diagram at the same time.

[0010] Furthermore, the link tracking module includes: An information parsing unit, used to parse the link tracking information of each microservice node respectively to obtain a link tracking ID and a service ID; the link tracking information also includes a service call timestamp, a response time, a call result, a summary of passed parameters, a summary of returned results, and a called interface name and an operation type; The link restoration unit is used to concatenate the link tracking information of each microservice node in the order of service calls according to the link tracking ID and service ID to form the complete link tracking information in the service call process, and obtain the corresponding complete call link; A consistency check unit is used to build a distributed consistency component using a stream computing engine, and to perform data consistency check on the complete link tracking information through the distributed consistency component; The visualization display unit is used to clean the logs and perform business logic processing on the complete link tracking information after the data consistency verification is correct. It analyzes the dependencies, call sequence, and time distribution between services based on the complete call link, and constructs the corresponding service call topology diagram through a graphical interface.

[0011] Furthermore, the system also includes a performance monitoring module for performing statistical analysis on the collected link tracking information to obtain the operating indicators of the system service cluster, and evaluating whether the operating indicators have reached the system bottleneck based on preset indicator thresholds. If the system bottleneck is reached, the service cluster is optimized for performance based on preset load balancing strategies and caching strategies.

[0012] In summary, the beneficial effects of the present invention are as follows: A link tracking method for a service cluster based on log burying provided by the present invention is provided. The method generates a link tracking ID through a snowflake algorithm at the starting point of the service call, and allocates a corresponding service ID to each microservice node participating in the service call in the service cluster, ensuring that a non-repeated ID can be generated even in a distributed environment; then, according to the preset burying logic, log burying is set at the entrance, exit and key logic processing point of each service; in the service call process, the link tracking information containing the link tracking ID and service ID in each microservice node is asynchronously recorded through the log burying, and the link tracking information is uploaded to the tracking data storage system through the message queue for storage, without manual intervention, ensuring the automatic collection and dissemination of data, as well as the integrity and immediacy of the data; finally, according to the link tracking ID and service ID in the link tracking information, the link tracking information of each microservice node is connected in series and the complete call link of the service call is restored, and a service call topology diagram is generated at the same time. The present invention uploads the data that needs to be tracked and analyzed by means of log burying, and the background performs unified log cleaning, business side logic processing, storage, and finally visual display, realizing the automatic collection and transmission of link data, being able to quickly and accurately track the real link information, and supporting the rapid retrieval and analysis of a large amount of real-time link data. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the following is a brief introduction to the drawings required for use in the embodiment of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work, and these are all within the protection scope of the present invention.

[0014] Figure 1 It is a flow chart of a link tracking method of a service cluster based on log embedding of the present invention; Figure 2 A diagram of the link tracking architecture of the present invention; Figure 3 It is the link tracking logical view of the present invention; Figure 4 This is a functional module diagram of a link tracking system of a service cluster based on log embedding according to the present invention. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. If there is no conflict, the various features of the present invention and the embodiments can be combined with each other, all within the scope of protection of the present invention.

[0016] As an important part of modern distributed system service governance, link tracking technology has emerged and developed along with the rise and popularization of microservice architecture and has gradually become an indispensable part. With the continuous expansion of enterprise-level application scale, traditional monolithic architecture has gradually exposed problems such as difficult maintenance, poor scalability, and low development efficiency. Under the microservice architecture, large applications are split into a series of small, independent service units, each of which is responsible for a specific business function and works together through API calls. This architectural design improves the flexibility, scalability and maintainability of the system, but also brings new challenges, especially in system monitoring and troubleshooting. Link tracking technology is a distributed monitoring technology born to address these challenges. Therefore, the present invention can provide developers with a global perspective to observe and understand the complete process of each request by recording, collecting and analyzing the call relationship between services in the entire distributed system. The core idea of ​​its link tracking is to generate a unique trace ID for each request, and pass and record this ID in the service node through which the request passes, while collecting information such as the call time, response status, and time consumption on the node. Through this information, the call link diagram of the entire request can be reconstructed, so that the service or node where the problem occurs can be quickly located when a problem occurs. The detailed implementation process of the present invention is shown in the following examples.

[0017] Example 1: Please see Figure 1 to Figure 3 , Figure 1 This is a flow chart of a link tracking method for a service cluster based on log tracking in Example 1 of the present invention. Figure 2 is an architecture diagram of link tracking implemented in the present invention, Figure 3 This is a link tracking logical view of an embodiment of the present invention. Figure 1 As shown, the present invention provides a link tracking method for a service cluster based on log embedding, including: S1: At the starting point of the service call, a link tracking ID is generated through the snowflake algorithm, and a corresponding service ID is assigned to each microservice node participating in the service call in the service cluster; S2: According to the preset tracking logic, set log tracking points at the entry, exit and key logic processing points of each service; S4: During the service call process, the link tracking information including the link tracking ID and the service ID of each microservice node is asynchronously recorded through the log embedding point, and the link tracking information is uploaded to the tracking data storage system through the message queue for storage; S4: Based on the link tracking ID and service ID in the link tracking information, the link tracking information of each microservice node is connected in series to restore the complete call link of the service call, and a service call topology diagram is generated at the same time.

[0018] Further, in the embodiment of the present invention, if Figure 2 As shown, the link architecture implemented in the embodiment of the present invention is a microservice architecture, which provides a unified log collection method (i.e., ghLog) on ​​the user access side, and log collection is performed in the form of stream data (Stream). The log processing backend provides a common infrastructure, and the processing method of specific services is expanded and implemented in the form of microservices. At the same time, the corresponding service data is stored in the sqlite database management system and the mysql database. Finally, the database is read through the visualization service backend for visualization processing and displayed to the user through a web browser.

[0019] Further, in the embodiment of the present invention, referring to Figure 3 The link tracking logic shown in the figure, the present invention sets a log point between the network modules of service A and the data plane, and samples the key information of the link called by the service. Among them, the configuration center of the control plane needs to pre-set the microservice basic framework and business logic of service A. After the service call starts, the sampled service call data is reported by the service gateway, and the data reported by the service network is uploaded to the stream computing engine through the message queue, and the consistency check is performed through the distributed consistency component. After the verification is correct, the data visualization processing is finally performed to display the link tracking data in the service cluster to the user.

[0020] Furthermore, in an embodiment of the present invention, at the starting point of the service call, the present invention first uses the snowflake algorithm to generate a globally unique ID, which is used as the unique identifier for this link tracking, namely, the link tracking ID (Trace ID). The snowflake algorithm is popular for its high performance and the uniqueness of the generated ID. It combines the timestamp, machine ID, and serial number to ensure that a non-duplicate ID can be generated even in a distributed environment. No matter how the request jumps between services, the entire call link can be accurately tracked through this Trace ID.

[0021] Furthermore, in the embodiment of the present invention, the preset tracking logic is specifically as follows: The link tracking ID is carried in the request header of the service call request. When passing from the current service to the next service, the service ID of the current service is attached to the request header, and the service ID of the next service to be called is recorded. Figure 2 As shown, the services that can be called by the present invention include service trace processing service, election trace processing service, log synchronization trace service, log cleaning service, sqlite storage service and MySQL storage service.

[0022] Specifically, the embodiment of the present invention pre-assigns a unique service ID to each microservice node participating in the service call to identify its position and role in the system. When a request is passed from one service to another, in addition to carrying the original Trace ID, the Service ID of the current service is also attached to the request header, and the Service ID of the next service to be called is recorded. This process is implemented through automated tracking logic without manual intervention, ensuring automatic data collection and dissemination.

[0023] Furthermore, in an embodiment of the present invention, step S2 records key information through preset code embedding points (i.e., log embedding points) at the entrance and exit of each service, as well as key logic processing points. This information includes, but is not limited to: service call timestamp, response time, call result (success / failure), summary of passed parameters, summary of returned results, etc. The log information not only contains the Trace ID and service ID, but also records the interface name, operation type, etc. of the call for subsequent analysis. The collection of relevant call data is completed through a lightweight asynchronous mechanism to reduce the impact on the main business logic.

[0024] Specifically, in the present invention, each time a service call is completed, the collected link tracking information will be uploaded to a centralized tracking data storage system, such as a distributed database or a big data platform, in real time or periodically. High-performance message queue technology is used here to buffer and reliably transmit these data to ensure the integrity and immediacy of the data. The storage system needs to have high concurrent writing capabilities and efficient query capabilities to support the rapid retrieval and analysis of a large amount of real-time link data.

[0025] Further, in the embodiment of the present invention, in step S4, the link tracking information of each microservice node is connected in series according to the link tracking ID and the service ID in the link tracking information to restore the complete call link of the service call, and a service call topology diagram is generated at the same time, including: Parse the link tracking information of each microservice node separately to obtain the link tracking ID and service ID; the link tracking information also includes the service call timestamp, response time, call result, passed parameter summary, return result summary, and called interface name and operation type; Based on the link tracking ID and service ID, the link tracking information of each microservice node is concatenated in the order of service calls to form the complete link tracking information in the service call process, and the corresponding complete call link is obtained; Use the stream computing engine to build a distributed consistency component, and use the distributed consistency component to perform data consistency verification on the complete link tracking information; After the data consistency check is correct, the complete link tracking information is log cleaned and the business logic is processed. The dependency, call sequence and time distribution between services are analyzed based on the complete call link, and the corresponding service call topology diagram is constructed through a graphical interface.

[0026] Specifically, after the relevant data of link tracking is collected, the backend system uses the Trace ID as a clue to connect the link information scattered in each service node and restore the complete path of the entire request. The service call topology diagram is displayed through a graphical interface to intuitively show the dependency relationship, call sequence, time consumption distribution, etc. between services.

[0027] Furthermore, in an embodiment of the present invention, the method also includes S6: performing statistical analysis on the collected link tracking information to obtain the operating indicators of the system service cluster, and evaluating whether the operating indicators have reached the system bottleneck based on a preset indicator threshold. If the system bottleneck is reached, the service cluster is optimized for performance based on a preset load balancing strategy and cache strategy.

[0028] The embodiment of the present invention uploads the data that needs to be tracked and analyzed by means of log embedding, and the background performs unified log cleaning, business logic processing, storage, and finally visual display, thereby realizing the automatic collection and transmission of link data, being able to quickly and accurately track the real link information, and being able to support the rapid retrieval and analysis of a large amount of real-time link data.

[0029] Example 2: Reference Figure 4 As shown, an embodiment of the present invention further provides a link tracking system for a service cluster based on log embedding points, including: The tracking configuration module is used to generate a link tracking ID through the snowflake algorithm at the starting point of the service call, and assign a corresponding service ID to each microservice node participating in the service call in the service cluster; The tracking point configuration module is used for preset tracking point logic, setting log tracking points at the entry, exit and key logic processing points of each service; A link tracking module is used to asynchronously record the link tracking information containing the link tracking ID and the service ID of each microservice node through the log embedding point during the service call process, and upload the link tracking information to the tracking data storage system through the message queue for storage; The link restoration module is used to concatenate the link tracking information of each microservice node according to the link tracking ID and service ID in the link tracking information and restore the complete call link of the service call, and generate a service call topology diagram at the same time.

[0030] Furthermore, in an embodiment of the present invention, the link tracking module includes: An information parsing unit, used to parse the link tracking information of each microservice node respectively to obtain a link tracking ID and a service ID; the link tracking information also includes a service call timestamp, a response time, a call result, a summary of passed parameters, a summary of returned results, and a called interface name and an operation type; The link restoration unit is used to concatenate the link tracking information of each microservice node in the order of service calls according to the link tracking ID and service ID to form the complete link tracking information in the service call process, and obtain the corresponding complete call link; A consistency check unit is used to build a distributed consistency component using a stream computing engine, and to perform data consistency check on the complete link tracking information through the distributed consistency component; The visualization display unit is used to clean the logs and perform business logic processing on the complete link tracking information after the data consistency verification is correct. It analyzes the dependencies, call sequence, and time distribution between services based on the complete call link, and constructs the corresponding service call topology diagram through a graphical interface.

[0031] Specifically, the link tracking module in the embodiment of the present invention uses Trace ID as a clue to connect the link information scattered in each service node in series, and restore the complete path of the entire request. The service call topology diagram is displayed through a graphical interface to intuitively display the dependency relationship, call sequence, time consumption distribution, etc. between services.

[0032] Furthermore, the system also includes a performance monitoring module for performing statistical analysis on the collected link tracking information to obtain the operating indicators of the system service cluster, and evaluating whether the operating indicators have reached the system bottleneck based on preset indicator thresholds. If the system bottleneck is reached, the service cluster is optimized for performance based on preset load balancing strategies and caching strategies.

[0033] Among them, the link tracking system of the present invention is not only used for fault location, but also can be used as an important tool for performance monitoring. By statistically analyzing the collected link data, system bottlenecks, services or interfaces with excessive resource usage can be identified, providing a basis for system optimization. For example, by analyzing indicators such as average response time, throughput, and error rate, it is possible to guide the adjustment of service load balancing strategies and cache strategy optimization.

[0034] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technologies therein with equivalents. However, these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A link tracking method for a service cluster based on log embedding, characterized in that: include: At the starting point of the service call, a link tracking ID is generated through the snowflake algorithm, and a corresponding service ID is assigned to each microservice node participating in the service call in the service cluster; According to the preset tracking logic, set log tracking points at the entry, exit and key logic processing points of each service; During the service call process, the link tracking information including the link tracking ID and the service ID of each microservice node is asynchronously recorded through the log embedding point, and the link tracking information is uploaded to the tracking data storage system through the message queue for storage; According to the link tracking ID and service ID in the link tracking information, the link tracking information of each microservice node is concatenated and restored to restore the complete call link of the service call, and a service call topology diagram is generated at the same time.

2. According to the link tracking method of the service cluster based on log embedding point according to claim 1, it is characterized in that: The preset tracking logic is specifically as follows: The link tracking ID is carried in the request header of the service call request. When passing from the current service to the next service, the service ID of the current service is attached to the request header, and the service ID of the next service to be called is recorded; the services include service trace processing service, election trace processing service, log synchronization trace service, log cleaning service, sqlite storage service and MySQL storage service.

3. According to the link tracking method of the service cluster based on log embedding point according to claim 1, it is characterized in that: According to the link tracking ID and service ID in the link tracking information, the link tracking information of each microservice node is connected in series and the complete call link of the service call is restored, and a service call topology diagram is generated at the same time, including: Parse the link tracking information of each microservice node separately to obtain the link tracking ID and service ID; the link tracking information also includes the service call timestamp, response time, call result, passed parameter summary, return result summary, and called interface name and operation type; Based on the link tracking ID and service ID, the link tracking information of each microservice node is concatenated in the order of service calls to form the complete link tracking information in the service call process, and the corresponding complete call link is obtained; Use the stream computing engine to build a distributed consistency component, and use the distributed consistency component to perform data consistency verification on the complete link tracking information; After the data consistency check is correct, the complete link tracking information is log cleaned and the business logic is processed. The dependency, call sequence and time distribution between services are analyzed based on the complete call link, and the corresponding service call topology diagram is constructed through a graphical interface.

4. According to claim 1, the link tracking method of the service cluster based on log embedding point is characterized in that: The method also includes: performing statistical analysis on the collected link tracking information to obtain the operating indicators of the system service cluster, and evaluating whether the operating indicators have reached the system bottleneck based on a preset indicator threshold. If the system bottleneck is reached, the service cluster is optimized according to the preset load balancing strategy and cache strategy.

5. A link tracking system for a service cluster based on log embedding, characterized in that: include: The tracking configuration module is used to generate a link tracking ID through the snowflake algorithm at the starting point of the service call, and assign a corresponding service ID to each microservice node participating in the service call in the service cluster; The tracking point configuration module is used for preset tracking point logic, setting log tracking points at the entry, exit and key logic processing points of each service; A link tracking module is used to asynchronously record the link tracking information containing the link tracking ID and the service ID of each microservice node through the log embedding point during the service call process, and upload the link tracking information to the tracking data storage system through the message queue for storage; The link restoration module is used to concatenate the link tracking information of each microservice node according to the link tracking ID and service ID in the link tracking information and restore the complete call link of the service call, and generate a service call topology diagram at the same time.

6. The link tracking system of the service cluster based on log embedding according to claim 5 is characterized in that: The link tracking module includes: An information parsing unit, used to parse the link tracking information of each microservice node respectively to obtain a link tracking ID and a service ID; the link tracking information also includes a service call timestamp, a response time, a call result, a summary of passed parameters, a summary of returned results, and a called interface name and an operation type; The link restoration unit is used to concatenate the link tracking information of each microservice node in the order of service calls according to the link tracking ID and service ID to form the complete link tracking information in the service call process, and obtain the corresponding complete call link; A consistency check unit is used to build a distributed consistency component using a stream computing engine, and to perform data consistency check on the complete link tracking information through the distributed consistency component; The visualization display unit is used to clean the logs and perform business logic processing on the complete link tracking information after the data consistency verification is correct. It analyzes the dependencies, call sequence, and time distribution between services based on the complete call link, and constructs the corresponding service call topology diagram through a graphical interface.

7. The link tracking system of the service cluster based on log embedding according to claim 5 is characterized in that: The system also includes a performance monitoring module for performing statistical analysis on the collected link tracking information, obtaining the operating indicators of the system service cluster, and evaluating whether the operating indicators have reached the system bottleneck based on preset indicator thresholds. If the system bottleneck is reached, the service cluster is optimized for performance based on preset load balancing strategies and caching strategies.

Images