An analysis system for tracking calls between microservices

Abstract

The present invention relates to an analysis system for tracking calls between microservices, belonging to the field of microservice monitoring technology. It includes a data acquisition module, a data transmission module, a data storage module, a data analysis module and a data display module; the data analysis module is used to perform call tracking analysis between microservices by calling the tracking data of the data storage module; the data acquisition module is used to collect tracking data based on the buried point log, wherein the tracking data includes service performance data on the microservice node and data describing the service call relationship; the data storage module is used to uniformly manage and store the collected tracking data, and provide a query interface based on user transaction requests. The present invention mines association rules on the collected tracking data through the data analysis module, thereby discovering service dependencies, identifying bottlenecks and performance problems, analyzing service call patterns, discovering abnormal behaviors and faults, and improving the availability and maintainability of the microservice architecture.

Description

Technical Field

[0001] The present invention belongs to the technical field of microservice monitoring, and in particular relates to an analysis system for tracking calls between microservices. Background Art

[0002] The rise of new technology architectures such as microservices, big data, and cloud computing has brought new technical directions to platform integration. Containerization is a technology that packages applications and their dependencies in an independent, portable container. It has the advantages of isolation, consistency, and flexibility, and can be deployed and expanded quickly. The microservice architecture is to miniaturize and isolate service-oriented integration, achieve comprehensive decoupling of business services, refine various services, form a one-to-one service set, and achieve independent deployment of functions. Since each application microservice runs completely independently and there are clear operating boundaries between services, these independent microservices are used to coordinate and communicate with each other through lightweight communication mechanisms to perform interactive scheduling, and finally achieve a complete combination of an application.

[0003] However, when microservices are applied, there are problems such as difficulty in locating errors, difficulty in troubleshooting, and difficulty in clarifying service call relationships. These problems become more complicated with the increase of service modules. Therefore, there is an urgent need to provide an analysis system for tracking calls between microservices to perform performance monitoring and call link tracking for all microservices. Summary of the invention

[0004] In order to solve the above problems existing in the prior art, the present invention provides an analysis system for tracking calls between microservices. The collected tracking data is mined through association rules through a data analysis module, so as to discover service dependencies, identify bottlenecks and performance issues, analyze service call patterns, discover abnormal behaviors and faults, and improve the availability, performance and maintainability of the microservice architecture.

[0005] The purpose of the present invention can be achieved through the following technical solutions:

[0006] The present invention provides an analysis system for tracking calls between microservices, including a data acquisition module, a data transmission module, a data storage module, a data analysis module and a data display module; the data acquisition module is electrically connected to the data transmission module, the data transmission module is electrically connected to the data storage module, the data storage module is electrically connected to the data analysis module, and the data analysis module is electrically connected to the data display module;

[0007] The data analysis module is used to perform call tracking analysis between microservices by calling the tracking data of the data storage module; the process of the data analysis module performing call tracking analysis includes the following steps:

[0008] S01, data preprocessing: normalize the collected tracking data;

[0009] S02. Constructing accident database: converting the pre-processed data to make the data conform to the form of association analysis;

[0010] S03, mining frequent item sets: using association analysis algorithm to mine frequent item sets that meet the support threshold according to the minimum support;

[0011] S04. Establish monitoring labels: Establish monitoring labels for the mined frequent itemsets, where the detection labels include fault labels and transaction labels;

[0012] The mining of frequent itemsets includes tracking data and transaction sets set by users.

[0013] As a preferred technical solution of the present invention, the process of the data analysis module performing call tracking analysis also includes the following steps:

[0014] S05, generating strong association rules: filtering out strong association rules from the frequent item set according to the set strong association rule conditions; the strong association rule conditions include support and confidence;

[0015] S06. Measuring strong association rules: Evaluate strong association rules by lift, leverage and confidence; lift , leverage ratio , confidence .

[0016] As a preferred technical solution of the present invention, the process of the data analysis module performing call tracking analysis also includes the following steps:

[0017] S07. Build a strong association rule base: Use strong association rules that meet the measurement indicators to build a strong association rule base to complete the analysis and monitoring of tracking data.

[0018] As a preferred technical solution of the present invention, the data acquisition module is used to collect tracking data based on the embedding log, wherein the tracking data includes service performance data on the microservice node and data describing the service call relationship.

[0019] As a preferred technical solution of the present invention, the data transmission module is used for asynchronously transmitting the collected tracking data.

[0020] As a preferred technical solution of the present invention, the data storage module is used to uniformly manage and store the collected tracking data, and provide a query interface based on user transaction requests.

[0021] As a preferred technical solution of the present invention, the data storage module matches the collected tracking data according to the classified transactions, calls and events, and constructs an item set according to the transaction ID for storage.

[0022] As a preferred technical solution of the present invention, the data display module is used for link query, log query, status display and alarm push, and displays the changes and status of the data in real time by visually displaying the data in the form of charts, graphs and tables.

[0023] The present invention also provides an analysis method for tracking calls between microservices, which is applied to the above-mentioned analysis system for tracking calls between microservices, and includes the following steps:

[0024] S11. Define tracking requirements: determine the key indicators, performance indicators and business processes to be tracked;

[0025] S12. Insert tracking code: Insert tracking code at key locations of each microservice according to tracking requirements and select integrated tracking tools;

[0026] S13. Generate a unique identifier: Generate a unique tracking identifier for each cross-service request to complete the tracking request in the entire call chain;

[0027] S14. Collect tracing data: Record key tracing data in each microservice and connect the entire call chain by passing tracing information across services;

[0028] S15, storing and analyzing tracking data: storing the tracking data in a data storage module, and analyzing and visualizing the data through a data analysis module and a data display module;

[0029] S16. Analyze and optimize performance: According to the analysis results of the data analysis module, identify potential performance bottlenecks and delays, and optimize them;

[0030] S17. Monitoring and alarm: By setting up monitoring and alarm mechanisms, relevant personnel can be notified in a timely manner when key indicators or performance indicators exceed the threshold.

[0031] Furthermore, in step S11, key indicators include request volume, response time and error rate; performance indicators include throughput, concurrency and latency; and business processes include the entire user request processing process.

[0032] The beneficial effects of the present invention are:

[0033] The present invention obtains the tracking data between microservices through the data acquisition module, and performs association rule mining on the stored tracking data through the data analysis module, thereby discovering the frequent item sets between the tracking data and the association relationship between transactions, calls and events. At the same time, through the mining of specific transaction data, the user demand status is monitored, service dependencies are discovered, bottlenecks and performance issues are identified, service call patterns are analyzed, abnormal behaviors and faults are discovered, and service design and architecture are optimized, thereby improving the availability, performance and maintainability of the microservice architecture. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to facilitate understanding by those skilled in the art, the present invention is further described below with reference to the accompanying drawings.

[0035] Figure 1 A schematic diagram of the structure of an analysis system for tracing calls between microservices provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation methods, structures, features and effects of the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0037] See also Figure 1 , an analysis system for tracking calls between microservices, including a data acquisition module, a data transmission module, a data storage module, a data analysis module and a data display module;

[0038] The data acquisition module is electrically connected to the data transmission module, the data transmission module is electrically connected to the data storage module, the data storage module is electrically connected to the data analysis module, and the data analysis module is electrically connected to the data display module.

[0039] The data collection module is used to collect tracking data based on the embedded point log, wherein the tracking data includes service performance data on the microservice node and data describing the service call relationship.

[0040] It is understandable that tracking logs refer to inserting code into the application to record the occurrence of specific events and related data, and write these data into log files or send them to log collection systems. By inserting tracking code in key code paths, various events and indicators of the application during operation can be captured. These events can be user operations, API calls, performance indicators, error information, etc.

[0041] The data transmission module is used for asynchronously transmitting the collected tracking data. It is understandable that the data transmission module sends the tracking data to the data storage module after collecting each service request.

[0042] The data storage module is used to uniformly manage and store the collected tracking data, and provide a query interface based on user transaction requests.

[0043] In this embodiment, the collected tracking data is matched according to the classified transactions, calls and events, and the item set is constructed according to the transaction ID for storage, wherein the transaction (Transaction) represents the business request initiated by the user, which is a global concept. The transaction includes a series of service calls in the request process, and each transaction has an ID, namely the transaction ID. Different service call processes under the same transaction are associated through the transaction ID, and the transaction ID must be guaranteed to be globally unique. The call (Span) represents the execution process of a service, that is, the request initiated by the user or other services reaches the application server, and the application server responds to it. The analysis system records the relevant information describing a service call by real-time monitoring of the application server component, including the server type, service information, timestamp information, exception information, etc., wherein the server type is used to identify the node of the system topology structure diagram. In addition, in order to support service call tracking, it is also necessary to add identification data such as the tracking context identifier and the sampling identifier. The event (Event) represents the process of interacting with other components during the service execution process, mainly including remote service calls and database access. A service execution process may contain many events, so for each event, an event number is required to identify the order of occurrence, and for two different categories of events, there is also an event type as a distinction. For remote service call events, the attribute fields include the address and parameters of the called service; for database access events, they mainly record the database type, database connection address and SQL statement.

[0044] It is understandable that the data storage module provides a query interface based on user transaction requests, that is, when the data analysis module calls the data storage module, it queries based on the item set constructed by the transaction ID.

[0045] The data analysis module is used to perform call tracking analysis between microservices by calling the tracking data of the data storage module.

[0046] It should be noted that the process of call tracking analysis by the data analysis module includes the following steps:

[0047] S01. Data preprocessing: Since the collected data have different orders of magnitude, the collected tracking data are normalized;

[0048] S02. Construct an accident database: convert the preprocessed data so that the data conforms to the form of association analysis, such as converting the fault type into a fault code.

[0049] S03. Mining frequent item sets: Use association analysis algorithms (such as Apriori algorithm or FP-Growth algorithm) to mine frequent item sets that meet the support threshold according to the minimum support, that is, combinations of fault data that often appear at the same time.

[0050] S04. Establish monitoring labels: Establish monitoring labels for the mined frequent itemsets, where the detection labels include fault labels and transaction labels.

[0051] It should be noted that transaction labels refer to monitoring object labels set by users according to their needs. In the frequent item set mining process, in addition to the mining of tracking data, it also includes special mining of transaction sets set by users to establish transaction labels.

[0052] S05. Generate strong association rules: Filter out strong association rules from the frequent item set according to the set strong association rule conditions; the strong association rule conditions include support and confidence, etc.

[0053] S06. Measuring strong association rules: Evaluate strong association rules by lift, leverage, and confidence.

[0054] Among them, the improvement , which is used to measure whether the probability of Y appearing when X appears is improved. When the lift is 1, X and Y are independent of each other; when the lift is greater than 1, X and Y are positively correlated, and the larger the lift, the stronger the positive correlation; when the lift is less than 1, X and Y are negatively correlated, and the smaller the lift, the stronger the negative correlation.

[0055] Leverage When the leverage ratio is 0, X and Y are independent of each other; when the leverage ratio is greater than 0, X and Y are positively correlated, and the larger the leverage ratio, the stronger the positive correlation; when the leverage ratio is less than 0, X and Y are negatively correlated, and the larger the leverage ratio, the stronger the negative correlation.

[0056] Confidence ,in is the confidence level, , when the confidence is 1, X and Y are independent of each other; when the confidence is greater than 1, X and Y are positively correlated, and the larger the confidence is, the stronger the positive correlation is; when the confidence is less than 1, X and Y are negatively correlated, and the smaller the confidence is, the stronger the negative correlation is.

[0057] S07. Build a strong association rule base: Use strong association rules that meet the measurement indicators to build a strong association rule base to complete the analysis and monitoring of tracking data.

[0058] It is understandable that frequent item sets in the tracking data and the association rules between transactions, calls and events are obtained through association rule mining, so as to realize the analysis of call tracking between microservices; in particular, by mining the selected transaction set, the personalized needs of users are met, which is not a single frequent item set mining or fault data mining.

[0059] The data display module is used for link query, log query, status display and alarm push. The data is visualized in the form of charts, graphs, tables, etc., and the data changes and status are displayed in real time to help users understand the operation of the system or business in a timely manner.

[0060] This embodiment also provides an analysis method for tracking calls between microservices, including the following steps:

[0061] S11. Define tracking requirements: Determine the key indicators, performance indicators, and business processes to be tracked.

[0062] It is understandable that key indicators include request volume, response time, error rate, etc.; performance indicators include throughput, concurrency, latency, etc.; business processes include the entire user request processing process, such as user login, order query, order placement, etc. By tracking the business process, we can understand the specific process of the request in the system, as well as the processing time and contribution rate of each microservice.

[0063] In addition to the above tracking content, other tracking requirements may also be considered, such as: Support for distributed tracking: Since microservice architectures are usually distributed, it is necessary to support cross-service tracking and link tracking. Logging and error tracking: It is necessary to record log information in the system and track and handle errors in a timely manner. Monitoring and alarm mechanism: It is necessary to set up a monitoring and alarm mechanism to promptly notify relevant personnel when key indicators or performance issues exceed thresholds.

[0064] S12. Insert tracing code: Insert tracing code at key locations of each microservice according to tracing requirements so that tracing identifiers can be passed between services.

[0065] It should be noted that if the tracing requirements cannot be met by simply inserting the tracing code, you can integrate the tracing tools, select the appropriate tracing tools, such as OpenTelemetry, Zipkin, or Jaeger, and integrate them into the microservice architecture to track request delivery and performance indicators, and quickly identify bottlenecks and errors in the system.

[0066] S13. Generate unique identifier: Generate a unique tracking identifier for each cross-service request to track the request throughout the entire call chain.

[0067] It is understood that the generation of tracking identifiers includes the following methods:

[0068] UUID (Universally Unique Identifier): UUID is a standardized unique identifier format that can be generated by various programming languages ​​and libraries. For example, in Java, you can use the java.util.UUID class to generate UUID. Each generated UUID is unique and suitable for tracking requests.

[0069] Distributed ID Generators: Generate unique identifiers using distributed ID generators, such as the Snowflake algorithm or Twitter's Snowflake algorithm. These algorithms combine elements such as timestamps and machine IDs to generate globally unique identifiers.

[0070] Custom identifier generation: Generate a unique identifier using a custom method. For example, combine a timestamp with a random number, or use specific rules to generate an identifier. The key to this method is to ensure that the generated identifier is unique enough and can be correctly passed and associated throughout the call chain.

[0071] It should be noted that in the above method for generating the identifier, it is ensured that the identifier is generated at the beginning of the request and is associated with the request.

[0072] S14. Collect tracing data: Record key tracing data in each microservice and connect the entire call chain by passing tracing information across services (i.e. passing tracing identifiers between microservices).

[0073] S15. Storing and analyzing tracking data: storing the tracking data in a data storage module, and analyzing and visualizing the data through a data analysis module and a data display module.

[0074] The data analysis module performs analysis including the following steps:

[0075] Data preprocessing: Since the order of magnitude of the collected data is different, the collected tracking data is normalized; Construct an accident database: Convert the preprocessed data so that the data conforms to the form of association analysis, such as converting the fault type into a fault code. Mining frequent item sets: Use an association analysis algorithm (such as the Apriori algorithm or the FP-Growth algorithm) to mine frequent item sets that meet the support threshold according to the minimum support, that is, the combination of fault data that often appears at the same time. Establish monitoring labels: Establish monitoring labels for the mined frequent item sets, where the detection labels include fault labels and transaction labels. Generate strong association rules: Filter out strong association rules from the frequent item sets according to the set strong association rule conditions; the strong association rule conditions include support and confidence, etc. Measure strong association rules: Evaluate strong association rules by lift, leverage and confidence. Build a strong association rule base: Use strong association rules that meet the measurement indicators to build a strong association rule base, so as to complete the analysis and monitoring of tracking data.

[0076] It is understandable that data visualization is performed through the data display module to visualize the trace data to better understand the performance of the entire call link. Various tools and libraries are used to generate charts, dashboards, or other forms of visual reports. For example, tools such as Elasticsearch, Grafana, and Kibana are used to display and analyze trace data.

[0077] S16. Analyze and optimize performance: Based on the analysis results of the data analysis module, identify potential performance bottlenecks and delays and optimize them.

[0078] It is understandable that by analyzing the trace data, performance bottlenecks can be identified. Check the requests with long service call times, check the performance issues of database queries, find out the potential causes of delays, and determine the specific links that cause performance degradation. According to the analysis results, take corresponding optimization measures. The optimization methods include:

[0079] Cache: cache data to reduce access to external resources; asynchronous processing: convert some time-consuming operations to asynchronous processing to improve concurrency performance; concurrency control: optimize the concurrency control strategy for operations such as database queries and resource access to avoid performance bottlenecks; service splitting: split overly large or complex services into smaller services to improve performance and scalability; optimize algorithms and data structures: for complex computing operations, optimize algorithms and data structures to reduce time complexity and space complexity.

[0080] It should be noted that after optimization, performance indicators are continuously monitored and iterative improvements are made to ensure timely detection and resolution of emerging performance issues and continuous optimization of the overall performance of the system.

[0081] S17. Monitoring and alarm: By setting up monitoring and alarm mechanisms, relevant personnel can be notified in a timely manner when key indicators or performance indicators exceed the threshold.

[0082] It is understandable that monitoring and alarming generally include the following processes: First, determine the indicators that need to be monitored, including server load, response time, memory usage, database query time, etc. In actual applications, select indicators that are closely related to performance according to the system requirements of the design. Select monitoring tools: Select monitoring tools suitable for the system, such as Prometheus, Grafana, Zabbix, Datadog, etc. Be sure to ensure that the selected tool supports the monitored indicators and provides flexible alarm configuration functions. Configure monitoring indicator collection: Collect the selected monitoring indicators by configuring monitoring tools in the system. Generally, you can collect indicators by adding indicator recording code to the application code, or use the agent or plug-in provided by the tool. Set alarm rules: Set corresponding alarm rules according to business needs and system performance goals. For example, when the server load exceeds the threshold, the response time exceeds a specific time, the database query time is too long, etc., the alarm is triggered. Make sure to set a reasonable threshold and appropriate alarm level. Set notification channels: Select appropriate notification channels to ensure that relevant personnel are notified in time when performance problems occur, such as email, SMS, instant messaging tools, etc. In addition, different notification methods can be set according to different alarm levels to ensure that key issues can be responded to in time. Test and verify: After deploying the monitoring and alarm mechanism, test and verify to ensure that the monitoring tool can correctly collect indicators and trigger corresponding alarms when the threshold is exceeded. Iterative improvement: Continuously monitor and adjust the alarm rules and thresholds to ensure the effectiveness of the monitoring and alarm mechanism. Optimize the alarm configuration according to the actual situation to avoid false positives and false negatives.

[0083] The present invention obtains the tracking data between microservices through the data acquisition module, and performs association rule mining on the stored tracking data through the data analysis module, thereby discovering the frequent item sets between the tracking data and the association relationship between transactions, calls and events. At the same time, through the mining of specific transaction data, the user demand status is monitored, service dependencies are discovered, bottlenecks and performance issues are identified, service call patterns are analyzed, abnormal behaviors and faults are discovered, and service design and architecture are optimized, thereby improving the availability, performance and maintainability of the microservice architecture.

[0084] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technical personnel in this field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. An analysis system for tracking calls between microservices, Features: It includes a data acquisition module, a data transmission module, a data storage module, a data analysis module and a data display module, wherein the data acquisition module is electrically connected to the data transmission module, the data transmission module is electrically connected to the data storage module, the data storage module is electrically connected to the data analysis module, and the data analysis module is electrically connected to the data display module; The data analysis module is used to perform call tracking analysis between microservices by calling the tracking data of the data storage module; The process of performing call tracking analysis by the data analysis module includes the following steps: Data preprocessing: normalize the collected tracking data; Constructing accident database: transforming preprocessed data to make it suitable for correlation analysis; Mining frequent item sets: Use association analysis algorithm to mine frequent item sets that meet the support threshold according to the minimum support; Establish monitoring labels: Establish monitoring labels for the mined frequent itemsets, where the detection labels include fault labels and transaction labels; The mining of frequent itemsets includes tracking data and transaction sets set by users; The process of the data analysis module performing call tracking analysis also includes the following steps: Generate strong association rules: Filter out strong association rules from frequent item sets according to the set strong association rule conditions; strong association rule conditions include support and confidence; Measuring strong association rules: Evaluate strong association rules by lift, leverage, and confidence; lift , leverage ratio , confidence , where X and Y represent different item sets; Build a strong association rule base: Use strong association rules that meet the measurement indicators to build a strong association rule base to complete the analysis and monitoring of tracking data; The acquisition of tracking data includes the following steps: Identify key metrics, performance indicators, and business processes to track; Insert tracing code at key locations of each microservice based on tracing requirements and select integrated tracing tools; Generate a unique tracking identifier for each cross-service request to complete the tracking request in the entire call chain; Record key tracing data in each microservice and connect the entire call chain by passing tracing information across services; The key indicators include request volume, response time and error rate; performance indicators include throughput, concurrency and latency; business processes include the entire user request processing process; The data collection module is used to collect tracking data based on the embedding log, wherein the tracking data includes service performance data on the microservice node and data describing the service call relationship; The data transmission module is used for asynchronously transmitting the collected tracking data; The data storage module is used to uniformly manage and store the collected tracking data and provide a query interface based on user transaction requests; The data storage module matches the collected tracking data according to the classified transactions, calls and events, and constructs an item set according to the transaction ID for storage.

2. According to claim 1, an analysis system for tracking calls between microservices, Features: The data display module is used for link query, log query, status display and alarm push, and displays data changes and status in real time by visually displaying data in the form of charts, graphs and tables.

3. An analysis method for tracking calls between microservices, applied to an analysis system for tracking calls between microservices as claimed in any one of claims 1 to 2, Features: The following steps are involved: S11. Define tracking requirements: determine the key indicators, performance indicators and business processes to be tracked; S12. Insert tracing code: Insert tracing code at key locations of each microservice according to tracing requirements and select integrated tracing tools; S13. Generate a unique identifier: Generate a unique tracking identifier for each cross-service request to complete the tracking request in the entire call chain; S14. Collect tracing data: Record key tracing data in each microservice and connect the entire call chain by passing tracing information across services; S15, storing and analyzing tracking data: storing the tracking data in a data storage module, and analyzing and visualizing the data through a data analysis module and a data display module; S16. Analyze and optimize performance: According to the analysis results of the data analysis module, identify potential performance bottlenecks and delays, and optimize them; S17, Monitoring and Alarm: By setting up monitoring and alarm mechanisms, relevant personnel are notified in a timely manner when key indicators or performance indicators exceed thresholds; In step S11, the key indicators include request volume, response time and error rate; Performance indicators include throughput, concurrency, and latency; business processes include the entire user request processing process.

Images