A full-link monitoring system and fault diagnosis method applied to a micro-service architecture

Abstract

The application discloses a kind of whole link monitoring system and fault diagnosis method applied to microservice architecture, it is related to microservice architecture technical field, including: link data acquisition module, for integrating TLOG component into each service, and when each service carries out request and response, the link data of each service is intercepted and captured;Link data processing module is used to receive the link data sent by each service using Kafka, generates link data table using Flink The latest link data table and standard link data table are obtained;Link data analysis module matches fault judgment mode according to comparison result, carries out fault root cause positioning;Fault data display module is used to show fault root cause positioning result.The application supports a variety of fault judgment mode and business scene adaptation strategy, through the mutual cooperation between each module, effectively solve the lag of fault discovery under microservice architecture, positioning difficulty and the deficiency of relying on artificial, realize the fault diagnosis effect of whole link.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of micro-service architecture, and particularly relates to a full-link monitoring system and a fault diagnosis method applied to micro-service architecture. BACKGROUND

[0002] With the continuous development of information technology, the micro-service architecture has been widely applied in the field of software development. The architecture splits a complex application into multiple small and autonomous services, so that the communication between services becomes complex and frequent, thereby greatly improving the flexibility and scalability of the system, enabling each service to be independently developed, deployed and maintained, and quickly responding to changes in market demand. However, the micro-service architecture still has some problems in complex communication between services, especially in fault troubleshooting and processing.

[0003] In the micro-service architecture, the traditional monitoring method usually only focuses on the health status of individual services, including: setting basic monitoring indicators (service response time, throughput, etc.), judging whether the service is normally running by monitoring the indicators; collecting and analyzing service logs, when a fault occurs, the operation and maintenance personnel will check the service log files and find possible abnormal information from them; setting a health check interface for a specific service, periodically calling the interface to confirm whether the service is alive, etc.

[0004] In the micro-service architecture, a request may pass through multiple services, and the fault reason may involve multiple services. The traditional monitoring method ignores the complete link information across services and only focuses on individual services, so that when a fault occurs, the operation and maintenance personnel cannot accurately locate in time, and the fault response often depends on manual troubleshooting and analysis, which is easy to cause fault processing lag, thereby affecting the stability and reliability of the system.

[0005] Based on this, the present application provides a full-link monitoring system and a fault diagnosis method applied to micro-service architecture, which can eliminate the drawbacks of the prior art. SUMMARY

[0006] The purpose of the present application is to provide a full-link monitoring system and a fault diagnosis method applied to micro-service architecture, to solve the problem that the traditional monitoring method applied to micro-service architecture in the background art only focuses on the health status of individual services and ignores the complete link information across services.

[0007] To achieve the above purpose, the present application provides the following technical scheme:

[0008] A full-link monitoring system applied to micro-service architecture, comprising:

[0009] The link data collection module is configured to integrate the TLOG component into each service and intercept and capture link data of each service when the service makes a request and a response. The link data collection module includes a TLOG component in the form of a software program and integrated into each service to capture link information when the service makes a request and a response, and an interceptor configured to perform an intercepting operation in the process of service request and response to ensure that the TLOG component can accurately obtain link data.

[0010] The link data processing module is configured to receive link data sent by each service using Kafka, generate a link data table using Flink, obtain a latest link data table and a standard link data table, and tag a business scenario according to a business scenario scale and a service scheduling depth in the process of generating the link data table, match an adaptive processing mode. The link data processing module includes Kafka, Flink, a timing task scheduling component, and a preset standard link matching unit.

[0011] The processing mode includes a first processing mode in which Flink directly generates a dynamic link data table as a latest link data table, and adapts a preset standard link data table, a second processing mode in which Flink aggregates data and saves the data to a temporary table, and a timing task scheduling component writes data in the temporary table to the latest link data table and the standard link data table, and a third processing mode in which Flink generates an enhanced table for storing hot and cold data in a hierarchical manner, hot data is processed using the first processing mode, and cold data is processed using the second processing mode.

[0012] The link data analysis module is configured to compare the latest link data table and the standard link data table, match an adaptive fault judgment mode according to a comparison result, and perform fault root cause positioning using the adaptive fault judgment mode. The link data analysis module has a machine learning model built in.

[0013] The fault data display module is configured to interact with the link data analysis module to obtain a fault root cause positioning result, and display the fault root cause positioning result in a visual form.

[0014] Preferably, the fault judgment mode includes a first fault judgment mode and a second fault judgment mode, the comparison result is topological consistency and error adaptation, and the first fault judgment mode is adapted, and the comparison result is topological inconsistency, and the second fault judgment mode is adapted.

[0015] The first fault judgment mode includes querying and analyzing an error type of a last error service, determining whether there is an upstream or downstream service that is affected by the last error service according to the error type, and performing multi-dimensional fault root cause positioning of the last error service or the last error service and the upstream or downstream service that is affected by the last error service in combination with a machine learning model.

[0016] The second fault judgment mode comprises: locating the first inconsistent service in the topology, combining a machine learning model, triggering multi-dimensional fault root cause positioning of the service itself, direct invoker and network between the service itself and the invoker.

[0017] Preferably, the timing task scheduling component takes the basic scheduling period as the reference, and corrects the basic scheduling period by a scheduling period correction value according to the future link fault risk level predicted by the current fault root cause and by the link data analysis module; the higher the fault risk level is, the larger the correction value is, and the smaller the corrected scheduling period is.

[0018] Preferably, the link data processing module is further configured to: in the process of marking the business scenario label according to the business scenario scale and the service scheduling depth, marking the business priority;

[0019] After marking the business scenario label according to the business scenario scale and the service scheduling depth and matching the adaptive processing mode, desensitizing the link data according to the data desensitization mode matched with each business scenario according to the business scenario label;

[0020] In the process of generating the enhanced table of the hierarchical storage cold and hot data by Flink, the cold and hot data determination rules are dynamically adjusted in combination with the business priority corresponding to the link data and the historical link data influence range;

[0021] The dynamic adjustment of the cold and hot data determination comprises: quantifying whether the preset business core data is contained and whether the access frequency is greater than the preset access frequency, and obtaining a basic determination score by weighted calculation; quantifying the business priority corresponding to the link data to obtain a business priority quantization weight coefficient; quantifying the historical link data influence range to obtain a historical link data influence range quantization coefficient; correcting the basic determination score by the business priority quantization weight coefficient and the historical link data influence range quantization coefficient as correction coefficients to obtain a final determination score; comparing the final determination score with a preset score, and if the final determination score is greater than the preset score, determining that the data is hot data, otherwise, determining that the data is cold data.

[0022] Preferably, the link data processing module is further configured to:

[0023] The back pressure index of the Flink job is monitored in real time, the current back pressure index is compared with a preset back pressure index threshold, whether a matched processing mode switching strategy is triggered is determined according to a comparison result, and when the matched processing mode switching strategy is triggered, a target switching processing mode is started. When the target switching processing mode is stable, the original processing mode is stopped, and the target processing mode is switched. The condition for triggering the matched processing mode switching strategy includes: if the current back pressure index is greater than the preset index threshold and the duration is greater than a first preset time period, the matched processing mode is switched according to a priority order of switching the second processing mode, the third processing mode and the improved third processing mode.

[0024] The improved third processing mode includes reducing a preset score in dynamic adjustment of a cold and hot data determination rule, and a score reduction range of the preset score is adjusted based on an absolute value of a difference between the back pressure index and the preset back pressure index threshold. The larger the absolute value of the difference, the greater the score reduction range.

[0025] If it is monitored that the current back pressure index is greater than the preset index threshold and the duration is greater than a second preset time period, the matched processing mode is directly switched according to the strategy of switching the improved third processing mode. The second preset time period is greater than the first preset time period.

[0026] If it is monitored that the current back pressure index is less than the preset back pressure index threshold, the priority order needs to be switched according to the condition for triggering the matched processing mode switching strategy, and the original processing mode is switched back in a reverse priority order.

[0027] Preferably, the link data processing module is further configured to: in a process of writing data in a temporary table into a latest link data table and a standard link data table by using a timing task scheduling component, setting a link timeout threshold for the temporary table, marking link data exceeding the link timeout threshold as incomplete link data, and taking the incomplete link data as an archive log without writing the incomplete link data into the latest link data table and the standard link data table.

[0028] Preferably, the link data analysis module includes a comparison unit and a mode selection unit. The comparison unit is configured to compare the latest link data table and the standard link data table to obtain a comparison result of whether a topology result is consistent. The mode selection unit is configured to match a fault judgment mode adapted to the comparison result, and to perform fault root cause positioning by using the adapted fault judgment mode.

[0029] The link data analysis module is further configured to: in a process of comparing the latest link data table and the standard link data table and matching the adapted fault judgment mode according to a comparison result, judging whether a branch structure exists in a topology for the standard link data table. When it is judged that the branch structure exists, a third fault judgment mode adapted to the comparison result and consistent with an error is selected to replace a first fault judgment mode.

[0030] The third fault judgment mode includes: querying the last service that reported an error and whether there is a service with a symmetrical counterpart in the current topology. If a service with a symmetrical counterpart is found in the current topology, the service log data of the service with a symmetrical counterpart and the error type of the last service that reported an error are combined to determine whether there are upstream or downstream services that affect the last service that reported an error. If no service with a symmetrical counterpart is found in the current topology, the error type of the last service that reported an error is directly used to determine whether there are upstream or downstream services that affect the last service that reported an error. Then, a machine learning model is used to perform multi-dimensional fault root cause localization of the last service that reported an error or the last service that reported an error and its upstream or downstream services that affected it.

[0031] Prioritized, the step of combining the service log data of the service symmetrical to the last error-reporting service and the error type of the last error-reporting service to comprehensively determine whether there are upstream and downstream services that affect the last error-reporting service includes: determining whether the symmetrical service meets a first condition based on the service log data of the service symmetrical to the last error-reporting service; the first condition includes: the existence of error information and the existence of a business relationship with the last error-reporting service; counting the number and proportion of services symmetrical to the last error-reporting service that meet the first condition; if the number of services symmetrical to the last error-reporting service that meet the first condition is greater than a preset number or the proportion of services symmetrical to the last error-reporting service that meet the first condition is greater than a preset proportion, then regardless of the error type of the last error-reporting service, it is directly determined that there are upstream and downstream services that affect the last error-reporting service.

[0032] Preferably, the link data analysis module is further configured to: monitor service metrics data through Prometheus during the fault root cause localization process, obtain the corresponding service's time consumption data, request parameters, memory and CPU resource metrics, match the feature weight ratio adapted to each business scenario according to the business scenario label, obtain the feature weight ratio matched to the current business scenario label, use a machine learning model, input the corresponding service's time consumption data, request parameters, memory and CPU resource metrics and service log data, and use the matched feature weight ratio to output the fault root cause localization.

[0033] A fault diagnosis method for a full-link monitoring system applied to a microservice architecture specifically includes the following steps:

[0034] Step S1: Integrate the TLOG component into each service, and intercept and capture the link data of each service when making requests and responses;

[0035] Step S2: Use Kafka to receive the link data sent by each service, use Flink to generate a link data table from the link data, and obtain the latest link data table and the standard link data table.

[0036] Step S3: Compare the latest link data table with the standard link data table, match the appropriate fault judgment mode according to the comparison result, and use the appropriate fault judgment mode to locate the root cause of the fault in multiple dimensions.

[0037] Step S4: Display the obtained multi-dimensional fault root cause localization to realize microservice fault localization and diagnosis.

[0038] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0039] 1. This invention utilizes the TLOG component to intercept and capture link data from various services, comprehensively acquiring link information. It employs Kafka to receive data and Flink to generate link data tables, improving the efficiency of link data processing. It compares the latest link data table with the standard link data table and matches fault judgment patterns based on the comparison results. Combined with machine learning models, it performs multi-dimensional fault root cause localization, achieving accurate fault localization and diagnosis. The fault root cause localization is visualized, making it convenient for maintenance personnel to view and handle faults, thus achieving automated fault diagnosis.

[0040] 2. This invention processes data according to the business scenario tag adaptation processing mode, making data processing more suitable for the needs of different business scenarios, realizing multi-dimensional troubleshooting of service faults, and combining precise analysis with business scenarios, thereby improving the accuracy of fault detection and early warning capabilities.

[0041] 3. This invention performs business priority tagging on link data and adjusts the rules for determining hot and cold data, and reasonably stores hot and cold data in layers to improve the efficiency of data processing and storage. It also sets a link timeout threshold to handle incomplete links, so as to avoid incomplete link data from interfering with the latest link data table and the standard link data table, and protect the security and privacy of link data in different business scenarios. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the structure of the end-to-end monitoring system of the present invention.

[0043] Figure 2 This is a schematic diagram of the end-to-end monitoring system of the present invention.

[0044] Figure 3 This is a schematic diagram illustrating the steps of the fault diagnosis method of the present invention.

[0045] Figure label annotations: Link data acquisition module 10, Link data processing module 20, Link data analysis module 30, Fault data display module 40. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0047] Example 1

[0048] In this embodiment, as Figure 1 and Figure 2 As shown, a full-link monitoring system applied to a microservice architecture includes:

[0049] Link data acquisition module 10 is used to integrate the TLOG component into each service and intercept and capture the link data of each service when making requests and responses. The link data acquisition module 10 includes the TLOG component and an interceptor. The TLOG component is integrated into each service in the form of a software program. When each service makes requests and responses, it captures link information. The TLOG component is a lightweight software library that can be embedded into each microservice in the form of code. It can generate a unique trace identifier (TraceID) for each external request and record link node data (Span) containing information such as TraceID, SpanID, parent SpanID, service name, timestamp, time consumption, and error code when the request traverses each service. The interceptor is used to perform interception operations during the service request and response process to ensure that the TLOG component can accurately obtain the link data. The interceptor and the TLOG component are connected through a code-level association relationship. When a service makes a request and response, the interceptor will trigger the TLOG component to capture data, such as user service link data, product service data, and order service data.

[0050] The link data processing module 20 is used to receive link data sent by various services using Kafka, generate link data tables using Flink, obtain the latest link data table and the standard link data table, and also to tag the link data table with business scenario tags according to the business scenario scale and service scheduling depth during the link data table generation process, matching and adapting the processing mode. The link data processing module 20 includes Kafka, Flink, a scheduled task component, and a preset standard link matching unit. Kafka is a high-throughput distributed message queue used to decouple data acquisition and processing, and Flink is a stream processing framework used for... Real-time processing of link data: The scheduled task component uses a basic scheduling cycle as a benchmark. The link data analysis module 30 predicts the future link failure risk level based on the current root cause of the failure and adjusts the basic scheduling cycle by matching the scheduling cycle correction value. The higher the failure risk level, the larger the correction value, and the smaller the adjusted scheduling cycle. This module uses the Kafka message queue as a data buffer and relay station. It uses Kafka to receive link data sent by various services and uses Flink to aggregate individual link data into a complete link data. It obtains the latest link data table and the standard link data table. The latest link data table can be generated in real time using FlinkStateful Functions, that is, using TraceID as the key, all spans belonging to the same Trace are aggregated in real time. When the last span of a Trace is received, the complete call chain is calculated directly in real time, generating a complete link data table. The preset standard link matching unit is used to dynamically match and obtain the preset standard link table in streaming computation, or the scheduled task component is used to write the link data stored in the temporary table into the latest link table and the standard link table. Figure 2 As shown, when using Flink to aggregate individual link data into a complete link data, a dynamic collection strategy can also be adopted, for example: normal requests are sampled at 1%; high latency / error requests are sampled at 100%;

[0051] The processing modes include a first processing mode, a second processing mode, and a third processing mode. In the first processing mode, Flink directly generates a dynamic link data table as the latest link data table, which is adapted to the preset standard link data table. In the second processing mode, Flink aggregates data and saves it to a temporary table. A timed task scheduling component is used to write the data in the temporary table to the latest link data table and the standard link data table. In the third processing mode, Flink generates an enhanced table that stores hot and cold data in a hierarchical manner. Hot data uses the first processing mode, and cold data uses the second processing mode.

[0052] In this embodiment, the first processing mode is adapted to the first business scenario, directly generating dynamic links and dynamically matching and comparing them with the preset standard link data table stored in memory. In this process, the latest link data table with a frequency greater than the preset frequency under the same business scenario conditions in the adjacent preset time period can be used as the standard link data table for periodic updates.

[0053] In this embodiment, the second processing mode is adapted to the second business scenario. Flink is used for streaming processing to aggregate, filter and transform data in real time, forming a complete link data and saving it to a temporary table. The scheduled task scheduling component is called to design and execute a scheduled task: the temporary link is processed to form the latest link and the standard link, and written to the latest link data table and the standard link data table respectively. The temporary table is cleaned to reduce the accumulation of temporary data and the occupation of storage space.

[0054] In this embodiment, the third processing mode is adapted to the third business scenario. Flink is used to generate an enhanced table for hierarchical storage of hot and cold data. A hybrid storage strategy is adopted. Hot data is link data that contains preset core business data and has an access frequency greater than the preset access frequency. Cold data is link data that does not contain preset core business data and has an access frequency no greater than the preset access frequency. The first processing mode is applied to hot data links, and the second processing mode is applied to cold data links. Hot and cold data are classified according to the access frequency and value of the link data.

[0055] The link data analysis module 30 is used to compare the latest link data table with the standard link data table, match an appropriate fault judgment mode based on the comparison results, and use the appropriate fault judgment mode to locate the root cause of the fault. The link data analysis module 30 includes a comparison unit and a mode selection unit, and has a built-in machine learning model. The machine learning model is a specially trained machine learning model, such as a CNN neural network model, a recurrent neural network model, a graph neural network model, or an ensemble of these models. The comparison unit is used to compare the latest link data table with the standard link data table to obtain a comparison result to determine whether the topology results are consistent. For example, if the standard link data table corresponds to the link as service A – service B – service C – service D, and... If the latest link data table corresponds to the link A-Service B-Service C, then the corresponding topology results are inconsistent. For example, if the standard link data table corresponds to the link A-Service B-Service C-Service D, while the latest link data table corresponds to the link A-Service B-Service C-Service D, and the link for Service D shows an internal error, then the corresponding topology results are consistent and an error is reported. The mode selection unit is used to match the fault judgment mode that is suitable for the comparison result, and to use the suitable fault judgment mode to locate the root cause of the fault. The fault judgment mode includes a first fault judgment mode and a second fault judgment mode. If the comparison result is that the topology is consistent and an error is reported, the first fault judgment mode is suitable. If the comparison result is that the topology is inconsistent, the second fault judgment mode is suitable.

[0056] The first fault diagnosis mode includes: querying and analyzing the error type of the last service that reported an error; determining whether there are any upstream or downstream services affecting it based on the error type; and generating an error type and upstream / downstream service mapping table based on historical error type statistics to complete the specific judgment. For example: an order request is made via Gateway→Order-Service→Payment-Service→Inventory-Service. Both the standard link data table and the latest link data table contain these four services, but the request fails. The error information and status codes of the link are checked to obtain the error type of the last service that reported an error. If the error type is 5xx Server Error, it indicates that the service itself failed to execute, and therefore it is considered faulty, with no upstream or downstream services affecting it. If the error type is Timeout or Error... Propagation may involve upstream and downstream services that affect it. By combining machine learning models, multi-dimensional root cause localization of the last faulty service or the last faulty service and its affected upstream and downstream services can be performed. That is, by using Prometheus monitoring data, multi-dimensional data of the last faulty service is obtained and input into a pre-trained machine learning model for fault diagnosis and localization to obtain the final root cause localization of the fault.

[0057] Furthermore, when considering whether upstream and downstream services directly affect the last service that reported an error, the scheduling depth of upstream and downstream services can be set in advance based on the average link impact range of service failures in various historical business scenarios. If it is determined that there are upstream and downstream services that affect the last service that reported an error, the scheduling depth of upstream and downstream services set in the corresponding business scenario is matched, and several upstream and downstream services with the corresponding service scheduling depth are obtained to perform multi-dimensional fault root cause localization of the last service that reported an error and several upstream and downstream services with the corresponding service scheduling depth that affected it.

[0058] The second fault diagnosis mode includes: locating the first inconsistent service in the topology, and combining it with a machine learning model to trigger multi-dimensional fault root cause localization of the service itself, the direct caller, and the network between the service and the caller. For example, if the standard link data table corresponds to the link A-service B-service C-service D, while the latest link data table corresponds to the link A-service B-service C, locate the first inconsistent service D in the topology. The fault usually lies in the first inconsistent service, i.e., service D itself. It may also be due to a problem with the caller C affecting the normal operation of service D, or there may be a problem with the intermediate network device between the first inconsistent service D and the caller service C. Then, parallel bidirectional detection is performed, and the multi-dimensional data corresponding to the service or network is input into the machine learning model to obtain the final fault root cause localization.

[0059] Furthermore, when considering whether the direct caller directly affects the first inconsistent service in the location topology, the service scheduling layer depth of the caller service can be set in advance based on the average link impact range of service failures in various historical business scenarios, and the caller service scheduling depth set in the corresponding business scenario can be matched to trigger multi-dimensional fault root cause location.

[0060] The fault data display module 40 is used to interact with the link data analysis module 30 to obtain the fault root cause location results and display the fault root cause location results in a visual form. The fault data display module 40 is a web page-based display component that can display fault information through intuitive charts, reports and other forms.

[0061] The link topology diagram displays nodes, links, and error markers. Each service (e.g., A, B, C, D) is represented by a circular node, with colors indicating status (e.g., green for normal, red for error, yellow for warning, gray for no call, etc.). Infrastructure such as databases and load balancers are represented by square nodes. Connections between nodes represent call relationships, line width indicates call frequency (e.g., wider lines indicate more frequent calls), and color indicates latency (e.g., red for over 2000ms, yellow for 1000-2000ms, green for under 1000ms). Error nodes display a red exclamation mark, and hovering the mouse displays the error type (e.g., D service: 5xx error). Clicking on a node allows viewing detailed metrics (CPU, memory, latency) and log snippets. This display component supports zooming, panning, and other operations.

[0062] The root cause display includes a root cause overview, indicator trends, log snippets, and handling suggestions. The root cause overview includes the root cause of the fault, the probability of the root cause, and the scope of impact. The indicator trends are displayed as a line chart showing the changes of the core indicators of the faulty service over the past hour. The log snippets can display the key logs of the faulty service. The handling suggestions are displayed in a list format, showing actionable suggestions. Each suggestion is marked with priority and operation steps, and the panel can be expanded and collapsed.

[0063] In this embodiment, the link data processing module 20 also includes a tagging unit and a pattern matching unit. Considering that the microservice architecture of applications in different business scenarios is quite different, such as some simple business scenarios with only 3 to 5 services, real-time processing can be performed directly. For some more complex but typical business scenarios, batch asynchronous processing can be used to dynamically match the preset standard link data table.

[0064] To further optimize link monitoring and avoid data accumulation between operators during the link data table generation process using Flink, the link data processing module 20 is also used for:

[0065] Real-time monitoring of back pressure metrics of Flink jobs; comparison of current back pressure metrics with preset back pressure metric thresholds; determination of whether to trigger an adaptive processing mode switching strategy based on comparison results; initiation of target processing mode switching when triggered; cessation of original processing mode operation and switching to target processing mode once target processing mode stabilizes; conditions for triggering the adaptive processing mode switching strategy include: if the current back pressure metric is greater than the preset metric threshold and the duration is greater than the first preset time period, then the matching processing mode is switched in the priority order of switching to the second processing mode, switching to the third processing mode, and switching to the improved third processing mode.

[0066] The improved third processing mode includes reducing the preset score in the dynamic adjustment of the hot and cold data judgment rules. The score reduction is adjusted based on the absolute value of the difference between the back pressure index and the preset back pressure index threshold. The larger the absolute value of the difference, the larger the score reduction.

[0067] If the current back pressure index is detected to be greater than the preset index threshold and the duration is greater than the second preset time period, the matching processing mode will be switched directly according to the strategy of switching to the improved third processing mode, and the second preset time period is greater than the first preset time period.

[0068] If the current back pressure index is detected to be less than the preset back pressure index threshold, the switching priority order in the trigger adaptation processing mode switching strategy conditions should be followed, and the original processing mode should be switched back according to the reverse priority.

[0069] Among them, such as Figure 1As shown, the link data processing module 20 is also used to: tag business priorities during the process of tagging business scenarios according to business scenario scale and service scheduling depth;

[0070] After tagging business scenarios with business scenario scale and service scheduling depth, and matching and adapting processing modes, the data anonymization method is matched with each business scenario according to the business scenario tag. The data anonymization method matched with the current business scenario tag is used to anonymize the link data. Considering that different business scenarios have different requirements for data protection, this invention presets adapted data anonymization methods for different business scenarios. For example, for the first business scenario, when collecting data using TLOG, the following anonymization methods are used: hashing user_id, retaining the first 6 digits of order num, and replacing the middle 4 digits of phone num with other fields. This operation can protect the security and privacy of link data under different business scenarios.

[0071] During the process of generating the enhanced table for hierarchical storage of hot and cold data by Flink, the rules for determining hot and cold data are dynamically adjusted based on the business priority corresponding to the link data and the influence range of historical link data.

[0072] Dynamic adjustment of hot and cold data judgment includes: quantifying whether it contains preset core business data and whether the access frequency is greater than the preset access frequency, and obtaining a basic judgment score through weighted calculation. For example, containing preset business data earns 100 points, and the access frequency being greater than the preset access frequency earns 100 points, with a weighted calculation of (0.5:0.5) to obtain 100 points; quantifying the business priority corresponding to the link data and obtaining the business priority quantification weight coefficient. For example, business priorities P1, P2, P3, and P4 can have corresponding quantification weight coefficient ranges of (0.8-1.0], (0.6-0.8], (0.4-0.6], and (0.1-0.4], respectively; quantifying the impact range of historical link data and obtaining the impact range of historical link data. The impact coefficient is determined by the number of core business links associated with the link data. If the link data is associated with at least three core business links, it is considered to have a broad impact, with a coefficient ranging from 1.6 to 2.0; if associated with two core business links, it is considered to have a medium impact, with a coefficient ranging from 1.2 to 1.5; and if associated with only one core business link, it is considered to have a moderate impact, with a coefficient ranging from 1.0 to 1.1. The business priority quantification weight coefficient and the historical link data impact range quantification coefficient are used as correction coefficients to adjust the base judgment score. The final judgment score is then compared with the preset score. If the final judgment score is greater than the preset score, it is considered hot data; otherwise, it is considered cold data.

[0073] The business scenario tag is an automatic classification identifier for each request link, used to distinguish different business characteristics. The link data processing module 20 has several service quantity thresholds and several service scheduling depth thresholds preset. For example, there are three business scenarios: the first business scenario with small business scale and low service scheduling depth, the second business scenario with medium business scale and medium service scheduling depth, and the third business scenario with large business scale and high service scheduling depth. The scale and depth can be divided according to the service quantity threshold and the service scheduling depth threshold, and can be adjusted according to actual needs.

[0074] Among them, such as Figure 1 As shown, the link data processing module 20 is also used to: set a link timeout threshold for the temporary table during the process of writing data from the temporary table to the latest link data table and the standard link data table using the timed task scheduling component; mark link data that exceeds the link timeout threshold as incomplete links and archive them as archive logs, without writing them to the latest link data table and the standard link data table.

[0075] In this embodiment, the module can prevent incomplete link data from interfering with the latest link data table and the standard link data table, ensuring data accuracy and validity, and archives incomplete links to facilitate subsequent analysis and query operations;

[0076] Among them, such as Figure 1 As shown, the link data analysis module 30 is also used to: in the process of comparing the latest link data table and the standard link data table and matching the appropriate fault judgment mode according to the comparison result, determine whether there is a branch structure in the topology for the standard link data table. When it is determined that there is a branch structure, select the third fault judgment mode that is consistent with the topology and the error is adapted to replace the first fault judgment mode.

[0077] The third fault diagnosis mode includes: querying the last service that reported an error and whether there is a service that is symmetrical to it in the current topology. If a service that is symmetrical to it is found in the current topology, the service log data of the service that is symmetrical to the last service that reported an error and the error type of the last service that reported an error are combined to determine whether there are any upstream or downstream services that affect the last service that reported an error. If no service that is symmetrical to it is found in the current topology, the error type of the last service that reported an error is directly used to determine whether there are any upstream or downstream services that affect the last service that reported an error. Then, the machine learning model is combined to perform multi-dimensional fault root cause localization of the last service that reported an error or the last service that reported an error and its upstream and downstream services that have an impact on it.

[0078] In this embodiment, the link data analysis module 30 further includes a branch structure judgment unit, used to determine whether a branch structure exists in the topology. Specifically, it includes: combining the service log data of the service symmetrical to the last error-reporting service and the error type of the last error-reporting service to comprehensively determine whether there are upstream and downstream services that affect the last error-reporting service. Specifically, it includes: judging whether the symmetrical service meets the first condition based on the service log data of the service symmetrical to the last error-reporting service; the first condition includes: the existence of error information and the existence of a business relationship with the last error-reporting service; counting the number and proportion of services symmetrical to the last error-reporting service that meet the first condition. If the number of services symmetrical to the last error-reporting service that meet the first condition is greater than a preset number or the proportion of services symmetrical to the last error-reporting service that meet the first condition is greater than a preset proportion, then regardless of the error type of the last error-reporting service, it is directly determined that there are upstream and downstream services that affect the last error-reporting service.

[0079] Based on the service log data of the service that is symmetrical to the last service that reported an error, determine whether the symmetrical service meets the second condition. The second condition includes: the existence of error information and the comprehensive correlation value between the service and the last service that reported an error is greater than a preset value. The comprehensive correlation value is calculated by weighting multiple dimensions such as data dependency correlation, business correlation, and resource correlation. If the last service that reported an error is a timeout, it can be determined that the current service timeout is likely caused by upstream service blockage, and it is determined that there is an upstream service that affects the last service that reported an error.

[0080] Among them, such as Figure 1 As shown, the link data analysis module 30 is also used for: monitoring service metrics data through Prometheus during the fault root cause localization process, obtaining the corresponding service's time consumption data, request parameters, memory and CPU resource metrics, matching the feature weight ratio adapted to each business scenario according to the business scenario label, obtaining the feature weight ratio matched to the current business scenario label, using a machine learning model, inputting the corresponding service's time consumption data, request parameters, memory and CPU resource metrics and service log data, and using the matched feature weight ratio to output the fault root cause localization. Prometheus is an open-source system monitoring and alerting toolkit used to collect service metrics data. This module can train a machine learning model on historically collected metrics data and service log data through historical business scenarios and different feature weight conditions, obtain feature weights with fault root cause localization accuracy greater than the preset accuracy, and determine the feature weight ratio adapted to each business scenario.

[0081] In this embodiment, the end-to-end monitoring system collects service metric data through Prometheus, combines it with business scenario label matching feature weight ratios, uses machine learning models to input features and apply weight ratios to output fault root cause localization, can investigate service faults from multiple dimensions such as metric data and log data, and combine it with business scenario for precise analysis, thereby improving the accuracy of fault detection and early warning capabilities, and improving fault diagnosis efficiency.

[0082] Furthermore, this invention can also set up multiple machine learning models, such as machine learning models with different structural complexities. For the last service that reported an error and its affected upstream and downstream services, a machine learning model with higher structural complexity is used to locate the root cause of the error, while a machine learning model with lower structural complexity is used to locate the root cause of the error.

[0083] Example 2

[0084] Unlike Embodiment 1, this invention also provides a fault diagnosis method for a full-link monitoring system applied to a microservice architecture, such as... Figure 3 As shown, the specific steps include:

[0085] Step S1: Integrate the TLOG component into each service, and intercept and capture the link data of each service when making requests and responses;

[0086] This step embeds the TLOG component into various services through code integration, and uses interceptors to trigger the TLOG component to capture data during service requests and responses.

[0087] Step S2: Use Kafka to receive link data sent by each service, use Flink to generate a link data table from the link data, and obtain the latest link data table and the standard link data table.

[0088] In this step, Kafka is deployed and run in the form of a cluster to receive link data sent by various services. Flink performs streaming processing on the data received by Kafka to generate a link data table. Flink Stateful Functions can be used to generate the latest link data table in real time, and the standard link data table can be dynamically matched and obtained in the streaming computation using a preset standard link matching unit, or the link data stored in the temporary table can be written to the latest link data table and the standard link data table using a scheduled task component.

[0089] Step S3: Compare the latest link data table with the standard link data table, match the appropriate fault judgment mode according to the comparison result, and use the appropriate fault judgment mode to locate the root cause of the fault in multiple dimensions.

[0090] In this step, the comparison operation can be carried out through database query and comparison. Based on the comparison results, an appropriate fault judgment mode is selected, such as the first fault judgment mode, the second fault judgment mode, etc. When locating the root cause of the fault, the machine learning model is combined, and the indicator data of the Prometheus collection service is used for auxiliary analysis.

[0091] Step S4: Display the obtained multi-dimensional fault root cause localization to realize microservice fault localization and diagnosis;

[0092] In this embodiment, the fault diagnosis method can achieve comprehensive collection, processing and analysis of inter-service link data in microservice architecture. It uses multiple fault judgment modes combined with machine learning models to locate the root cause of faults in multiple dimensions and visualizes the results. This effectively solves the problems of delayed fault discovery, difficulty in fault location and reliance on manual intervention in traditional monitoring methods, and improves the real-time performance and automation of fault diagnosis.

[0093] The present invention also provides a computer-readable storage medium, which includes a stored computer program, including but not limited to various media capable of storing program code such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk. The computer program, when running, controls the computer device containing the computer-readable storage medium to execute a fault diagnosis method. The present invention also provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the fault diagnosis method.

[0094] In summary, this invention integrates the TLOG component into various microservices, achieving non-intrusive data collection across the entire data chain. It utilizes Kafka and Flink to build a scalable real-time data processing pipeline, and can automatically adapt processing modes based on business scenario tags. By comparing real-time and standard data chains, it matches and executes multiple fault diagnosis modes, combines machine learning models for multi-dimensional fault root cause localization, and finally presents the localization results clearly through a visual interface. This system and method effectively solve the pain points of delayed fault discovery, difficult fault localization, and reliance on manual intervention in microservice architectures, achieving end-to-end fault diagnosis and improving operational efficiency.

[0095] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A full-link monitoring system applied to a microservice architecture, characterized in that, include: The link data acquisition module (10) is used to integrate the TLOG component into each service and to intercept and capture the link data of each service when each service makes a request and a response. The link data acquisition module (10) includes a TLOG component and an interceptor. The TLOG component is in the form of a software program and is integrated into each service. It is used to capture link information when each service makes a request and a response. The interceptor is used to perform interception operations during the service request and response process to ensure that the TLOG component can accurately obtain the link data. The link data processing module (20) is used to receive link data sent by various services using Kafka, generate a link data table from the sent link data using Flink, obtain the latest link data table and the standard link data table, and also to tag the business scenario according to the business scenario scale and service scheduling depth during the generation of the link data table, and match the adaptation processing mode. The link data processing module (20) includes Kafka, Flink, a timed task scheduling component and a preset standard link matching unit. The processing modes include: the first processing mode is to directly generate a dynamic link data table as the latest link data table by Flink, which is adapted to the preset standard link data table; the second processing mode is to aggregate data by Flink and save it to a temporary table, and use a timed task scheduling component to write the data in the temporary table into the latest link data table and the standard link data table; the third processing mode is to generate an enhanced table by Flink to store hot and cold data in a hierarchical manner, with hot data using the first processing mode and cold data using the second processing mode. The link data analysis module (30) is used to compare the latest link data table and the standard link data table, match the appropriate fault judgment mode according to the comparison result, and use the appropriate fault judgment mode to locate the root cause of the fault. The link data analysis module (30) has a built-in machine learning model. The fault data display module (40) is used to interact with the link data analysis module (30) to obtain the fault root cause location results and display the fault root cause location results in a visual form.

2. The end-to-end monitoring system for microservice architecture according to claim 1, characterized in that, The fault judgment mode includes a first fault judgment mode and a second fault judgment mode. The comparison result is that the topology is consistent and the error is adapted to the first fault judgment mode. The comparison result is that the topology is inconsistent and the second fault judgment mode is adapted. The first fault judgment mode includes: querying and analyzing the error type of the last faulty service, judging whether there are upstream and downstream services that affect it based on the error type, and combining machine learning models to perform multi-dimensional fault root cause localization of the last faulty service or the last faulty service and its upstream and downstream services that have an impact. The second fault diagnosis mode includes: locating the first inconsistent service in the topology, and using a machine learning model to trigger multi-dimensional fault root cause localization of the service itself, the direct caller, and the network between the service and the caller.

3. The end-to-end monitoring system for microservice architecture according to claim 1, characterized in that, The scheduled task component uses the basic scheduling cycle as a benchmark, and the link data analysis module adjusts the basic scheduling cycle by matching the scheduling cycle correction value based on the future link failure risk level predicted by the current failure root cause. The higher the fault risk level, the larger the correction value, and the smaller the corrected scheduling cycle.

4. The end-to-end monitoring system for microservice architecture according to claim 1, characterized in that, The link data processing module (20) is also used to: tag business priorities during the process of tagging business scenarios according to business scenario scale and service scheduling depth; After tagging business scenarios with business scenario scale and service scheduling depth, and matching and adapting processing modes, the data desensitization method is matched with each business scenario according to the business scenario tag, and the link data desensitization is performed using the data desensitization method matched with the current business scenario tag. During the process of generating the enhanced table for hierarchical storage of hot and cold data by Flink, the rules for determining hot and cold data are dynamically adjusted based on the business priority corresponding to the link data and the influence range of historical link data. The dynamic adjustment of hot and cold data determination includes: quantifying whether it contains preset core business data and whether the access frequency is greater than the preset access frequency, and obtaining a basic determination score through weighted calculation; quantifying the business priority corresponding to the link data and obtaining the business priority quantification weight coefficient; quantifying the influence range of historical link data and obtaining the historical link data influence range quantification coefficient; using the business priority quantification weight coefficient and the historical link data influence range quantification coefficient as correction coefficients to correct the basic determination score and obtain the final determination score; comparing the final determination score with the preset score, if the final determination score is greater than the preset score, it is identified as hot data, otherwise it is cold data.

5. A full-link monitoring system for microservice architecture according to claim 4, characterized in that, The link data processing module (20) is also used for: Real-time monitoring of back pressure metrics of Flink jobs; comparison of current back pressure metrics with preset back pressure metric thresholds; determination of whether to trigger an adaptive processing mode switching strategy based on comparison results; initiation of target processing mode switching when triggered; cessation of original processing mode operation and switching to target processing mode once target processing mode stabilizes; conditions for triggering the adaptive processing mode switching strategy include: if the current back pressure metric is greater than the preset metric threshold and the duration is greater than the first preset time period, then the matching processing mode is switched in the priority order of switching to the second processing mode, switching to the third processing mode, and switching to the improved third processing mode. The improved third processing mode includes reducing the preset score in dynamically adjusting the hot and cold data judgment rules. The score reduction is adjusted based on the absolute value of the difference between the back pressure index and the preset back pressure index threshold. The larger the absolute value of the difference, the larger the score reduction. If the current back pressure index is detected to be greater than the preset index threshold and the duration is greater than the second preset time period, the matching processing mode will be switched directly according to the strategy of switching the improved third processing mode, where the second preset time period is greater than the first preset time period. If the current back pressure index is detected to be less than the preset back pressure index threshold, the priority order of the switching strategy conditions for triggering the adaptation should be followed, and the original processing mode should be switched back according to the reverse priority.

6. The end-to-end monitoring system for microservice architecture according to claim 1, characterized in that, The link data processing module (20) is also used to: set a link timeout threshold for the temporary table during the process of writing data from the temporary table to the latest link data table and the standard link data table using the timed task scheduling component; mark the link data that exceeds the link timeout threshold as an incomplete link and archive it as an archive log, without writing it to the latest link data table and the standard link data table.

7. A full-link monitoring system for microservice architecture according to claim 2, characterized in that, The link data analysis module (30) includes a comparison unit and a mode selection unit. The comparison unit is used to compare the latest link data table and the standard link data table to obtain the comparison result of whether the topology results are consistent. The mode selection unit is used to match the fault judgment mode that is suitable for the comparison result and use the suitable fault judgment mode to locate the root cause of the fault. The link data analysis module (30) is also used to: in the process of comparing the latest link data table and the standard link data table and matching the appropriate fault judgment mode according to the comparison result, determine whether there is a branch structure in the topology for the standard link data table. When it is determined that there is a branch structure, select the third fault judgment mode that is consistent with the topology and the error is adapted to replace the first fault judgment mode. The third fault judgment mode includes: querying the last fault-reporting service and whether there is a service that is symmetrical to it in the current topology. If a service that is symmetrical to it is found in the current topology, the service log data of the service that is symmetrical to the last fault-reporting service and the error type of the last fault-reporting service are combined to make a comprehensive judgment on whether there are upstream and downstream services that affect the last fault-reporting service. If no service with a symmetrical relationship is found in the current topology, determine whether there are upstream or downstream services that affect the last service that reported an error based on the error type of the last service that reported an error. Then, combine the machine learning model to perform multi-dimensional root cause localization of the last service that reported an error or the upstream and downstream services that affected the last service that reported an error.

8. A full-link monitoring system for microservice architecture according to claim 7, characterized in that, The step of combining the service log data of the service corresponding to the last error-reporting service and the error type of the last error-reporting service to determine whether there are upstream and downstream services that affect the last error-reporting service includes: determining whether the queried symmetrical service meets a first condition based on the service log data of the service corresponding to the last error-reporting service; the first condition includes: the existence of error information and the existence of a business relationship with the last error-reporting service; counting the number and proportion of symmetrical services that meet the first condition; if the number of symmetrical services that meet the first condition is greater than a preset number or the proportion of symmetrical services that meet the first condition is greater than a preset proportion, then regardless of the error type of the last error-reporting service, it is directly determined that there are upstream and downstream services that affect the last error-reporting service.

9. A full-link monitoring system for microservice architecture according to claim 1, characterized in that, The link data analysis module (30) is also used to: monitor service metrics data through Prometheus during the fault root cause localization process, obtain the corresponding service's time consumption data, request parameters, memory and CPU resource metrics, match the feature weight ratio with each business scenario according to the business scenario label, obtain the feature weight ratio matched with the current business scenario label, use the machine learning model, input the corresponding service's time consumption data, request parameters, memory and CPU resource metrics and service log data, and use the matched feature weight ratio to output the fault root cause localization.

10. A fault diagnosis method for a full-link monitoring system applied to a microservice architecture according to any one of claims 1-9, characterized in that, Specifically, the following steps are included: Step S1: Integrate the TLOG component into each service, and intercept and capture the link data of each service when making requests and responses; Step S2: Use Kafka to receive the link data sent by each service, use Flink to generate a link data table from the link data, and obtain the latest link data table and the standard link data table. Step S3: Compare the latest link data table with the standard link data table, match the appropriate fault judgment mode according to the comparison result, and use the appropriate fault judgment mode to locate the root cause of the fault in multiple dimensions. Step S4: Display the obtained multi-dimensional fault root cause localization to realize microservice fault localization and diagnosis.

Images