Micro-service system full-link monitoring method based on Skywalking

Abstract

The invention relates to the technical field of full-link monitoring, and discloses a Skwalking-based micro-service system full-link monitoring method, which comprises the following steps of: 1, automatically collecting full-link tracking data without invasion, loading an Agent probe through a bytecode enhancement technology when a micro-service instance is started, and loading the Agent probe through a bytecode enhancement technology; and the Agent probe automatically intercepts service calling under the condition of not modifying a service code. According to the technical scheme, a Kafka cluster is introduced between a data acquisition end and a processing platform to serve as a buffer channel, and an L1 and L2 distributed OAP processing architecture capable of being horizontally expanded is designed, so that the technical effect of guaranteeing the integrity and real-time performance of mass monitoring data in a super-large-scale micro-service environment is achieved; compared with a transmission architecture in which a monitoring probe is directly connected with a central server in the prior art, the transmission architecture solves the problem that the accuracy of a monitoring result is damaged due to the fact that data congestion, delay and even packet loss are easy to occur during data peak flood due to the lack of an effective buffering and peak clipping mechanism.

Description

Technical Field

[0001] This invention relates to the field of end-to-end monitoring technology, specifically to an end-to-end monitoring method for microservice systems based on Skywalking. Background Technology

[0002] With the rapid development of information technology, microservice architecture, with its characteristics of high cohesion, low coupling, independent deployment, and elastic scalability, has become the technical architecture for modern enterprises to build complex and large-scale application systems. This architecture breaks down a large monolithic application into a set of small, autonomous services, each built around independent business capabilities and interacting and collaborating through lightweight communication mechanisms.

[0003] While microservice architecture brings agility in development and deployment, it significantly increases the complexity of system operation and maintenance. In a distributed environment, a complete business request often needs to be completed across multiple microservice instances, and the call relationships between services constitute a complex and dynamically changing network topology. Performance fluctuations or failures in any microservice node or network link in the system can trigger a chain reaction, leading to delays or even interruptions in business processes. Therefore, establishing a comprehensive, real-time, and accurate monitoring system is crucial to ensuring the stability, reliability, and high-performance operation of microservice systems.

[0004] Application performance management systems (APMs) are the mainstream technical solution to address the aforementioned challenges. By deploying probes in each microservice instance, APM systems can collect massive amounts of telemetry data, including service call chains, performance metrics, and logs, providing operations personnel with insights into the system's operational status. However, in practice, especially in ultra-large-scale deployment scenarios with tens of thousands of services and instances, the application of existing monitoring methods and APM tools still faces a series of severe challenges:

[0005] The surge in the number of services and instances causes the system to generate a large amount of data instantly. Traditional direct-connection transmission architectures lack buffering and peak-shaving capabilities, making them prone to congestion, latency, and packet loss, which affects the accuracy of monitoring results.

[0006] The solution collects basic performance data of a single micro-service node, but does not capture the full picture of business request flow. When system performance degrades or services are abnormal, operations and maintenance personnel cannot trace the call chain, making it difficult to quickly locate problems and increasing the time spent on troubleshooting.

[0007] The monitoring scope is limited to the performance of underlying technical components, and it cannot be linked to technical indicators and business processes. The operations and maintenance team finds it difficult to assess the impact of technical problems on the business, and it is not easy to respond to faults efficiently and prioritize them.

[0008] Fixed threshold alarm rules do not match changes in business load, often resulting in false alarms or missed alarms. A single notification method is not enough to ensure that information is delivered. The lack of a closed-loop mechanism leads to untimely problem detection, chaotic processing procedures, and lack of supervision.

[0009] To address these issues, this invention proposes a full-link monitoring method for microservice systems based on Skywalking. Summary of the Invention

[0010] To address the shortcomings of existing technologies, this invention provides a full-link monitoring method for microservice systems based on Skywalking, thereby resolving the problems mentioned in the background section.

[0011] To achieve the above objectives, the present invention provides the following technical solution: a full-link monitoring method for microservice systems based on Skywalking, comprising the following steps:

[0012] Step 1: Automated and non-intrusive collection of end-to-end tracing data. When the microservice instance starts, the Agent probe is loaded through bytecode enhancement technology. The Agent probe automatically intercepts service calls, database accesses and message queue interactions without modifying the business code, generates and transmits tracing IDs and segment IDs for each request, and captures end-to-end context information.

[0013] Step 2: Reliably buffer and efficiently transmit the end-to-end context information. The end-to-end context information collected by the Agent probe is used as monitoring data and uniformly sent to the pre-deployed Kafka cluster. The Kafka cluster's queue mechanism is used to buffer and smooth out peaks and valleys in the monitoring data, and the replication mechanism is used to perform persistent backup of the monitoring data.

[0014] Step 3: Perform layered processing and persistent storage on the monitoring data transmitted by the Kafka cluster. First, the L1 layer OAP nodes pull data from the Kafka cluster and perform preliminary aggregation. Then, the L2 layer OAP nodes obtain the data after preliminary aggregation and perform in-depth analysis to calculate service performance indicators and build a service call link topology. Finally, the final indicators generated by the analysis, the aggregated tracking data, and the service call link topology are uniformly stored in the Elasticsearch cluster.

[0015] Step 4: Associate and map the service call chain topology diagram with the core business scenario, provide management functions to allow users to predefine the core business scenario, and bind the core business scenario with one or more sets of technical components selected from the service call chain topology diagram;

[0016] Step 5: Based on the correlation between the service performance indicators and the core business scenarios, execute intelligent alarms and automated operation and maintenance closed loop. Execute alarm rules according to the data stored in the Elasticsearch cluster. When an alarm is triggered, accurately push alarm content containing information about the affected core business scenarios according to the preset notification strategy, and automatically create an operation and maintenance work order.

[0017] Preferably, the automated, non-intrusive collection of end-to-end tracking data specifically includes:

[0018] In the container orchestration environment, the Sidecar pattern is adopted, and the Agent probe is deployed as an independent container in the same Pod as the business microservice container, and mounted to the business microservice through a shared volume.

[0019] Preferably, the layered processing specifically includes:

[0020] The OAP nodes in the L1 layer are dedicated to pulling raw monitoring data from the Kafka cluster at high speed and performing data format parsing and lightweight aggregation calculations.

[0021] The OAP node in the L2 layer is dedicated to acquiring the pre-processed data from the L1 layer and performing resource-intensive deep analysis tasks such as building a service call chain topology and calculating the percentile value of service response time.

[0022] Preferably, the automated operation and maintenance closed loop specifically comprises:

[0023] When an alarm is triggered, an operation and maintenance work order is automatically created in the external operation and maintenance work order system via the API interface. The data structure of the operation and maintenance work order includes an alarm rule description, fault context information, the name of the affected core business scenario, and a deep link to the monitoring system's visualization interface.

[0024] Preferably, the persistent storage specifically includes:

[0025] The OAP nodes of the L2 layer will write the service performance metrics generated after in-depth analysis, the aggregated tracing data, and the service call chain topology diagram into the Elasticsearch cluster according to the predefined data model and indexing strategy to optimize the performance of subsequent data retrieval and visualization.

[0026] Preferably, step 1 further includes:

[0027] Sub-step 1.1: Perform initialization determination of the tracing context. When the Agent probe intercepts a request to enter the current microservice instance, it first uses the tracing context extraction algorithm to parse the header information of the request to check whether a tracing context exists. The tracing context includes the global tracing ID and the parent segment ID.

[0028] Sub-step 1.2: Generate and transmit a tracing identifier. If the initialization result of the tracing context is that no tracing context exists, determine that the current request is the starting point of a new tracing link, and generate the identifier according to the unique identifier generation function. Generate a brand new global tracking ID, and at the same time generate a root segment ID. If the determination result is that it exists, directly extract the global tracking ID and the parent segment ID to generate a new child segment ID to establish a call hierarchy relationship.

[0029] in, For global tracking ID, Functions for generating unique identifiers This is the current timestamp. For machine identifiers, For process identifier, For serial number;

[0030] Sub-step 1.3: Capture and encapsulate the context information of the segment. After determining the global tracking ID and the segment ID, record the start timestamp of the current operation. After the operation is completed, record the end timestamp. The segmented time is obtained through the segmented time calculation formula:

[0031] ,

[0032] Simultaneously, the segmentation error flag is determined based on preset error judgment conditions:

[0033] ≥ ,

[0034] Finally, the global tracking ID, segment ID, parent segment ID, segment time, segment error flag, and service and endpoint names are encapsulated together into structured end-to-end context information.

[0035] in, To divide the time into segments, For the start timestamp of the operation, This is the timestamp indicating the end of the operation. The response status code for the operation. The preset error status code threshold, This is the segmentation error flag.

[0036] Preferably, step 2 further includes:

[0037] Sub-step 2.1 involves executing batch aggregation and transmission decisions for monitoring data. The Agent probe aggregates the generated end-to-end context information into monitoring data batches in local memory and determines whether to transmit the monitoring data batches based on preset batch transmission trigger conditions.

[0038] ,

[0039] in, The size of the current monitoring data batch in bytes. This is the preset maximum batch size threshold. This is the current system timestamp. The timestamp of the last successful send. The preset maximum transmission time interval, Send flag;

[0040] Sub-step 2.2: Select the target partition and send the monitoring data batch. When the sending flag is true, the Agent probe acts as the producer and uses the partition selection algorithm. The system calculates the target partition index to which the monitoring data batch should be sent, then sends the monitoring data batch to the target partition of the specified topic in the Kafka cluster, and finally forms a persistent message record to be confirmed in the target partition. For the target partition index, For hash functions, The tracking ID is extracted from the batch of monitoring data. Total number of partitions for a theme;

[0041] Sub-step 2.3: Perform synchronous confirmation of persistent backup. Upon receiving the persistent message record to be confirmed, the leader replica of the target partition in the Kafka cluster immediately replicates the record synchronously to all follower replicas, based on the number of replica confirmations.

[0042] ≥ The system determines whether the write operation was successful. Once the write operation is successful, the pending persistent message record is converted into an acknowledged persistent message record, which can then be consumed by the L1 layer OAP node. This represents the number of copies that have been successfully synchronized and have received confirmation responses. The minimum number of synchronized replicas required to ensure no data loss. This is the final confirmation status received by the producer indicating a successful write operation.

[0043] Preferably, step 3 further includes:

[0044] Sub-step 3.1 involves performing preliminary metric aggregation on the service endpoint. The L1 layer OAP nodes consume the confirmed persistent message records generated in step two from the Kafka cluster in consumer group mode, and aggregate the message records belonging to the same service and the same endpoint within a preset time window. This aggregation operation uses a sliding window counting algorithm to count the total number of requests within the window. Number of erroneous requests The final result is a preliminary aggregated dataset containing server endpoint identifiers, time window identifiers, total number of requests, and total number of errors. The error is obtained by accumulating the segmentation error flags in the message records.

[0045] Sub-step 3.2: Calculate service performance metrics and construct a topology graph. The L2 layer OAP nodes obtain the initially aggregated data and calculate the service error rate using the following formula:

[0046] ,

[0047] The final service error rate metric is calculated, and the topology update judgment criteria are also applied:

[0048] ,

[0049] This determines whether to update the service call chain topology diagram, ultimately generating the final metric and the updated service call chain topology diagram, wherein... To improve service error rate, The total number of errors in the initially aggregated data. Total number of requests This refers to the source and target service pair parsed from the service endpoint identifier. For the existing set of topological edges, Update the topology flag;

[0050] Sub-step 3.3: Perform indexed persistent storage of monitoring data. The OAP node of the L2 layer maps the final metric and the service call chain topology to a data model predefined for the metric data and topology data. Based on the indexing strategy optimized for time series data, the mapped data is written to the Elasticsearch cluster for efficient retrieval of association mapping and intelligent alerts.

[0051] Preferably, step 4 further includes:

[0052] Sub-step 4.1: Create a core business scenario object. Receive the business scenario name input by the user through the management function, generate a unique core business scenario identifier, and initialize a core business scenario data structure containing the identifier and an empty list of technical components. This data structure provides a carrier for subsequent technical component binding.

[0053] Sub-step 4.2 involves associating and binding the technical components, visually presenting the generated service call chain topology diagram on the management interface, and receiving one or more sets of technical components selected by the user from the service call chain topology diagram, based on the component validity determination criteria:

[0054] ,

[0055] The identifiers of the selected technical components are added to the list of technical components within the core business scenario data structure, ultimately forming a binding relationship between the core business scenario identifiers and the list of technical component identifiers. The technology components selected by the user, This refers to the entire set of vertices in the service call link topology graph. Add a function to the list. List of technical components;

[0056] Sub-step 4.3: Calculate and store the health score of the business scenario. Based on the established binding relationship, extract the service performance indicators of all technical components associated with the core business scenario, and calculate the health score using the business scenario health score calculation formula.

[0057] ,

[0058] To obtain a quantified business scenario health score, the business scenario health score is associated with the core business scenario identifier and stored in the Elasticsearch cluster for execution of alert rules. For business scenario health scoring, The total number of bound technical components, For the first The preset weighting coefficients of the technical components For the first Service error rate of technical components.

[0059] Preferably, step 5 further includes:

[0060] Sub-step 5.1: Execute alarm rule determination based on business scenario. The system periodically retrieves the health score of the business scenario from the Elasticsearch cluster, and adopts an alarm trigger condition determination algorithm:

[0061] ,

[0062] The health score of the aforementioned business scenario is determined to generate an alarm trigger flag, wherein, Assess the health of the current business scenario. This is the minimum health score threshold preset for this business scenario. This is the alarm trigger flag;

[0063] Sub-step 5.2: Aggregate and generate structured alarm content. When the alarm trigger flag is 1, the system retrieves the specific service performance indicators and related technical components that triggered the alarm from the Elasticsearch cluster based on the core business scenario identifier associated with the business scenario health score, and generates the alarm content using the alarm content generation function.

[0064] ,

[0065] The system aggregates and generates a structured alert that includes the name of the affected core business scenario, the current business scenario health score, abnormal service performance metric values, and root cause technical component information. This refers to the structured alarm content. For content generation functions, Identifiers for core business scenarios. For business scenario health scoring, These are abnormal indicator values. Information on root technology components;

[0066] Sub-step 5.3 involves executing strategic notifications and automated work order creation. Based on the importance level of the core business scenario in the structured alarm content, the system matches and selects the corresponding notification channel from the preset notification strategy library, accurately pushes the structured alarm content through the notification channel, calls the external operation and maintenance work order system API, and automatically creates a traceable operation and maintenance work order with the structured alarm content as input parameters, so as to form an automated operation and maintenance closed loop of discovery, alarm, dispatch, and processing.

[0067] This invention provides a method for end-to-end monitoring of microservice systems based on Skywalking. It offers the following advantages:

[0068] 1. This invention adopts a technical solution that introduces a Kafka cluster as a buffer channel between the data acquisition end and the processing platform, and designs a horizontally scalable distributed OAP processing architecture of L1 and L2. This achieves the technical effect of ensuring the integrity and real-time performance of massive monitoring data in a large-scale microservice environment. Compared with the existing technology that uses a transmission architecture that directly connects monitoring probes to a central server, this invention solves the problem that the lack of effective buffering and peak shaving mechanisms can easily lead to data congestion, delays, and even packet loss when facing data floods, thereby compromising the accuracy of monitoring results.

[0069] 2. This invention employs a Sidecar mode to deploy Agent probes and utilizes bytecode enhancement technology to automatically intercept and capture complete service call chain information containing a unique tracking ID. This achieves the technical effect of clearly reproducing the entire flow of any request in a distributed system. Compared with the existing technology of isolated collection of basic performance indicators of a single microservice node, this invention solves the problem that when the system malfunctions, maintenance personnel cannot trace the complete service call chain, resulting in low efficiency in locating the root cause of the fault.

[0070] 3. This invention adopts a technical solution that provides management functions to allow users to predefine core business scenarios and binds multiple service interfaces or database instances that constitute the scenario to the microservice topology links automatically discovered by the APM system. This achieves the technical effect of directly mapping the performance indicators of the underlying technical components to the upper-layer core business processes. Compared with the existing technical solutions where the monitoring dimension is limited to the performance of the underlying technical components, this invention solves the problem that the operation and maintenance team cannot intuitively assess the scope and severity of the impact of technical failures on key businesses, thus making it difficult to carry out efficient fault response and priority ranking.

[0071] 4. This invention adopts a technical solution that constructs a set of multi-dimensional dynamic alarm rules and can automatically convert alarm information into traceable operation and maintenance work orders containing detailed fault context according to alarm level and attribution. This achieves the technical effect of forming an automated operation and maintenance closed loop of "discovery-alarm-dispatch-processing-closure". Compared with the existing technical solutions that use fixed threshold alarm rules and have a single notification method, this invention solves the shortcomings of frequent false alarms or missed alarms caused by fixed rules, and the disconnect between alarm events and subsequent processing procedures, which ultimately leads to untimely problem discovery, chaotic response and processing procedures and a lack of effective supervision. Attached Figure Description

[0072] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0073] To enable those skilled in the art to understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort should fall within the scope of protection of the present invention.

[0074] The present invention will now be described in detail with reference to the accompanying drawings:

[0075] Example 1:

[0076] Please see the appendix Figure 1 This invention provides a method for end-to-end monitoring of a microservice system based on Skywalking, comprising the following steps:

[0077] Step 1: Automated and non-intrusive collection of end-to-end tracing data. When the microservice instance starts, the Agent probe is loaded through bytecode enhancement technology. The Agent probe automatically intercepts service calls, database accesses and message queue interactions without modifying the business code, generates and transmits tracing IDs and segment IDs for each request, and captures end-to-end context information.

[0078] Sub-step 1.1: Perform initialization judgment of tracing context. When the Agent probe intercepts a request to enter the current microservice instance, it first uses the tracing context extraction algorithm to parse the header information of the request to check whether there is a tracing context. The tracing context includes the global tracing ID and the parent segment ID.

[0079] Sub-step 1.2: Generate and pass the tracing identifier. If the initialization result of the tracing context is that no tracing context exists, determine that the current request is the starting point of a new tracing link, and generate the identifier based on the unique identifier generation function. Generate a brand new global tracing ID, and at the same time generate a root segment ID. If the determination result is that it exists, directly extract the global tracing ID and the parent segment ID, and generate a new child segment ID to establish a call hierarchy relationship.

[0080] in, For global tracking ID, Functions for generating unique identifiers This is the current timestamp. For machine identifiers, For process identifier, For serial number;

[0081] Sub-step 1.3: Capture and encapsulate the segment's context information. After determining the global tracking ID and segment ID, record the start timestamp of the current operation. After the operation is completed, record the end timestamp. The segmented time is obtained through the segmented time calculation formula:

[0082] ,

[0083] Simultaneously, the segmentation error flag is determined based on preset error judgment conditions:

[0084] ≥ ,

[0085] Ultimately, the global tracking ID, segment ID, parent segment ID, segment time, segment error flag, and service and endpoint names are encapsulated together into structured end-to-end context information.

[0086] in, To divide the time into segments, For the start timestamp of the operation, This is the timestamp indicating the end of the operation. The response status code for the operation. The preset error status code threshold, This is a segmentation error flag;

[0087] Step 2: Reliable buffering and efficient transmission of end-to-end context information. The end-to-end context information collected by the Agent probe is used as monitoring data and uniformly sent to a pre-deployed Kafka cluster. The Kafka cluster's queuing mechanism is used to buffer and smooth out peaks and valleys in the monitoring data, and the replication mechanism is used to perform persistent backup of the monitoring data.

[0088] Sub-step 2.1 involves executing batch aggregation and sending decisions for monitoring data. The Agent probe aggregates the generated end-to-end context information into monitoring data batches in local memory and determines whether to send the monitoring data batches based on preset batch sending trigger conditions.

[0089] ,

[0090] in, The size of the current monitoring data batch in bytes. This is the preset maximum batch size threshold. This is the current system timestamp. The timestamp of the last successful send. The preset maximum transmission time interval, Send flag;

[0091] Sub-step 2.2: Select the target partition and send the monitoring data batch. When the sending flag is true, the Agent probe acts as the producer and uses the partition selection algorithm: This process calculates the target partition index to which the monitoring data batch should be sent, then sends the monitoring data batch to the target partition of the specified topic in the Kafka cluster, and finally forms a persistent message record awaiting acknowledgment within the target partition. For the target partition index, For hash functions, The tracking ID is extracted from the batch of monitoring data. Total number of partitions for a theme;

[0092] Sub-step 2.3 involves performing synchronous confirmation of persistent backups. Upon receiving the persistent message record to be confirmed, the leader replica of the corresponding target partition in the Kafka cluster immediately replicates the record synchronously to all follower replicas, based on the number of replica confirmations.

[0093] ≥ This is used to determine whether the write operation was successful. Once the write operation is successful, the pending persistent message record is transformed into an acknowledged persistent message record, which can then be consumed by the L1 layer OAP nodes. This represents the number of copies that have been successfully synchronized and have received confirmation responses. The minimum number of synchronized replicas required to ensure no data loss. This is the final successful write confirmation status received by the producer.

[0094] Step 3: Perform layered processing and persistent storage on the monitoring data transmitted by the Kafka cluster. First, the L1 layer OAP nodes pull data from the Kafka cluster and perform preliminary aggregation. Then, the L2 layer OAP nodes obtain the data after preliminary aggregation and perform in-depth analysis to calculate service performance indicators and build a service call link topology. Finally, the final indicators generated by the analysis, the aggregated tracking data, and the service call link topology are uniformly stored in the Elasticsearch cluster.

[0095] Sub-step 3.1 involves performing preliminary metric aggregation on the service endpoint. The L1 layer OAP nodes consume the confirmed persistent message records generated in step two from the Kafka cluster in consumer group mode. Aggregation is performed on message records belonging to the same service and the same endpoint within a preset time window. This aggregation operation uses a sliding window counting algorithm to count the total number of requests within the window. Number of erroneous requests The final result is a preliminary aggregated dataset containing server endpoint identifiers, time window identifiers, total number of requests, and total number of errors. The information is obtained by accumulating the segmentation error flags in the message log.

[0096] Sub-step 3.2: Calculate service performance metrics and construct a topology graph. The L2 layer OAP nodes obtain the initially aggregated data and calculate the service error rate using the formula:

[0097] ,

[0098] The final service error rate metric is calculated, and the topology update judgment criteria are also applied:

[0099] ,

[0100] This determines whether to update the service call chain topology diagram, ultimately generating the final metrics and the updated service call chain topology diagram. To improve service error rate, This represents the total number of errors in the initially aggregated data. Total number of requests This refers to the source and target service pair parsed from the service endpoint identifier. For the existing set of topological edges, Update the topology flag;

[0101] Sub-step 3.3 executes the indexing and persistent storage of monitoring data. The L2 layer OAP node maps the final metrics and service call chain topology to a data model predefined for metric data and topology data. Based on the indexing strategy optimized for time series data, the mapped data is written to the Elasticsearch cluster for efficient retrieval of correlation mapping and intelligent alerts.

[0102] Step 4: Associate and map the service call chain topology diagram with the core business scenarios, provide management functions to allow users to predefine core business scenarios, and bind the core business scenarios with one or more sets of technical components selected from the service call chain topology diagram;

[0103] Sub-step 4.1: Create a core business scenario object. Receive the business scenario name input by the user through the management function and generate a unique core business scenario identifier. This identifier is used to initialize a core business scenario data structure containing the identifier and an empty list of technical components. The data structure provides a carrier for subsequent technical component binding.

[0104] Sub-step 4.2 involves associating and binding the technical components, visually presenting the generated service call chain topology diagram on the management interface, and receiving one or more sets of technical components selected by the user from the service call chain topology diagram, based on the component validity determination criteria:

[0105] ,

[0106] The identifiers of the selected technical components are added to the list of technical components within the core business scenario data structure, ultimately forming a binding relationship between the core business scenario identifier and the list of technical component identifiers. The technology components selected by the user, This refers to the entire set of vertices in the service call link topology graph. Add a function to the list. List of technical components;

[0107] Sub-step 4.3 calculates and stores the health score of the business scenario. Based on the established binding relationship, it extracts the service performance indicators of all technical components associated with the core business scenario and calculates the health score using the business scenario health score calculation formula.

[0108] ,

[0109] To obtain a quantified business scenario health score, this score is associated with a core business scenario identifier and stored in an Elasticsearch cluster for use in executing alert rules. For business scenario health scoring, The total number of bound technical components, For the first The preset weighting coefficients of the technical components For the first Service error rate of technical components;

[0110] Step 5: Based on the correlation between service performance metrics and core business scenarios, execute intelligent alerts and automated operation and maintenance closed loop. Based on the data stored in the Elasticsearch cluster, execute alert rules. When an alert is triggered, according to the preset notification strategy, accurately push alert content containing information about the affected core business scenarios and automatically create operation and maintenance work orders.

[0111] Sub-step 5.1 involves executing alarm rule determination based on business scenarios. The system periodically retrieves the business scenario health score from the Elasticsearch cluster, and uses an alarm trigger condition determination algorithm:

[0112] ,

[0113] The health score of the business scenario is determined to generate an alarm trigger flag. Assess the health of the current business scenario. This is the minimum health score threshold preset for this business scenario. This is the alarm trigger flag;

[0114] Sub-step 5.2 aggregates and generates structured alarm content. When the alarm trigger flag is 1, the system retrieves the specific service performance indicators and related technical components that triggered the alarm from the Elasticsearch cluster based on the core business scenario identifier associated with the business scenario health score, and generates the alarm content using the alarm content generation function.

[0115] ,

[0116] The system aggregates and generates a structured alert that includes the name of the affected core business scenario, the current business scenario health score, abnormal service performance metric values, and root cause technical component information. For structured alarm content, For content generation functions, Identifiers for core business scenarios. For business scenario health scoring, These are abnormal indicator values. Information on root technology components;

[0117] Sub-step 5.3 executes strategic notification and automated work order creation. Based on the importance level of the core business scenario in the structured alarm content, the system matches and selects the corresponding notification channel from the preset notification strategy library, accurately pushes the structured alarm content through the notification channel, calls the external operation and maintenance work order system API, and automatically creates a traceable operation and maintenance work order with the structured alarm content as input parameters, so as to form an automated operation and maintenance closed loop of discovery, alarm, dispatch and processing.

[0118] Through automated, non-intrusive data collection technology, it is possible to comprehensively capture the complete path and performance details of each user request flowing through a complex microservice network without intruding on or modifying the source code of any existing business system. This provides a detailed and accurate raw data foundation for the monitoring system, fundamentally solving the problem that traditional monitoring cannot effectively track distributed links due to the lack of a global perspective.

[0119] By introducing a distributed message queue as an intermediate buffer layer for data transmission, the invention successfully decouples the data acquisition end from the back-end processing platform, effectively copes with the surge in instantaneous massive data due to the rapid increase in service scale, ensures the integrity and high reliability of monitoring data during transmission, avoids data loss caused by system congestion, and guarantees the accuracy of subsequent analysis results.

[0120] By constructing a layered distributed processing architecture, the invention decomposes the processing pressure of massive amounts of raw monitoring data into different computing levels, effectively separating lightweight aggregation tasks from resource-intensive in-depth analysis tasks. This greatly improves the efficiency of data processing and the horizontal scalability of the monitoring platform, ensuring that the system can perform real-time and efficient analysis and storage of massive amounts of data.

[0121] By establishing a mapping relationship between the underlying technical topology and the upper-level core business scenarios, the invention gives intuitive business meaning to purely technical performance indicators, enabling the operations and maintenance team to quickly determine the actual impact range and severity of any technical failure on critical business processes. This solves the problem that traditional monitoring is limited to the technical level and cannot provide business value guidance for fault response.

[0122] By implementing intelligent alerts based on business impact and driving automated operation and maintenance processes, the invention transforms the monitoring system from a passive observation tool into a proactive management and control platform that can actively discover problems, accurately push alerts, and automatically trigger response processes, forming a traceable, manageable, and supervised operation and maintenance closed loop, significantly improving the timeliness of fault response and the standardization of handling processes.

[0123] Example 2:

[0124] In this embodiment, taking the typical core business scenario of "user querying electricity bill" as an example, the complete request chain of this business scenario requires the coordinated work of multiple technical components such as the business linking platform, the accounting front-end microservice, the electricity bill query microservice, the penalty query microservice, and the data microservice.

[0125] The specific implementation steps of the method of the present invention are as follows:

[0126] Step 1: Automated and non-intrusive collection of end-to-end tracking data

[0127] When a user initiates an electricity bill inquiry request through channels such as WeChat or Alipay, the request first passes through the enterprise network's security isolation device to reach the business connection platform. The business connection platform immediately invokes the billing front-end microservice. At this time, the Skywalking Agent probe deployed in the billing front-end microservice Pod automatically intercepts this inbound request according to the technical solution in step one of this invention. Without modifying any business code, the probe generates a globally unique tracking ID for the request and records the context information of the current segment, including the request arrival time and request parameters. Next, when the billing front-end microservice calls the downstream electricity bill inquiry microservice, the Agent probe again captures the information of this outbound call and injects the context information including the tracking ID into the call request, passing it to the electricity bill inquiry microservice. Upon receiving the request, the Agent probe in the electricity bill inquiry microservice identifies the tracking ID and associates the processing within the service onto the same call chain.

[0128] Step 2: Reliable Buffering and Efficient Transmission of End-Link Context Information

[0129] In step one above, the Agent probes on each microservice instance encapsulate the captured end-to-end context information, including call relationships, latency, and parameters, into structured monitoring data. According to the technical solution in step two of this invention, the monitoring data is uniformly and asynchronously sent to a highly available Kafka cluster. Leveraging its high throughput, the Kafka cluster smoothly receives and buffers the instantaneous data surges generated by all 3800+ Pods. Simultaneously, utilizing the persistence and replication mechanisms of message queues ensures that each piece of monitoring data is transmitted to the backend processing platform at least once, in an orderly and reliable manner, preventing data loss even in the event of temporary failures of backend processing nodes.

[0130] Step 3: Layered Processing and Persistent Storage of Monitoring Data Transmitted from the Kafka Cluster. The L1 layer OAP nodes of the Skywalking cluster act as consumers, rapidly pulling reliable monitoring data from the Kafka cluster for initial data cleaning, format parsing, and lightweight aggregation. The post-processed data is then passed to the L2 layer OAP nodes. The L2 layer nodes, based on the technical solution in Step 3 of this invention, perform deep analysis and computation tasks. At a certain point in this embodiment, the L2 layer nodes calculated that the average response time of the electricity bill query microservice significantly increased from the usual stable 100 milliseconds to 300 milliseconds. All final metrics derived through deep analysis, aggregated link tracing data, and dynamically constructed service topology graphs are uniformly and persistently stored in the Elasticsearch cluster.

[0131] Step Four: Mapping the Service Call Link Topology to Core Business Scenarios. In the early stages of implementing this invention's method, the operations team, through the system's visual management interface and based on the technical solution in Step Four, defined the core business scenario of "user electricity bill inquiry." Simultaneously, the key technical components necessary for this scenario, including specific interfaces of the billing front-end microservice, the electricity bill inquiry microservice, and the late payment penalty inquiry microservice, were associated and bound to this business scenario. Therefore, when the performance indicators of the electricity bill inquiry microservice deteriorate in Step Three, the system can immediately map this technical issue to specific business impacts.

[0132] Step 5: Based on the correlation between service performance indicators and core business scenarios, execute a pre-set targeted alarm rule within the intelligent alarm and automated operation and maintenance closed-loop system: "When the average response time of the electricity bill query microservice exceeds 200 milliseconds and this state lasts for more than 5 seconds, a critical alarm is triggered." When the response time calculated in Step 3 meets this rule, the process in Step 5 of this invention is automatically activated. The system first sends an alarm SMS to the operation and maintenance team leader according to the preset notification policy, with the content: "[Critical Alarm] The core business 'user electricity bill query' is affected. The response time of the component 'electricity bill query microservice' exceeds the threshold, with a current average of 300ms. Please investigate immediately!" At the same time, the system automatically creates an operation and maintenance work order in the internal IT service management system through the API interface. The work order contains all relevant fault context information and directly assigns this work order to the operation and maintenance personnel responsible for the microservice.

[0133] Upon receiving the work order, the responsible operations and maintenance personnel clicked the embedded link within the work order to directly access the visual monitoring interface of this invention. The interface provided a clear view of the end-to-end call chain details for the "user electricity bill inquiry" service scenario, quickly pinpointing an anomaly in a specific instance of the electricity bill inquiry microservice. Further drill-down analysis revealed that the root cause was a JVM memory overflow in that instance, preventing it from providing normal service. The operations and maintenance personnel immediately took measures, optimizing some database query statements and scaling up resources for the electricity bill inquiry microservice. After the problem was resolved, the service's average response time quickly returned to normal, the alarm was automatically cleared, and the operations and maintenance personnel closed the work order, forming a complete automated processing loop.

[0134] Example 3: Deep Customization and Refined Management of Monitoring Platform

[0135] This embodiment, based on a Skywalking-based end-to-end monitoring method for microservice systems, provides a detailed and optimized implementation scheme for user interaction, alarm logic, and data collection strategies of the monitoring platform, specifically including:

[0136] 1. Enhanced Implementation of Visual Monitoring Display

[0137] This embodiment provides a highly interactive, multi-dimensional, visual monitoring dashboard for the "service call chain topology diagram" and "core business scenarios" generated in claims 3 and 4. Users can perform the following operations on the monitoring platform interface:

[0138] The system displays a global technical topology map by default, showing all microservices and their dependencies. Users can select predefined core business scenarios from the drop-down menu in the dashboard. After selection, the topology map will automatically filter and focus, highlighting the technical components bound to that business scenario, and using color coding to intuitively indicate the health status of each component, achieving a smooth transition from a complex technical view to a clear business view.

[0139] On the dashboard, when a user clicks on any service node or edge, a sidebar pops up displaying a smooth curve chart showing the key performance indicators of that service or call relationship in real time. Users can also choose to "compare with the same period yesterday" or "compare with the same period last week," and the system will retrieve historical data stored in Elasticsearch and overlay it on the same chart, making it easy for users to quickly identify whether performance anomalies are sporadic fluctuations or recurring issues.

[0140] When a service node turns red on the topology map, the user can simply double-click the node. The system will automatically redirect to the distributed tracing query interface, pre-filtering a list of all request chains marked as faulty that flowed through this service within that time window. Operations personnel no longer need to manually set query conditions to directly review the specific call stack that caused the problem, greatly improving troubleshooting efficiency.

[0141] 2. Optimization of Alarm Strategy Implementation

[0142] This embodiment enhances the "intelligent alarm" mechanism described in claims 5 and 10, providing flexible and intelligent alarm strategy configuration to cope with complex production environments:

[0143] In addition to fixed threshold alerts based on business scenario health scores, this embodiment introduces a dynamic baseline algorithm. The system periodically analyzes historical performance metrics stored in Elasticsearch, automatically learning and generating a performance baseline that fluctuates over time for each service's core metrics. When real-time metrics deviate from this baseline by more than a preset percentage, an alert is triggered. This feature effectively adapts to the tidal effect of business activity, avoiding false alarms caused by normal traffic increases during peak business periods.

[0144] The alarm rule engine in this embodiment supports configuring composite alarm rules based on AND / OR logic. For example, a rule can be set as: "When the error rate of the order service is higher than 5% for 5 consecutive minutes and the average response time exceeds 800 milliseconds, a critical alarm will be triggered." In addition, alarm escalation strategies are supported, such as: "If a warning-level alarm is not confirmed and identified within 15 minutes, it will be automatically escalated to a critical alarm, and a higher-level responsible person will be notified."

[0145] When a core underlying service fails, it can trigger a chain reaction of alarms from numerous upstream services. This embodiment analyzes the service call topology to automatically identify the common root cause of the alarms. The system will converge multiple alarm events triggered by the same fault point during notification, sending a single root cause alarm that lists all affected downstream services in the details. This effectively avoids the disruption to operations and maintenance personnel caused by an "alarm storm."

[0146] 3. Implementation of refined blacklist and whitelist management

[0147] To optimize system performance and reduce monitoring costs, this embodiment adds management capabilities to the "automated non-intrusive data acquisition" process described in claims 1 and 2:

[0148] In the monitoring platform's management interface, users can configure a blacklist for data collection. Blacklist rules can be based on service names, host IP address ranges, HTTP request URL prefixes, or message queue topics. When the agent probe intercepts a request matching the blacklist rules, it will be allowed directly without generating any tracking data. This effectively eliminates interference from non-business traffic such as health checks and heartbeat detection, reducing the load on Kafka and Elasticsearch.

[0149] In contrast to blacklists, users can configure whitelists to track specific critical requests. For example, a rule can be set to "only track requests whose HTTP request headers contain user-level=VIP". Furthermore, to balance comprehensive monitoring with system overhead under heavy traffic, this embodiment supports setting differentiated sampling rates for different services or endpoints. For example, the core "payment service" can be set to 100% full sampling, while the non-critical "log service" can be set to 10% sampling.

[0150] Example 4: Monitoring and Visualization Implementation for Core Business Scenarios

[0151] This embodiment, based on the methods of claims 1-10, focuses on how to transform technical monitoring data into information that is business-valuable and easy to understand, and is a deeper implementation of the solutions of claims 4 and 9.

[0152] 1. Business semantic escaping of link data

[0153] To address the issue of technical link diagrams being difficult for non-technical personnel to understand, this embodiment establishes a mapping and escaping mechanism for business semantics:

[0154] Create semantic mapping rules: In the management function provided in claim 4, in addition to binding technical components with core business scenarios, users are allowed to add business aliases for specific technical endpoints or database operations.

[0155] When visualizing the details of the call chain, the system queries this mapping rule. For technical nodes with aliases, the business aliases will be used first or displayed simultaneously during the display. Therefore, an original technical call chain will be presented as a readable business process on the interface:

[0156] Gateway → Create User Order → Deduct Product Inventory → Record Operation Log;

[0157] This kind of escaping greatly lowers the barrier for business analysts, product managers, and even management to understand the system's operating status.

[0158] 2. Visualized Link Aggregation for Core Business Scenarios

[0159] Based on the semantic escaping described above, this embodiment provides an aggregated visualization view specifically designed for core business scenarios, which aggregates fragmented single request traces into a macroscopic business process view:

[0160] When a user selects to view a core business scenario, the system will query Elasticsearch to retrieve all link data belonging to that scenario within that time period. Then, it will perform aggregation analysis based on the business aliases in the links to generate a "business process topology diagram" with business steps as nodes and business flow relationships as edges.

[0161] In the business process topology diagram:

[0162] The nodes in the diagram represent specific business steps (such as "verify refund application", "execute refund operation", "send notification SMS"). The nodes will aggregate and display the performance metrics for that business step within a selected time range.

[0163] The edges connecting nodes represent the flow relationships between business steps, and the number of flows and average time consumption can also be displayed on the edges.

[0164] Through the view, users can clearly see the overall health of the business processes. Which business steps take the longest or have the lowest success rate can be quickly identified. For example, if the "processing a refund" step takes significantly longer than other steps, the operations team can immediately drill down to analyze the underlying technical reasons, thus pinpointing the technical root cause from a business issue.

[0165] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for end-to-end monitoring of a microservice system based on Skywalking, characterized in that, Includes the following steps: Step 1: Automated and non-intrusive collection of end-to-end tracing data. When the microservice instance starts, the Agent probe is loaded through bytecode enhancement technology. The Agent probe automatically intercepts service calls, database accesses and message queue interactions without modifying the business code, generates and transmits tracing IDs and segment IDs for each request, and captures end-to-end context information. Step 2: Reliably buffer and efficiently transmit the end-to-end context information. The end-to-end context information collected by the Agent probe is used as monitoring data and uniformly sent to the pre-deployed Kafka cluster. The Kafka cluster's queue mechanism is used to buffer and smooth out peaks and valleys in the monitoring data, and the replication mechanism is used to perform persistent backup of the monitoring data. Step 3: Perform layered processing and persistent storage on the monitoring data transmitted by the Kafka cluster. First, the L1 layer OAP nodes pull data from the Kafka cluster and perform preliminary aggregation. Then, the L2 layer OAP nodes obtain the data after preliminary aggregation and perform in-depth analysis to calculate service performance indicators and build a service call link topology. Finally, the final indicators generated by the analysis, the aggregated tracking data, and the service call link topology are uniformly stored in the Elasticsearch cluster. Step 4: Associate and map the service call chain topology diagram with the core business scenario, provide management functions to allow users to predefine the core business scenario, and bind the core business scenario with one or more sets of technical components selected from the service call chain topology diagram; Step 5: Based on the correlation between the service performance indicators and the core business scenarios, execute intelligent alarms and automated operation and maintenance closed loop. Execute alarm rules according to the data stored in the Elasticsearch cluster. When an alarm is triggered, accurately push alarm content containing information about the affected core business scenarios according to the preset notification strategy, and automatically create an operation and maintenance work order.

2. The method for end-to-end monitoring of a microservice system based on Skywalking according to claim 1, characterized in that, The automated, non-intrusive collection of end-to-end tracking data specifically includes: In the container orchestration environment, the Sidecar pattern is adopted, and the Agent probe is deployed as an independent container in the same Pod as the business microservice container, and mounted to the business microservice through a shared volume.

3. The method for end-to-end monitoring of a microservice system based on Skywalking according to claim 1, characterized in that, The layered processing specifically includes: The OAP nodes in the L1 layer are dedicated to pulling raw monitoring data from the Kafka cluster at high speed and performing data format parsing and lightweight aggregation calculations. The OAP node in the L2 layer is dedicated to acquiring the pre-processed data from the L1 layer and performing resource-intensive deep analysis tasks such as building a service call chain topology and calculating the percentile value of service response time.

4. The method for end-to-end monitoring of a microservice system based on Skywalking according to claim 1, characterized in that, The automated operation and maintenance closed loop is specifically as follows: When an alarm is triggered, an operation and maintenance work order is automatically created in the external operation and maintenance work order system via the API interface. The data structure of the operation and maintenance work order includes an alarm rule description, fault context information, the name of the affected core business scenario, and a deep link to the monitoring system's visualization interface.

5. The method for end-to-end monitoring of a microservice system based on Skywalking according to claim 1, characterized in that, The persistent storage specifically refers to: The OAP nodes of the L2 layer will write the service performance metrics generated after in-depth analysis, the aggregated tracing data, and the service call chain topology diagram into the Elasticsearch cluster according to the predefined data model and indexing strategy to optimize the performance of subsequent data retrieval and visualization.

6. The method for end-to-end monitoring of a microservice system based on Skywalking according to claim 1, characterized in that, Step 1 further includes: Sub-step 1.1: Perform initialization determination of the tracing context. When the Agent probe intercepts a request to enter the current microservice instance, it first uses the tracing context extraction algorithm to parse the header information of the request to check whether a tracing context exists. The tracing context includes the global tracing ID and the parent segment ID. Sub-step 1.2: Generate and transmit a tracing identifier. If the initialization result of the tracing context is that no tracing context exists, determine that the current request is the starting point of a new tracing link, and generate the identifier according to the unique identifier generation function. Generate a brand new global tracking ID, and at the same time generate a root segment ID. If the determination result is that it exists, directly extract the global tracking ID and the parent segment ID to generate a new child segment ID to establish a call hierarchy relationship. in, For global tracking ID, Functions for generating unique identifiers This is the current timestamp. For machine identifiers, For process identifier, For serial number; Sub-step 1.3: Capture and encapsulate the context information of the segment. After determining the global tracking ID and the segment ID, record the start timestamp of the current operation. After the operation is completed, record the end timestamp. The segmented time is obtained through the segmented time calculation formula: ， Simultaneously, the segmentation error flag is determined based on preset error judgment conditions: ≥ ， Finally, the global tracking ID, segment ID, parent segment ID, segment time, segment error flag, and service and endpoint names are encapsulated together into structured end-to-end context information. in, To divide the time into segments, For the start timestamp of the operation, This is the timestamp indicating the end of the operation. The response status code for the operation. The preset error status code threshold, This is the segmentation error flag.

7. The method for end-to-end monitoring of a microservice system based on Skywalking according to claim 1, characterized in that, Step 2 further includes: Sub-step 2.1 involves executing batch aggregation and transmission decisions for monitoring data. The Agent probe aggregates the generated end-to-end context information into monitoring data batches in local memory and determines whether to transmit the monitoring data batches based on preset batch transmission trigger conditions. ， in, The size of the current monitoring data batch in bytes. This is the preset maximum batch size threshold. This is the current system timestamp. The timestamp of the last successful send. The preset maximum transmission time interval, Send flag; Sub-step 2.2: Select the target partition and send the monitoring data batch. When the sending flag is true, the Agent probe acts as the producer and uses the partition selection algorithm. The system calculates the target partition index to which the monitoring data batch should be sent, then sends the monitoring data batch to the target partition of the specified topic in the Kafka cluster, and finally forms a persistent message record to be confirmed in the target partition. For the target partition index, For hash functions, The tracking ID is extracted from the batch of monitoring data. Total number of partitions for a theme; Sub-step 2.3: Perform synchronous confirmation of persistent backup. Upon receiving the persistent message record to be confirmed, the leader replica of the target partition in the Kafka cluster immediately replicates the record synchronously to all follower replicas, based on the number of replica confirmations. ≥ The system determines whether the write operation was successful. Once the write operation is successful, the pending persistent message record is converted into an acknowledged persistent message record, which can then be consumed by the L1 layer OAP node. This represents the number of copies that have been successfully synchronized and have received confirmation responses. The minimum number of synchronized replicas required to ensure no data loss. This is the final confirmation status received by the producer indicating a successful write operation.

8. The method for end-to-end monitoring of a microservice system based on Skywalking according to claim 1, characterized in that, Step 3 further includes: Sub-step 3.1 involves performing preliminary metric aggregation on the service endpoint. The L1 layer OAP nodes consume the confirmed persistent message records generated in step two from the Kafka cluster in consumer group mode, and aggregate the message records belonging to the same service and the same endpoint within a preset time window. This aggregation operation uses a sliding window counting algorithm to count the total number of requests within the window. Number of erroneous requests The final result is a preliminary aggregated dataset containing server endpoint identifiers, time window identifiers, total number of requests, and total number of errors. The error is obtained by accumulating the segmentation error flags in the message records. Sub-step 3.2: Calculate service performance metrics and construct a topology graph. The L2 layer OAP nodes obtain the initially aggregated data and calculate the service error rate using the following formula: ， The final service error rate metric is calculated, and the topology update judgment criteria are also applied: ， This determines whether to update the service call chain topology diagram, ultimately generating the final metric and the updated service call chain topology diagram, wherein... To improve service error rate, The total number of errors in the initially aggregated data. Total number of requests This refers to the source and target service pair parsed from the service endpoint identifier. For the existing set of topological edges, Update the topology flag; Sub-step 3.3: Perform indexed persistent storage of monitoring data. The OAP node of the L2 layer maps the final metric and the service call chain topology to a data model predefined for the metric data and topology data. Based on the indexing strategy optimized for time series data, the mapped data is written to the Elasticsearch cluster for efficient retrieval of association mapping and intelligent alerts.

9. A method for end-to-end monitoring of a microservice system based on Skywalking according to claim 1, characterized in that, Step 4 further includes: Sub-step 4.1: Create a core business scenario object. Receive the business scenario name input by the user through the management function, generate a unique core business scenario identifier, and initialize a core business scenario data structure containing the identifier and an empty list of technical components. This data structure provides a carrier for subsequent technical component binding. Sub-step 4.2 involves associating and binding the technical components, visually presenting the generated service call chain topology diagram on the management interface, and receiving one or more sets of technical components selected by the user from the service call chain topology diagram, based on the component validity determination criteria: ， The identifiers of the selected technical components are added to the list of technical components within the core business scenario data structure, ultimately forming a binding relationship between the core business scenario identifiers and the list of technical component identifiers. The technology components selected by the user, This refers to the entire set of vertices in the service call link topology graph. Add a function to the list. List of technical components; Sub-step 4.3: Calculate and store the health score of the business scenario. Based on the established binding relationship, extract the service performance indicators of all technical components associated with the core business scenario, and calculate the health score using the business scenario health score calculation formula. ， To obtain a quantified business scenario health score, the business scenario health score is associated with the core business scenario identifier and stored in the Elasticsearch cluster for execution of alert rules. For business scenario health scoring, The total number of bound technical components, For the first The preset weighting coefficients of the technical components For the first Service error rate of technical components.

10. A method for end-to-end monitoring of a microservice system based on Skywalking according to claim 1, characterized in that, Step 5 further includes: Sub-step 5.1: Execute alarm rule determination based on business scenario. The system periodically retrieves the health score of the business scenario from the Elasticsearch cluster, and adopts an alarm trigger condition determination algorithm: ， The health score of the aforementioned business scenario is determined to generate an alarm trigger flag, wherein, Assess the health of the current business scenario. This is the minimum health score threshold preset for this business scenario. This is the alarm trigger flag; Sub-step 5.2: Aggregate and generate structured alarm content. When the alarm trigger flag is 1, the system retrieves the specific service performance indicators and related technical components that triggered the alarm from the Elasticsearch cluster based on the core business scenario identifier associated with the business scenario health score, and generates the alarm content using the alarm content generation function. ， The system aggregates and generates a structured alert that includes the name of the affected core business scenario, the current business scenario health score, abnormal service performance metric values, and root cause technical component information. This refers to the structured alarm content. For content generation functions, Identifiers for core business scenarios. For business scenario health scoring, These are abnormal indicator values. Information on root technology components; Sub-step 5.3 involves executing strategic notifications and automated work order creation. Based on the importance level of the core business scenario in the structured alarm content, the system matches and selects the corresponding notification channel from the preset notification strategy library, accurately pushes the structured alarm content through the notification channel, calls the external operation and maintenance work order system API, and automatically creates a traceable operation and maintenance work order with the structured alarm content as input parameters, so as to form an automated operation and maintenance closed loop of discovery, alarm, dispatch, and processing.

Images