Track-based monitoring alarm clustering method
By defining a standardized alarm model in the microservice architecture and using trace_id to transmit information, combined with European-style distance and hierarchical clustering technology, the problem of duplication of alarm information and root causes in the microservice architecture is solved, and efficient alarm management and rapid fault location are achieved.
Patent Information
- Application Number
- CN202510679536.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-26
- Publication Date
- 2025-08-26
AI Technical Summary
In the microservice architecture, due to the complex dependencies between services, the real root of the failure is often hidden in the multi-layer call chain, and it is difficult for the existing technology to effectively track it, resulting in a large number of repeated, irrelevant or similar alarm information flooding the monitoring system, reducing the work efficiency of operation and maintenance personnel and the difficulty of fault location.
By defining the standardized alarm model of service interface signature, unique identifier, call hierarchy, exception information and call chain records, the interceptor is used to monitor microservice requests, record exception information and generate structured aggregated alarms, use trace_id for information transmission and completion, and combine with European-style distance and hierarchical clustering technology to generate a tree map of the alarm to identify the source of the exception.
It improves the integrity and management efficiency of alarm information, reduces duplicate recording, quickly locates the root cause of failures, reduces operation and maintenance costs, and improves fault response speed and accuracy.
Smart Images
Figure CN120540939A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of real-time monitoring of microservices, and in particular to a monitoring alarm clustering method based on trace. Background Art
[0002] The microservices architecture breaks down a single application into multiple independent services, each responsible for a specific function. While this offers advantages in flexibility and scalability, it also introduces increased complexity. The inter-service dependencies mean that a failure in one service can quickly impact the entire architecture, generating a large number of related alerts. In a microservices scenario, service calls are closely interconnected, and an anomaly in a downstream service can trigger alerts in multiple upstream services. This phenomenon results in a large number of duplicate alerts, reducing the signal-to-noise ratio and potentially misleading fault location. As the number of alerts increases, quickly locating and resolving faults becomes significantly more difficult.
[0003] Monitoring systems are flooded with repetitive, irrelevant, or similar alerts, leaving operations and maintenance personnel feeling fatigued and confused. Due to the tight coupling between microservices, the true root cause of a fault may be hidden somewhere in the call chain. This situation not only increases the workload of operations and maintenance personnel but also increases the time and difficulty of fault location. As the volume of alert messages continues to rise, operations and maintenance teams are often forced to sift through the vast amount of data to identify key information, leading to rapid and unnecessary reactions under anxiety and pressure.
[0004] Existing technologies have the following shortcomings: Due to the complex dependencies between microservices, existing monitoring and alarm management technologies often hide the true root cause of faults in multi-layered call chains, making them difficult to track effectively. Monitoring data for different services may be distributed across different systems, lacking a unified view, making it difficult for operations personnel to conduct comprehensive analysis. Generated alarms may also lack unified context, making it difficult to form a complete fault analysis. When handling alarms, operations personnel rely on their own experience to determine the priority and handling method of alarms, which not only reduces processing efficiency but also causes important alarms to be overlooked. Continuous and high-frequency alarms can easily fatigue operations personnel, reducing their work efficiency and judgment, leading to multiple operations personnel repeatedly handling alarms with different symptoms but the same root cause.
[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the Invention
[0006] The purpose of the present invention is to provide a monitoring alarm clustering method based on trace to solve the problems in the above background technology.
[0007] In order to achieve the above object, the present invention provides the following technical solution: a monitoring alarm clustering method based on trace, which specifically includes the following steps:
[0008] Step 1: Define a standardized alert model for the service interface's signature, unique identifier, call hierarchy, exception information, and call chain record fields. Use interceptors to monitor all service interface requests in the microservice architecture, centrally capture business logic exceptions in the service process, and record them in log files.
[0009] Step 2: Define the message and use the context information to complete the missing trace_id and interface name. Use Redis to check for repeated alarms within the time window.
[0010] Step 3: Aggregate similar alarm information within the time window and group it by trace_id, extract and sort the trace_level, identify the anomaly source and result, and merge the stack_list to finally generate a structured aggregated alarm.
[0011] Preferably, the information structure of the alarm model is defined, and its specific information fields are defined as:
[0012] S1: interface_sign is the signature information of the specific method of the service interface;
[0013] S2: trace_id is a unique identifier generated during each service call;
[0014] S3: trace_level is the calling level of the specific method of the current service interface. Set the trace_level of the root method to 0. Each time a method is called to a lower-level service, the corresponding value of trace_level increases.
[0015] S4: msg is a character string indicating the cause of the exception;
[0016] S5: stack_list is a detailed record of the call chain and stores the information of each call process in a table format.
[0017] Preferably, an interceptor is used to monitor requests of all service interfaces in the microservice architecture, and the timestamp, interface name, request parameters and user identity of the request are recorded and stored in the log document. The trace_id is passed from service A to service B through the service call link, and information is recorded during the processing of service B to keep the call chain complete. The business logic exceptions of the service process are captured centrally and recorded in the log document, and the interface_sign and trace_id are combined to correspond to msg.
[0018] Preferably, define msg to include error type, error description and timestamp and uniformly adopt JSON format. Missing trace_id and interface name are completed by context information. Define a five-minute time window C, check whether there are duplicate records of the same alarm within the time window C. When the same trace_id already exists within the time window C, directly skip the record of the alarm corresponding to the trace_id, use Redis to record the checked trace_id and its timestamp, and when a new alarm arrives, check whether the trace_id exists in Redis. When the trace_id exists and is within the time window C, no further record is made.
[0019] Preferably, a five-minute time window B is defined, and alarms within the time window B are considered similar alarms. The trace_id, trace_level, timestamp, msg, and stack_list of the alarm information are extracted. Each trace_id represents a complete call chain of a request. All alarm records are grouped according to trace_id, and multiple alarms of the same request are processed centrally. Alarm records with the same trace_id are sorted according to trace_level. Each time a downward call is made, the level value increases by 1. The time interval between adjacent alarms is extracted, and the number of severity levels of alarm records with the same trace_id is recorded. The similarity between alarms is calculated using Euclidean distance, and a similarity matrix is constructed. The specific formula is:
[0020]
[0021] Where d(j,k) represents the Euclidean distance between alarms, j i represents the characteristics of the jth alarm at the ith severity level, k i represents the feature of the kth alarm at the i-th severity level, n represents the number of severity levels of alarm records with the same trace_id, f(t) represents the time interval between adjacent alarms, δ represents the alarm information with the same trace_id, extracts the interface_sign of the lowest and highest levels, the affected service at the lowest level = min(trace_level) and represents the source of the abnormal cause, the abnormal cause = the msg corresponding to max(trace_level) and represents the final result of the abnormal cause, for the alarm records with the same trace_id, extracts the corresponding stack_list and merges them into a complete list, and adds tags indicating the importance and hierarchical relationship in the call chain, uses the linkage function in the SciPy library to perform hierarchical clustering, and uses the dendrogram function to draw the final dendrogram of the aggregated alarms.
[0022] Preferably, specific alarm rules are formulated for each cluster based on the final aggregated alarm, long-term alarm trends are displayed through a visual dashboard and abnormal levels are observed in a timely manner, and the silhouette coefficient is used to evaluate the clustering effect and reflect the distinction between each level.
[0023] In the above technical solution, the technical effects and advantages provided by the present invention are:
[0024] By defining the information structure, a comprehensive data description is provided, so that each level of alarm has a clear context. By using interceptors in microservices, key information can be collected before the request reaches each service interface, effectively improving the timeliness and accuracy of problem diagnosis. With trace_id as the core, the call information is transmitted between various services to ensure the integrity and consistency of the alarm records. The warehouse centrally captures business logic anomalies in the service process, reducing the risk of missed reports or delayed responses, ensuring that operation and maintenance personnel can quickly intervene and handle them. The use of context information to complete the missing trace_id and interface name improves the integrity of the alarm information and provides sufficient information to help diagnose even in complex call scenarios. Analysis: By defining time windows C and B, efficient management and deduplication of alarms are achieved, effectively preventing repeated recording of the same alarm and improving the efficiency and accuracy of alarm management. The similarity between alarms is calculated through Euclidean distance, and hierarchical clustering technology is used to efficiently cluster alarm records to form a hierarchical tree diagram of alarms and extract important abnormal information. Through efficient alarm recording and classification, the time for manual analysis and processing is reduced, thereby reducing operation and maintenance costs. Transparent alarm information and tracking links can quickly locate faults, improve the overall fault response speed, and avoid multiple operation and maintenance personnel from repeatedly processing alarms with different appearances but the same root cause. Alarm correlation analysis and root cause tracing help operation and maintenance personnel quickly locate the root cause of the problem. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction to the drawings required for use in the embodiments will be given below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.
[0026] Figure 1 The present invention is a flowchart of a monitoring alarm clustering method based on trace. DETAILED DESCRIPTION
[0027] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these example embodiments are provided so that the description of this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art.
[0028] The present invention provides Figure 1 The monitoring alarm clustering method based on trace shown in the figure specifically includes the following steps:
[0029] Step 1: Define a standardized alert model for the service interface's signature, unique identifier, call hierarchy, exception information, and call chain record fields. Use interceptors to monitor all service interface requests in the microservice architecture, centrally capture business logic exceptions in the service process, and record them in log files.
[0030] Define the information structure of the alarm model, and its specific information fields are defined as:
[0031] S1: interface_sign is the signature information of the specific method of the service interface;
[0032] S2: trace_id is a unique identifier generated during each service call;
[0033] S3: trace_level is the calling level of the specific method of the current service interface. Set the trace_level of the root method to 0. Each time a method is called to a lower-level service, the corresponding value of trace_level increases.
[0034] S4: msg is a character string indicating the cause of the exception;
[0035] S5: stack_list is a detailed record of the call chain and stores the information of each call process in a table format.
[0036] Use interceptors to monitor requests to all service interfaces in the microservice architecture, and record the request timestamp, interface name, request parameters, and user identity in the log file. Pass the trace_id from service A to service B through the service call link, and record information during the processing of service B to keep the call chain complete. Centrally capture business logic exceptions in the service process and record them in the log file, combining interface_sign and trace_id to correspond to msg.
[0037] By recording detailed interface signatures, unique identifiers, call levels, exception information, and call chain records, it helps development and operation teams quickly understand and trace the source of problems. Interceptors can be used to monitor service calls in real time, capture and record business logic exceptions in a timely manner, and reduce losses that may be caused by delayed processing. By standardizing the fields of the alarm model, the format of log records is unified.
[0038] Step 2: Define the message and use the context information to complete the missing trace_id and interface name. Use Redis to check for repeated alarms within the time window.
[0039] Define msg to include the error type, error description, and timestamp in a unified JSON format. Complete missing trace_ids and interface names with context information to ensure that each call record contains complete information. Define a five-minute time window C and check whether there are duplicate records of the same alarm within time window C. If the same trace_id already exists within time window C, skip recording the alarm corresponding to the trace_id to avoid repeated alarms. Use Redis to record the trace_id and its timestamp. When a new alarm arrives, check whether the trace_id exists in Redis. If the trace_id exists and is within time window C, no record is made.
[0040] By using contextual information to complete missing trace_ids and interface names, we ensure that each alarm record contains key information, thereby improving the efficiency of debugging and troubleshooting. By leveraging Redis's efficient storage and fast access features, we can check whether the same alarm exists within the set time window, avoiding interference caused by multiple alarms triggered by the same problem. By quickly checking the uniqueness of alarms and reducing unnecessary alarm records, operation and maintenance personnel can focus their attention on the problems that really need to be handled, thereby improving the overall fault response speed. The unified use of JSON format for msg enhances data consistency and readability.
[0041] Step 3: Aggregate similar alarm information within the time window and group it by trace_id. Extract and sort the trace_level, identify the anomaly source and result, and merge the stack_list to generate a structured aggregated alarm.
[0042] Define a five-minute time window B. Check if the alarms within time window B are considered similar alarms. Extract the trace_id, trace_level, timestamp, msg, and stack_list of the alarm information. Each trace_id represents the complete call chain of a request. Group all alarm records by trace_id and centrally process multiple alarms for the same request. Alarm records with the same trace_id are sorted by trace_level. The level value increases by 1 for each downward call. Extract the time interval between adjacent alarms and record the severity of the alarm records with the same trace_id. Use Euclidean distance to calculate the similarity between alarms and construct a similarity matrix. The specific formula is:
[0043]
[0044] Where d(j,k) represents the Euclidean distance between alarms, j i represents the characteristics of the jth alarm at the ith severity level, k i represents the feature of the kth alarm at the i-th severity level, n represents the number of severity levels of alarm records with the same trace_id, f(t) represents the time interval between adjacent alarms, δ represents the alarm information with the same trace_id, extracts the interface_sign of the lowest and highest levels, the affected service at the lowest level = min(trace_level) and represents the source of the abnormal cause, the abnormal cause = the msg corresponding to max(trace_level) and represents the final result of the abnormal cause, for the alarm records with the same trace_id, extracts the corresponding stack_list and merges them into a complete list, and adds tags indicating the importance and hierarchical relationship in the call chain, uses the linkage function in the SciPy library to perform hierarchical clustering, and uses the dendrogram function to draw the final dendrogram of the aggregated alarms.
[0045] Based on the final aggregated alarm, specific alarm rules are formulated for each cluster. The long-term alarm trend is displayed through a visual dashboard and the abnormal level is observed in a timely manner. The silhouette coefficient is used to evaluate the clustering effect and reflect the differentiation between each level.
[0046] By aggregating alarm information with the same trace_id, the root cause of the anomaly can be traced more accurately, the specific service where the problem occurred and its impact can be identified, and the stack_list of multiple alarms can be merged to provide a complete call chain, allowing development and operation and maintenance personnel to quickly understand the problem occurrence process. By calculating the Euclidean distance between alarms, the similarity between alarms can be quantified, helping the team identify and handle similar problems. Hierarchical clustering methods can be used to effectively classify alarm information. Alarm trends and anomaly levels can be intuitively observed through visual dashboards, allowing timely responses. The silhouette coefficient is used to evaluate clustering effects to ensure the effectiveness and reliability of clustering results, thereby improving the accuracy of alarm processing.
[0047] Example 1: This implementation is based on a specific scenario of a microservice system. Currently, a user initiates a request to service A, service A internally calls service B, and service B calls service C. For example, service A is for user login, service B is for user identity verification, and service C is for user information query. Assuming that service B currently throws an exception due to a database failure, setting service A as the root call, its trace_level is 0. Service B is called by service A, then its trace_level is 1, and the trace_id of service A is used to keep the call chain complete. Even if C is successfully called, its trace_level is 2, but service B has already thrown an exception, the msg field in the alarm information of service B includes: error_type, error_description, and error_timestamp. Assuming that tr is missing in the current service B, ace_id and interface name. A time window is used to check whether the trace_id and its timestamp exist in Redis. If the same trace_id exists in Redis, the alarm record is skipped to avoid duplicate alarms. All alarm records for service B are grouped by trace_id, and relevant information is extracted. Its trace_id continues to use the trace_id of service A, and the trace_levels are 2, 3, and 1. Sorting by trace_level yields the following information: the sorted trace_levels are 1, 2, and 3. The lowest-level interface_sign is AuthService.validateUser, which is assumed to be extracted from the context. The lowest-level affected service has min(trace_level) = 1. The cause of the exception is User validation failed due to database error, and the final result of the exception is System unable to process request. When the time interval between adjacent alarms in service B is 1, the Euclidean distance between alarm A and alarm B is 1. A similarity matrix is constructed, and hierarchical clustering is performed using the SciPy library.
[0048] By defining the information structure, a comprehensive data description is provided, so that each level of alarm has a clear context. By using interceptors in microservices, key information can be collected before the request reaches each service interface, effectively improving the timeliness and accuracy of problem diagnosis. With trace_id as the core, the call information is transmitted between various services to ensure the integrity and consistency of the alarm records. The warehouse centrally captures business logic anomalies in the service process, reducing the risk of missed reports or delayed responses, ensuring that operation and maintenance personnel can quickly intervene and handle them. The use of context information to complete the missing trace_id and interface name improves the integrity of the alarm information and provides sufficient information to help diagnose even in complex call scenarios. Analysis: By defining time windows C and B, efficient management and deduplication of alarms are achieved, effectively preventing repeated recording of the same alarm and improving the efficiency and accuracy of alarm management. The similarity between alarms is calculated through Euclidean distance, and hierarchical clustering technology is used to efficiently cluster alarm records to form a hierarchical tree diagram of alarms and extract important abnormal information. Through efficient alarm recording and classification, the time for manual analysis and processing is reduced, thereby reducing operation and maintenance costs. Transparent alarm information and tracking links can quickly locate faults, improve the overall fault response speed, and avoid multiple operation and maintenance personnel from repeatedly processing alarms with different appearances but the same root cause. Alarm correlation analysis and root cause tracing help operation and maintenance personnel quickly locate the root cause of the problem.
[0049] The above description is merely illustrative of certain exemplary embodiments of the present invention. It goes without saying that those skilled in the art will be able to modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and description are illustrative in nature and should not be construed as limiting the scope of protection of the claims.
Claims
1. A monitoring alarm clustering method based on trace, characterized by: The following steps are involved: Step 1: Define a standardized alert model for the service interface's signature, unique identifier, call hierarchy, exception information, and call chain record fields. Use interceptors to monitor all service interface requests in the microservice architecture, centrally capture business logic exceptions in the service process, and record them in log files. Step 2: Define the message and use the context information to complete the missing trace_id and interface name. Use Redis to check for repeated alarms within the time window. Step 3: Aggregate similar alarm information within the time window and group it by trace_id, extract and sort the trace_level, identify the anomaly source and result, and merge the stack_list to finally generate a structured aggregated alarm.
2. The trace-based monitoring alarm clustering method according to claim 1, characterized in that: In step 1, the specific information fields of the standardized alarm model are defined as: S1: interface_sign is the signature information of the specific method of the service interface; S2: trace_id is a unique identifier generated during each service call; S3: trace_level is the calling level of the specific method of the current service interface. Set the trace_level of the root method to 0. Each time a method is called to a lower-level service, the corresponding value of trace_level increases. S4: msg is a character string indicating the cause of the exception; S5: stack_list is a detailed record of the call chain and stores the information of each call process in a table format.
3. The trace-based monitoring alarm clustering method according to claim 1, characterized in that: In the step 1, an interceptor is used to monitor requests of all service interfaces in the microservice architecture, and the timestamp, interface name, request parameters and user identity of the request are recorded and stored in the log file. The trace_id is passed from service A to service B through the service call link, and information is recorded during the processing of service B to keep the call chain complete. Business logic exceptions of the service process are captured centrally and recorded in the log file, and are mapped to msg in combination with interface_sign and trace_id.
4. The trace-based monitoring alarm clustering method according to claim 1, characterized in that: In the step 2, define msg to include the error type, error description and timestamp and uniformly adopt JSON format. The missing trace_id and interface name are completed by context information. A five-minute time window C is defined. Check whether there are duplicate records of the same alarm within the time window C. When the same trace_id already exists within the time window C, directly skip the record of the alarm corresponding to the trace_id. Use Redis to record the checked trace_id and its timestamp. When a new alarm arrives, check whether the trace_id exists in Redis. If the trace_id exists and is within the time window C, no further record is made.
5. The trace-based monitoring alarm clustering method according to claim 1, characterized in that: In the step 3, a five-minute time window B is defined, and the alarms within the time window B are checked to be considered similar alarms, and the trace_id, trace_level, timestamp, msg and stack_list of the alarm information are extracted. Each trace_id represents the complete call chain of a request, all alarm records are grouped according to trace_id, and multiple alarms of the same request are processed centrally. For the alarm records with the same trace_id, they are sorted according to trace_level. Each time a downward call is made, the level value increases by 1, and the time interval between adjacent alarms is extracted. The number of severity levels of alarm records with the same trace_id is recorded, the similarity between alarms is calculated using Euclidean distance, and a similarity matrix is constructed to extract the interface_si of the lowest level and the highest level. gn, the lowest-level affected service = min(trace_level) and indicates the source of the exception cause. The exception cause = max(trace_level) corresponding msg and indicates the final result of the exception cause. For alarm records with the same trace_id, the corresponding stack_list is extracted and merged into a complete list. Tags indicating the importance and hierarchical relationship in the call chain are added. Hierarchical clustering is performed using the linkage function in the SciPy library, and the dendrogram function is used to draw the final dendrogram of the aggregated alarms.
6. The trace-based monitoring alarm clustering method according to claim 5, characterized in that: The specific formula for calculating the similarity between alarms using Euclidean distance is: Where d(j,k) represents the Euclidean distance between alarms, j i represents the characteristics of the jth alarm at the ith severity level, k i represents the feature of the kth alarm at the i-th severity level, n represents the number of severity levels of alarm records with the same trace_id, f(t) represents the time interval between adjacent alarms, and δ represents the alarm information with the same trace_id.
7. The trace-based monitoring alarm clustering method according to claim 1, characterized in that: In step three, specific alarm rules are formulated for each cluster based on the final aggregated alarm. The long-term alarm trend is displayed through a visual dashboard and the abnormal level is observed in a timely manner. The silhouette coefficient is used to evaluate the clustering effect and reflect the distinction between each level.