A method and system for end-to-end hierarchical monitoring of distributed applications

Abstract

This invention discloses a method and system for hierarchical monitoring of the entire chain of distributed applications, relating to the field of computer system operation and maintenance monitoring technology. The method includes: constructing a full-stack dependency topology configuration model containing three component types: the application itself, infrastructure components, and related systems; sequentially executing first-level monitoring of the target application itself, second-level monitoring of infrastructure components, and third-level monitoring of related systems; summarizing the monitoring results at each level and performing collaborative diagnosis; determining multi-dimensional indicators based on a differentiated set of key indicators and mapping component status levels; triggering a full-chain veto based on key dependency tags and key indicator anomalies; dynamically correcting the component score coefficient bidirectionally based on the status duration; calculating a weighted comprehensive score and mapping it to a health level; and executing hierarchical alarms based on the health level and the veto trigger situation. This invention achieves end-to-end availability assessment and accurate fault location across the entire chain of distributed systems.

Description

Technical Field

[0001] This invention relates to the field of computer system operation and maintenance monitoring technology, specifically to a distributed application end-to-end hierarchical monitoring method and system. Background Technology

[0002] With the deepening of enterprise digital transformation, distributed architecture has become the standard architecture for core business systems in fields such as finance, e-commerce, and government affairs. A typical transaction usually requires the collaborative completion of dozens or even hundreds of distributed application services, databases, caching middleware, message middleware, and upstream and downstream related business systems. Failure in any link may lead to business failure, fund errors, or compliance risks. Financial regulatory agencies must control the failure recovery time and failure recovery point targets of core transaction systems within minutes or even seconds. Therefore, accurate, comprehensive, and real-time proactive detection and assessment of the full-link availability of distributed applications has become a key prerequisite for ensuring business continuity and meeting regulatory compliance requirements.

[0003] Chinese invention patent CN108270840B discloses a business monitoring scheme. This scheme configures a resource processing interface for a resource system within the business system, processes business data by calling this interface to obtain processing results, configures a running weight for the resource system based on these results, and monitors the resource system according to these running weights. However, this prior art has the following obvious shortcomings: 1) The monitoring dimension is singular, lacking a systematic and hierarchical monitoring architecture covering the entire chain. This solution only focuses on the processing results when the business system calls external resource systems, adjusting the running weight of the resource system based on the processing results such as success, failure, and timeout. Its monitoring objects are limited to the external resource systems that the business system depends on. However, in real distributed system operation and maintenance scenarios, a complete business transaction not only depends on upstream related systems, but also deeply depends on the application itself and various types of infrastructure components such as databases, caching middleware, and message middleware. This solution neither involves proactive availability detection of infrastructure components such as databases, caching, and message middleware, nor includes the application itself in the monitoring scope, nor establishes a hierarchical monitoring architecture covering the entire chain from "the application itself to infrastructure components to related systems". This singular monitoring dimension makes it impossible for operation and maintenance personnel to obtain an end-to-end full-chain availability view. When a business transaction fails, it is difficult to quickly locate whether the failure occurred in the application itself, a certain type of infrastructure component, or upstream and downstream related systems, which seriously prolongs the fault location time.

[0004] 2) The lack of a refined collaborative diagnostic mechanism makes it impossible to achieve accurate fault determination and hierarchical alarm linkage. Although the solution proposes an idea of ​​dynamically adjusting the running weight based on the processing results, its monitoring and decision-making mechanism is relatively crude: it only adjusts the running weight by adding or subtracting based on the processing results such as success, failure, and timeout, and judges whether the resource system has failed based on whether the running weight is lower than a certain threshold. This mechanism has the following shortcomings: First, it fails to distinguish the essential differences in fault modes and key indicators among different component types (databases, caches, message middleware, and related systems), making it impossible to accurately determine the characteristics of each component by setting differentiated key indicators. Second, its weight adjustment is based solely on simple addition and subtraction operations on the result of a single processing step, without introducing a dynamic correction mechanism for fault duration. This fails to reflect the cumulative effect of faults, as there is a fundamental difference in severity between brief fluctuations and persistent faults, but this solution does not differentiate between them, easily leading to false alarms or missed alarms. Third, its alarm mechanism is based solely on triggering the running weight threshold, lacking a tiered alarm escalation strategy linked to fault duration. When the fault continues to worsen, dynamic escalation of alarm levels cannot be achieved. Finally, this solution lacks a complete component state transition model and state recovery observation mechanism. When component states oscillate frequently at threshold boundaries, it cannot effectively suppress fluctuations, affecting the stability of diagnostic conclusions. The root cause of these shortcomings is that this existing technology only proposes isolated weight adjustment rules, failing to construct a collaborative diagnostic system that deeply couples differentiated indicator determination, state transition management, bidirectional duration correction, and tiered alarm linkage. Summary of the Invention

[0005] To address the aforementioned shortcomings in existing technologies, this invention provides a method and system for hierarchical monitoring of the entire chain of distributed applications.

[0006] To address the aforementioned technical problems, this invention provides a method for hierarchical monitoring of the entire chain of distributed applications, comprising the following steps: Step S1: Construct a full-stack dependency topology configuration model.

[0007] A full-stack dependency topology configuration model is pre-built and stored for each target application. This model includes at least the following configuration information: application identifier, component identifier, component type and subtype, probe connection parameters, business probe interface Uniform Resource Locator (URL), weight coefficient, key dependency markers, and a set of key metrics. The component type includes its own application type, infrastructure component type, and associated system type. The infrastructure component type further includes database type, caching middleware type, and message middleware type. The subtype is used to identify specific component types for routing to the corresponding probe executor. The set of key metrics includes the names of key metrics and anomaly detection rules used to make a veto decision for the component.

[0008] The authentication credentials for database type, cache middleware type, and message middleware type components in the full-stack dependency topology configuration model are managed uniformly in the backend configuration file. When loading the configuration, the corresponding credentials are automatically injected into the probe parameters according to the component type.

[0009] Step S2: Receive the scheduling trigger instruction and load the configuration model.

[0010] The system receives monitoring trigger commands from the upper-layer scheduling platform via a standardized interface. These commands at least include a target application identifier. Based on the target application identifier, the system loads the corresponding full-stack dependency topology configuration model from the configuration storage into a memory cache. The configuration version number in the memory cache is used for subsequent configuration change detection and hot updates.

[0011] Step S3: Perform the first level of monitoring, namely, monitoring the availability of your own application.

[0012] Based on the configuration information of the target application itself in the full-stack dependency topology configuration model, a probe request is initiated to the probe interface exposed by the target application itself. After receiving the request, the target application's probe interface flows through the target application's business call chain step by step, aggregating the availability status of each module within the target application, and then returns a standardized response.

[0013] Upon receiving the response, determine the first-level monitoring result of the target application: If the Hypertext Transfer Protocol status code is a success response code and the overall status field in the response body is normal, and the response time does not exceed the preset first time threshold, then the target application status is determined to be normal. If the Hypertext Transfer Protocol status code is a success response code but the response time exceeds the first time threshold, then the target application is determined to be in a state of performance degradation. If the Hypertext Transfer Protocol status code is not a success response code, or the overall status field in the response body is unavailable, or the request timed out, or a connection error occurred, then the target application is determined to be unavailable.

[0014] Step S4: Perform the second level of monitoring, namely infrastructure component monitoring.

[0015] After the first level of monitoring is completed, based on the configuration information of the infrastructure component type in the full-stack dependency topology configuration model, the following three types of sub-monitoring tasks are executed in parallel to detect the availability of each infrastructure component. Each detection operation reuses the long connection or connection pool that has been established with the target component.

[0016] 1) Database monitoring: Based on the configuration information of the database type component in the full-stack dependency topology configuration model, including database subtype, network address, port, and automatically injected authentication credentials, a probe structured query statement is sent to the target database instance through an established long connection or connection pool, and a response is received.

[0017] If the structured query statement is successfully executed and the execution time does not exceed the preset second time threshold, the database component is determined to be in normal status. If the structured query statement is successfully executed but the execution time exceeds the second time threshold, the database component is determined to be in a state of performance degradation. If the probe fails to execute a structured query, throws any exception, times out, or experiences a connection error, the database component is deemed unavailable.

[0018] 2) Monitoring of cache middleware: Based on the configuration information of the cache middleware type component in the full-stack dependency topology configuration model, including the cache middleware subtype, network address, port, and automatically injected authentication credentials, a probe command is sent to the target cache instance through an established long connection or connection pool and a response is received.

[0019] If the probe command responds successfully and the response time does not exceed the preset third time threshold, the cache middleware component is determined to be in normal status. If the probe command responds successfully but the response time exceeds the third time threshold, or if the read / write integrity verification fails, the cache middleware component is determined to be in a state of performance degradation. If the probe command does not respond, returns an error, times out, or causes a connection error, the cache middleware component is determined to be unavailable.

[0020] 3) Message middleware monitoring: Based on the configuration information of the message middleware type component in the full-stack dependency topology configuration model, including message middleware subtype, cluster address, and automatically injected authentication credentials, a metadata retrieval request is sent to the target message cluster through an established long connection or connection pool, and a response is received.

[0021] If the metadata is successfully retrieved and the retrieval time does not exceed the preset fourth time threshold, the message middleware component is determined to be in normal status. If metadata is successfully retrieved but the retrieval time exceeds the fourth time threshold, or if message sending and receiving verification fails, the message middleware component is determined to be in a state of performance degradation. If metadata retrieval fails, the request times out, or a connection error occurs, the message middleware component is determined to be unavailable.

[0022] Step S5: Perform the third level of monitoring, namely, monitoring of related system services.

[0023] After the second level of monitoring is completed, iterate through the configurations of type "associated system" in the full-stack dependency topology configuration model and perform business-level availability detection for each associated system: Based on the Uniform Resource Locator (URL) of the business probe interface recorded in the configuration information of the associated system, a probe request is initiated using the Hypertext Transfer Protocol (HTTP) GET method. The business probe interface is provided by the associated system and follows dual security constraints: internally, the business probe interface performs only read-only query operations by default; if write capability verification is required, insert, update, or delete operations are allowed, but the data objects for these operations must be and are limited to a pre-established dedicated probe table, and access to any business data tables is strictly prohibited.

[0024] After receiving the standardized response returned by the business detection interface, determine the third-level monitoring result of the associated system: If the Hypertext Transfer Protocol status code is a success response code, the overall availability conclusion field in the response body is normal, the status of each key sub-check item is normal, and the response time does not exceed the preset fifth time threshold, then the status of the associated system is determined to be normal. If the Hypertext Transfer Protocol status code is a success response code and the overall availability conclusion field is normal, but the response time exceeds the fifth time threshold or there is a non-critical sub-check item with a performance degradation status, then the associated system status is determined to be performance degradation. If the Hypertext Transfer Protocol status code is not a success response code, or the overall availability conclusion field is unavailable, or any key sub-check item is unavailable, or the request times out, or a connection error occurs, or the response format does not conform to the preset specification, then the associated system is determined to be unavailable.

[0025] Step S6: Real-time detection and hot update of configuration changes.

[0026] After startup, the system continuously sends long polling requests to the configuration management terminal, each request carrying the configuration version number currently in the memory cache. Upon receiving the long polling request, the configuration management terminal compares the configuration version number with the latest configuration version number in the storage layer. If the version numbers are the same, the long polling request is suspended until it times out and returns an empty response. Upon receiving the empty response, the next round of long polling requests is initiated immediately. If the version numbers are different, the configuration management terminal will immediately return the complete configuration data of the latest version, mark the configuration in the memory cache as pending update, wait for the current probe batch to complete, replace the configuration object reference in memory with an atomic operation, and the new configuration will take effect on subsequent probe batches. The probe batch that is being executed will still use the old configuration until it is completed.

[0027] It should be noted that step S6 and the monitoring execution processes of steps S2 to S5 are independent of each other and run in parallel. The monitoring execution processes of steps S2 to S5 are repeatedly executed in units of probe batches, while the long polling listening in step S6 runs continuously after startup, detecting configuration changes in real time and completing configuration switching at the boundary of the probe batch. The two do not block each other.

[0028] Step S7: Result aggregation and collaborative diagnosis.

[0029] Summarize the results from the first-level monitoring, second-level monitoring, and third-level monitoring, and perform the following collaborative diagnostic processing: 1) Multi-dimensional indicator judgment and component status level mapping.

[0030] Multi-dimensional indicator assessment is performed based on the differentiated key indicator sets corresponding to each component type. Different component types have differentiated sets of key indicators: For database-type components, the key metric is the success rate of probing structured query statements. This metric is considered abnormal when the probing structured query statement fails to execute or throws any exception. For caching middleware type components, the key metric is the success rate of probe command response. This metric is considered abnormal when the probe command does not respond, returns an error, or a connection error occurs. For message middleware type components, the key metric is the metadata retrieval success rate. This metric is considered abnormal when the metadata retrieval request times out, returns an exception, or a connection exception occurs. For components of the associated system type, key indicators include the Hypertext Transfer Protocol status code, the overall availability conclusion field, and the status of key sub-check items. When the Hypertext Transfer Protocol status code is not a success response code, the overall availability conclusion field is unavailable, or any key sub-check item is unavailable, the indicator is considered abnormal.

[0031] Based on the results of the above key indicators, and combined with auxiliary indicators (including response time and format compliance), the detection results of each component are mapped to the component status level, which includes normal state, performance degradation state and unavailable state.

[0032] 2) End-to-end veto decision.

[0033] Check whether each component is a critical dependency and whether there are any anomalies in key metrics: When any component is marked as a critical dependency and its key metrics are abnormal, a full-link veto is triggered, forcing the overall availability score of the entire link to the lowest value and the health level to the severe abnormality level. When there are no critical dependent components or key indicators that are abnormal, a veto will not be triggered, and the overall availability score of the entire chain will be calculated according to the normal process.

[0034] 3) Two-way dynamic correction of component state duration.

[0035] A state duration counter is maintained for each detection component, recording the duration of the component's current state since the most recent state change. The component score coefficient is then dynamically adjusted bidirectionally based on this state duration. For components in a state of performance degradation, their score coefficient decreases progressively as the duration of performance degradation increases: when the duration of performance degradation does not exceed the first duration threshold, the first score coefficient is maintained; when the duration of performance degradation exceeds the first duration threshold but does not exceed the second duration threshold, the score coefficient is reduced to the second score coefficient; when the duration of performance degradation exceeds the second duration threshold, the score coefficient is reduced to the third score coefficient. For components in an unavailable state, their score coefficient is zero, and a persistent failure penalty value is applied based on the duration of unavailability: no penalty is applied when the unavailability duration does not exceed the third duration threshold; a first penalty coefficient is applied when the unavailability duration exceeds the third duration threshold but does not exceed the fourth duration threshold; a second penalty coefficient is applied when the unavailability duration exceeds the fourth duration threshold but does not exceed the fifth duration threshold; and a third penalty coefficient is applied when the unavailability duration exceeds the fifth duration threshold. When a component recovers from an abnormal state to a normal state, it does not immediately return to the full score coefficient, but enters a recovery observation period. During the recovery observation period, it participates in the scoring calculation with a preset transition score coefficient. After the recovery observation period ends and the component remains stable in a normal state, the score coefficient returns to the full score coefficient.

[0036] 4) Recovery of the state during the observation period.

[0037] If the component malfunctions again during the recovery observation period, the execution state will be rolled back: If a component changes from a normal state to a performance degradation state during the recovery observation period, the score coefficient immediately reverts to the score coefficient corresponding to the performance degradation state, and the state duration counter starts from the cumulative duration of the last time the component entered the performance degradation state. If a component changes from a normal state to an unavailable state during the recovery observation period, the score coefficient is immediately reset to zero, and the state duration counter starts counting from the cumulative duration of the component's last unavailable state.

[0038] 5) Weighted aggregation and comprehensive score calculation.

[0039] Based on the duration-adjusted score coefficients of each component and the weight coefficients of each component defined in the full-stack dependency topology configuration model, a basic weighted total score is calculated. Then, a persistent failure penalty value is calculated. The persistent failure penalty value is subtracted from the basic weighted total score, and the larger value is compared with zero to obtain the overall end-to-end availability score. If an end-to-end veto is triggered, the overall score is directly forced to zero.

[0040] The comprehensive score is mapped to a corresponding health level based on the preset scoring range. The health level includes at least a healthy level, a sub-healthy level, an abnormal level, and a severely abnormal level.

[0041] Step S8: Hierarchical alarm handling.

[0042] Obtain the current health level, veto trigger conditions, and duration of status information for each component from the diagnostic results, and execute the following alert decisions: 1) Alarm classification determination.

[0043] The alert level is determined based on the health status and the triggering of a veto: If the health level is severely abnormal or a full-link veto is triggered, a Level 1 emergency alarm will be triggered. If the health level is abnormal, a Level 2 critical alert will be triggered. If the health level is sub-healthy and the duration of performance degradation of any component exceeds the first duration threshold, a third-level warning alarm will be triggered.

[0044] 2) Alarm suppression handling.

[0045] For the same type of fault in the same component, only the first alarm is pushed within the preset suppression window period. Subsequent identical alarms are suppressed within the suppression window period and are not pushed again, but the status information is still continuously updated.

[0046] 3) Alarm escalation handling.

[0047] The alarm level is automatically escalated based on the duration of the fault: When the duration of the fault corresponding to the first-level emergency alarm exceeds the first escalation threshold, the alarm notification scope will be expanded to include first-level management personnel. When the duration of the fault corresponding to the first-level emergency alarm exceeds the second escalation threshold, the alarm notification scope will be expanded to second-level management personnel and the emergency response process will be triggered.

[0048] 4) Differentiated push notifications across multiple channels.

[0049] Alarm notifications are pushed through different channels based on different alarm levels: Level 1 emergency alarms are pushed through telephone, SMS and instant messaging channels; Level 2 serious alarms are pushed through SMS and instant messaging channels; Level 3 warning alarms are pushed through instant messaging channels and the work order system.

[0050] Step S9: Monitoring report generation and abnormal event persistence.

[0051] Output a structured end-to-end availability monitoring report. The report should include at least the following information: monitoring timestamp, target application identifier, overall end-to-end availability score and health level, whether a veto was triggered and the reason for the veto, detailed monitoring results for each level and component, and alarm trigger records. The monitoring report is returned to the upper-layer scheduling platform in a preset format and persistently stored in the database.

[0052] Meanwhile, when any component is first detected to have a performance degradation or unavailability, an abnormal event record is automatically created, recording the start time of the abnormality. During the duration of the abnormal state, new records are not created repeatedly in subsequent detection cycles; only the duration information of the abnormal event is updated. When the component's state returns to normal, the abnormal event record is written back, supplementing the end time of the abnormality and the total duration of the abnormality, and the record is marked as a recovered state.

[0053] Accordingly, the present invention also provides a distributed application end-to-end hierarchical monitoring system, comprising: The full-stack dependency topology configuration module is used to pre-build and store a full-stack dependency topology configuration model for each target application. The full-stack dependency topology configuration model includes at least the following configuration information: application identifier, component identifier, component type and subtype, probe connection parameters, business probe interface Uniform Resource Locator, weight coefficient, key dependency marker, and key indicator set. The component type includes its own application type, infrastructure component type, and associated system type. The infrastructure component type further includes database type, cache middleware type, and message middleware type.

[0054] The probe scheduling engine receives monitoring trigger commands from the upper-layer scheduling platform through a standardized interface. Based on the target application identifier carried in the monitoring trigger command, it loads the corresponding full-stack dependency topology configuration model from the full-stack dependency topology configuration module into the memory cache, and schedules probe executors at all levels to execute monitoring tasks according to the configuration information of the full-stack dependency topology configuration model. The probe scheduling engine is also used to continuously send long polling requests to the full-stack dependency topology configuration module after startup to detect configuration changes, and replace the configuration object reference in memory with atomic operations at the probe batch boundary to achieve hot update of configuration.

[0055] The self-application detector is used to initiate a probe request to the probe interface exposed by the target application itself based on the configuration information of the self-application type in the full-stack dependency topology configuration model, receive the response and determine the first-level monitoring result of the target application.

[0056] The infrastructure probe executor set includes a first probe executor plugin for multiple database subtypes, a second probe executor plugin for multiple cache middleware subtypes, and a third probe executor plugin for multiple message middleware subtypes. Each probe executor plugin is used to perform probe operations by reusing the long-established connection or connection pool between the infrastructure type component and the target component according to the configuration information of the infrastructure type component in the full-stack dependency topology configuration model, so as to obtain the second-level monitoring results.

[0057] The associated system business detection executor is used to traverse the configurations of type associated system in the full-stack dependency topology configuration model, call the business detection interface exposed by each associated system in the Hypertext Transfer Protocol GET method to perform business-level availability detection, and obtain the third-level monitoring results; the business detection interface follows dual security constraints: only read-only query operations are performed, or change operations are strictly limited to a pre-established detection-specific table.

[0058] The results aggregation and diagnosis module is used to summarize the first-level monitoring results, the second-level monitoring results, and the third-level monitoring results. It performs multi-dimensional indicator judgment, component status level mapping, end-to-end veto judgment, bidirectional dynamic correction of component score coefficients, recovery observation period status rollback, weighted aggregation calculation, and continuous fault penalty to generate a comprehensive end-to-end availability score and health level.

[0059] The alarm notification module is used to receive the current health level, veto trigger status, and duration of status of each component from the result aggregation and diagnosis module, and push alarm notifications through differentiated alarm channels according to the preset hierarchical alarm strategy, alarm suppression rules, and alarm escalation rules.

[0060] The monitoring report generation module is used to output structured end-to-end availability monitoring reports and persistently store monitoring results, component details, and abnormal events in the database.

[0061] Compared with the prior art, this application has the following beneficial effects: 1. Full-link monitoring has been implemented. Existing technologies only monitor external resource systems called by business systems, adjusting the operational weight of these resource systems based on the success, failure, or timeout results of resource processing interface calls. This approach does not involve monitoring other types of components such as the application itself, databases, caching middleware, and message middleware; essentially, it is a single-layer, single-type resource monitoring solution.

[0062] This application establishes a three-tiered monitoring architecture covering the application itself, infrastructure components (databases, caching middleware, message middleware), and related systems. The first tier monitors the application's own process liveness and basic service responsiveness; its probe interface can flow sequentially through the business call chain to aggregate the status of internal modules. The second tier monitors various infrastructure components in parallel, reusing long connections or connection pools to perform lightweight probe operations. The third tier monitors the business probe interfaces of related systems to perform business-level availability verification. These three tiers of monitoring, progressing from the inside out, constitute a complete end-to-end availability detection system.

[0063] The beneficial effects of this architecture design are that it expands the monitoring dimension from a single external resource system to all component types that the entire distributed application chain depends on, enabling operations and maintenance personnel to obtain a comprehensive end-to-end availability view, fundamentally solving the "blind men and the elephant" monitoring dilemma of existing technologies.

[0064] 2. Collaborative diagnosis has been achieved. Existing diagnostic decision-making mechanisms are relatively simplistic: they merely adjust operational weights based on processing results and determine whether a resource system has failed based on whether the weight falls below a certain threshold. This mechanism suffers from the following drawbacks: it fails to distinguish the fundamental differences in failure modes among different component types; it does not introduce dynamic correction for failure duration; it lacks state transition management and recovery observation mechanisms; and alarms are triggered solely based on weight thresholds, lacking a tiered linkage strategy. Essentially, weight adjustment, fault determination, and alarm triggering are three loosely coupled and even independent processes.

[0065] This application constructs a deeply coupled collaborative diagnostic mechanism consisting of "differentiated indicator judgment + end-to-end veto + two-way duration correction + hierarchical alarm linkage". Specifically: (1) Regarding the determination of differentiated indicators, this application defines a set of differentiated key indicators for different types of components. For databases, it is the success rate of SQL execution; for caches, it is the success rate of PING command response; for message middleware, it is the success rate of metadata retrieval; and for related systems, it is the HTTP status code, the overall field, and the status of key sub-items. This enables the veto function to accurately adapt to the unique failure modes of each component and avoid misjudgment by a "one-size-fits-all" approach.

[0066] (2) Regarding the end-to-end veto mechanism, this application is the first to deeply bind the veto decision with the core dependency mark. The end-to-end veto is triggered only when a component marked as a "core dependency" experiences an abnormal key indicator. This design ensures a second-level response to core failures and avoids unnecessary business interruptions caused by brief fluctuations in non-core components.

[0067] (3) Regarding the bidirectional correction of duration, this application maintains a state duration counter for each component, implements a progressive degradation strategy of "the longer the duration, the lower the score" for performance degradation states, and implements a continuous fault penalty for unavailable states. More importantly, this application pioneers a complete recovery observation period mechanism. After a component recovers from an abnormal state, it does not immediately return to full score, but enters an observation period with a transitional score coefficient. It only returns to full score after the observation period expires and remains stable. If an abnormality occurs again during the observation period, a state rollback is executed, with the duration calculated from the cumulative duration. This mechanism enables the diagnostic score to transition smoothly in both the direction of fault deterioration and recovery, effectively suppressing state jitter.

[0068] (4) Regarding the hierarchical alarm linkage, the alarm level is driven by the health level, the trigger condition of the veto, and the duration of the fault. Severe abnormality or veto triggers an emergency alarm, abnormal level triggers a severe alarm, and sub-health and duration exceeding the threshold trigger a warning alarm. At the same time, the duration of the fault drives the alarm to automatically escalate. The alarm module also has a suppression mechanism to avoid alarm storms.

[0069] Existing weight adjustments only reflect the instantaneous results of a single call, lacking awareness of the cumulative effects of faults, and even more so lacking the ability to observe recovery and suppress jitter. The four mechanisms mentioned above in this application achieve deep coupling through unified scheduling timing, shared state counters, and standardized data interfaces, forming a complete diagnostic closed loop.

[0070] 3. Improved security of business detection Existing technologies process business data by calling resource processing interfaces, which essentially involves initiating real business requests and manipulating real business data. This approach completely ignores the data boundary constraints of probe operations. For scenarios requiring business-level availability verification (such as verifying whether related systems can perform core business operations like queries and writes), directly adopting existing technologies would result in probe requests being completely mixed with real business requests at the data level, leading to serious data pollution and compliance risks.

[0071] This application pioneered a "dual security constraint" for the business probing interface: the first constraint prioritizes read-only queries, with the interface internally executing only read-only operations such as SELECT and GET by default; the second constraint strictly limits change operations to a dedicated probing table. If write capability verification is required, INSERT, UPDATE, and DELETE operations are permitted but must be performed exclusively on the pre-established dedicated probing table, and access to any business data tables is strictly prohibited. To ensure the reliable enforcement of this constraint, this application also provides two technical enforcement measures: database permission isolation (configuring a dedicated account with restricted permissions for the probing interface at the DBMS level) and runtime verification probes (intercepting and examining the target table name for each write operation at the application layer).

[0072] The beneficial effects of this design are: it elevates "security probes" from a best practice that relies on developers' self-discipline to a systematic approach with technical constraints, blocking unauthorized writing to business data from both the physical and code execution levels, fundamentally solving the long-standing problem of blurred boundaries between probe behavior and business data, and enabling business-level availability probes to be implemented securely and compliantly for the first time in highly regulated industries such as finance.

[0073] 4. Enabled dynamic updates of configuration management. Existing technologies do not involve a structured dependency topology configuration model, nor do they address dynamic configuration update mechanisms. Their monitoring objects and policies are essentially statically configured; any adjustments require modifying the configuration and restarting the service.

[0074] This application establishes a structured full-stack dependency topology configuration model. This model uses components as the basic unit, explicitly defining the type, subtype, probe connection parameters, weight coefficients, key dependency tags, and key indicator sets for each component. Based on this model, this application also designs a long-polling hot update mechanism that allows seamless application of new configurations without restarting the probe scheduling engine: the engine continuously sends long-polling requests to the configuration management end, carrying the current configuration version number; when the configuration changes, the management end immediately returns the new configuration; the engine waits for the current probe batch to complete, then replaces the configuration object reference in memory with an atomic operation. The new configuration takes effect for subsequent probe batches, and the currently executing task continues to use the old configuration until completion. The old and new configuration objects coexist briefly during the switching process, and the old configuration is automatically released by the memory reclamation mechanism after all tasks holding references to it have finished.

[0075] The advantages of this design are: the latency for detecting configuration changes is on the order of seconds, the switching granularity is at the batch level, atomic operations ensure thread safety, and the monitoring task is not interrupted throughout the process, thus meeting the rigid requirements of enterprise-level operation and maintenance systems for dynamic configuration activation.

[0076] 5. Achieve decoupling and concurrent scheduling The existing monitoring technology relies on the synchronous calls of the business system to the resource processing interface. The monitoring logic is tightly coupled with the business logic and does not involve concurrency scheduling strategies.

[0077] The detection scheduling engine in this application is decoupled from the upper-layer scheduling platform through standardized interfaces (HTTPAPI, message queues, gRPC, etc.), and can be integrated into automated operation and maintenance platforms, timed scheduling systems, event-driven platforms, or manually triggered consoles. Internally, the engine implements a hierarchical concurrent scheduling strategy: the first level prioritizes its own application monitoring, the results of which influence the scheduling decisions of subsequent levels; the second level, infrastructure monitoring, uses parallel scheduling, with monitoring tasks for various databases, caches, and message middleware submitted to the thread pool simultaneously to shorten the overall monitoring time; the third level, monitoring of related systems, executes after the second level completes. The engine also features global timeout control and a configurable retry mechanism.

[0078] 6. Improved data audit management capabilities Existing technologies do not address the structured, persistent storage of monitoring data or audit traceability mechanisms.

[0079] This application employs a three-tiered storage structure: a master monitoring results table, a component monitoring details table, and an anomaly event table. The overall results of each monitoring session, detailed detection information for each component, and anomaly events are all fully recorded and associated with configuration version numbers to ensure historical data interpretability. The anomaly event log supports complete lifecycle management from anomaly initiation and continuous updates to recovery and shutdown, enabling fault tracing, trend analysis, and compliance auditing.

[0080] The technical contribution of this application lies not only in proposing the above-mentioned innovative technical features, but also in the fact that these features do not operate in isolation, but form an organic whole that is interdependent and synergistic.

[0081] The full-stack dependency topology configuration model provides a "genetic map" for differentiated indicator judgment and veto power. It is precisely because of this precisely configured full-stack dependency topology model that the diagnostic engine can automatically match differentiated key indicator sets (database probe SQL, cached PING commands, message middleware metadata retrieval) for different types of components. This allows veto power to accurately and effectively determine the unique failure modes of each component, completely avoiding "one-size-fits-all" misjudgments. Simultaneously, the "core dependency marker" in the topology model assigns business priority to veto power; only failures of core components trigger a full-link emergency response, while failures of non-core components are handled according to normal procedures, ultimately achieving precise alignment between business impact and alarm levels.

[0082] The duration correction mechanism injects "time-dimensional" decision-making wisdom into veto and alarm linkage. While veto itself is based on an instantaneous judgment of a single probe result, the duration correction mechanism allows the consequences of a veto to be dynamically amplified over time—the duration counter of the DOWN state continuously accumulates, and the alarm module automatically escalates the alarm level accordingly; when a component recovers, the recovery observation period mechanism prevents the emergency state from being immediately lifted due to brief network jitter. This ensures that diagnostic conclusions possess both the timeliness of a second-level response and a global perspective over time.

[0083] Tiered alarm linkage transforms diagnostic conclusions into precise operational actions. Alarm levels are no longer isolated scoring maps, but are determined by a comprehensive result of the core / non-core roles defined by the full-stack topology, whether a veto is triggered, and duration adjustments. This is matched with differentiated push channels (telephone / SMS / instant messaging / work order system) and a duration-based automatic escalation strategy. This design ensures precise alignment between alarms and business impact—emergency faults are detected within seconds, persistent faults are escalated step-by-step, and brief fluctuations are automatically suppressed.

[0084] A unified collaborative workflow ensures seamless cooperation between modules. The entire diagnostic process follows a strict timeline: detection result aggregation → status tracker update (including duration counter and consecutive success counter) → comprehensive score calculation (including veto decision, duration correction, and recovery observation period decision) → alarm decision (including grading decision, suppression decision, and escalation decision) → result persistence. The diagnostic engine acts as the sole decision-making center, the status tracker as the passively updated data source, and the alarm module as the executor of the diagnostic results. There are no mutual queries or cross-calls between modules; instead, collaboration is achieved through the unified scheduling of the diagnostic engine.

[0085] The configuration hot update mechanism and the monitoring execution process run in parallel without blocking each other. The long-polling listening thread continuously detects configuration changes and completes the atomic switch at the batch boundary. The monitoring task is uninterrupted throughout the process, and the old and new configurations transition smoothly.

[0086] In summary, compared to the closest existing technology, this application represents a significant advancement across various technical dimensions, including monitoring architecture, diagnostic mechanisms, business probe security, configuration management, scheduling architecture, and data persistence. More importantly, these technical features are not simply stacked together; rather, they are deeply coupled through a unified collaborative working sequence and data flow protocol, forming a complete technical closed loop: "Topology configuration definition → multi-dimensional indicator-driven veto → time-dimensional dynamic correction → fault level-linked action." It is this systematic collaborative integration that enables this application to address a series of comprehensive pain points in existing technologies, such as single monitoring dimensions, crude diagnostic mechanisms, data pollution risks from business probes, rigid configuration management, and alarm storms, resulting in an overall technical performance improvement of "1+1>2." Attached Figure Description

[0087] Figure 1 This is an overall flowchart of the method described in Embodiment 1; Figure 2 This is a flowchart for the first level of monitoring (monitoring the availability of the application itself); Figure 3 Flowchart for Level 2 monitoring (infrastructure component monitoring); Figure 4 Flowchart for Level 3 monitoring (related system business monitoring); Figure 5 Flowchart for result aggregation and collaborative diagnosis; Figure 6 This is an architecture diagram of the system described in Example 2. Detailed Implementation

[0088] To better illustrate this invention, the following terms are first defined and explained: The full-stack dependency topology configuration model is a structured data model used to describe all components, their attributes, relationships, monitoring strategies, and diagnostic rules that a specific target application depends on at runtime. This model, with components as the basic unit, is the foundation for the hierarchical monitoring and collaborative diagnostics implemented in this invention.

[0089] Tiered monitoring is a monitoring strategy that proceeds step by step from the inside out, following a pre-defined sequence: first, the availability of the target application itself is probed, i.e., level one monitoring; then, the various infrastructure components it depends on are probed in parallel, i.e., level two monitoring; and finally, the upstream and downstream related business systems are probed, i.e., level three monitoring.

[0090] Collaborative diagnostics is a diagnostic method that deeply couples multiple aspects, such as multi-dimensional differentiated indicator judgment, end-to-end veto mechanism, bidirectional dynamic correction based on fault duration, and hierarchical alarm linkage, through unified scheduling timing and data flow protocol, thereby forming a complete decision-making closed loop from fault discovery, root cause location to state recovery.

[0091] The business probe interface is a standardized application programming interface provided by the associated system, adhering to the principle of dual security constraints. This interface receives probe requests from the monitoring system, internally aggregates the health status of itself and its downstream dependencies, and returns a standardized, comprehensive availability conclusion.

[0092] Dual security constraints are a core security principle designed to ensure that business-level probing operations do not contaminate business data. It includes two constraints: the first constraint is to prioritize read-only queries, meaning that the interface internally only performs database query operations by default; the second constraint is to strictly limit change operations to dedicated probing tables, meaning that any data objects that are written, modified, or deleted must be and only limited to pre-established dedicated probing tables that are physically isolated from business data.

[0093] The state recovery observation period is a stability mechanism used to suppress frequent fluctuations in system state. When a component recovers from an abnormal state, i.e., a performance degradation or unavailability state, to a normal state, its score coefficient does not immediately return to full score. Instead, it enters a preset observation period, during which it participates in the scoring as a transition coefficient. Only after continuous stability during the observation period does it return to the full score coefficient.

[0094] Long polling hot update is a dynamic update mechanism that allows configuration changes to take effect without restarting the service. The probe scheduling engine continuously sends long polling hypertext transfer protocol requests to the configuration management end to detect configuration changes in real time, and replaces the configuration object reference in memory through atomic operations after the current probe batch is completed, thereby achieving a smooth transition and immediate effect of the configuration.

[0095] Example 1 like Figure 1 As shown in the figure, this embodiment provides a distributed application end-to-end hierarchical monitoring method, which is uniformly scheduled and executed by the detection scheduling engine, and includes the following steps.

[0096] Step S1: Construct a full-stack dependency topology configuration model.

[0097] A full-stack dependency topology configuration model is pre-built and stored for each target application. This full-stack dependency topology configuration model is the foundation for the hierarchical monitoring and collaborative diagnosis of this invention, and it includes at least the following core configuration information: application identifier, used to uniquely identify a target application; component identifier, used to uniquely identify a specific component that the target application depends on; component type, which includes its own application type, infrastructure component type, and related system type, wherein the infrastructure component type further includes database type, caching middleware type, and message middleware type; subtype, used to identify the specific component type, such as MySQL, Oracle, Redis, Memcached, Kafka, or RabbitMQ, so as to route to the corresponding dedicated probe executor during probing; probe connection parameters, including network address, port, etc., for databases. For components such as caching and message queues, authentication credentials (username and password) are not stored in plaintext in the front-end configuration model, but are managed uniformly in the back-end configuration file. The detection scheduling engine automatically injects these credentials into the detection parameters based on the component type when loading the configuration. A Uniform Resource Locator (URL) for the business detection interface is used to configure the access address of the business detection interface for components of the associated system type. A weight coefficient is used to represent the component's impact on end-to-end availability in subsequent comprehensive scoring. A critical dependency flag is used to mark whether the component is a critical dependency; this flag will affect the triggering of a veto in the end-to-end process. A set of key metrics contains the names of key metrics and anomaly judgment rules used to determine whether the component is subject to a veto; different types of components have different sets of key metrics.

[0098] Step S2: Receive the scheduling trigger instruction and load the configuration model.

[0099] The detection scheduling engine receives monitoring trigger commands from the upper-layer scheduling platform through a standardized interface. This standardized interface can be a Hypertext Transfer Protocol Application Programming Interface, a message queue, or a gRPC remote procedure call. This invention decouples itself from the upper-layer scheduling platform through this interface. The scheduling platform can be an automated operation and maintenance platform, a timed scheduling system, an event-driven platform, or a manually triggered console.

[0100] The monitoring trigger command must contain at least one target application identifier. Upon receiving the command, the engine loads the corresponding full-stack dependency topology configuration model from the configuration store into the memory cache based on the target application identifier. The configuration version number in this memory cache is used for subsequent configuration change detection and hot updates.

[0101] Step S3: Perform the first level of monitoring, namely, monitoring the availability of your own application.

[0102] like Figure 2As shown, based on the configuration information of the application itself in the full-stack dependency topology configuration model, the probe scheduling engine schedules the application's own probe to initiate a Hypertext Transfer Protocol (HTTP) probe request to the probe interface exposed by the target application itself, such as the interface with the path / api / probe / internal. After receiving the request, the target application's own probe interface will flow through the target application's business call chain step by step, aggregating the availability status of its internal modules, including the network layer, business logic layer, and data access layer, and finally returning a standardized JS object simplified response, which contains at least one field representing the overall status.

[0103] Upon receiving the response, the first-level monitoring result of the target application is determined according to the following rules: If the HIP status code is a success response code 200, and the overall status field in the response body is UP, indicating normal operation, and the response time does not exceed the preset first timeout threshold (default value is 3000 milliseconds), then the target application is determined to be in a normal, i.e., UP state. If the HIP status code is a success response code 200, and the overall status field in the response body is UP, but the response time exceeds the first timeout threshold of 3000 milliseconds, then the target application is determined to be in a performance degradation, i.e., DEGRADED state. If the HIP status code is a non-successful response code (not 200), or the overall status field in the response body is DOWN, indicating unavailability, request timeout, or connection error, then the target application is determined to be in an unavailable, i.e., DOWN state.

[0104] Step S4: Perform the second level of monitoring, namely infrastructure component monitoring.

[0105] like Figure 3 As shown, after the first-level monitoring is completed, based on the configuration information of infrastructure components in the full-stack dependency topology configuration model, the probe scheduling engine schedules various infrastructure probe executors in parallel to execute the following three types of sub-monitoring tasks. All probe operations reuse long-established connections or connection pools with the target components to reduce resource overhead.

[0106] First, database monitoring. Based on the configuration information of the database type component, including the database subtype, network address, port, and authentication credentials automatically injected by the engine, the database probe executor sends probe structured query statements to the target database instance through established long connections or connection pools. For example, it sends SELECT1 to MySQL and receives the response. The status determination rules are as follows: If the probe structured query statement executes successfully and the execution time does not exceed the preset second time threshold (default value is 500 milliseconds), the database component is determined to be in a normal state, i.e., UP state. If the probe structured query statement executes successfully but the execution time exceeds the second time threshold of 500 milliseconds, the database component is determined to be in a performance degradation state, i.e., DEGRADED state. If the probe structured query statement fails to execute, throws any structured query language exception, times out, or experiences a network connection exception, the database component is determined to be in an unavailable state, i.e., DOWN state. Among these, when the probe structured query statement fails to execute or throws any exception, its key indicator, namely the success rate of the probe structured query statement execution, is determined to be abnormal.

[0107] Second, cache middleware monitoring. Based on the configuration information of the cache middleware type component, including the cache middleware subtype, network address, port, and automatically injected authentication credentials, the cache probe executor sends probe commands to the target cache instance through established long connections or connection pools, such as sending a PING command to Redis, and receives the response. The status determination rules are as follows: If the probe command responds successfully and the response time does not exceed the preset third time threshold (default value is 200 milliseconds), the cache middleware component is determined to be in normal status, i.e., UP state. If the probe command responds successfully but the response time exceeds the third time threshold of 200 milliseconds, or if read / write integrity verification fails (e.g., after writing a probe-specific key-value pair, it cannot be read correctly), the cache middleware component is determined to be in performance degradation status, i.e., DEGRADED state. If the probe command has no response, returns an error, times out, or experiences a connection exception, the cache middleware component is determined to be in unavailable status, i.e., DOWN state. Among these, when the probe command has no response, returns an error, or experiences a connection exception, its key indicator, namely the probe command response success rate, is determined to be abnormal.

[0108] Third, message middleware monitoring. Based on the configuration information of the message middleware type component, including the message middleware subtype, cluster address, and automatically injected authentication credentials, the message probe executor sends a metadata retrieval request to the target message cluster through an established long connection or connection pool, such as retrieving metadata for a specified topic in Kafka, and receives the response. The status determination rules are as follows: If the metadata retrieval is successful and the retrieval time does not exceed the preset fourth time threshold (default value is 3000 milliseconds), the message middleware component is determined to be in normal status, i.e., UP state. If the metadata retrieval is successful but the retrieval time exceeds the fourth time threshold of 3000 milliseconds, or if message sending and receiving verification fails (e.g., failing to send and consume a test message to a probe-specific topic), the message middleware component is determined to be in performance degradation status, i.e., DEGRADED state. If the metadata retrieval fails, the request times out, or a connection exception occurs, the message middleware component is determined to be in unavailable status, i.e., DOWN state. Among these, when the metadata retrieval request times out, returns an exception, or a connection exception occurs, its key indicator, namely the metadata retrieval success rate, is determined to be abnormal.

[0109] Step S5: Perform the third level of monitoring, namely, monitoring of related system services.

[0110] like Figure 4 As shown, after the second-level monitoring is completed, the configurations of type associated system in the full-stack dependency topology configuration model are traversed, and business-level availability detection is performed on each associated system.

[0111] Based on the Uniform Resource Locator (URL) of the business probe interface recorded in the configuration information of the associated system, a probe request is initiated using the Hypertext Transfer Protocol (HTTP) GET method. This business probe interface is provided by the associated system and adheres to a strict set of dual security constraints. The first constraint prioritizes read-only queries, meaning the interface internally only performs read-only operations such as SELECT and GET by default. The second constraint strictly limits modification operations to dedicated probe tables. If write capability verification is required, INSERT, UPDATE, or DELETE operations are permitted, but the data objects for these operations must be and only limited to pre-established dedicated probe tables, such as tables containing the identifiers healthcheck or probe. Access to any business data tables is strictly prohibited.

[0112] To ensure the reliable enforcement of the aforementioned dual security constraints, this invention provides two technical enforcement methods that can be used in combination. The first is database permission isolation, which involves configuring a dedicated account with restricted permissions for the probe interface at the database management system level. This account has full read and write permissions only for the probe-specific tables, and read-only permissions or no permissions at all for business data tables. The second is runtime verification probes, which involve injecting a verification module at the application layer of the related system using aspect-oriented programming or middleware technology. This verification module intercepts all database write operations and verifies whether the target table name is in the probe-specific table whitelist; if not, it immediately intercepts and throws an exception.

[0113] After receiving the standardized response from the business probe interface, the third-level monitoring result of the associated system is determined according to the following rules: If the Hypertext Transfer Protocol (HTTP) status code is a success response code 200, the overall availability conclusion field in the response body is UP, all key sub-check items are UP, and the response time does not exceed the preset fifth time threshold (default value is 5000 milliseconds), then the associated system is determined to be in normal condition (UP). If the HTTP status code is 200 and the overall availability conclusion field is UP, but the response time exceeds the fifth time threshold of 5000 milliseconds or a non-key sub-check item is DEGRADED, then the associated system is determined to be in a performance degradation state (DEGRADED). If the HTTP status code is not 200, or the overall availability conclusion field is DOWN, or any key sub-check item is DOWN, or the request times out, or a connection error occurs, or the response format does not conform to the preset JS object simplified pattern specification, then the associated system is determined to be in unavailable state (DOWN). Among them, when the Hypertext Transfer Protocol status code is not 200, the overall availability conclusion field is DOWN, or any key sub-check item is DOWN, the corresponding key indicator is judged as abnormal.

[0114] If the business detection interface is not configured, a warning will be issued during the configuration verification phase, and the associated system will be skipped and marked as not configured during monitoring.

[0115] Step S6: Real-time detection and hot update of configuration changes.

[0116] To meet the requirements of enterprise-level operation and maintenance systems for dynamic configuration updates, this invention designs a long-polling hot update mechanism that allows for seamless application of new configurations without restarting the detection and scheduling engine. This mechanism operates independently and in parallel with the monitoring execution process.

[0117] After startup, the detection scheduling engine continuously sends long polling HIP requests to the configuration management system. Each long polling request carries the configuration version number currently in the memory cache. Upon receiving the long polling request, the configuration management system compares this configuration version number with the latest configuration version number in the storage layer.

[0118] If the version numbers are the same, the long polling request is suspended until a timeout occurs (default 60 seconds), after which an empty response (HTTP status code 304) is returned. Upon receiving the empty response, the engine immediately initiates the next round of long polling requests. If the version numbers are different, the configuration management system immediately returns the latest complete configuration data with an HTTP status code of 200. Upon receiving the new configuration, the engine marks the configuration in its memory cache as pending update and waits for the current probe batch to complete before performing the configuration switch.

[0119] The aforementioned probe batch refers to a complete end-to-end hierarchical monitoring cycle executed by the engine for a single target application, encompassing the entire process from receiving trigger commands, progressively probing, result aggregation, to alarm notification. A batch represents the smallest granularity for hot update switching, ensuring that configuration changes do not take effect during the probe process.

[0120] The engine performs atomic operations at batch boundaries to replace configuration object references in memory. Specifically, internally, the engine maintains a thread-safe atomic configuration object reference for each application. When the new configuration is ready, the engine atomically updates the reference to point to the new configuration object. All newly started probe tasks retrieve the latest configuration through the read method of this atomic reference. Probe tasks submitted to the thread pool but not yet executed, because they already hold references to the old configuration object, continue to execute according to the old configuration. The replaced old configuration object continues to exist in memory until all in-process tasks holding references to it have completely finished, at which point it is automatically reclaimed by the runtime environment's garbage collection mechanism. This mechanism ensures thread safety without locking, and the transition between the old and new configurations is smooth, with monitoring tasks remaining uninterrupted throughout the process.

[0121] Furthermore, when the detection scheduling engine crashes and restarts due to a malfunction, the configuration version number in memory is lost. During the startup initialization phase, the engine initializes the configuration version number in memory to zero and proactively initiates a synchronization request to the configuration management terminal to pull the latest configuration. After loading it into the memory cache, it starts a new round of long polling listening based on this version number, thus achieving automatic recovery after the crash.

[0122] Step S7: Result aggregation and collaborative diagnosis.

[0123] like Figure 5As shown, the process first determines key performance indicators (KPIs) differentiated by component type and maps them to status levels. Then, it checks for anomalies in the KPIs of critical dependent components to determine whether to trigger a full-link veto. Next, the core algorithm performs bidirectional dynamic correction based on the duration of each component's status: imposing time penalties on performance degradation and unavailability, and introducing a recovery observation period mechanism after recovery to suppress jitter. Finally, a weighted aggregation is used to calculate a comprehensive score and map it to a health level, forming a complete decision-making closed loop from fault detection to assessment.

[0124] The first-level, second-level, and third-level monitoring results are aggregated, and the following collaborative diagnostic processing is performed by the result aggregation and diagnosis module. This module achieves deep coupling of multi-dimensional indicator judgment, end-to-end veto, bidirectional duration correction, and hierarchical alarm linkage.

[0125] First, multi-dimensional indicator judgment and component status level mapping. Multi-dimensional indicator judgment is performed based on the differentiated key indicator sets corresponding to each component type. Different component types have different sets of key indicators. For database components, the key indicator is the success rate of probing structured query statements; this indicator is considered abnormal when the probing structured query statement fails to execute or throws any exception. For caching middleware components, the key indicator is the success rate of probing command responses; this indicator is considered abnormal when the probing command has no response, returns an error, or a connection exception occurs. For message middleware components, the key indicator is the success rate of metadata retrieval; this indicator is considered abnormal when the metadata retrieval request times out, returns an exception, or a connection exception occurs. For related system components, key indicators include the Hypertext Transfer Protocol status code, the comprehensive availability conclusion field, and the status of key sub-check items; this indicator is considered abnormal when the Hypertext Transfer Protocol status code is not 200, the comprehensive availability conclusion field is DOWN, or any key sub-check item status is DOWN.

[0126] Based on the results of the above key indicators, and combined with auxiliary indicators such as response time and format compliance, the detection results of each component are mapped to one of the following three component status levels: normal state, i.e., UP state; performance degradation state, i.e., DEGRADED state; unavailable state, i.e., DOWN state.

[0127] Second, the end-to-end veto mechanism. This involves checking whether each component is marked as a critical dependency and whether any critical metrics are abnormal. If any component is marked as a critical dependency in the full-stack dependency topology configuration model, and its critical metrics are determined to be abnormal in this round of probing, an end-to-end veto is triggered. In this case, the overall end-to-end availability score is forcibly set to the lowest value of zero, and the health level is forcibly set to the severe anomaly level. If no critical dependent components have abnormal critical metrics, no veto is triggered, and the overall end-to-end availability score is calculated according to the normal process.

[0128] Third, bidirectional dynamic correction of component state duration. A state duration counter is maintained for each probe component. This counter records the duration of the component's current state since the most recent state change, and is accumulated in monitoring cycles. The component score coefficient is dynamically corrected bidirectionally based on this state duration.

[0129] For components in a state of performance degradation, their score coefficient decreases progressively in three levels as the duration of performance degradation increases. When the performance degradation duration T is less than T1 (where T1 is the first duration threshold, defaulting to 3 minutes), the component's score coefficient remains at the first score coefficient of 0.5. When the performance degradation duration T is greater than or equal to T1 and less than T2 (where T2 is the second duration threshold, defaulting to 10 minutes), the component's score coefficient decreases to the second score coefficient of 0.3. When the performance degradation duration T is greater than or equal to T2, the component's score coefficient further decreases to the third score coefficient of 0.

[0130] For components in an unavailable state, their base score coefficient is zero, and the system applies a persistent failure penalty value based on the duration of unavailability. This penalty value is used to further lower the overall score, and its calculation follows four rules: When the unavailability duration T satisfies T < T3, where T3 is the third duration threshold with a default value of 5 minutes, no penalty is applied, i.e., the penalty coefficient is 0. When the unavailability duration T satisfies T > or equal to T3 and less than T4, where T4 is the fourth duration threshold with a default value of 15 minutes, a first penalty coefficient of 0.3 is applied. When the unavailability duration T satisfies T > or equal to T4 and less than T5, where T5 is the fifth duration threshold with a default value of 30 minutes, a second penalty coefficient of 0.6 is applied. When the unavailability duration T satisfies T > or equal to T5, a third penalty coefficient of 1.0 is applied.

[0131] When a component recovers from an abnormal state (i.e., a performance degradation or unavailability) to a normal state, it does not immediately revert to the full score of 1.0. Instead, it enters a recovery observation period, with a default duration of 2 minutes. During this recovery observation period, the component's score is fixed at a transitional score of 0.7. Only after the recovery observation period expires and the component has remained consistently stable in a normal state does the score revert to the full score of 1.0.

[0132] Fourth, state rollback during the recovery observation period. If a component experiences another anomaly during the recovery observation period, state rollback will be executed, with the following specific rules: If a component changes from a normal state to a performance degradation state during the recovery observation period, its score coefficient will immediately roll back from 0.7 to 0.5, and its state duration counter will start counting from the cumulative duration of the component's last entry into the performance degradation state, rather than starting from zero. If a component changes from a normal state to an unavailable state during the recovery observation period, its score coefficient will immediately return to zero, and its state duration counter will start counting from the cumulative duration of the component's last entry into the unavailable state, rather than starting from zero.

[0133] For an unavailability state triggered by a veto, recovery requires meeting the following confirmation conditions: N consecutive successful probes (N = 3 by default), meaning that the component's key metrics are all normal for three consecutive monitoring periods, and auxiliary metrics have all recovered to within the preset warning threshold. After meeting these conditions, the component's state changes from unavailable to normal and enters the recovery observation period. For partial recovery paths from unavailability to performance degradation (i.e., key metrics have returned to normal but auxiliary metrics still exceed the threshold), the component state is directly converted to performance degradation, the score coefficient is restored to 0.5, and the previous unavailability duration is retained and continues to be included in the continuous failure penalty value.

[0134] Fifth, weighted aggregation and comprehensive score calculation. The system calculates the end-to-end availability comprehensive score according to the following steps: First, calculate the basic weighted total score, which is equal to the sum of the products of the weight coefficients of all components and the score coefficient of that component after duration correction. The sum of the weight coefficients of all components is normalized to 100%. Second, calculate the persistent failure penalty value, which is equal to the sum of the products of the weight coefficients of all components currently unavailable and the duration penalty coefficient of that component at the current moment. Third, calculate the final comprehensive score, which is equal to the basic weighted total score minus the persistent failure penalty value; if the result is less than zero, it is set to zero.

[0135] Specifically, when a full-link veto is triggered, the normal calculation process described above is bypassed, and the overall score is directly and forcibly reset to zero.

[0136] The comprehensive score is mapped to a corresponding health level based on a preset scoring range, with the specific mapping rules as follows: When the comprehensive score is greater than or equal to 90 and less than or equal to 100, the health level is healthy. When the comprehensive score is greater than or equal to 60 and less than 90, the health level is sub-healthy. When the comprehensive score is greater than or equal to 30 and less than 60, the health level is abnormal. When the comprehensive score is less than 30 or a veto is triggered, the health level is severely abnormal.

[0137] To more clearly illustrate the deep coupling between veto power, duration adjustment, and tiered alarm linkage, the following example illustrates the complete process of a core database component from failure to full recovery. Assume the probe is executed once per minute, and this database component is marked as a critical dependency.

[0138] During the first probing cycle, at time T0, the database's probing structured query failed. The status tracker updated the component's status to unavailable, and the duration counter reached 0 minutes. The diagnostic engine triggered a veto, assigning a comprehensive score of 0, classifying the health level as severely abnormal, and transmitting the faulty component information and duration to the alarm module. The alarm module then triggered a P0 emergency alarm.

[0139] At time T0 + 5 minutes, the component remains unavailable, and the status tracker updates the duration counter to 5 minutes. The diagnostic engine maintains a veto and a comprehensive score of 0, and the health level remains critically abnormal. It then passes the 5-minute duration to the alarm module. The alarm module maintains the P0 alarm, but since the 5-minute duration does not meet the escalation threshold, it maintains the current notification range.

[0140] At time T0 + 12 minutes, the component remains unavailable, and the status tracker updates the duration counter to 12 minutes. The diagnostic engine maintains a veto and a comprehensive score of 0, and the health level remains critically abnormal. It then passes the 12-minute duration to the alarm module. Because the duration exceeds the first escalation threshold of 10 minutes, the alarm module escalates the P0 alarm, expanding the notification scope to include technical managers.

[0141] At time T0 plus 16 minutes, the component detected that the structured query statement had been executed successfully, key metrics returned to normal, and the status tracker updated the consecutive success counter to 1. The diagnostic engine determined that the veto condition was no longer met, and the comprehensive score was recalculated, but the component remained in the pre-observation transition phase. The alarm module maintained the alarms, waiting for the consecutive success count to reach the target.

[0142] At time T0+18 minutes, the component detection continued to succeed, and the status tracker updated the consecutive success counter to 2. The diagnostic engine scoring section recovered. The alarm module continued to maintain alarms.

[0143] At time T0 + 20 minutes, the component successfully detected for the third consecutive time, meeting the condition of three consecutive successes. The status tracker changed the component status to normal and entered the recovery observation period. The diagnostic engine removed the veto power, and the comprehensive score recovery calculation was performed. The component's score coefficient was 0.7, which is the transition coefficient for the observation period. The alarm module cleared the P0 alarm and sent a recovery notification.

[0144] At time T0+22 minutes, the component was successfully detected and the observation period was completed. The status tracker marked the component status as normal and the observation period as expired. The diagnostic engine score returned to normal, and the component's score coefficient returned to 1.0. The alarm module issued no alarms, and the system returned to normal.

[0145] The above process fully demonstrates the following key collaborative points: A veto is triggered immediately at time T0, ensuring a second-level response to emergency faults. The duration counter for the unavailability state continuously accumulates, driving automatic alarm escalation at T0 + 12 minutes, demonstrating the driving role of duration correction in alarm linkage. At T0 + 20 minutes, although the component has recovered, it is not immediately at full strength, but enters an observation period with a transition coefficient of 0.7, demonstrating the suppression of state jitter during the recovery observation period. Throughout the process, the triggering, maintenance, escalation, and deactivation of alarms are all driven by the health level and duration information uniformly output by the diagnostic engine, and the various modules work collaboratively through standardized data interfaces.

[0146] Step S8: Hierarchical alarm handling.

[0147] The alarm notification module obtains the health level, veto trigger status, and duration of each component's status from the diagnostic results and then makes the following alarm decisions.

[0148] First, alarm classification is determined based on the health level and the triggering of a veto. If the health level is severely abnormal or a full-link veto is triggered, a Level 1 emergency alarm (P0) is triggered. If the health level is abnormal, a Level 2 severe alarm (P1) is triggered. If the health level is sub-healthy and the duration of performance degradation of any component exceeds the first duration threshold of 3 minutes, a Level 3 warning alarm (P2) is triggered.

[0149] Second, alarm suppression processing. For the same type of fault in the same component, within a preset suppression window period (default 5 minutes), only the first alarm is pushed. Subsequent identical alarms are suppressed within this suppression window period and are not pushed again, but the status information continues to be updated. Alarm suppression only affects the push behavior of alarm notifications and does not affect the status recording of abnormal event trigger flags.

[0150] Third, alarm escalation handling. Alarm levels are automatically escalated based on the duration of the fault. When the duration of a Level 1 emergency alarm exceeds the first escalation threshold (default 10 minutes), the alarm notification scope is expanded to Level 1 management personnel, such as technical managers. When the duration of a Level 1 emergency alarm exceeds the second escalation threshold (default 30 minutes), the alarm notification scope is expanded to Level 2 management personnel, such as department heads, and the emergency response process is triggered.

[0151] Fourth, differentiated push notifications through multiple channels. Alarm notifications are pushed through differentiated channels based on different alarm levels. Level 1 emergency alarms (P0 level) are pushed via telephone, SMS, and instant messaging. Level 2 critical alarms (P1 level) are pushed via SMS and instant messaging. Level 3 warning alarms (P2 level) are pushed via instant messaging and the work order system.

[0152] Step S9: Monitoring report generation and abnormal event persistence.

[0153] The monitoring report generation module outputs a structured end-to-end availability monitoring report. This report includes at least the following information: monitoring timestamp, target application identifier, overall end-to-end availability score and health level, whether a veto was triggered and the reason for the veto, detailed monitoring results for each level and component, including status, latency, key indicator judgment results, fault location summary, and alarm trigger records. This monitoring report, in a preset format such as a simplified JS object notation, is returned to the upper-layer scheduling platform and persistently stored in the database.

[0154] For data persistence, the system employs a three-tier storage structure. The first tier is the monitoring results master table, recording the overall results of each end-to-end monitoring session, including monitoring time, application identifier, comprehensive score, health level, veto flag, and configuration version number. The second tier is the component monitoring details table, recording detailed detection information for each component in each monitoring session, including component identifier, status level, time consumed, key indicator judgment results, and configuration version number; it has a one-to-many relationship with the monitoring results master table. The third tier is the abnormal event table, specifically recording abnormal events discovered during the monitoring process.

[0155] The exception event table supports complete exception lifecycle management, with the following write and update timing: When any component is first detected to have a performance degradation or unavailability state, the system automatically creates a new exception event record, recording the start time of the exception. During the duration of the same exception state, the system does not repeatedly create new records in subsequent detection cycles, but only updates the duration information of the current exception state of the component. When the component state recovers to normal, the system writes back the exception event record, supplementing the exception end time and the total duration of the exception, and marks the record as recovered. When an exception event is first recorded, if the exception has met the preset alarm triggering conditions, the trigger flag is set to triggered. If the conditions have not yet been met, it is marked as not triggered, and updated to triggered and an alarm is issued when the duration reaches the threshold in a subsequent cycle. Alarm suppression does not affect the status record of the trigger flag.

[0156] Example 2 This embodiment provides a distributed application end-to-end hierarchical monitoring system, which is used to implement the method described in Embodiment 1, such as... Figure 6As shown, the system uses a probe scheduling engine as its core, receiving trigger commands from the upper-layer scheduling platform through standardized interfaces. The engine loads the configuration model from the full-stack dependency topology configuration module and schedules its own application probes, infrastructure probe executors (including various plugins such as databases, caches, and message middleware), and related system business probe executors to execute three levels of monitoring tasks. All probe results are aggregated to the result aggregation and diagnosis module, where a comprehensive score and health level are generated after collaborative diagnosis. The alarm notification module pushes tiered alarms based on the diagnostic results, and the monitoring report generation module is responsible for persistent storage of the results. Simultaneously, the engine maintains synchronization with the configuration module through a long polling mechanism, enabling hot configuration updates. The system architecture clearly demonstrates the data flow and call relationships between the modules.

[0157] The system includes the following functional modules: The full-stack dependency topology configuration module is used to pre-build and store the full-stack dependency topology configuration model as described in step S1 of Example 1 for each target application. This module includes the following sub-components: A configuration interface layer, providing a visual interface for operations personnel to perform CRUD operations on configurations, and supporting dependency topology visualization, template import, and cross-environment batch migration. A configuration verification engine, used to verify the matching of required fields, data formats, component subtypes, and probe executor plugins in the configuration, and can perform optional connectivity pre-checks on the business probe interface addresses of related systems. A configuration storage layer, supporting relational databases such as MySQL or PostgreSQL, or configuration centers such as Etcd, Apollo, or Nacos as storage backends. A configuration version manager, used to automatically generate version numbers and save configuration snapshots each time a configuration is released, supporting historical version tracing, rollback, and version difference comparison. A configuration distribution and synchronizer, used to respond to long polling requests from the probe scheduling engine, performing version number comparison and new configuration distribution.

[0158] The probe scheduling engine, as the core execution hub of the system, receives monitoring trigger commands from the upper-layer scheduling platform through a standardized interface. Based on the target application identifier carried in the monitoring trigger command, it loads the corresponding full-stack dependency topology configuration model from the full-stack dependency topology configuration module into the memory cache, and schedules probe executors at each level to execute monitoring tasks according to the configuration information of the configuration model. This probe scheduling engine also continuously sends long-polling requests to the full-stack dependency topology configuration module after startup to detect configuration changes, and replaces the configuration object reference in memory with atomic operations at the probe batch boundary, achieving hot configuration updates as described in step S6 of embodiment one. Internally, the engine maintains a configurable thread pool. Monitoring tasks at each level are submitted to the thread pool asynchronously for execution, employing the following scheduling strategies: Level 1 monitoring, i.e., monitoring of the application itself, is executed first, and its results affect the scheduling decisions of subsequent levels. If the application itself is unavailable and marked as a critical dependency, a veto can be triggered, and subsequent levels can be skipped. Level 2 monitoring, i.e., monitoring of infrastructure components, uses parallel scheduling; monitoring tasks for each database, cache, and message middleware are submitted to the thread pool simultaneously. Level 3 monitoring, i.e., monitoring of related systems, is executed after Level 2 is completed. The engine also features a global timeout control, which defaults to 30 seconds, and a configurable retry mechanism, which retry once by default with a 2-second interval.

[0159] The self-application detector is used to initiate a Hypertext Transfer Protocol (HTTP) probe request to the probe interface exposed by the target application itself based on the configuration information of the self-application type in the full-stack dependency topology configuration model, receive the response and determine the first-level monitoring result of the target application. The specific implementation is as described in step S3 of embodiment one.

[0160] The infrastructure probing executor set includes first-level probing executor plugins for various database subtypes, such as MySQL and Oracle; second-level probing executor plugins for various caching middleware subtypes, such as Redis and Memcached; and third-level probing executor plugins for various message middleware subtypes, such as Kafka, RabbitMQ, and RocketMQ. Each probing executor plugin is used to perform probing operations based on the configuration information of the infrastructure type components in the full-stack dependency topology configuration model, reusing established long connections or connection pools with the target components to obtain second-level monitoring results. The specific implementation is as described in step S4 of Example 1.

[0161] The associated system business detection executor is used to traverse the configurations of type associated system in the full-stack dependency topology configuration model. It calls the exposed business detection interfaces of each associated system using the Hypertext Transfer Protocol (HTTP) GET method to perform business-level availability detection and obtain third-level monitoring results. The specific implementation is as described in step S5 of Example 1. This business detection interface follows the dual security constraints and returns a JS object in simplified format according to the standardized interface contract. For scenarios where the interface is unavailable, the following fallback processing is performed. If the interface's Uniform Resource Locator (URL) is configured but the request times out (exceeding 5000 milliseconds) or returns a status code other than 200, the associated system is determined to be unavailable. Returning status code 429, i.e., rate limiting, indicates performance degradation. If the response format does not conform to the preset pattern, it is determined to be unavailable.

[0162] The results aggregation and diagnosis module is used to summarize the first-level, second-level, and third-level monitoring results. It performs multi-dimensional indicator judgment, component status level mapping, end-to-end veto judgment, and bidirectional dynamic correction of component score coefficients. This includes recovery observation period status rollback, weighted aggregation calculation, and continuous fault penalty, generating a comprehensive end-to-end availability score and health level. The specific implementation is as described in step S7 of Example 1. This module maintains an independent status tracker for each component, recording the status level, duration counter, continuous success counter, and recovery observation period flag.

[0163] The alarm notification module receives the current health level, veto trigger status, and duration of each component's status from the result aggregation and diagnosis module. Based on preset hierarchical alarm strategies, alarm suppression rules, and alarm escalation rules, it pushes alarm notifications through differentiated alarm channels, as described in step S8 of Embodiment 1. This module includes an alarm hierarchical submodule, an alarm suppression submodule, an alarm escalation submodule, and a multi-channel push submodule.

[0164] The monitoring report generation module is used to output a structured end-to-end availability monitoring report and persistently store the monitoring results, component details and abnormal events in the database, as described in step S9 of Embodiment 1.

[0165] In the system described in this invention, each functional module forms a diagnostic closed loop through a unified collaborative working sequence, with the data flow as follows: After collecting all probe results, the probe scheduling engine transmits the probe status of each component to the corresponding status tracker in the result aggregation and diagnostic module. The status tracker updates the status, duration, and continuous success counter. The diagnostic engine reads the updated status information and calculates the score and health level. The alarm module receives the diagnostic results and executes alarm decisions. The reporting module persists all data. The diagnostic engine is the sole decision-making center, the status tracker is a passively updated data source, and the alarm module is the executor of the diagnostic results. There are no mutual queries or cross-calls between the modules, ensuring the consistency of decisions.

[0166] Example 3 This embodiment uses a core accounting system in the financial sector as a typical application scenario to describe in detail the implementation process of the technical solution of this invention. In this scenario, the target application is the core accounting system, and its full-stack dependency topology configuration is as follows: Application Type: None, Subtype: Core Accounting System (self), Weight: 20, Key Dependency: Yes, Key Metric: Comprehensive Status Field. Database Type: MySQL, Subtype: Accounting Database, Weight: 30, Key Dependency: Yes, Key Metric: Success Rate of Detecting Structured Query Statements. Cache Type: Redis, Subtype: Transaction Flow Cache, Weight: 15, Key Dependency: No, Key Metric: PING Command Response Success Rate. Message Queue Type: Kafka, Subtype: Reconciliation Message Queue, Weight: 10, Key Dependency: No, Key Metric: Success Rate of Topic Metadata Acquisition. Related System Type: None, Subtype: People's Bank of China Payment System, Weight: 15, Key Dependency: Yes, Key Metrics: Hypertext Transfer Protocol Status Code and Comprehensive Availability Conclusion Field. Related System Type: None, Subtype: Internal Risk Control System, Weight: 10, Key Dependency: Yes, Key Metrics: Hypertext Transfer Protocol Status Code and Comprehensive Availability Conclusion Field. The authentication credentials for the database, cache, and message queue components are managed uniformly in the backend configuration file, and the detection scheduling engine automatically injects them when loading the configuration.

[0167] During the monitoring process, the following typical failure scenarios exist: Scenario A involves a core database failure triggering a veto and an emergency alert. During a probing cycle, when the MySQL probe executor sends a SELECT1 probe statement to the accounting database, it throws a connection exception, with the exception message being "Communicationslinkfailure". The result aggregation and diagnostic module determines that the key indicator for this database component, namely the success rate of probing structured query statements, is abnormal, and its status is set to unavailable. Because this MySQL component is marked as a critical dependency in the configuration model, a full-link veto is triggered. The overall score is forcibly set to zero, and the health level is forcibly set to severe anomaly. The alert notification module immediately pushes a Level 1 emergency alert via telephone, SMS, and WeChat, clearly indicating that the faulty component is the MySQL accounting database and the cause of the anomaly is a connection error.

[0168] Scenario B describes the performance degradation of a caching component and its complete recovery and rollback process. Within a probing period, the Redis probe detects a PING command response time of 300 milliseconds, exceeding the preset third time threshold of 200 milliseconds, but the command executes successfully. Therefore, the component is judged to be in a performance degradation state, with an initial score coefficient of 0.5, and the state duration counter begins counting. If this performance degradation state persists for more than 3 minutes, its score coefficient automatically drops to 0.3. If, at the 6th minute, the component's response returns to normal, with the time returning to 50 milliseconds, its state changes back to normal, but immediately enters a 2-minute recovery observation period, with the score coefficient adjusted to 0.7. If, at the 7th minute of the observation period, the component's response deteriorates again, with the time exceeding 200 milliseconds again, a state rollback is triggered. Its state immediately reverts to performance degradation, the score coefficient rolls back to 0.5, and the state duration counter starts counting from the cumulative duration of the previous performance degradation, i.e., 7 minutes. If the component recovers again at the 12th minute and passes the 2-minute recovery observation period completely, the score coefficient will eventually recover to 1.0.

[0169] Scenario C involves a persistent failure of a related system triggering an alarm escalation. When the related system business probe executor calls the business probe interface representing the People's Bank of China's payment system (https: / / pbc-payment / business-probe), a request timeout occurs exceeding 5000 milliseconds, determining the related system to be unavailable. Since this component is a critical dependency, a veto and a P0 emergency alarm are triggered. As the failure continues, the alarm module monitors the duration of this unavailability. When the duration exceeds 10 minutes, the alarm module automatically expands the alarm notification scope to technical managers. When the duration exceeds 30 minutes, the notification scope is further expanded to department heads, triggering a pre-defined emergency response process and generating a high-priority work order.

[0170] Example 4 This embodiment uses an e-commerce order system outside the financial sector as another application scenario to illustrate the cross-industry versatility of the present invention. In this scenario, the target application is an order service, and its full-stack dependency topology configuration can be mapped as follows: a MySQL order database, corresponding to the accounting database in a financial scenario, with a weight of 30 and marked as a key dependency; a Redis shopping cart cache, corresponding to the transaction log cache, with a weight of 15 and marked as a key dependency; a Kafka order event queue, corresponding to the reconciliation message queue, with a weight of 10 and marked as a key dependency; and an associated system representing a third-party payment interface, corresponding to the People's Bank of China payment system, with a weight of 15 and marked as a key dependency. When implementing the present invention, all monitoring logic, hierarchical scheduling strategies, collaborative diagnostic rules, scoring algorithms, state transition mechanisms, and data persistence structures are completely consistent with Embodiment 3. The only difference is the specific component information in the full-stack dependency topology configuration model, such as IP addresses, ports, and component names, which need to be replaced and configured according to the actual environment of the e-commerce system. This shows that the present invention does not depend on a specific business domain and can be applied indiscriminately to any distributed system with multiple component dependencies.

[0171] Example 5 This embodiment details a specific implementation of the business detection interface in this invention. This interface is provided by the associated system being called, such as the People's Bank of China payment system in Embodiment 3, and is a concrete manifestation of its adherence to the dual security constraint principle.

[0172] This business probe interface, deployed as a standalone application programming interface endpoint at the path / business-probe, is located in the associated system. When the probe scheduling engine calls this interface via the Hypertext Transfer Protocol GET method, the internal execution logic is as follows.

[0173] First, the interface performs a series of internal health checks, which include at least the validation of its own core dependent components. For example, it attempts to perform a read-only heartbeat query, such as SELECT1FROMDUAL, from the database to verify database readability and sends a lightweight probe request to its own downstream dependencies.

[0174] To implement dual security constraints, the implementation of this interface includes the following technical safeguards.

[0175] First, database permission isolation. When connecting to the database, this interface does not use the application's own general account, but a specially created database account with restricted permissions, such as probe_user. This account is configured at the database management system level to have SELECT, INSERT, UPDATE, and DELETE permissions for dedicated probe tables, such as probe_healthcheck; and only SELECT permissions or no permissions at all for business data tables. This prevents accidental modification of business data by the probe logic from the database management system kernel level, and cannot be bypassed even if there are defects in the application code.

[0176] Second, runtime verification of the probe. In the application layer code of the interface, a verification module is injected through aspect-oriented programming or middleware technology. This module maintains a whitelist of dedicated table names for probing, containing only `probe_healthcheck`, and intercepts all upcoming database write operations, including INSERT, UPDATE, and DELETE. After parsing the target table name, it compares it with the whitelist. If the target table is not in the whitelist, the write operation is immediately intercepted and a predefined exception is thrown, while an audit log containing information such as time, operation type, and target table name is generated. If the target table is in the whitelist, the operation is allowed.

[0177] After completing all internal checks, the interface aggregates the status of each sub-check item and generates a response body in a simplified JS object format according to a standardized contract. For example, a successful probe response includes the following fields: The overall availability conclusion field has a value of UP, indicating normal operation. The timestamp field has a value of 2026-01-20T10:23:45Z, in ISO8601 format. The message field has a value of an empty string. The details array contains four sub-check items named database_read, database_write, cache, and downstream_payment, all with a status of UP and delays of 8 milliseconds, 15 milliseconds, 3 milliseconds, and 45 milliseconds respectively. All of these message fields are empty.

[0178] The definitions of each field are as follows: The `overall` field is a string type, required, and can be UP, DEGRADED, or DOWN. The aggregation rule is: if any sub-item is DOWN, then `overall` is DOWN; if no sub-item is DOWN but DEGRADED exists, then `overall` is DEGRADED; if all are UP, then `overall` is UP. The `timestamp` field is the response generation time in ISO8601 format. The `message` field is an optional supplementary description and must not contain any business-sensitive data. The `details` array must contain at least one sub-check item element. Each element contains `name` (required), `status` (required), `latency_ms` (optional), and `message` (optional). The response body must not contain any business-sensitive data. Finally, the interface returns a Hypertext Transfer Protocol (HTTP) response with a 200 status code to the probe scheduling engine.

[0179] For historical detection records accumulated in the detection-specific table, the associated system performs periodic cleanup through a background scheduled task, which is executed every 10 minutes by default, deleting all records whose timestamps exceed the preset expiration time and are 5 minutes old by default.

[0180] It should be noted that the specific values ​​described in the above embodiments, such as various time consumption thresholds, duration thresholds, weight coefficients, and score coefficients, are all illustrative examples. Those skilled in the art can flexibly configure and adjust them according to the actual system's performance requirements and operation and maintenance strategies. These adjustments do not depart from the core ideas and protection scope of this invention.

[0181] Based on the description of the above embodiments, the core of the technical solution claimed by this invention lies in providing a systematic, end-to-end availability assurance method and system that starts from full-stack dependent topology configuration, through hierarchical monitoring and collaborative diagnosis, and ultimately achieves accurate alarms and intelligent operation and maintenance. Any equivalent substitutions, modifications, or improvements made to the technical solution of this invention without departing from the spirit and principles defined by this invention, such as adjusting specific threshold parameters, replacing similar databases or middleware components, or changing the specific channels for alarm push, shall fall within the protection scope of this invention.

Claims

1. A method for hierarchical monitoring of the entire chain of distributed applications, characterized in that, include: Build and store a full-stack dependency topology configuration model, which includes each component type and its key indicator set and key dependency tags. The component types include their own application types, infrastructure component types and related system types. Receive monitoring trigger command and load the corresponding configuration model according to the target application identifier; Based on the configuration information of its own application type in the configuration model, perform the first level of monitoring on the target application itself; After the first level of monitoring is completed, the second level of monitoring is executed in parallel based on the configuration information of the infrastructure component types in the configuration model to detect the availability of various infrastructure components. After the second-level monitoring is completed, business-level availability detection is performed based on the configuration information of the associated system type in the configuration model to obtain the third-level monitoring results; Summarize monitoring results at all levels and perform collaborative diagnosis: Perform multi-dimensional indicator judgment and map component status level based on the differentiated key indicator set corresponding to each component type; when any component is marked as a critical dependency and the key indicator is abnormal, trigger a full-link veto; maintain a status duration counter for each component and perform bidirectional dynamic correction of the component score coefficient based on the status duration; calculate a comprehensive score and map it to the health level based on the corrected score coefficient and component weight coefficient. The alarm level is determined based on the health level and the triggering of the veto, and alarm processing is executed accordingly. Output monitoring reports and persist them to memory.

2. The distributed application end-to-end hierarchical monitoring method according to claim 1, characterized in that, The method also includes real-time detection and hot updating of configuration changes: After startup, it continuously sends long polling requests to the configuration management terminal, and the long polling requests carry the configuration version number in the current memory cache. The configuration management terminal compares the configuration version number with the latest configuration version number in the storage layer. If the version numbers are different, it immediately returns the complete configuration data of the latest version. After the current probe batch completes, the configuration object reference in memory is replaced with an atomic operation. The new configuration takes effect on subsequent probe batches, while probe batches that are currently being executed continue to use the old configuration until they are completed. The configuration change perception and hot update process and the monitoring execution process run independently and in parallel.

3. A distributed application end-to-end hierarchical monitoring method according to claim 1 or 2, characterized in that, The full-stack dependency topology configuration model includes at least the following configuration information: application identifier, component identifier, component type and subtype, probe connection parameters, business probe interface Uniform Resource Locator, weight coefficient, key dependency marker, and key indicator set; the infrastructure component type further includes database type, cache middleware type, and message middleware type; the subtype is used to identify specific component types to route to the corresponding probe executor; the key indicator set includes the key indicator names and anomaly judgment rules on which the component is vetoed.

4. A distributed application end-to-end hierarchical monitoring method according to claim 1 or 2, characterized in that, The first level of monitoring specifically includes: Based on the configuration information of the type "self-application" in the configuration model, a probe request is initiated to the probe interface exposed by the target application itself and a response is received. If the Hypertext Transfer Protocol status code is a success response code and the overall status field in the response body is normal, and the response time does not exceed the preset first time threshold, then the target application is determined to be in normal status. If the Hypertext Transfer Protocol status code is a success response code but the response time exceeds the first time threshold, it is determined to be a performance degradation. If the Hypertext Transfer Protocol status code is not a success response code, or the overall status field in the response body is unavailable, or the request timed out, or a connection error occurred, then it is determined to be unavailable.

5. A distributed application end-to-end hierarchical monitoring method according to claim 1 or 2, characterized in that, In Level 2 monitoring: The specific steps for probing database type components are as follows: send a probe structured query statement to the target database instance through an established long connection or connection pool and receive the response; if the execution is successful and the time taken does not exceed the preset second time threshold, it is considered normal; if the execution is successful but the time taken exceeds the second time threshold, it is considered performance degradation; if the execution fails, throws an exception, times out, or a connection exception occurs, it is considered unavailable. The detection of cache middleware type components is as follows: send a probe command to the target cache instance through an established long connection or connection pool and receive a response; if the response is successful and the time consumed does not exceed the preset third time consumption threshold, it is determined to be normal; if the response is successful but the time consumed exceeds the third time consumption threshold or the read / write integrity verification fails, it is determined to be performance degradation; if there is no response, an error is returned, the connection times out, or a connection abnormality occurs, it is determined to be unavailable. The detection of message middleware type components is as follows: send a metadata retrieval request to the target message cluster through an established long connection or connection pool and receive a response; if the retrieval is successful and the time taken does not exceed the preset fourth time threshold, it is determined to be normal; if the retrieval is successful but the time taken exceeds the fourth time threshold or the message sending and receiving verification fails, it is determined to be performance degradation; if the retrieval fails, the request times out, or a connection abnormality occurs, it is determined to be unavailable.

6. A distributed application end-to-end hierarchical monitoring method according to claim 1 or 2, characterized in that, In the third-level monitoring, the business detection interface of the associated system follows dual security constraints: by default, the interface only performs read-only query operations; if write capability needs to be verified, insert, update or delete operations are allowed, but the data objects operated on must be and are limited to the pre-established detection-specific table. The determination of the third-level monitoring result is as follows: If the Hypertext Transfer Protocol (HTTP) status code is a successful response code, and the overall availability conclusion field in the response body is normal, and the status of each key sub-check item is normal, and the response time does not exceed the preset fifth time threshold, then the associated system is determined to be in normal status; if the HTTP status code is a successful response code and the overall availability conclusion field is normal, but the response time exceeds the fifth time threshold or there is a non-key sub-check item with a performance degradation status, then it is determined to be performance degradation; if the HTTP status code is not a successful response code, or the overall availability conclusion field is unavailable, or any key sub-check item is unavailable, or the request times out, or a connection error occurs, or the response format does not conform to the preset specification, then it is determined to be unavailable.

7. A distributed application end-to-end hierarchical monitoring method according to claim 1 or 2, characterized in that, The component score coefficient is dynamically adjusted bidirectionally based on the state duration, specifically as follows: For components in a state of performance degradation, the score coefficient decreases progressively as the duration of performance degradation increases; For components that are unavailable, the score coefficient is zero, and a persistent failure penalty value is applied based on the duration of unavailability. When a component recovers from an abnormal state to a normal state, it enters a recovery observation period. During the recovery observation period, it participates in the scoring calculation with a preset transition score coefficient. After the observation period ends and the condition remains stable, it recovers to the full score coefficient.

8. A distributed application end-to-end hierarchical monitoring method according to claim 7, characterized in that, This also includes regression in status during the recovery observation period: If a component changes from a normal state to a performance degradation state during the recovery observation period, the score coefficient will be rolled back to the score coefficient corresponding to the performance degradation state, and the state duration counter will be counted from the cumulative duration of the last time the component entered the performance degradation state. If a component changes from a normal state to an unavailable state during the recovery observation period, the score coefficient is reset to zero, and the state duration counter starts counting from the cumulative duration of the component's last unavailable state.

9. A distributed application end-to-end hierarchical monitoring method according to claim 1 or 2, characterized in that, The comprehensive score is calculated and mapped to the health level, specifically as follows: Calculate the base weighted total score, which is equal to the sum of the products of the weight coefficients of all components and the score coefficient of that component after duration correction; Calculate the persistent failure penalty value, which is equal to the sum of the products of the weight coefficients of all components currently in an unavailable state and the duration penalty coefficient of that component at the current moment; The overall score is equal to the basic weighted total score minus the persistent failure penalty value. If the result is less than zero, it is set to zero. When a full-link veto is triggered, the overall score is directly set to zero. The comprehensive score is mapped to a corresponding health level based on a preset scoring range. The health level includes at least a healthy level, a sub-healthy level, an abnormal level, and a severely abnormal level.

10. A distributed application end-to-end hierarchical monitoring system, characterized in that, include: The full-stack dependency topology configuration module is used to pre-build and store a full-stack dependency topology configuration model for each target application. The model includes each component type and its key indicator set and key dependency tags. The component types include the application type itself, infrastructure component types and related system types. The detection scheduling engine is used to receive monitoring trigger instructions through a standardized interface, load the corresponding configuration model from the full-stack dependency topology configuration module into the memory cache according to the target application identifier, and schedule the detection executors at all levels to execute monitoring tasks according to the configuration information. The self-application detector is used to perform first-level monitoring on the target application itself based on the configuration information of the self-application type in the configuration model. The infrastructure probe executor collection includes multiple probe executor plugins for database types, cache middleware types, and message middleware types, which are used to execute second-level monitoring in parallel and obtain second-level monitoring results. The associated system business detection executor is used to call the business detection interface exposed by the associated system to perform business-level availability detection based on the configuration information of the associated system type in the configuration model, and obtain the third-level monitoring results; The results aggregation and diagnosis module is used to summarize monitoring results at all levels, perform multi-dimensional indicator judgment, component status level mapping, full-link veto judgment, bidirectional dynamic correction of component score coefficient, weighted aggregation calculation, and continuous fault penalty, and generate comprehensive score and health level. The alarm notification module is used to push alarm notifications through differentiated alarm channels based on the health level and the triggering of the veto. The monitoring report generation module is used to output structured monitoring reports and persist them.

Images