Method for testing performance of service continuity in master-backup switching

Abstract

The application provides a service continuity performance test method in master-back switching, and belongs to the technical field of data information processing. The method comprises the following steps: defining a switching pipeline; using infrastructure to manage infrastructure resources of master and back-up sites, and uniformly managing application configurations through a configuration management database; integrating a data consistency check task in the switching pipeline, and performing data consistency check at key nodes; if the check fails, the pipeline is blocked; using a double-track intelligent decision engine to decide a switching time in combination with a preset service time window and a risk strategy; switching part of non-key business traffic from the master site to the back-up site, synchronously running an automated business verification script, and after verification, gradually expanding the proportion of switched business traffic until complete switching; and taking security scanning and compliance checkpoints as quality access control and embedding them in each standard stage of the switching pipeline, and if the current stage fails to pass the security scanning and compliance checkpoints, the next stage of the switching pipeline is blocked.

Description

Technical Field

[0001] This application belongs to the field of data information processing technology, and specifically relates to a method for testing service continuity performance during primary / standby switchover. Background Technology

[0002] Critical information systems in the banking industry, particularly core accounting, payment, and credit management systems, have extremely high requirements for continuous operation, data consistency, and disaster recovery capabilities. According to relevant regulations, commercial banks should establish a comprehensive business continuity management system, regularly conduct off-site disaster recovery switchover drills and continuity stress tests, and ensure that critical business operations can be restored within a specified timeframe in the event of regional disasters, infrastructure failures, or major security incidents, without data loss or inconsistency.

[0003] Traditional off-site disaster recovery switchover or continuous stress testing typically relies on manually developing switchover plans, manually executing multiple switchover steps, and manually verifying the results, which has the following technical drawbacks: First, it relies heavily on manual intervention. The switchover process involves multiple stages, including infrastructure preparation, configuration verification, data synchronization confirmation, service startup and shutdown, traffic switching, and business verification. Each stage requires manual operation by operations and maintenance personnel or execution through scattered scripts. Taking a typical off-site switchover of a core system as an example, the operation steps usually exceed 200, involving multiple teams such as the system group, network group, database group, application group, and business verification group. Manual operation is complex and prone to errors, and the preparation period for a single switchover drill often lasts for several weeks.

[0004] Second, the process is lengthy and lacks standardization. Existing switchover processes are typically executed by multiple teams in a decentralized manner, with the connection between steps relying on offline communication and coordination, making it difficult to form a reusable and repeatable standard process. Each time a drill is conducted, the team needs to re-examine the switchover steps, reconfirm dependencies, and redistribute execution responsibilities, making it difficult to accumulate and pass on drill experience. Furthermore, the switchover processes executed by different teams and individuals vary, making it difficult to ensure the consistency of switchover quality.

[0005] Third, ensuring environmental consistency is difficult. Computing resources, storage resources, network configurations, middleware versions, and application artifacts at the primary and backup sites are prone to migration. Since primary and backup sites are typically maintained by different teams, and the disaster recovery site is often in standby mode, frequent changes to the primary site often fail to be synchronized to the disaster recovery site in a timely manner, leading to the exposure of hidden faults after the switchover. Common migration issues include inconsistent operating system kernel parameters, inconsistent middleware configuration file versions, inconsistent database parameters, and inconsistent application artifact versions. These migration issues are difficult to detect under normal circumstances, but can cause service startup failures or operational anomalies during a switchover.

[0006] Fourth, data consistency verification has a single dimension. Traditional solutions often only verify the replication link status or a small amount of table data, lacking a comprehensive consistency verification mechanism covering the physical, logical, business, and metadata layers. For example, only verifying whether the database replication latency is within the threshold, without verifying key business semantics such as total score balance, internal account netting, and serial number continuity at the business level, may lead to hidden errors in business data after the switchover, affecting the accuracy of accounting.

[0007] Fifth, the switching decision relies on human experience. The timing and conditions for switching are usually determined manually based on experience, making it difficult to integrate multi-dimensional information such as system health, business metrics, and replication latency in real time. In the event of sudden failures, the response speed and accuracy of manual decision-making cannot be guaranteed, and the different judgment standards of different decision-makers may lead to decision delays or errors.

[0008] Sixth, there is a lack of end-to-end observability and automatic rollback capabilities. Traditional switchovers lack end-to-end monitoring of switchover status, anomaly location, and the scope of business impact. Monitoring data is scattered across multiple independent tools, requiring operations personnel to switch between multiple consoles, making it difficult to gain a holistic understanding of the switchover process. Once an anomaly occurs, the rollback process also relies on manual intervention, which is risky, time-consuming, and may introduce new data inconsistency issues.

[0009] Therefore, there is an urgent need for a method for switching between primary and backup sites in a different location for critical information systems in the banking industry, which can achieve process standardization, environmental consistency, multi-dimensional data verification, intelligent decision-making, process observability, and automated rollback. Summary of the Invention

[0010] The purpose of this application is to provide a method for testing service continuity performance during primary / standby switchover.

[0011] According to a first aspect of this application, a method for testing service continuity performance during primary / standby switchover is provided, comprising: The entire process of switching between primary and backup sites of the business system is modeled as multiple different standardized stages, and these multiple different standardized stages are defined as a switching pipeline that is executed repeatedly. Infrastructure as Code is used to manage the infrastructure resources of primary and backup sites, and application configurations are managed uniformly through a configuration management database to ensure the consistency of environment and configuration before and after the primary and backup site switchover. Integrate a data consistency verification task into the switching pipeline, perform data consistency verification at critical nodes, and block the pipeline if the verification fails. Collect real-time monitoring data from primary and backup sites, utilize a dual-track intelligent decision engine, and combine preset business time windows and risk strategies to determine the timing of the switchover and trigger the switchover pipeline. After the switchover pipeline is triggered, a portion of the non-critical business traffic will be switched from the primary site to the disaster recovery site. Simultaneously, an automated business verification script will be run. After the verification is passed, the proportion of business traffic switched will be gradually increased until the switchover is complete. The unified observability platform enables real-time tracking and visualization of application performance, infrastructure status, and / or business transaction links during the switchover process. Security scans and compliance checkpoints are built into each standard stage of the switchover pipeline as quality gates. If the current stage fails to pass the security scan and compliance checkpoint, it will block the entry into the next stage of the switchover pipeline. The preset rollback pipeline is a twin of the forward switching pipeline. When the rollback trigger index is detected to exceed the rollback threshold, the preset rollback pipeline is automatically triggered to restore the traffic and environment to the state before the switchover.

[0012] In an optional implementation, the use of Infrastructure as Code (IaC) to manage the infrastructure resources of the primary and backup sites, and the unified management of application configurations through a configuration management database to ensure consistency of environment and configuration before and after the primary and backup site switchover, includes: The primary and backup dual-site mirroring is modeled using Infrastructure as Code (ICC). The infrastructure resources of the primary and backup sites are uniformly described using a declarative template, and the common definitions and site differences of the primary and backup sites are separated by site variable parameterization in the declarative template. Calculate hash fingerprints for the deployment artifact packages and application configuration items, and write the hash fingerprints as unique identifiers for the deployment artifact packages and configuration items into the configuration management database. When the primary and backup sites load the deployment artifact packages and configuration items, they compare the fingerprints. If the fingerprints do not match, the switchover is blocked. The application configuration is divided into three layers: baseline configuration, site configuration, and runtime configuration. A complete configuration list for the primary and backup sites is generated in advance before the switch. The baseline configuration of the primary and backup sites is completely consistent and centrally managed for version control. The site configuration is independently versioned according to the primary and backup site dimensions. The runtime configuration is dynamically injected when the primary and backup sites switch. Periodically scan the differences between the actual operating infrastructure resources and the declared state of the configuration management database of the primary and backup sites, and automatically trigger configuration write-back or alarm blocking for any unauthorized changes detected based on the differences between the infrastructure resources and the declared state of the configuration management database. Before the pipeline switching execution phase, a consistency check is automatically performed, and the pipeline enters the next phase after the consistency check passes. The consistency check includes at least one of the following: declarative template version number consistency check, fingerprint comparison, and configuration item difference whitelist check. Changes to configuration items or infrastructure resources are recorded in the version control system via transaction commits. After generating a unique change identifier, it is used as a rollback anchor point along with a timestamp snapshot. When a pipeline switchover fails, either the primary or backup site is rolled back to its pre-switch state based on the rollback anchor point.

[0013] In an optional implementation, integrating a data consistency verification task into the switching pipeline, performing data consistency verification at critical nodes, and blocking the pipeline if the verification fails, includes: Verify the integrity of the underlying database replication link and logs of the primary and backup sites; Verify that the data content of the databases of the primary and backup sites is consistent; Verify the data correctness at the business semantic level of the databases of the primary and backup sites; Verify the consistency of the table structures in the databases of the primary and backup sites.

[0014] In an optional implementation, the dual-track intelligent decision engine includes an arbitration fusion unit, a main-track rule engine, and a secondary-track machine learning auxiliary model; the process of collecting real-time monitoring data from primary and backup sites, utilizing the dual-track intelligent decision engine, and combining preset business time windows and risk strategies to determine the switching timing includes: The main track rule engine first checks the triggering status of the forced blocking rule and the forced triggering rule. If neither the forced blocking rule nor the forced triggering rule is triggered, it calculates the switching necessity score and the switching risk score based on real-time monitoring data. Then, it matches the switching necessity score and the switching risk score with the preset decision matrix, and finally outputs the main decision instruction after making corrections using the preset business time window. If the main decision instruction output by the main track rule engine is a deterministic instruction, the arbitration fusion unit determines the switching information based on the main decision instruction; If the main decision instruction output by the main track rule engine is a non-deterministic instruction, the arbitration fusion unit combines the predicted success probability of switching, the predicted rollback probability, the optimal switching timing, and the non-deterministic instruction output by the secondary track machine learning auxiliary model to determine the switching information.

[0015] In an optional implementation, the switching necessity score is obtained by weighted fusion of the current overall health of the main site, the intensity of the deterioration trend, the predicted failure probability, and the intensity of external triggering signals; the switching risk score is obtained by weighted fusion of data synchronization delay risk, disaster recovery site's own health risk, current business load risk, upstream and downstream dependent system risk, historical switching failure rate under the same conditions, and business time window risk.

[0016] In an optional implementation, the step of collecting real-time monitoring data from primary and backup sites, utilizing a dual-track intelligent decision engine, and combining preset business time windows and risk strategies to determine the switching timing, further includes: The real-time monitoring data is then normalized.

[0017] In an optional implementation, the main track rule engine first checks the triggering status of the forced blocking rule and the forced triggering rule. If neither the forced blocking rule nor the forced triggering rule is triggered, then it calculates the switching necessity score and the switching risk score based on real-time monitoring data, including: The real-time monitoring data is processed in parallel using four analysis methods: threshold analysis, trend analysis, predictive analysis, and anomaly detection, to output the current overall health of the main site, the intensity of the deterioration trend, and the predicted failure probability, which are used to calculate the switching necessity score. The threshold analysis involves setting two levels of thresholds for each indicator: an alarm threshold and a circuit breaker threshold. When an indicator exceeds the alarm threshold, it enters an alarm state; when it exceeds the circuit breaker threshold, it enters a dynamic state. The trend analysis is based on extracting N time series sampling points in the most recent T minutes using a sliding window, calculating the trend slope using least squares linear regression, and calculating the goodness of fit. If the goodness of fit exceeds the set goodness threshold, the trend deterioration score is calculated based on the ratio of the trend slope to the preset deterioration threshold. The predictive analysis adopts a dual-track parallel design. The first prediction track extrapolates the remaining time to reach the circuit breaker threshold based on the linear regression model of trend analysis and maps it to the predicted failure probability. The second prediction track uses a two-layer LSTM time series prediction model to output the predicted values ​​and upper bounds of the confidence intervals for each future time step. The highest value of the first prediction track and the second prediction track is taken as the final predicted failure probability. The anomaly detection includes single-index step change detection and multi-index associated anomaly detection. Single-index step change detection compares the difference in mean between two adjacent sliding windows and triggers a change alarm. Multi-index associated anomaly detection matches the current multi-index state combination in real time based on a predefined anomaly index combination pattern rule base, outputs the anomaly detection result, and determines the triggering state of the forced triggering rule.

[0018] In an optional implementation, the four analysis methods dynamically switch between three triggering modes based on the current state of the business system. The three triggering modes include: normal inspection mode, switch preparation mode, and event triggering mode. The normal inspection mode is the default operating state, and the relevant thresholds in this mode use standard values. The handover preparation mode indicates that there is a potential handover risk or that a handover plan is imminent. In this mode, the relevant thresholds are tightened based on the standard values. The event-triggered mode is a mode for setting up sudden events. In this mode, the relevant thresholds are tightened based on the values ​​used in the switching preparation mode.

[0019] In an optional implementation, the step of collecting real-time monitoring data from primary and backup sites, utilizing a dual-track intelligent decision engine, and combining preset business time windows and risk strategies to determine the switching timing, further includes: Set different switching risk scores and mandatory blocking rules for different business time windows.

[0020] In an optional implementation, the step of switching a portion of non-critical business traffic from the primary site to the disaster recovery site, simultaneously running an automated business verification script, and gradually increasing the proportion of switched business traffic until a complete switch is achieved after successful verification includes: A preset grayscale step is obtained, and the step is dynamically selected according to the switching risk level. Different grayscale steps correspond to different traffic ratios. For each grayscale level, execute a sequence of atomic operations including traffic distribution weight adjustment, session persistence processing, health check injection, and data layer read / write separation confirmation; After each grayscale step transition is completed, the observation window is entered. If the automatic rollback condition is triggered in the observation window, the transition pipeline is immediately interrupted and a rollback is executed. At the end of each grayscale step, four types of verification scripts are automatically executed in parallel: core accounting transaction verification, batch and reconciliation verification, concurrency and capacity verification, and cross-site data consistency verification.

[0021] In an optional implementation, after the predefined rollback pipeline is triggered, it is executed in the following order: Capture on-site data at abnormal moments and associate it with the identifier of the pipeline being switched and store it. By modifying the global load balancer's resolution policy, domain name resolution is gradually switched back to the main site. By updating the gateway's routing rules, traffic entering the gateway is proportionally switched back. For established long connections, the connection eviction mechanism forces clients to reconnect to the main site. Set the service instances started on the disaster recovery site to no longer receive residual traffic, restore the service instances on the main site that were stopped or downgraded during the switchover to normal operation, and restore the list of service instances in the service discovery registry to the snapshot version before the switchover. Leveraging the infrastructure-as-code and configuration management database capabilities, changes to application configurations, network policies, and middleware configurations during the switchover process can be rolled back to the versions before the switchover. For scenarios where no business writes have occurred at the disaster recovery site, the synchronization direction of the primary and backup databases will be restored to primary to backup. For scenarios where business writes have occurred at the disaster recovery site, incremental data will be flowed back to the primary site through transaction log parsing, and data consistency verification will be performed. The automated business verification script is invoked to compare the deviations of the key indicators after rollback with the corresponding indicators before the switch. Once all indicators converge to the set range of the corresponding indicators, the rollback is deemed successful.

[0022] A second aspect of this application provides a service continuity performance testing apparatus during primary / standby switchover, comprising: The pipeline definition module is configured to model the entire process of switching between primary and backup sites of the business system into multiple different standardized stages, and define the multiple different standardized stages as a switching pipeline that is executed repeatedly. The management module is configured to manage the infrastructure resources of the primary and backup sites using Infrastructure as Code (IaC), and to uniformly manage application configurations through a configuration management database, ensuring the consistency of the environment and configuration before and after the primary and backup site switchover. The verification module is configured to integrate data consistency verification tasks into the switching pipeline, perform data consistency verification at critical nodes, and block the pipeline if the verification fails. The decision-making module is configured to collect real-time monitoring data from the primary and backup sites, utilize a dual-track intelligent decision engine, and combine preset business time windows and risk strategies to determine the timing of the switchover and trigger the switchover pipeline. The traffic switching module, when configured to trigger the switching pipeline, will switch a portion of the non-critical business traffic from the main site to the disaster recovery site, and simultaneously run an automated business verification script. After the verification is passed, the proportion of business traffic switched will be gradually increased until the switch is complete. The observation module is configured to track and visualize application performance, infrastructure status and / or business transaction links in real time during the switchover process through a unified observability platform. Security scans and compliance checkpoints are built into each standard stage of the switchover pipeline as quality gates. If the current stage fails to pass the security scan and compliance checkpoint, it will block the entry into the next stage of the switchover pipeline. The rollback module is configured as a predefined rollback pipeline twin with the forward switching pipeline. When the rollback trigger indicator is detected to exceed the rollback threshold, the preset rollback pipeline is automatically triggered to restore the traffic and environment to the state before the switch.

[0023] Compared with related technologies, the technical solution of this application has at least the following advantages: This application constructs a standardized switchover pipeline, decomposing the entire remote switchover process into automated stages such as environment preparation, data synchronization and verification, service switching, business verification, traffic switching, observation, and rollback. It achieves version control and drift detection of the primary and backup site environments and configurations through Infrastructure as Code and a configuration management database. At critical switchover nodes, it performs four-layer data consistency verification: physical layer, logical layer, business layer, and metadata layer. It constructs a dual-track intelligent decision engine with a main track of rules and a secondary track of machine learning-assisted models, and an arbitration fusion unit adjudicating the decision. This engine automatically determines the switchover timing based on real-time monitoring data and preset risk strategies. It employs a gray-scale step-by-step strategy to execute gradual traffic switching while simultaneously running automated business verification scripts. A unified observability platform enables end-to-end tracking of the entire switchover process and embedding security and compliance access control. A predefined rollback pipeline, twinned with the configuration and forward switchover pipeline, enables automatic rollback in abnormal scenarios. After the switchover is completed, the switchover process is iteratively optimized based on a full-process data review. This application realizes the process-oriented, automated, intelligent and observable nature of business continuity stress testing for important information systems in the banking industry, which significantly improves switchover reliability, shortens RTO, and reduces switchover risk. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating the service continuity performance testing method during primary / standby switchover provided in this application.

[0025] Figure 2 This is a detailed flowchart illustrating the service continuity performance testing method during primary / standby switchover provided by this application.

[0026] Figure 3 This is a flowchart illustrating the intelligent decision engine in the business continuity performance testing method for primary / standby switchover provided in this application.

[0027] Figure 4 This is a structural block diagram of the service continuity performance testing device during primary / standby switchover provided by this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] To address the problems in existing technologies for off-site disaster recovery switching and continuous stress testing of critical information systems in the banking industry, such as high reliance on manual processes, lengthy and unstandardized processes, difficulty in ensuring environmental consistency, single-dimensional data consistency verification, reliance on human experience for switching decisions, and lack of full-process observability and automatic rollback capabilities, this application aims to provide a business continuity performance testing solution for primary-backup switching. This solution achieves process-oriented, automated, intelligent, and observable business continuity stress testing of critical information systems by constructing a standardized switching pipeline, implementing version control of infrastructure and configurations, adopting four-layer data consistency verification, building a dual-track intelligent decision engine, implementing progressive traffic switching and business verification, establishing a full-link observability and security compliance embedded mechanism, configuring a predefined rollback pipeline, and conducting post-switching review and process optimization. This improves the reliability, security, and efficiency of switching.

[0030] Based on the above analysis, and referring to... Figure 1 As shown, this application exemplarily provides a service continuity performance testing method during primary / standby switchover, including: In step S101, the entire process of switching between primary and backup sites of the business system is modeled as multiple different standardized stages, and the multiple different standardized stages are defined as a switching pipeline that is executed repeatedly. In step S102, the infrastructure resources of the primary and backup sites are managed using Infrastructure as Code (ICC), and the application configuration is uniformly managed through the configuration management database to ensure the consistency of the environment and configuration before and after the primary and backup site switchover. In step S103, a data consistency verification task is integrated into the switching pipeline. Data consistency verification is performed at critical nodes, and the pipeline is blocked if the verification fails. In step S104, real-time monitoring data of the primary and backup sites are collected. Using the dual-track intelligent decision engine, combined with the preset business time window and risk strategy, the timing of the switchover is determined and the switchover pipeline is triggered. In step S105, after the switchover pipeline is triggered, a portion of the non-critical business traffic is switched from the main site to the disaster recovery site. The automated business verification script is run synchronously. After the verification is passed, the proportion of the switched business traffic is gradually increased until the switchover is complete. In step S106, the application performance, infrastructure status and / or business transaction links during the switching process are tracked and visualized in real time through a unified observability platform. Security scanning and compliance checkpoints are built into each standard stage of the switching pipeline as quality gates. If the current stage fails to pass the security scanning and compliance checkpoints, it will block the entry into the next stage of the switching pipeline. In step S107, a predefined rollback pipeline, which is a twin of the forward switching pipeline, is preset. When the rollback trigger index is detected to exceed the rollback threshold, the preset rollback pipeline is automatically triggered to restore the flow and environment to the state before the switch.

[0031] The following example uses a banking system for reference. Figure 2 As shown, the method provided in this application is described in detail, and its steps are as follows: S201, Switching Blueprints and Pipeline Definitions The entire offsite migration process is modeled as standardized stages including environment preparation, data synchronization and verification, service switching, business verification, traffic switching, observation, and rollback. These stages are defined as a repeatable migration pipeline within the Continuous Integration / Continuous Deployment (CI / CD) platform. The migration blueprint is a top-level abstraction of the entire offsite migration process, describing the task node list, execution order, parallel strategies, timeout settings, failure handling rules, rollback trigger conditions, and the responsible team and emergency contact person for each stage. The migration blueprint is stored in the version control system as a declarative configuration file, supporting version management and difference comparison.

[0032] S202, Version Control of Infrastructure and Configuration Infrastructure as Code (IaC) is used to manage the infrastructure resources of primary and backup sites, and application configurations are managed uniformly through the Configuration Management Database (CMDB) to ensure consistency between the environment and configuration before and after the switchover.

[0033] In some optional embodiments, Infrastructure as Code (IaC) is used to manage the infrastructure resources of the primary and backup sites, and application configurations are uniformly managed through a configuration management database to ensure consistency of environment and configuration before and after the primary and backup site switchover, including: The primary and backup dual-site mirroring is modeled using Infrastructure as Code (ICC). The infrastructure resources of the primary and backup sites are uniformly described using a declarative template, and the common definitions and site differences of the primary and backup sites are separated by site variable parameterization in the declarative template. Calculate hash fingerprints for the deployment artifact packages and application configuration items, and write the hash fingerprints as unique identifiers for the deployment artifact packages and configuration items into the configuration management database. When the primary and backup sites load the deployment artifact packages and configuration items, they compare the fingerprints. If the fingerprints do not match, the switchover is blocked. The application configuration is divided into three layers: baseline configuration, site configuration, and runtime configuration. A complete configuration list for the primary and backup sites is generated in advance before the switch. The baseline configuration of the primary and backup sites is completely consistent and centrally managed for version control. The site configuration is independently versioned according to the primary and backup site dimensions. The runtime configuration is dynamically injected when the primary and backup sites switch. Periodically scan the differences between the actual operating infrastructure resources and the declared state of the configuration management database of the primary and backup sites, and automatically trigger configuration write-back or alarm blocking for any unauthorized changes detected based on the differences between the infrastructure resources and the declared state of the configuration management database. Before the pipeline switching execution phase, a consistency check is automatically performed, and the pipeline enters the next phase after the consistency check passes. The consistency check includes at least one of the following: declarative template version number consistency check, fingerprint comparison, and configuration item difference whitelist check. Changes to configuration items or infrastructure resources are recorded in the version control system via transaction commits. After generating a unique change identifier, it is used as a rollback anchor point along with a timestamp snapshot. When a pipeline switchover fails, either the primary or backup site is rolled back to its pre-switch state based on the rollback anchor point.

[0034] In this embodiment, step S202 includes the following sub-steps: S2021, Infrastructure as Code Dual-Site Mirroring Model: A declarative template is used to uniformly describe the resource topology of the primary and backup sites, including computing, storage, network, and middleware. The template separates common definitions and site differences between the primary and backup sites (such as IP ranges, primary environment identifiers, and backup environment identifiers) through site variable parameterization, ensuring that the primary and backup sites have the same structural origin and controllable parameters. The declarative template is described using Infrastructure as Code tools such as Terraform or Ansible, and the template files are managed under a version control system, with each change generating a unique version number.

[0035] S2022, Unified Artifacts and Configuration Fingerprinting: An immutable hash fingerprint (SHA-256) is calculated for all deployment artifact packages and configuration items, and this fingerprint is written to the Configuration Management Database (CMDB) as a unique identifier. During primary and standby site loading, the fingerprints are compared; if the fingerprints do not match, the switchover pipeline is blocked. This fingerprinting mechanism ensures the immutability and traceability of deployment artifacts and configurations. Any minor change will result in a fingerprint change, which will be detected by the consistency check gate.

[0036] S2023, Three-Layer Separation Model: Application configuration is divided into three layers: Baseline configuration, Site-specific configuration, and Runtime configuration. Baseline configuration consists of configuration items that are completely identical between the primary and standby servers, with centralized version management. Site configuration includes differences between the primary and standby servers (such as database connection strings and VIP addresses), and is independently versioned by site. Runtime configuration consists of configuration items dynamically injected by the orchestration engine during failover (such as the currently active site identifier and read / write separation strategy). A complete configuration list for the target site is generated before failover to ensure the completeness and correctness of configuration items during the failover.

[0037] S2024, Bidirectional Drift Detection: Periodically scans for differences between the actual running resources of the primary and backup sites and the declared state of the CMDB (Configuration Management Database), generating drift reports. Drift detection covers multiple dimensions, including operating system parameters, middleware configurations, network policies, security group rules, and application configuration files. Unauthorized changes detected automatically trigger configuration rewrites or alarm blocking, ensuring convergence of the declared states of the environments and repositories on both sides before the switchover. The drift detection frequency is a full scan once daily, increasing to an incremental scan every 4 hours within 48 hours before the switchover.

[0038] S2025. Pre-switch consistency verification access control: In the pre-switch execution phase of the pipeline, the following verifications are automatically performed, and the next phase can only proceed if all of them pass: Template version number consistency verification, confirming that the IaC template version number loaded by the primary and backup sites is consistent; Artifact fingerprint comparison, confirming that the hash of the artifact deployed on the primary and backup sides is equal to the hash declared in the repository; Configuration item difference whitelist verification, only allowing expected differences in the site configuration layer, and all differences between the baseline configuration and the runtime configuration are considered as verification failures.

[0039] S2026, Transactional Changes and Rollback Anchors: Each configuration or infrastructure change is recorded in the repository via a transactional commit, generating a unique change ID and timestamp snapshot as a rollback anchor. In the event of a failover, either the primary or standby side can be precisely rolled back to its pre-switch state based on the anchor. The transactional change mechanism ensures the atomicity and rollback capability of all change operations, avoiding inconsistencies caused by partial successes and partial failures in the environment.

[0040] S203, Data Synchronization and Consistency Guarantee This step comprises two levels: the first level is routine data synchronization, where the primary site continuously synchronizes data to the disaster recovery site through underlying database replication mechanisms (such as Oracle DataGuard, MySQL master-slave replication, etc.) to maintain near real-time consistency between the disaster recovery site and the primary site; the second level is pre-switch data consistency verification. At key nodes after the decision engine triggers the switch pipeline but before the actual traffic switch is executed, data consistency verification is performed across four layers: physical layer, logical layer, business layer, and metadata layer. If verification fails, the pipeline is blocked to prevent switchover from occurring due to inconsistent data. Therefore, in some embodiments, integrating data consistency verification tasks into the switch pipeline and performing data consistency verification at key nodes, with pipeline blocked if verification fails, includes: Verify the integrity of the underlying database replication link and logs of the primary and standby sites, including replication position verification, replication delay verification, and log continuity verification. The replication position verification verifies the SCN of Oracle DataGuard, the LSN of DM database, and the GTID of MySQL to confirm that the position applied by the standby database catches up with the latest commit position of the primary database. Verify that the data content of the databases of the primary and backup sites is consistent; Verify the data correctness at the business semantic level of the databases of the primary and backup sites; Verifying the consistency of table structures in the databases of the primary and backup sites includes ensuring consistency of DDL versions, indexes, constraints, and triggers between the primary and backup sites, as well as synchronizing the current value of sequences and auto-increment values ​​between the primary and backup sites.

[0041] The four-layer verification mechanism is explained in detail below, including the following steps: S2031, Physical Layer Consistency Verification: Verifies the integrity of the underlying replication link and logs between the primary and standby databases. Specifically, this includes: Replication site verification, such as verifying the SCN (System Change Number) of Oracle DataGuard, the LSN (Log Sequence Number) of DM database, and the GTID (Global Transaction Identifier) ​​of MySQL, to confirm whether the applied site of the standby database has caught up with the latest commit site of the primary database; Replication latency verification, such as verifying the RPO (Recovery Point Objective) metric, confirms whether the transaction latency between the primary and backup is within the set latency threshold, such as setting the latency threshold to 1 second; Log continuity verification, such as verifying whether archived logs or redo logs are continuous without gaps and whether there are any data gaps that have not been transmitted.

[0042] S2032, Logical Layer Consistency Verification: Verifies whether the data content of the primary and backup databases is consistent. Specifically, this includes: row count verification, comparing the total number of rows in key tables (such as account tables and transaction log tables) using COUNT(*); checksum comparison, calculating the MD5 hash of tables or partitions and comparing the primary and backup data; and sampled data comparison, sampling and comparing field-level content by primary key, suitable for fast verification of very large tables.

[0043] S2033. Business Layer Consistency Verification: This verifies the correctness of data at the business semantic level, which is the most critical dimension for the banking industry. Business layer verification includes, but is not limited to: general ledger balance verification (verifying whether the debit and credit entries in the general ledger are balanced); internal account netting verification (verifying whether the netting of internal accounts is zero); and sequence number continuity verification (verifying whether transaction serial numbers and voucher numbers are consecutive and without skipped numbers). Business layer consistency verification is directly related to the correctness of accounting and is a key guarantee for the normal operation of business after the switchover.

[0044] S2034, Referential Integrity and Metadata Consistency Verification: Verifies table structure consistency, including but not limited to: DDL version, indexes, constraints, and trigger consistency between primary and backup; verifies sequence and auto-increment value consistency, ensuring that the current value of the verification sequence is synchronized between primary and backup. Metadata consistency verification avoids application anomalies caused by differences in table structure.

[0045] S204, Intelligent Switching Decision and Triggering Based on the analysis of real-time monitoring data such as system health, business indicators, and data synchronization latency of the main production site and disaster recovery site, combined with preset business time windows and risk strategies, the intelligent decision engine automatically judges or assists in the decision-making of the best switching time and triggers the switching pipeline.

[0046] In some embodiments, the dual-track intelligent decision engine includes an arbitration fusion unit, a main-track rule engine, and a secondary-track machine learning auxiliary model; the process of collecting real-time monitoring data from the main site and disaster recovery site, and using the dual-track intelligent decision engine in conjunction with preset business time windows and risk strategies to determine the timing of the switchover includes: The main track rule engine first checks the triggering status of the forced blocking rule and the forced triggering rule. If neither the forced blocking rule nor the forced triggering rule is triggered, it calculates the switching necessity score and the switching risk score based on real-time monitoring data. It then matches the switching necessity score and the switching risk score with a preset decision matrix and makes corrections using a preset business time window before finally outputting a preliminary decision instruction. If the preliminary decision instruction output by the main track rules engine is a deterministic instruction, the arbitration fusion unit determines the switching information based on the preliminary decision instruction; If the initial decision instruction output by the main track rules engine is a non-deterministic instruction, the arbitration fusion unit combines the predicted success probability of switching, the predicted rollback probability, the optimal switching timing, and the non-deterministic instruction output by the secondary track machine learning auxiliary model to determine the switching information.

[0047] The process of collecting real-time monitoring data from the primary site and disaster recovery site, utilizing a dual-track intelligent decision engine, and combining preset business time windows and risk strategies to determine the timing of the switchover also includes: The real-time monitoring data is then normalized.

[0048] See Figure 3 As shown, step S204 includes the following sub-steps: S2041. Multi-source real-time monitoring data collection and normalization: Collect at least one of the following indicators: system health indicators (CPU, memory, disk I / O, network bandwidth, node liveness), middleware health indicators (database connection pool, log backlog, thread pool), application health indicators (TPS, transaction volume, transaction error rate, key interface success rate), business health indicators (key transaction success rate, accounting entry volume, reconciliation consistency), data synchronization indicators (replication location, replication latency, link status, log status, gap count), and external indicators (management announcements, duty notifications, maintenance announcements, alarm work orders).

[0049] Because the indicators have different dimensions, normalization can be performed to facilitate weighted summation and comparison. Normalization methods include the following: (1) Fixed-range indicator normalization: Applicable to indicators with clearly defined upper and lower limits, such as CPU utilization (0-100%), disk utilization (0-100%), memory utilization (0-100%), etc. The normalization formula is X_norm = (X - X_min) / (X_max - X_min), where X_norm is the normalized value of the current indicator, X is the original value of the current indicator, and X_min and X_max are the maximum and minimum indicator values ​​of this type of fixed-range indicator. For indicators that are "lower is healthier", the value is inverted during mapping to unify the semantics of "higher is healthier": Health_score = 1 - (X - X_min) / (X_max - X_min), where Health_score is the normalized value of this type of indicator. For example, CPU utilization from 0% to 100% is mapped to health from 1.0 to 0.0. This method needs to be used in conjunction with outlier filtering to remove or truncate outliers that exceed the theoretical lower and lower limits.

[0050] (2) Deviation index normalization: Applicable to indicators without clear upper and lower limits but with historical baseline distribution, such as network latency, TPS, transaction response time, etc. The deviation index is calculated by standard deviation (Z-Score) of the current value relative to the historical mean, and then mapped to the [0,1] interval by the Sigmoid function. The specific steps are as follows: First, calculate the standard deviation deviation X_zscore = (X_current - μ) / σ, where X_current is the current value, μ is the moving average of the past 1 hour, and σ is the moving standard deviation of the past 1 hour; then, map by the Sigmoid function, and the mapped health score Health_score = 1 / (1 + exp(X_zscore)). When the current value is equal to the historical mean, the health score is about 0.5 (neutral), when the current value is much lower than the historical mean, the health score approaches 1.0, and when the current value is much higher than the historical mean, the health score approaches 0.0.

[0051] (3) Special handling of latency indicators: For data latency indicators (such as master-slave replication latency), a reverse mapping based on a preset tolerance limit is adopted: Risk_score = min(1.0, X_current / X_tolerance), where Risk_score is the reverse mapping result, i.e., the normalized value, X_current is the current value of the indicator, and X_tolerance is the tolerance limit. For example, when the current replication latency is 45 seconds and the tolerance limit is 60 seconds, Risk_score = min(1.0, 45 / 60) = 0.75, and the corresponding data latency health score DataLag_HealthScore = 1 - 0.75 = 0.25.

[0052] (4) Abnormal range quantification: Define four ranges: "normal", "alarm", "critical", and "circuit break". When the indicator value is completely within the normal range, the health score is 1.00; when it exceeds the alarm threshold but does not reach the critical threshold, it is 0.5; when it is close to the circuit break threshold and needs immediate attention, it is 0.2; and when it exceeds the circuit break threshold and the service is damaged, it is 0.

[0053] (5) Time series smoothing: For high-frequency metrics (TPS, latency, error rate) collected at the second level, EMA (exponential moving average) smoothing is performed before normalization to eliminate noise data caused by GC pauses, instantaneous network jitter, etc., ensuring that the decision engine makes judgments based on stable trends rather than instantaneous fluctuations. The EMA calculation formula is EMA_t = α1·X_t + (1-α1)·EMA_(t-1), where EMA_t is the exponential moving average, EMA_(t-1) is the exponential moving average at time t-1, α1 = 2 / (Q+1), Q is the window size, and X_t is the metric value. Taking a collection frequency of 1 time / second and a 60-second window as an example, Q=60, α1≈0.032, the maximum impact of a single spike on EMA_t is about 3.2%, which is insufficient to trigger a decision.

[0054] S2042, Multidimensional Analysis and Quantification The normalized indicator vector enters the multi-dimensional analysis engine, which processes it in parallel using four analysis methods to comprehensively depict the system situation from four dimensions: current state, trend of change, future prediction, and abnormal patterns. Therefore, in some embodiments, the main track rule engine first checks the triggering status of the forced blocking rule and the forced triggering rule. If neither the forced blocking rule nor the forced triggering rule is triggered, the switching necessity score and the switching risk score are calculated based on real-time monitoring data, including: The real-time monitoring data is processed in parallel using four analysis methods: threshold analysis, trend analysis, predictive analysis, and anomaly detection, to output the current overall health of the main site, the intensity of the deterioration trend, and the predicted failure probability, which are used to calculate the switching necessity score. The threshold analysis involves setting two levels of thresholds for each set indicator: an alarm threshold and a circuit breaker threshold. When an indicator exceeds the alarm threshold, it enters an alarm state; when it exceeds the circuit breaker threshold, it enters a dynamic state. The trend analysis is based on extracting N time series sampling points in the most recent T minutes using a sliding window, calculating the trend slope using least squares linear regression, and calculating the goodness of fit. If the goodness of fit exceeds the set goodness threshold, the trend deterioration score is calculated based on the ratio of the trend slope to the preset deterioration threshold. The predictive analysis is a dual-track parallel design. The first predictive track extrapolates the remaining time to reach the circuit breaker threshold based on the linear regression model of trend analysis and maps it to the predicted failure probability. The second predictive track uses a two-layer LSTM time series prediction model to output the predicted values ​​and upper bounds of the confidence intervals for each future time step. The highest value of the first and second predictive tracks is taken as the final predicted failure probability. The anomaly detection includes single-index step change detection and multi-index associated anomaly detection. Single-index step change detection compares the difference in mean between two adjacent sliding windows and triggers a change alarm. Multi-index associated anomaly detection matches the current multi-index state combination in real time based on a predefined anomaly index combination pattern rule base, outputs the anomaly detection result, and determines the triggering state of the forced triggering rule. The anomaly detection result is written into the anomaly detection rule base.

[0055] The four analysis methods described above are explained in detail below with examples.

[0056] (1) Threshold Analysis: Determining the current state range of each set indicator (used to determine whether an alarm or circuit breaker is needed) and identifying existing problems is the most direct means of state perception. Each indicator has at least two preset threshold levels: Warning Threshold and Critical Threshold, defined by the risk strategy configuration file. Taking the core trading error rate as an example, when the error rate exceeds the circuit breaker threshold by 5%, it enters the CRITICAL state, the health score is directly lowered to the lowest level, and a P1 level alarm is triggered; when the error rate exceeds the warning threshold by 1% but does not reach the circuit breaker threshold, it enters the WARNING state, the health score is linearly interpolated proportionally in the range of [0.0, 0.5], and a P2 level alarm is triggered; otherwise, it enters the NORMAL state, and the health score is 1.0. The warning threshold triggers "early warning perception" to enter the switching preparation analysis mode, and the circuit breaker threshold triggers "action decision" to directly affect the SNS score (switching necessity score) and the judgment of the Hard Trigger forced trigger rule.

[0057] (2) Trend analysis: Identify “pre-deterioration” scenarios where the current set indicator value has not exceeded the threshold but continues to deteriorate, providing a basis for preventive switching. The trend slope calculation method based on sliding window linear regression is as follows: First, extract N time series sampling points (t, x) from the most recent T minutes (default T=15 minutes), where t is the time corresponding to the sampling point and x is the sampling value; then, use least squares linear regression to calculate the trend slope k = [N·Σ(t·x) - Σt·Σx] / [N·Σt² - (Σt)²], k>0 indicates an increase in the indicator (for an indicator where "higher is worse", it means deterioration), k<0 indicates a decrease in the indicator; next, calculate the goodness of fit R² = 1 - SS_res (residual sum of squares) / SS_tot (total sum of squares), and only when R²≥0.70 is the trend considered significant and reliable, R²<0.70 is considered random fluctuation, and the trend deterioration score TrendScore=0; finally, perform trend deterioration score mapping, when k≤0, TrendScore=0.0, k in (0, When the trend deterioration threshold k_warn is in the range k_warn, TrendScore = k / k_warn; when k ≥ k_warn, TrendScore = 1.0. The trend deterioration threshold k_warn is configured by the risk strategy.

[0058] For example: The current CPU utilization is 72% (not exceeding the alarm threshold of 90%), but the slope k=1.8% / min in the last 15 minutes, R²=0.88 (significant trend), and the trend predicts that it will exceed the alarm threshold in about 10 minutes. TrendScore=1.8 / 2.0=0.90 (high trend deterioration score), triggering the switch to the prepared analysis mode for early intervention.

[0059] (3) Predictive analysis: Based on current trends and historical patterns, quantitatively predict the trend of each indicator and the probability of exceeding the threshold within the next T minutes, thus advancing the decision-making time from "response that has already occurred" to "predictive intervention". The prediction model adopts a dual-track parallel design, with the two tracks (the first prediction track and the second prediction track) complementing each other, and taking the high-risk value.

[0060] The first prediction trajectory is a trend extrapolation prediction, suitable for short-term decision-making time windows such as 0-30 minutes. Based on the linear regression model obtained from trend analysis, it directly extrapolates to predict the index value at time t+Δt: X(t+Δt) = X(t) + k·Δt, where X(t) is the index value at time t, and Δt is the time elapsed since time t. The remaining time to reach the alarm threshold is calculated as T_warn = (X_warn - X_current) / k, and the remaining time to reach the circuit breaker threshold is calculated as T_critical = (X_critical - X_current) / k. The failure probability based on the remaining time is mapped as P_trend = max(0, 1 -T_critical / T_horizon), where X_warn is the alarm threshold, X_current is the current value of the index, X_critical is the circuit breaker threshold, and T_horizon is the decision-making time window, meaning the closer to the circuit breaker, the higher the predicted failure probability.

[0061] The second prediction trajectory uses an LSTM (Long Short-Term Memory) time-series prediction model, suitable for predictions within a medium-term decision-making window, such as 0-120 minutes. The model input is a multi-dimensional normalized index time-series matrix (dimension: time step × number of indices) spanning the past two hours. The architecture consists of a two-layer LSTM (128 hidden units) plus a fully connected output layer. The output is the predicted value for each future time step and the upper bound of the 95% confidence interval. The fault probability P_lstm is calculated as P_lstm = P(X(t+Δt) > X_critical, Δt∈[0, T_horizon]), and the probability is included when the upper bound of the prediction confidence interval exceeds the circuit breaker threshold. The model training data sources include process feature data and result labels from previous switchover drills, chaotic engineering fault injection experimental data, and historical alarm and recovery event data (anonymized).

[0062] The dual-track fusion output is the predicted failure probability, PredictedFailureProbability = max(P_trend, P_lstm), where P_trend is the output of the first predicted track and P_lstm is the output of the second predicted track. The highest value of the two tracks is taken to ensure that no risk signal is missed. This predicted failure probability is directly used as the input component for calculating the SNS.

[0063] (4) Anomaly Detection: Identifies unusual anomalies that cannot be captured by threshold and trend methods, including single-indicator step change detection and multi-indicator correlation anomaly detection, covering the blind spots of threshold analysis. Anomaly detection includes two detectors: The first detector is a single-metric step jump detection. Its mechanism compares the difference in mean values ​​between two adjacent sliding windows (each 5 minutes) Δμ = |μ_window2 - μ_window1|. If Δμ > δ_jump (the jump detection threshold), a jump alarm is triggered, where μ_window1 and μ_window2 are the means of the two sliding windows, respectively. Applicable scenarios include a sudden increase in database connection count from 200 to 800 within 1 minute (connection leak), a sudden increase in message queue backlog from near zero to 100,000 messages (consumer stoppage), and a sudden drop in TPS from normal levels to 10% (core link failure).

[0064] The second detector is a multi-metric correlation anomaly detection mechanism. It uses a predefined rule base of anomaly metric combinations to match current multi-metric status combinations in real time. If a match is found, an anomaly is detected. Examples of anomaly metric combination rule bases include, but are not limited to: application-layer suspension rules (CPU health ≥ 0.8, memory health ≥ 0.8, TPS health ≤ 0.3, and error rate health ≤ 0.2, meaning resources are normal but business is paralyzed, corresponding to application-layer deadlock or memory overflow faults); network fragmentation rules (local service health ≥ 0.7, cross-IDC network latency health ≤ 0.1, and replication latency health ≤ 0.2, meaning local is normal but network is interrupted, corresponding to network fragmentation between IDCs); and storage degradation rules (disk IO health ≤ 0.2, database response latency health ≤ 0.3, and application error rate health ≤ 0.5, meaning chain failures caused by storage layer performance degradation). The rule base update mechanism involves extracting newly discovered anomaly patterns after each switchover drill or production fault review, manually reviewing them, and then writing them into the rule base. The rule base itself is subject to version control.

[0065] Anomaly detection results are output in a structured format of {anomaly flag: Boolean value, anomaly type code, confidence level}, and input into the mandatory rule layer for mandatory triggering rule Hard Trigger judgment.

[0066] S20421. Analysis Timing of Three Triggering Modes In some embodiments, the four analysis methods dynamically switch between three triggering modes based on the current state of the business system. The three triggering modes include: normal inspection mode, switching preparation mode, and event triggering mode. The normal inspection mode is the default operating state, and the relevant thresholds in this mode use standard values. The handover preparation mode indicates that there is a potential handover risk or that a handover plan is imminent. In this mode, the relevant thresholds are tightened based on the standard values. The event-triggered mode is a mode for setting up sudden events. In this mode, the relevant thresholds are tightened based on the values ​​used in the switching preparation mode.

[0067] In other words, the four analysis methods in this embodiment do not operate at a fixed frequency, but rather dynamically switch between the three trigger modes according to the current state of the system: The first trigger mode is a routine inspection type, running continuously 24 / 7. Entry conditions are the system's default operating state, performing a comprehensive four-dimensional analysis every 60 seconds per round, a threshold quick check every 10 seconds per round, using standard thresholds without tightening, and fully executing all four analysis methods. Results are written to the time series database. Its main functions are to establish healthy baselines for each indicator (μ and σ are dynamically updated over time), accumulate historical time series data required for trend and predictive analysis, and maintain continuous updates of current SNS / SRS scores for real-time display on the large screen.

[0068] The second trigger mode is the switchover preparation mode, which is entered before the planned switchover. Entry conditions are met if any of the following conditions are met: the time window from the planned switchover is ≤T-4 hours (triggered by switchover plan registration), a P2 level or higher alarm is received (triggered by threshold analysis), or the SNS score is ≥0.50 for three consecutive rounds during routine inspections. The analysis frequency is increased to 10 seconds / round for comprehensive four-dimensional analysis and 5 seconds / round for rapid threshold checks. Alarm thresholds are tightened by 20%, and circuit breaker thresholds are tightened by 10% (earlier alarm triggering, providing more buffer time for preparation). Additional actions include initiating disaster recovery side capacity preheating verification, RPO entering encrypted monitoring (collection frequency increased to 1 time / second), performing lightweight data consistency rapid verification (row count + location), and pushing a "Entering switchover preparation status" notification to on-duty personnel. Its main function is to identify and eliminate potential risks during the switchover timing in advance, ensuring the quality of the switchover window.

[0069] The third trigger mode is event-triggered, driven by external events, and has the highest priority. Entry conditions include receiving any of the following event signals: external events (management announcements, maintenance instructions, high-risk alarm work orders), anomaly detection (multi-indicator correlation anomaly rule hit), and tiered jump detection (significant jump in key indicators). The trigger response immediately initiates a complete four-dimensional analysis (without waiting for the next cycle), and after triggering, maintains a high-frequency monitoring state for 5 seconds per cycle, dynamically tightening thresholds according to event level (P1 event threshold tightens by 50%, P2 event threshold tightens by 30%, and P3 event threshold tightens by 15%). Its main function is to respond to sudden events, quickly output switching decisions, and minimize response latency.

[0070] S20422, Quantization output of the dual-score model After completing one round of analysis using the four analytical methods described above, two core quantitative scores are output: a switching necessity score and a switching risk score. These two branches serve as the coordinate axis inputs to the decision matrix. The switching necessity score is obtained by weighted fusion of the main site's current overall health, the intensity of the deterioration trend, the predicted failure probability, and the strength of external trigger signals. The switching risk score is obtained by weighted fusion of data synchronization delay risk, the disaster recovery site's own health risk, current business load risk, upstream and downstream dependent system risk, historical switching failure rate under similar conditions, and business time window risk.

[0071] (1) Switch Necessity Score (SNS): SNS measures the urgency of whether a switch is currently necessary, comprehensively reflecting the current state of the main site, future degradation trends, and external driving signals. The calculation formula is: SNS = α·(1 - PrimaryHealth_Score) + β·TrendDeteriorationScore +γ·PredictedFailureProbability + δ·ExternalTriggerSignal.

[0072] The default weights can be α=0.40, β=0.25, γ=0.25, δ=0.10, Σ(α+β+γ+δ)=1.00. These weights can be adjusted through the risk strategy configuration file. PrimaryHealth_Score represents the overall health of the main site, calculated by weighting the following: core transaction success rate health score (weight 0.35, highest weight, directly reflecting user perception), response latency P99 health score (weight 0.20, a key dimension of user experience), core service process survival health score (weight 0.15, basic availability guarantee), database service health score (weight 0.15, core of data processing), system resource health score (weight 0.10, weighted average of CPU + memory + disk), and external dependency service health score (weight 0.05, clearing gateway, payment pre-processing, etc.). TrendDeteriorationScore represents the future deterioration trend, i.e., the intensity of the deterioration trend. It is obtained based on the trend deterioration score obtained from trend analysis, such as by fusing and normalizing the trend deterioration scores corresponding to multiple indicators to obtain the intensity of the deterioration trend. The assignment rules for the ExternalTriggerSignal strength are as follows: 1.00 (highest) for management forced commands, 0.80 for maintenance active switching commands, 0.60 for P1 level fault work orders, 0.30 for P2 level fault work orders, and 0.00 for no external signal. PredictedFailureProbability is the predicted failure probability.

[0073] (2) Switch Risk Score (SRS): SRS measures the cost and risk of the current switch, comprehensively reflecting constraints such as disaster recovery readiness, data security, and business impact. The calculation formula is: SRS = w1·DataLagRisk + w2·StandbyHealthRisk + w3·BusinessLoadRisk+ w4·DependencyRisk + w5·HistoricalRisk + w6·TimeWindowRisk; The default weights are w1=0.25, w2=0.20, w3=0.20, w4=0.15, w5=0.10, w6=0.10, and Σ(w1+ w2+w3+ w4+ w5+ w6)=1.00. The risk of data synchronization latency is calculated as follows: DataLagRisk = min(1.0, CurrentRPO_seconds / RPO_tolerance_seconds), where CurrentRPO_seconds and RPO_tolerance_seconds are the current measured RPO value and the RPO tolerance limit, respectively. For example, if the current replication latency is 45 seconds and the tolerance limit is 60 seconds, DataLagRisk = 0.75. The health risk of the disaster recovery site itself is calculated as: StandbyHealthRisk = 1 - StandbyHealth_Score. The StandbyHealth_Score structure is the same as PrimaryHealth_Score, but it is collected specifically for the disaster recovery side. The current business load risk is calculated as: BusinessLoadRisk = Current_TPS / Peak_TPS_Historical. This means that the closer the current load is to the historical peak, the higher the risk of traffic fluctuations during the switchover. Current_TPS is the current TPS, and Peak_TPS_Historical is the historical peak TPS. The upstream and downstream dependency system risk is calculated as: DependencyRisk = 1 - The weighted average of the health scores of upstream and downstream dependent systems is used. The list of dependent systems includes online core, large and small value payment gateways, accounting system, channel front-end, etc. The historical drill risk, namely the failure rate of switching under the same historical conditions, comes from the historical switching review database described above. It queries the switching failure rate under similar historical conditions (combination buckets of time window type, load level, and RPO level). The business time window risk, TimeWindowRisk, is obtained by querying the business time window risk matrix at the current time.

[0074] S20423, Dual-Indicator Decision Matrix The main rule engine calculates the SNS (Switch Necessity) and SRS (Switch Risk) and matches them with a pre-built two-dimensional decision matrix to output preliminary decision instructions. This two-dimensional decision matrix uses SNS and SRS as its two axes, with each matrix element representing the decision instruction corresponding to that axis. The matrix reflects the decision-making principle of decisively executing the switching pipeline when necessity is high and risk is low, and of weighing whether to execute the switching pipeline when necessity and risk are both high. The decision matrix is ​​shown in Table 1.

[0075] Table 1 High [0.65, 1] AUTO REC DEFER [0.35, 0.65] REC REC HOLD Low [0, 0.35) HOLD HOLD HOLD The meanings and initial actions of each preliminary decision instruction are as follows: AUTO indicates that the automatic switching conditions are met, enters the mandatory rule verification layer (Hard Block check), and after passing, directly calls the pipeline API and synchronously notifies the management; REC indicates that a switch is recommended, requiring manual confirmation, generating a decision basis report, pushing an approval work order, starting a timeout timer (e.g., 15 minutes), and automatically escalating after the timeout; DEFER indicates that the necessity is high but the risk is temporarily too high, pushing a high necessity and high risk warning notification, outputting risk mitigation suggestions, and re-entering the analysis cycle for continuous evaluation after a period of time, such as 5 minutes; HOLD indicates that there is no need to switch, maintaining the current analysis mode, updating the monitoring dashboard, and not triggering any switch-related operations.

[0076] S2043, Combining Business Time Window with Risk Strategy The core idea is that the initial decision instructions output by the decision matrix can be modified by both the business time window filtering layer and the risk strategy verification layer to form the final main decision instructions of the main track rule engine. Time window risk is directly used as a component of SRS (w6·TimeWindowRisk) in quantitative calculation, while some time windows are subject to mandatory blocking coverage (Hard Block) to ensure the bottom-line constraint of banking business continuity.

[0077] S20431, Business Time Window Strategy Matrix The time-dependent nature of banking operations means that the switching risks vary significantly across different time periods, necessitating refined modeling. This application's embodiment categorizes the 24 hours of the day and key calendar features to form a time window risk matrix. Please refer to Table 2.

[0078] Table 2 02:00-06:00 Midnight Low 0.05 Forced Overwrite (Optimal Window) 06:00-09:00 before morning rush hour 0.35 AUTO requires a total SRS score of <0.4. 09:00-12:00 Morning peak 0.70 Only Hard Trigger can trigger 12:00-14:00 midday 0.50 Only Hard Trigger can trigger 14:00-18:00 Afternoon peak 0.70 Only Hard Trigger can trigger 18:00-24:00 Market close settlement 0.50 Only Hard Trigger can trigger 24:00-02:00 End-of-day batch 0.95 Hard Block The day before the end of the month Monthly Management Report +0.40 Hard Block The last day of the year Year-end settlement +0.40 Hard Block Important protection period Management Requirements +0.40 Hard Block Major holidays statutory holidays +0.40 Hard Block The TimeWindowRisk upper limit is truncated to 1.0 (including stacking).

[0079] S20432, Mandatory Rule Layer – Hard Constraints The mandatory rule layer is the last line of defense for decision security, and its priority is higher than the main decision instructions output based on the decision matrix. It is divided into two categories: hard block and hard trigger, which respectively protect the boundaries of "cannot be cut" and "must be cut".

[0080] Hard Block Rule Set – If any of the following conditions are met, the current state will be immediately replaced with a BLOCK decision instruction. Please refer to Table 3.

[0081] Table 3 HB-001 The measured RPO for disaster recovery data synchronization is > 600 seconds. The risk of data loss exceeds the acceptable limit. HB-002 The disaster recovery core database service is unreachable (three consecutive probe failures). Disaster recovery core dependencies unavailable HB-003 Disaster recovery overall health score (StandbyHealth_Score) < 0.50 Disaster recovery itself is in a sub-healthy state. HB-004 The data consistency check has any items that failed. Switching will introduce data consistency risks. HB-005 Currently in the mandatory blocking time window Bottom-line protection during critical business periods HB-006 The last switchover exercise ended in failure and a post-mortem analysis has not yet been completed. Historical risks have not been cleared, and switching back is not allowed. The processing flow for BLOCK decision commands is as follows: immediately abort all switchover operations and lock the pipeline entry point; push BLOCK alarms (including the blocking rule number and release condition description) to the operations and maintenance team and management; continuously monitor the blocking conditions, and re-enter the decision loop after the conditions are automatically released. The manual BLOCK overwrite mechanism (limited to extreme failure scenarios) requires both the operator and the shift supervisor to complete the overwrite authorization operation. The overwrite reason, authorizing person information, and timestamp are written to an immutable audit database. After overwrite, the most basic data security checks HB-001 and HB-002 must still be passed.

[0082] Hard Trigger Rule Set – If any of the following conditions are met and HB is not triggered, a hard AUTO decision instruction will be enforced. Please refer to Table 4.

[0083] Table 4 HT-001 The main site's core transaction success rate is less than 95%, lasting for more than 2 minutes. A serious malfunction has occurred in the transaction layer. HT-002 The main site's core database was completely unavailable for more than 30 seconds. Data storage core unavailable HT-003 The main site experienced an IDC-level network outage (confirmed by an alarm from the IDC monitoring system). Overall infrastructure failure HT-004 The main site's overall health score is less than 0.20 for more than 3 minutes. Severe degradation across multiple dimensions HT-005 Received a forced switchover order from the management authority Management compliance requirements are the highest priority. Security verification after forced triggering rules: Forced blocking rules can be verified first, such as HB-001 to HB-004 (data security baseline cannot be exempted). If all HB-001 to HB-004 pass, management is notified and the switchover pipeline is directly triggered; if any of HB-001 to HB-004 fails, the critical event response plan is triggered: the highest level event response (P0) is triggered, the emergency command group is automatically established, decision-making notifications are pushed, and a detailed list of gaps between primary and backup side failures is output.

[0084] S20433, Risk Strategy Configuration Management Risk strategies are versioned and stored in the version control system as configuration files, allowing adjustments to decision-making behavior through configuration changes without modifying the decision engine code. The risk strategy configuration file (e.g., risk_strategy_v2.3.yaml) contains the following configuration items: decision engine mode, SNS weight configurations, decision matrix quadrant boundary thresholds, additional preconditions for AUTO commands to take effect, REC work order manual approval timeout, contact person upgrade after timeout, forced blocking rule switch, and the number of hours to enter enhanced observation mode before switching windows.

[0085] Configuration change control: Any change to the risk policy configuration must go through CR (Code Review) and follow an independent approval process. The effective time of the change, the content of the change and the approver are all written into the audit log, which is aligned with the change transaction mechanism in step S202.

[0086] S2044, Core Principles of Intelligent Decision Engine The architecture design of the Intelligent Switch Decision Engine (ISDE) needs to simultaneously satisfy three mutually constraining objectives. The first objective is the interpretability and compliance of the decision. Switching operations in a bank's core system are managed activities, and each switch decision must provide clear justification to management and auditing departments, explaining "why the switch was triggered at this time." Pure machine learning black-box models cannot meet this requirement; therefore, the intelligent switch decision engine can be rule-driven to ensure transparent and traceable decision logic. The second objective is the continuous improvement of decision accuracy. Manually written rules cannot perceive complex multi-dimensional combinations of features. Within the ambiguous range of rule engine judgments, the empirical patterns contained in historical switch data are often more accurate than static rules. Therefore, machine learning models can be introduced to provide probabilistic auxiliary judgments where rule engines fall short. The third objective is the stability and security of the architecture. Machine learning models may produce incorrect predictions due to insufficient training data or feature drift. The erroneous output of the model should not directly affect core switch operations; therefore, machine learning (ML) models can only participate in decision-making under constrained conditions, not dominate it.

[0087] Based on the above three objectives, ISDE adopts a dual-track architecture in which the main track rule engine and the ML auxiliary model run in parallel, and the final decision is output through an arbitration fusion unit.

[0088] Therefore, in some embodiments, the dual-track intelligent decision engine includes an arbitration fusion unit, a main-track rule engine, and a secondary-track machine learning auxiliary model; the process of collecting real-time monitoring data from the main site and disaster recovery site, and using the dual-track intelligent decision engine in conjunction with preset business time windows and risk strategies to determine the switching timing includes: The main track rule engine first checks the triggering status of the forced blocking rule and the forced triggering rule. If neither the forced blocking rule nor the forced triggering rule is triggered, it calculates the switching necessity score and the switching risk score based on real-time monitoring data. Then, it matches the switching necessity score and the switching risk score with the preset decision matrix, and finally outputs the main decision instruction after making corrections using the preset business time window. If the main decision instruction output by the main track rule engine is a deterministic instruction, the arbitration fusion unit determines the switching information based on the main decision instruction; If the main decision instruction output by the main track rule engine is a non-deterministic instruction, the arbitration fusion unit combines the predicted success probability of switching, the predicted rollback probability, the optimal switching timing, and the non-deterministic instruction output by the secondary track machine learning auxiliary model to determine the switching information.

[0089] In this embodiment, ISDE is divided into four layers from top to bottom according to the data flow: input layer, processing layer, arbitration layer, and output layer, from external input to final output.

[0090] S20441, Input Layer The input layer of this dual-track architecture receives all quantitative results generated from the pre-analysis step S204 as raw material for the decision engine. This includes, but is not limited to, the following five types of inputs: First, the SNS switching necessity score, reflecting a weighted sum of the current deterioration level of the main site, the intensity of the trend deterioration, the predicted failure probability, and external triggering signals. The value ranges from [0,1], with higher values ​​indicating a greater necessity for switching. Second, the SRS switching risk score, reflecting a weighted sum of data synchronization delay risk, disaster recovery health status, business load level, dependent system risk, historical failure rate, and time window risk. The value ranges from [0,1] as well, with higher values ​​indicating a greater cost and benefit from the current switching. The greater the risk, the better; thirdly, the time window signal, which includes the time window type corresponding to the current moment, the time window risk score, and a Boolean flag indicating whether the mandatory blocking period has been hit, derived from the query results of the business time window strategy matrix; fourthly, the mandatory blocking rule status set, which includes the hit check results of six mandatory blocking rules from HB-001 to HB-006 and five mandatory triggering rules from HT-001 to HT-005, serving as the last line of defense for decision security; and fifthly, historical data and context, which includes the analysis mode at the current running moment, the switching success rate under similar historical conditions, and the number of days since the last switching exercise, mainly for use by ML-assisted models.

[0091] S20442, Processing Layer – Dual-track Parallel Computing The processing layer is the core of ISDE, consisting of two independent and parallel computing units: the main track rule engine and the ML auxiliary model sub-track. The two tracks are logically completely decoupled, do not block each other, and their respective computational results are then merged into the arbitration fusion unit.

[0092] The main rule engine is the core of the decision-making process, bearing responsibility for decisions in all scenarios, and its output is binding on any switching decisions. The main track executes in a strict three-step sequence: The first step is the pre-validation of mandatory blocking rules. The rule engine first checks the trigger status of all mandatory rules one by one. This step is the highest priority checkpoint in the entire decision-making process. If any mandatory blocking rule is triggered, regardless of the SNS and SRS score calculation results, the decision engine immediately outputs the BLOCK command to stop all subsequent decision-making steps. If a mandatory trigger rule is triggered and all mandatory blocking rules are not triggered, the decision engine directly outputs the FORCE_AUTO command, also skipping subsequent steps. Only when all mandatory blocking rules are not triggered will the main track continue to execute the second step. The second step of the main track is decision matrix mapping. Using SNS scores as the vertical axis (divided into three intervals: high [0.65, 1.0], medium [0.35, 0.65], and low [0, 0.35)) and SRS scores as the horizontal axis (also divided into high, medium, and low intervals), a 3×3 decision matrix is ​​formed. The rule engine finds the corresponding matrix cell based on the current SNS and SRS score combination to obtain the initial decision instruction. The third step of the main track is time window constraint correction. The initial conclusion of the decision matrix needs further correction through time window additional constraint rules. If the current time is during a peak business period, the additional constraint stipulates that only HardTrigger can trigger the switch during that period. If the matrix outputs AUTO but the current time is during a peak business period, it automatically downgrades to REC. If the current time is during a forced blocking period, an additional layer of time protection is added outside of Hard Block. After time window correction, the main track rule engine outputs the final main decision instruction, which is one of five types: AUTO, REC, DEFER, HOLD, or BLOCK.

[0093] The ML auxiliary model's sub-track is not an independent decision-maker, but rather a tool to enhance the accuracy of the rule engine in fuzzy regions (i.e., when the main track outputs a REC instruction). The sub-track uses an XGBoost gradient boosting tree classifier, taking, for example, a 35-dimensional structured feature vector as input, and outputs three probabilistic predictions: the probability of a successful switchover (P_success), the probability of triggering a rollback within a certain period after the switchover (P_rollback), and the optimal switchover timing under the current trend. In some embodiments, this optimal switchover timing is represented as the estimated number of minutes from the current optimal switchover timing (T_optimal). The activation of the sub-track has a clear precondition: the sub-track result is only adopted by the arbitrator fusion unit when the main track's rule engine outputs a REC instruction. When the main track outputs AUTO, HOLD, DEFER, or BLOCK, even if the sub-track has completed its calculations, its result does not participate in the final decision and is only used for subsequent review. This design ensures that the influence of the ML model is strictly constrained and will not interfere with deterministic scenarios.

[0094] S20443, Arbitration Fusion Device The arbitration fusion unit is the convergence point of the dual-track computation results. It is responsible for merging the output of the main track rule engine and the output of the ML auxiliary model sub-track into a single final decision instruction according to clearly defined priority rules. Its arbitration logic is as follows: when the main track outputs a non-REC conclusion (AUTO / BLOCK / FORCE_AUTO / DEFER / HOLD), the arbitration fusion unit directly adopts the main track conclusion as the final decision. The ML sub-track result does not change any conclusion and is only appended to the decision report for reference. This ensures the absolute authority of the rule engine in deterministic scenarios. When the main track outputs REC, the arbitration fusion unit adopts the probabilistic output of the ML sub-track and modifies the conclusion according to the following rules: If P_success is greater than or equal to 0.85 and P_rollback is less than or equal to 0.10, it indicates that the ML model has high confidence in this switchover, and the arbitration fusion unit upgrades REC to AUTO; if P_success is greater than or equal to 0.70 but does not reach the high confidence threshold, REC remains unchanged, and the ML probability report is attached to the approval work order for manual decision-making reference; if T_optimal is greater than 0, it indicates that the ML model judges that the current timing is not optimal and suggests waiting for several minutes until the conditions are better, then the arbitration fusion unit downgrades the REC instruction to a DEFER instruction and explains the reason for the suggestion to wait in the notification; if none of the above conditions are met, the DEFER instruction is also output, indicating that the ML model does not support performing the switchover at the current time.

[0095] While generating the final decision instruction, the arbitration fusion engine also needs to generate a decision basis report. The report records the complete reasoning chain of this round of decision-making in the form of a structured document, including intermediate conclusions of each analysis dimension, details of each sub-item of SNS / SRS, the results of each Hard rule verification, ML model version number and output value, and a description of the generation path of the final conclusion, ensuring that each decision-making step can be manually reviewed and managed.

[0096] S20444, Output Layer The output layer transforms the final decision (also known as switching information) of the arbitration fusion unit into four types of outputs, which flow to different downstream stages. The first type is the final decision instruction, which flows to the execution branch processing logic. Based on the five instruction types (AUTO / REC / DEFER / HOLD / BLOCK), it triggers different operations such as pipeline API calls, generating approval work orders, sending early warning notifications, updating the monitoring dashboard, or locking the pipeline entry point. The second type is the pipeline trigger parameter package (generated only for AUTO and FORCE_AUTO), which transmits execution parameters such as the initial traffic ratio, stepping strategy, stepping interval, and observation window duration to the subsequent gradual traffic switching stage in a standardized JSON format, as well as three pre-set rollback trigger thresholds for the subsequent rollback stage: error rate, response latency, and TPS reduction. The third type is the decision basis report, accessible externally via a URL for display on the operations and maintenance approval platform, reference in the manual confirmation interface, and use in post-event review and analysis. The fourth category is audit records, which are written to WORM (Write Once Read Many) immutable storage in the form of structured logs with complete fields. The content covers all inputs, intermediate conclusions, final instructions, execution results and operator information of this round of decision-making, meeting the retention requirements of management compliance, and serving as the data source for incremental training of ML models in the subsequent review phase.

[0097] The dual-track architecture of this application has three core design advantages. First, the rule engine ensures that the bottom line of decision-making is not compromised. Forced blocking and forced triggering rules solidify the safety boundaries of the switching operation in a coded form. These boundaries will not be violated by any prediction results from the ML model, ensuring behavioral determinism in extreme scenarios. Second, the ML model achieves digital accumulation of experience. After each switching exercise and production switch, the 35-dimensional feature vector of the switching process and the final result label are fed back into the training dataset. The model continuously absorbs new switching experience through incremental learning, causing the accuracy of decisions in the REC fuzzy range to steadily improve with the number of exercises, forming a virtuous cycle. Third, the clear priority rules of the arbitration fusion unit ensure orderly and uncluttered collaboration between the two tracks. The rule engine is not interfered with by ML in deterministic scenarios, while ML only has a voice in rule-fuzzy scenarios. The clear boundaries of authority and responsibility prevent ML from overstepping its authority to interfere with deterministic decisions and also prevent the rule engine from making suboptimal choices in fuzzy scenarios.

[0098] S205, Gradual Traffic Switching and Service Verification In some embodiments, a portion of non-critical business traffic is switched from the primary site to the disaster recovery site, and an automated business verification script is run synchronously. After successful verification, the proportion of switched business traffic is gradually increased until the switch is complete, including: A preset grayscale step is obtained, and the step is dynamically selected according to the switching risk level. Different grayscale steps correspond to different traffic ratios. For each grayscale level, execute a sequence of atomic operations including traffic distribution weight adjustment, session persistence processing, health check injection, and data layer read / write separation confirmation; After each grayscale step transition is completed, the observation window is entered. If the automatic rollback condition is triggered in the observation window, the transition pipeline is immediately interrupted and a rollback is executed. At the end of each grayscale step, four types of verification scripts are automatically executed in parallel: core accounting transaction verification, batch and reconciliation verification, concurrency and capacity verification, and cross-site data consistency verification.

[0099] In this embodiment, a canary release strategy is used for traffic switching. First, a small amount of non-critical business traffic is switched to the disaster recovery site, while automated business verification scripts (including core accounting transactions and reconciliation) are run. After successful verification, the traffic proportion is gradually increased until the switch is complete. This step includes the following sub-steps: S2051, Traffic Switching The traffic switching process is defined in three layers: the traffic ingress layer (intelligent DNS / API gateway) is responsible for distributing user requests proportionally; the service layer (application cluster / microservice) is responsible for handling business logic to ensure session affinity or distributed session synchronization; and the data layer (database) is responsible for storage and transaction processing to ensure consistency between primary and backup data. The switching target is a gradual transition from "full primary site" to "full disaster recovery site," with a mixed operating state in between (partial traffic on the disaster recovery site and partial traffic on the primary site).

[0100] S2052, Gray-scale stepping strategy S20521 Traffic Ratio Definition: A preset grayscale ladder is used, with each ladder corresponding to a traffic ratio (%). The default ladder is 1%→5%→10%→25%→50%→75%→100%. The ladder is dynamically selected by the decision engine based on the switching risk level. For example, in high-risk scenarios, the 75% step is skipped and the step directly reaches 100%, while in low-risk scenarios, a 2% fine-grained step is added.

[0101] S20522 Switching Execution Order: For each grayscale step, execute the following sequence of atomic operations (ensuring rollback capability): Traffic distribution weight adjustment: Modify the traffic distribution weight of the API gateway to forward a specified proportion of user requests to the service instance of the disaster recovery site; Session persistence processing: If the business requires session persistence, mark it through a global session middleware (such as Redis shared session) to ensure that the same session always resides on the same site; Health check injection: Activate probes on the disaster recovery service instance to continuously monitor the response status of newly received traffic; Data layer read / write separation confirmation: If the disaster recovery database has been upgraded to a read / write master database, dynamically update the read / write separation rules configured in the data source.

[0102] S20523 Time Window and Pause Strategy: After each grayscale transition is completed, an observation window (e.g., 5-10 minutes, which can be shortened or extended by the decision engine based on business timeframes) is entered by default. If an automatic rollback condition is triggered within the observation window, the pipeline immediately interrupts the current grayscale transition and executes the rollback step. If the decision engine outputs a REC command (manual assistance), an approval work order is generated after each grayscale transition, and the next grayscale transition can only proceed after confirmation by operations and maintenance personnel.

[0103] S2053, Automated Business Verification Script At the end of each grayscale step (after traffic stabilizes), the following set of verification scripts is executed automatically in parallel, and the verification results are directly fed back to the decision engine and the end-to-end observation module.

[0104] S20531 Core Accounting Transaction Verification: The verification asset is an automated test client deployed at the disaster recovery site (simulating real channel requests), covering high-frequency core transactions (such as transfers, balance inquiries, periodic account openings, payroll disbursements, etc.). The verification method involves sending tagged transactions (containing unique serial numbers), processing them at the disaster recovery site, and comparing the system return codes, response times, and written database data with the reconciliation platform's verification rules. The passing standard is that all transaction return codes are successful, the average response time is ≤120% of the baseline value, and the written database data is consistent with the primary site backup copy within seconds.

[0105] S20532 Batch and Reconciliation Verification: Verification of assets involves triggering a batch processing simulator to run pre-defined end-of-day tasks (such as accruing interest and setting aside reserves), and performing T+0 real-time reconciliation and T+1 end-of-day reconciliation. The verification method compares the batch processing output results (such as reports and general ledgers) of the disaster recovery site with the baseline results synchronously executed on the main site. The passing criteria are zero reconciliation discrepancies and batch processing time not exceeding twice the baseline time.

[0106] S20533 Concurrency and Capacity Verification: Verify the asset by using load testing tools (such as JMeter or Locust) to simulate a set multiple (e.g., 1.5 times) of the current gray-scale traffic ratio on the disaster recovery site to assess the service limits. The passing criteria are: CPU utilization ≤ 80%, database connection pool utilization ≤ 70%, error rate < 0.5%, and no timeout accumulation.

[0107] S20534 Data Consistency Verification (Cross-Site): The verification method involves executing a difference comparison script between the primary and backup sites after each gray-scale switch (e.g., MD5 verification of specific table rows, row count comparison, timestamp comparison). The passing standard is a difference of 0 (for scenarios that allow eventual consistency, the difference items must be within the preset tolerance list).

[0108] S2054 Automatic Rollback Conditions and Triggers During any gray-scale step, if any of the following conditions are met, the rollback of this step will be automatically triggered, the traffic ratio will be rolled back to the state before this gray-scale step, and the reason for failure will be recorded to the audit chain: any core transaction verification fails (error rate > 0.1%); reconciliation difference exceeds the tolerance threshold (e.g., amount difference > 0 yuan); disaster recovery site monitoring indicators (response time / error rate) trigger the forced blocking rule set; data consistency verification finds non-tolerable differences; operations and maintenance personnel actively click the "rollback" button in the switch panorama (manual intervention in REC mode); timeout (verification not completed within the preset observation window).

[0109] S206, End-to-End Observability and Security Compliance Embedded Throughout the switchover process, a unified monitoring platform enables real-time tracking and visualization of application performance, infrastructure status, and / or business transaction links. Security scans and compliance checkpoints are integrated into the pipeline stages, requiring successful completion before proceeding to the next stage. The core design goals of this step include three aspects: observability fusion, which unifies the collection, correlation, and visualization of application layer, infrastructure layer, business transaction layer, and network layer data during the switchover process to form a switchover dashboard; security and compliance shift to the left, embedding security scans and compliance checks as quality gates into every critical stage of the switchover, achieving strong constraint control where failure to pass is not permitted; and end-to-end traceability, automatically recording all operations, inspection results, indicator data, and decision-making basis throughout the switchover process, forming an audit evidence chain that meets financial management requirements.

[0110] S2061, Multidimensional Data Acquisition Layer of the Unified Observability Platform Based on open standard protocols such as OpenTelemetry, multiple types of telemetry data from primary and backup sites are collected in a unified manner. The collected data covers six main categories: First, application performance metrics, including JVM, thread pool, connection pool, TPS, response time, and error rate, collected via Prometheus Exporter to assess application health; second, infrastructure metrics, including CPU, memory, disk I / O, network traffic, and container status, collected via Prometheus Exporter to assess resource saturation; third, business transaction traces, targeting key transaction call chains such as core accounting, payment, and reconciliation, using SkyWalking probes for end-to-end trace tracking; fourth, logs, including application logs, system logs, middleware logs, and database logs, collected via LogEasy for anomaly analysis and root cause localization; fifth, business metrics, including transaction success rate, account reconciliation rate, single transaction time, and settlement latency, obtained through business data tracking combined with streaming computing, used for business continuity assessment; and sixth, network and topology data, including DNS resolution and cross-site link quality, collected via SNMP and network probes for validating traffic switching. The data acquisition agent has the capabilities of resuming interrupted downloads, local caching, and compressed transmission, ensuring that data is not lost in network jitter scenarios during switching.

[0111] S2062, Data Association and Processing Layer The collected multi-source data enters the streaming processing pipeline to complete key processing: a unified tagging system, which assigns unified tags such as switch_id (this switch ID), stage (pipeline stage), site (primary / backup site), and business_domain (business domain) to all data to achieve cross-dimensional association; TraceID integration, with unified injection of TraceIDs from the front-end entry point to the back-end database, and association with the execution ID of the switch pipeline; metric aggregation calculation, which calculates derived metrics such as windowed success rate and error rate mutations in real time; and anomaly detection, which performs real-time anomaly detection on key metrics based on dynamic baselines and triggers alarm events.

[0112] S2063, Switch cockpit visualization layer The failover dashboard offers five types of visualizations: a pipeline stage view, showing the current stage, execution time of each stage, and access control status; a primary / standby dual-site comparison view, displaying key indicators of the primary and backup sites side-by-side for a clear comparison before and after the failover; a business heatmap, displaying the health status of each core business unit by transaction; a link topology view, dynamically showing the traffic distribution ratio between the primary and backup sites, call chains, and abnormal nodes; and a data synchronization view, displaying real-time database master-slave latency and data reconciliation discrepancies. The dashboard supports drill-down, allowing users to drill down from the global view to specific transactions, call chains, and logs to pinpoint problems.

[0113] S2064, Pipeline Feedback Loop Layer The observability platform pushes key metrics to the pipeline orchestrator via an open API: after each stage is completed, the pipeline actively calls the platform API to obtain the health score (0-100) for that stage. If the score is lower than the threshold, it will block the entry into the next stage. When the platform detects a serious abnormal event, it will actively notify the pipeline to trigger the automatic rollback mentioned above. All metric snapshots and alarm events are archived as pipeline products as original evidence for post-event review and audit.

[0114] S207, Automated Rollback Mechanism In some embodiments, after the predefined rollback pipeline is triggered, it is executed in the following order: Capture on-site data at abnormal moments and associate it with the identifier of the pipeline being switched and store it. By modifying the global load balancer's resolution policy, domain name resolution is gradually switched back to the main site. By updating the gateway's routing rules, traffic entering the gateway is proportionally switched back. For established long connections, the connection eviction mechanism forces clients to reconnect to the main site. Set the service instances started on the disaster recovery site to no longer receive residual traffic, restore the service instances on the main site that were stopped or downgraded during the switchover to normal operation, and restore the list of service instances in the service discovery registry to the snapshot version before the switchover. Leveraging the infrastructure-as-code and configuration management database capabilities, changes to application configurations, network policies, and middleware configurations during the switchover process can be rolled back to the versions before the switchover. For scenarios where no business writes have occurred at the disaster recovery site, the synchronization direction of the primary and backup databases will be restored to primary to backup. For scenarios where business writes have occurred at the disaster recovery site, incremental data will be flowed back to the primary site through transaction log parsing, and data consistency verification will be performed. The automated business verification script is invoked to compare the deviations of the key indicators after rollback with the corresponding indicators before the switch. Once all indicators converge to the set range of the corresponding indicators, the rollback is deemed successful.

[0115] Predefined rollback trigger indicators (such as a surge in business error rate or unavailability of critical services) are pre-defined. When the monitoring system detects that the indicator exceeds the threshold, the predefined rollback pipeline is automatically triggered to quickly restore traffic and the environment to their pre-switch state. The predefined rollback pipeline is an automated execution process built in conjunction with the forward switchover pipeline from the initial design stage, managed in a versioned manner, and capable of being triggered independently. It is not a temporary emergency script, but a twin that grows alongside the switchover pipeline, adhering to the design principle that every forward switchover action must have a corresponding reverse rollback action. Structurally, the rollback pipeline includes six key elements: trigger layer, decision layer, orchestration layer, execution layer, verification layer, and recording layer. Its execution follows the principle of "traffic first, service follow, data fallback, and state convergence," unfolding sequentially or in parallel.

[0116] S2071, Pre-rollback checks and site preservation The first action after starting the rollback pipeline is not an immediate rollback, but rather the preservation of evidence of the current abnormal state. This includes capturing on-site data such as application logs, thread stacks, slow database queries, network packet captures, and container memory dumps at the moment of the anomaly, and archiving this data in the evidence preservation repository using the switch ID as a tag. This prevents the abnormal situation from being destroyed and the root cause from being untraceable after the rollback. Simultaneously, the current availability status of the primary site (i.e., the rollback target site) is confirmed, including service liveness, database writability, and network connectivity, ensuring that the primary site has the capacity to handle the rollback. If the primary site itself is unavailable, the process switches to a branch waiting for the primary site to recover or requiring special handling.

[0117] S2072, Fast Traffic Switchback This phase is the highest priority and most directly impactful part of the rollback pipeline, aiming to restore business traffic that has been switched to the disaster recovery site to the primary site in the shortest possible time. Rapid traffic rollback includes: gradually switching domain name resolution back to the primary site by modifying the global load balancer's DNS resolution policy; proportionally switching back traffic entering the API gateway by updating its routing rules; and forcing clients to reconnect to the primary site via a connection eviction mechanism for established long-lived connections. The traffic rollback employs a reverse canary rollout strategy compared to the forward switchover—for example, rolling back 70%, observing for 1 minute, and then rolling back to 100% to avoid sudden overload on the primary site.

[0118] S2073, Service Layer Rollback Simultaneously or immediately following the traffic rollback, the application service layer is rolled back, including: setting service instances started on the disaster recovery site to "maintenance mode" or taking them offline to prevent them from continuing to receive residual traffic; restoring service instances on the primary site that may have been stopped or downgraded during the switchover to normal operating status; and restoring the list of service instances in the service discovery registry (such as Nacos) to the snapshot version before the switchover.

[0119] S2074, Configuration and Infrastructure Rollback This phase leverages the Infrastructure as Code and Configuration Management Database (CMDB) capabilities to revert changes to application configurations, network policies (such as security groups and ACL rules), and middleware configurations (such as message queue subscription relationships and scheduled task policies) during the switchover process to their previous versions. The process includes pulling configuration snapshots from the CMDB or configuration center and distributing them in batches to target nodes using Ansible. For irreversible infrastructure changes, a resource rebuild process is triggered, and a pre-backed-up resource list is used for restoration.

[0120] S2075, Data Layer Rollback and Consistency Recovery Different data layer rollback strategies are adopted for different data scenarios. For scenarios where no business writes have occurred at the disaster recovery site (i.e., the anomaly occurred before the traffic switch or the write database was not reached during the canary phase), the data layer does not need to roll back; it only needs to restore the synchronization direction of the primary and backup databases to primary → backup. For scenarios where business writes have occurred at the disaster recovery site, reverse data backflow is required. This involves backflowing incremental data generated during the disaster recovery site switchback to the primary site through transaction log parsing, and performing data consistency checks during the backflow process to avoid data loss or duplication. For scenarios that cannot be automatically backflowed (such as transaction data involving complex business logic), the pipeline freezes relevant accounts, generates a discrepancy list, and submits it to the manual reconciliation processing channel. At the same time, distributed transactions need to trigger reverse compensation actions, caching systems (such as Redis) need to invalidate or preheat the cache, and messages that have not yet been consumed in the message queue need to be redirected or replayed.

[0121] S2076, Rollback Verification and Business Smoke Testing After all traffic, services, configurations, and data have been rolled back, the rollback pipeline enters the verification phase. This phase calls the automated business verification scripts mentioned above to perform end-to-end smoke tests on key business paths such as core accounting transactions, payment transactions, query transactions, and reconciliation transactions. Simultaneously, using the unified observability platform described above, the deviations of key metrics after the rollback (TPS, error rate, account reconciliation rate, etc.) from the baseline before the switchover are compared. All metrics must converge to the baseline's set range, such as ±5%, to determine a successful rollback. If verification fails, the pipeline enters the "secondary handling" branch, which includes retrying the rollback, upgrading to a full rollback, or transferring to a manual emergency handling process.

[0122] S2077, State Convergence and External Notification After a successful rollback is determined, the pipeline execution status convergence actions are as follows: the high-density monitoring sampling mode enabled during the switchover process is deactivated and normal monitoring is restored; the operation and maintenance freeze policy is lifted and the normal change and release process is restored; a rollback completion notification is sent to relevant business departments, customer service center, and management reporting system through the unified notification center; the switchover dashboard status is updated to "rollback completed" and all process data is retained for subsequent review and analysis.

[0123] S208, Post-Switch Review and Process Optimization The entire switchover process, including metrics, logs, and events, was collected and analyzed to generate a switchover performance report. Based on the post-mortem results, the switchover pipeline template, decision-making rules, and contingency plans were iteratively optimized. The post-mortem included: deviation analysis of actual and planned time consumption at each stage of the switchover; data consistency verification results; the consistency between the decision engine's decision-making basis and actual results; the prediction accuracy of the ML-assisted model; rollback triggering situations (if any); and a list of abnormal events and root cause analysis. The post-mortem results were stored in a structured report format and used as data for incremental training of the ML model, continuously improving the accuracy of the decision engine. Simultaneously, new abnormal patterns discovered during the post-mortem process were manually reviewed and added to the aforementioned anomaly detection rule base, enabling continuous evolution of the rule base.

[0124] This application has the following advantages over the prior art: First, it improves reliability and success rate. The standardized switchover pipeline breaks down the entire switchover process into automated stages, reducing human error; real-time business verification ensures business continuity; and a four-layer data consistency verification mechanism comprehensively guarantees data correctness from the physical, logical, business, and metadata layers. Through this application, the first-time success rate of switchover drills is significantly improved, and the problem of switchover failures due to human error is effectively resolved.

[0125] Second, it significantly reduces RTO. Automated execution drastically reduces switchover time from several hours in traditional manual switchovers to minutes; the intelligent decision engine accelerates response speed, enabling it to identify risks and trigger switchover preparation at the early stages of a failure. In typical implementation scenarios, the achieved RTO can be controlled within 1 hour, which is better than management requirements.

[0126] Third, it reduces risks and costs. End-to-end observability makes switching risks visible, allowing operations and maintenance personnel to monitor the switching status and the scope of anomalies in real time; routine drills reduce the cost of a single switching operation; and predefined rollback pipelines ensure rapid recovery capabilities in abnormal scenarios, reducing rollback time from hours to minutes.

[0127] Fourth, enhanced compliance. All operations are traceable and auditable, and the decision-making basis report fully records the reasoning chain of each round of decision-making; compliance checks are automated and embedded in each stage of the pipeline, avoiding the lag and omissions of manual compliance review; WORM immutable storage meets the requirements of financial management for the retention of audit evidence.

[0128] Fifth, promote continuous improvement. Based on the data review of each switchover, continuously optimize the switchover process and strategies; the ML-assisted model continuously absorbs new switchover experience through incremental learning, and the decision accuracy steadily improves with the number of drills; the anomaly detection rule base continues to evolve with the review, covering more and more anomaly patterns.

[0129] Accordingly, see Figure 4 As shown, in a second aspect, this application provides a service continuity performance testing apparatus during primary / standby handover, comprising: Pipeline definition module 401 is configured to model the entire process of remote switching between primary and backup sites of the business system into multiple different standardized stages, and define the multiple different standardized stages as a switching pipeline that is executed repeatedly. Management module 402 is configured to manage the infrastructure resources of the primary and backup sites using Infrastructure as Code, and to uniformly manage application configurations through a configuration management database to ensure the consistency of the environment and configuration before and after the primary and backup site switchover. The verification module 403 is configured to integrate a data consistency verification task in the switching pipeline, perform data consistency verification at critical nodes, and block the pipeline if the verification fails. Decision module 404 is configured to collect real-time monitoring data from primary and backup sites, utilize a dual-track intelligent decision engine, and combine preset business time windows and risk strategies to determine the timing of the switchover and trigger the switchover pipeline. The traffic switching module 405 is configured to switch a portion of non-critical business traffic from the main site to the disaster recovery site after triggering the switching pipeline. Simultaneously, an automated business verification script is run. After the verification is passed, the proportion of business traffic switched is gradually increased until the switch is complete. The observation module 406 is configured to track and visualize application performance, infrastructure status and / or business transaction links in real time during the switchover process through a unified observability platform. Security scanning and compliance checkpoints are built into each standard stage of the switchover pipeline as quality gates. If the current stage fails to pass the security scanning and compliance checkpoints, it will block the entry into the next stage of the switchover pipeline. The rollback module 407 is configured as a predefined rollback pipeline twin with the forward switching pipeline. When the rollback trigger index is detected to exceed the rollback threshold, the preset rollback pipeline is automatically triggered to restore the traffic and environment to the state before the switch.

[0130] The aforementioned service continuity performance testing device and the aforementioned service continuity performance testing method during primary / standby handover are consistent. For details, please refer to the description of the service continuity performance testing method during primary / standby handover above, which will not be repeated here.

[0131] Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for testing service continuity performance during primary / standby switchover, characterized in that, include: The entire process of switching between primary and backup sites of the business system is modeled as multiple different standardized stages, and these multiple different standardized stages are defined as a switching pipeline that is executed repeatedly. Infrastructure as Code is used to manage the infrastructure resources of primary and backup sites, and application configurations are managed uniformly through a configuration management database to ensure the consistency of environment and configuration before and after the primary and backup site switchover. Integrate a data consistency verification task into the switching pipeline, perform data consistency verification at critical nodes, and block the pipeline if the verification fails. Collect real-time monitoring data from primary and backup sites, utilize a dual-track intelligent decision engine, and combine preset business time windows and risk strategies to determine the timing of the switchover and trigger the switchover pipeline. After the switchover pipeline is triggered, a portion of the non-critical business traffic will be switched from the primary site to the disaster recovery site. Simultaneously, an automated business verification script will be run. After the verification is passed, the proportion of business traffic switched will be gradually increased until the switchover is complete. The unified observability platform enables real-time tracking and visualization of application performance, infrastructure status, and / or business transaction links during the switchover process. Security scans and compliance checkpoints are built into each standard stage of the switchover pipeline as quality gates. If the current stage fails to pass the security scan and compliance checkpoint, it will block the entry into the next stage of the switchover pipeline. The preset rollback pipeline is a twin of the forward switching pipeline. When the rollback trigger index is detected to exceed the rollback threshold, the preset rollback pipeline is automatically triggered to restore the traffic and environment to the state before the switchover.

2. The method according to claim 1, characterized in that, The method of using Infrastructure as Code (ICC) to manage the infrastructure resources of primary and backup sites, and using a configuration management database to uniformly manage application configurations, ensures the consistency of the environment and configuration before and after the primary and backup site switchover, including: The primary and backup dual-site mirroring is modeled using Infrastructure as Code (ICC). The infrastructure resources of the primary and backup sites are uniformly described using a declarative template, and the common definitions and site differences of the primary and backup sites are separated by site variable parameterization in the declarative template. Calculate hash fingerprints for the deployment artifact packages and application configuration items, and write the hash fingerprints as unique identifiers for the deployment artifact packages and configuration items into the configuration management database. When the primary and backup sites load the deployment artifact packages and configuration items, they compare the fingerprints. If the fingerprints do not match, the switchover is blocked. The application configuration is divided into three layers: baseline configuration, site configuration, and runtime configuration. A complete configuration list for the primary and backup sites is generated in advance before the switch. The baseline configuration of the primary and backup sites is completely consistent and centrally managed for version control. The site configuration is independently versioned according to the primary and backup site dimensions. The runtime configuration is dynamically injected when the primary and backup sites switch. Periodically scan the differences between the actual operating infrastructure resources and the declared state of the configuration management database of the primary and backup sites, and automatically trigger configuration write-back or alarm blocking for any unauthorized changes detected based on the differences between the infrastructure resources and the declared state of the configuration management database. Before the pipeline switching execution phase, a consistency check is automatically performed, and the pipeline enters the next phase after the consistency check passes. The consistency check includes at least one of the following: declarative template version number consistency check, fingerprint comparison, and configuration item difference whitelist check. Changes to configuration items or infrastructure resources are recorded in the version control system via transaction commits. After generating a unique change identifier, it is used as a rollback anchor point along with a timestamp snapshot. When a pipeline switchover fails, either the primary or backup site is rolled back to its pre-switch state based on the rollback anchor point.

3. The method according to claim 1, characterized in that, The integration of a data consistency verification task into the switching pipeline, performing data consistency verification at critical nodes, and halting the pipeline if verification fails, includes: Verify the integrity of the underlying database replication link and logs of the primary and backup sites; Verify that the data content of the databases of the primary and backup sites is consistent; Verify the data correctness at the business semantic level of the databases of the primary and backup sites; Verify the consistency of the table structures in the databases of the primary and backup sites.

4. The method according to claim 1, characterized in that, The dual-track intelligent decision engine includes an arbitration fusion unit, a main track rule engine, and a secondary track machine learning auxiliary model. The real-time monitoring data collected from the primary and backup sites is used to determine the switching timing using a dual-track intelligent decision engine, combined with preset business time windows and risk strategies. This includes: The main track rule engine first checks the triggering status of the forced blocking rule and the forced triggering rule. If neither the forced blocking rule nor the forced triggering rule is triggered, it calculates the switching necessity score and the switching risk score based on real-time monitoring data. Then, it matches the switching necessity score and the switching risk score with the preset decision matrix, and finally outputs the main decision instruction after making corrections using the preset business time window. If the main decision instruction output by the main track rule engine is a deterministic instruction, the arbitration fusion unit determines the switching information based on the main decision instruction; If the main decision instruction output by the main track rule engine is a non-deterministic instruction, the arbitration fusion unit combines the predicted success probability of switching, the predicted rollback probability, the optimal switching timing, and the non-deterministic instruction output by the secondary track machine learning auxiliary model to determine the switching information.

5. The method according to claim 4, characterized in that, The switching necessity score is obtained by weighted fusion of the current overall health of the main site, the intensity of the deterioration trend, the predicted failure probability, and the intensity of external triggering signals; the switching risk score is obtained by weighted fusion of data synchronization delay risk, disaster recovery site's own health risk, current business load risk, upstream and downstream dependent system risk, historical switching failure rate under the same conditions, and business time window risk.

6. The method according to claim 4, characterized in that, The process of collecting real-time monitoring data from primary and backup sites, utilizing a dual-track intelligent decision engine, and combining preset business time windows and risk strategies to determine the timing of switchover, also includes: The real-time monitoring data is then normalized.

7. The method according to claim 4, characterized in that, The main rule engine first checks the triggering status of the forced blocking rule and the forced triggering rule. If neither the forced blocking rule nor the forced triggering rule is triggered, it calculates the switching necessity score and the switching risk score based on real-time monitoring data, including: The real-time monitoring data is processed in parallel using four analysis methods: threshold analysis, trend analysis, predictive analysis, and anomaly detection, to output the current overall health of the main site, the intensity of the deterioration trend, and the predicted failure probability, which are used to calculate the switching necessity score. The threshold analysis involves setting two levels of thresholds for each set indicator: an alarm threshold and a circuit breaker threshold. When an indicator exceeds the alarm threshold, it enters an alarm state; when it exceeds the circuit breaker threshold, it enters a dynamic state. The trend analysis is based on extracting N time series sampling points in the most recent T minutes using a sliding window, calculating the trend slope using least squares linear regression, and calculating the goodness of fit. If the goodness of fit exceeds the set goodness threshold, the trend deterioration score is calculated based on the ratio of the trend slope to the preset deterioration threshold. The predictive analysis adopts a dual-track parallel design. The first prediction track extrapolates the remaining time to reach the circuit breaker threshold based on the linear regression model of trend analysis and maps it to the predicted failure probability. The second prediction track uses a two-layer LSTM time series prediction model to output the predicted values ​​and upper bounds of the confidence intervals for each future time step. The highest value of the first prediction track and the second prediction track is taken as the final predicted failure probability. The anomaly detection includes single-index step change detection and multi-index associated anomaly detection. Single-index step change detection compares the difference in mean between two adjacent sliding windows and triggers a change alarm. Multi-index associated anomaly detection matches the current multi-index state combination in real time based on a predefined anomaly index combination pattern rule base, outputs the anomaly detection result, and determines the triggering state of the forced triggering rule.

8. The method according to claim 7, characterized in that, The four analysis methods dynamically switch between three triggering modes based on the current state of the business system. The three triggering modes include: routine inspection mode, switch preparation mode, and event triggering mode. The normal inspection mode is the default operating state, and the relevant thresholds in this mode use standard values. The handover preparation mode indicates that there is a potential handover risk or that a handover plan is imminent. In this mode, the relevant thresholds are tightened based on the standard values. The event-triggered mode is a mode for setting up sudden events. In this mode, the relevant thresholds are tightened based on the values ​​used in the switching preparation mode.

9. The method according to claim 4, characterized in that, The process of collecting real-time monitoring data from primary and backup sites, utilizing a dual-track intelligent decision engine, and combining preset business time windows and risk strategies to determine the timing of switchover, also includes: Set different switching risk scores and mandatory blocking rules for different business time windows.

10. The method according to claim 1, characterized in that, The step of switching a portion of non-critical business traffic from the primary site to the disaster recovery site, simultaneously running an automated business verification script, and gradually increasing the proportion of switched business traffic until the switch is complete after successful verification includes: A preset grayscale step is obtained, which is dynamically selected according to the switching risk level, and different grayscale steps correspond to different traffic ratios. For each grayscale level, execute a sequence of atomic operations including traffic distribution weight adjustment, session persistence processing, health check injection, and data layer read / write separation confirmation; After each grayscale step transition is completed, the observation window is entered. If the automatic rollback condition is triggered in the observation window, the transition pipeline is immediately interrupted and a rollback is executed. At the end of each grayscale step, four types of verification scripts are automatically executed in parallel: core accounting transaction verification, batch and reconciliation verification, concurrency and capacity verification, and cross-site data consistency verification.

Images