Cross-data center fault isolation and switching method and system in multi-tenant environment

Abstract

The invention relates to the technical field of data center high availability, in particular to a cross-data center fault isolation and switching method and system in a multi-tenant environment. The resource mapping table is constructed by taking the tenants as the minimum control units, the affected tenants are accurately identified, logic isolation is executed, the problem of waste of full-tenant service migration resources caused by node-level or cluster-level switching in the prior art is avoided, and the fault influence range is minimized. The optimal data center is dynamically selected by combining the multi-factor objective function with the tenant SLA level, the defects that an existing scheduling strategy is opaque and tenant priorities cannot be distinguished can be overcome, and it is ensured that key tenant services are preferentially recovered. And finally, through combination of a gray takeover mechanism and health feedback confirmation and multi-dimensional recovery verification after migration, the condition that an existing recovery mechanism is extensive can be changed, the fault processing precision and the resource scheduling efficiency of the data center in a multi-tenant environment are remarkably enhanced, and the service continuity is guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of high availability of data centers, and in particular to a method and system for cross-data center fault isolation and switching in a multi-tenant environment. Background Art

[0002] Multi-tenant architectures are widely used in modern data center operations. In this architecture, independent service systems for different tenants often run on a shared pool of physical resources. While this model effectively improves resource utilization, it also introduces new challenges. When a physical resource or node fails, due to a lack of tenant-level awareness and response capabilities, the failure is likely to affect multiple unrelated tenants, triggering widespread cascading problems.

[0003] Existing cross-data center disaster recovery solutions have many shortcomings in practical applications.

[0004] First, using node-level or cluster-level failover often triggers the migration of entire services even if only a few tenants are affected. This not only results in significant resource waste but also increases the risk of SLA (Service Level Agreement) breaches.

[0005] Second, due to the lack of an effective impact domain identification mechanism, the system cannot accurately determine which tenants truly rely on the failed resources. It can only adopt a "wide coverage" preventive migration strategy, which results in the forced interruption or migration of services for unrelated tenants.

[0006] Third, the scheduling strategy lacks transparency and flexibility. The current platform finds it difficult to differentiate tenant priorities or service levels, and the goal of "prioritizing key tenants" cannot be achieved during the switching scheduling process, resulting in inefficient recovery of core businesses.

[0007] Fourth, the recovery mechanism is relatively crude, and the service verification mechanism after migration is weak, which easily leads to problems such as false service recovery and data inconsistency, seriously affecting business continuity.

[0008] Therefore, there is an urgent need for a disaster recovery solution that can start from the "tenant granularity", identify and locally isolate the scope of fault impact, and perform target selection and service reconstruction based on resource status and service level, so as to effectively improve the accuracy and response efficiency of disaster recovery to meet the high availability requirements of modern data centers. Summary of the Invention

[0009] The purpose of the present invention is to provide a cross-data center fault isolation and switching method and system in a multi-tenant environment to solve the above problems.

[0010] To achieve the above objectives, in one aspect of the present invention, a method for cross-data center fault isolation and switching in a multi-tenant environment is provided, comprising the following steps:

[0011] Build a tenant resource mapping table to record the relationship between tenants and their bound computing, storage, and network resources;

[0012] Collect key performance indicators of tenant running instances and calculate health scores;

[0013] Detecting a failure event, identifying an affected resource set, and determining an affected tenant set by matching the tenant resource mapping table;

[0014] Performing a logical isolation operation on the affected tenant set;

[0015] Based on multiple factors such as network access latency, switching cost, and resource availability, we construct an objective function, calculate the candidate data center scores, and select the optimal target data center.

[0016] Perform cross-data center migration of tenant services, using a grayscale takeover mechanism to gradually redirect requests, and confirm the status of the new instance through health feedback;

[0017] Perform recovery verification on the migrated services. After verification, update the resource mapping table and release the old resources.

[0018] Furthermore, in the method described above, the health scoring model is: calculating the tenant health score by at least one indicator including CPU occupancy, memory occupancy, disk I / O anomaly ratio and network anomaly ratio, and the weight of each indicator is configured according to the tenant's SLA level, where the lower the value, the closer to the failure boundary.

[0019] Furthermore, in the method described, the objective function is: calculating the score of the candidate data center based on at least one factor including network access delay, switching cost, and resource availability, selecting the candidate data center that minimizes the objective function value as the optimal target data center, and the weight of each factor is configured according to the tenant's SLA level.

[0020] Furthermore, in the method described above, the logical isolation operation includes: disconnecting the data path between the tenant instance and the shared component, freezing the configuration write permission, marking the tenant status as isolated, and triggering a disaster recovery snapshot backup.

[0021] Furthermore, in the method described, the grayscale takeover mechanism gradually directs requests to the new instance of the target data center according to a preset ratio, and the diversion ratio includes 10%, 30%, 50% and up to 100%. At each stage, health feedback is used to confirm whether the service accessibility, performance response, error rate and system resource occupancy of the new instance meet the preset thresholds.

[0022] Furthermore, in the method described above, the service recovery verification includes: API service reachability verification, configuration consistency verification, key log loading correctness verification and response rate verification within 30 seconds.

[0023] Furthermore, in the method described above, the migration modes of cross-data center migration include hot migration, cold migration, and active-active switching, and support asynchronous or incremental replication to reduce switching time.

[0024] In addition, in another aspect of the present invention, a cross-data center fault isolation and switching system in a multi-tenant environment is provided, which is characterized by comprising:

[0025] Resource mapping module, used to build and update tenant resource mapping tables;

[0026] Health score module, used to collect tenant status and calculate health score;

[0027] Fault identification module, used to detect fault events and identify affected tenants;

[0028] Isolation control module, used to perform tenant-level logical isolation operations;

[0029] Scheduling decision module, used to select the optimal target data center based on a multi-factor objective function;

[0030] Migration execution module, used to perform cross-data center service migration and implement grayscale takeover;

[0031] and a verification and convergence module for service recovery verification and resource status update.

[0032] Furthermore, in the system, the scheduling decision module includes a state acquisition unit and a policy scoring unit. The state acquisition unit obtains the network delay, switching cost and resource availability of the candidate data centers, and the policy scoring unit calculates the score based on the objective function and selects the optimal data center.

[0033] Furthermore, in the system, the migration execution module includes a health feedback unit for monitoring the health status of the new instance in real time and controlling the diversion ratio, and supporting interruption switching and rollback of traffic in the event of an abnormality.

[0034] Compared with the existing technology, the present invention has at least the following technical effects: the present invention constructs a resource mapping table with tenants as the minimum control unit, accurately identifies the affected tenants and performs logical isolation, thereby avoiding the problem of resource waste in the migration of all tenant services caused by node-level or cluster-level switching in the existing technology, thereby minimizing the scope of the fault impact. The use of multi-factor objective functions combined with tenant SLA levels to dynamically select the optimal data center can solve the defects of the existing scheduling strategy being opaque and unable to distinguish tenant priorities, and ensure that key tenant services are restored first. Finally, through the combination of the grayscale takeover mechanism and health feedback confirmation, as well as the multi-dimensional recovery verification after migration, the extensive status of the existing recovery mechanism can be changed, effectively avoiding false service recovery and data inconsistency, improving the flexibility, strategy and controllability of disaster recovery switching, significantly enhancing the fault handling accuracy and resource scheduling efficiency of the data center in a multi-tenant environment, and ensuring business continuity. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A method flow chart of a cross-data center fault isolation and switching method in a multi-tenant environment according to an embodiment of the present invention;

[0036] Figure 2 This is a flowchart of the overall design of a cross-data center fault isolation and switching method for a multi-tenant environment in one embodiment of the present invention;

[0037] Figure 3 The figure is a schematic diagram of the optimal target data center selection process in the event of a multi-tenant cross-data center failure in one embodiment of the present invention. DETAILED DESCRIPTION

[0038] The following is a more detailed description of a method and system for cross-data center fault isolation and switching in a multi-tenant environment, with reference to a schematic diagram. This diagram illustrates a preferred embodiment of the present invention. It should be understood that those skilled in the art may modify the present invention described herein while still achieving the beneficial effects of the present invention. Therefore, the following description should be understood as generally known to those skilled in the art and is not intended to limit the present invention.

[0039] For the sake of clarity, not all features of actual embodiments are described. In the following description, well-known functions and structures are not described in detail because they would obscure the present invention with unnecessary detail. It should be understood that in the development of any actual embodiment, numerous implementation details must be made to achieve the developer's specific goals, such as adapting from one embodiment to another to accommodate system or business constraints. Furthermore, it should be understood that such development work may be complex and time-consuming, but is nevertheless a routine undertaking for those skilled in the art.

[0040] The following paragraphs describe the present invention in more detail by way of example with reference to the accompanying drawings. The advantages and features of the present invention will become more apparent from the following description. It should be noted that the drawings are greatly simplified and not to exact scale, and are provided solely for the purpose of assisting in the description of the embodiments of the present invention.

[0041] Based on the teachings of this specification, those skilled in the art may form new technical solutions by cross-combining different implementation methods without generating technical contradictions. Such variations should be deemed to fall within the scope of protection of this patent.

[0042] Example 1

[0043] In a multi-tenant cloud environment, independent service systems of different tenants operate in a shared pool of physical resources. While this model improves resource utilization, when physical resources or nodes fail, existing cross-data center disaster recovery solutions lack tenant-level fault detection and handling capabilities, often causing failures to impact multiple tenants and triggering widespread cascading problems. For example, existing solutions often rely on node- or cluster-level failover, triggering the migration of entire services even if only a few tenants are affected, resulting in wasted resources and increased SLA risks. Furthermore, they lack precise impact domain identification, flexible scheduling strategies, and reliable recovery verification mechanisms, making it difficult to meet the differentiated disaster recovery needs of a multi-tenant environment.

[0044] In view of this, if Figure 1 and Figure 2 As shown, this embodiment proposes a cross-data center fault isolation and switching method for a multi-tenant environment, which achieves accurate fault handling and efficient service recovery by building mechanisms such as resource mapping, health scoring, and multi-factor scheduling. The method includes the following steps:

[0045] S1: Build a tenant resource mapping table to record the relationship between tenants and their bound computing, storage, and network resources.

[0046] S2: Collect key performance indicators of tenant running instances and calculate health scores.

[0047] S3: Detect a failure event, identify an affected resource set, and determine an affected tenant set by matching the tenant resource mapping table.

[0048] S4: Perform a logical isolation operation on the affected tenant set.

[0049] S5: Construct an objective function based on multiple factors such as network access latency, switching cost, and resource availability, calculate the candidate data center scores, and select the optimal target data center.

[0050] S6: Perform cross-data center migration of tenant services, use a grayscale takeover mechanism to gradually divert requests, and confirm the status of the new instance through health feedback.

[0051] S7: Perform recovery verification on the migrated service. After verification, update the resource mapping table and release the old resources.

[0052] For step S1, the generation of the tenant resource mapping table is strictly bound to the system life cycle and the tenant deployment process. During the system initialization phase or after the new tenant deployment is completed, the mapping table generation process will be automatically triggered to generate the corresponding mapping relationship based on the tenant's resource configuration information.

[0053] Specifically, during the system initialization phase, the platform will automatically scan the registered tenant information and its initial resource configuration to generate the corresponding mapping relationship. When the new tenant completes the deployment, the system will immediately trigger the mapping table creation process based on the resource application record of its service instance (such as computing nodes, storage volumes, and network links). For example, when tenant A deploys an e-commerce service in data center DC1, the system will automatically bind the server node S1, storage volume V1, and network IP address L1 it applied for to the tenant identifier A, forming an initial mapping table M(A) = {S1, V1, L1}. The mapping is stored in the form of key-value pairs to ensure the simplicity and queryability of the data structure.

[0054] Furthermore, the mapping table is standardized using the mathematical expression M(i) = {r_j}, where i is the tenant's unique identifier (such as the tenant ID or name), and {r_j} constitutes the set of resources bound to the tenant. Resource types include computing resources, storage resources, and network resources.

[0055] In this embodiment, computing resources include physical server nodes, virtual machine instances, etc. For example, the application service of tenant B runs on node S2 of data center DC2, and this node is included in the mapping table as r_j; storage resources include disk volumes, distributed storage partitions, etc. For example, the data file of tenant C is stored in storage volume V2, and V2 will be recorded as a resource item in the mapping table; network resources involve network links, IP addresses, load balancers, etc. For example, the service of tenant D communicates with the outside world through network link L2, and L2 becomes an indispensable component in the mapping table.

[0056] It should be noted that in order to adapt to the elastic changes of resources and platform operation and maintenance requirements in a multi-tenant environment, this mapping table has the ability to be dynamically updated in both directions.

[0057] The first is tenant-driven updates. Specifically, when a tenant performs elastic scaling due to traffic growth (e.g., adding a new server instance), the system captures the change event through the resource scheduling module and automatically adds the newly allocated resources to the corresponding tenant's mapping table. For example, if tenant A adds server node S1-1 during a promotion, the mapping table will be updated in real time to M(A) = {S1, V1, L1, S1-1}.

[0058] The second is platform operation and maintenance-driven updates. Specifically, if the platform makes operation and maintenance adjustments to physical resources (such as migrating storage volume V1 to a new physical device), the resource management module will synchronously refresh the resource physical location information in the mapping table to ensure that the mapping relationship is consistent with the actual deployment status. This dynamic update mechanism avoids the invalidation of resource binding relationships caused by static mapping, laying the foundation for accurate matching during subsequent fault identification - when a resource fails, the system can accurately screen out the tenants that are truly affected through the real-time updated mapping table, avoiding misjudgments or omissions caused by lagging mapping relationships.

[0059] For step S2, the system uses a timed polling method to collect key performance indicators of the tenant's running instance, for example, the default collection period is 5 minutes (the frequency can be adjusted according to the tenant's SLA level). In this embodiment, the key performance indicators include at least one of CPU utilization (U_cpu), memory occupancy (U_mem), disk I / O anomaly ratio (I_io), and network anomaly ratio (E_net). Among them, CPU utilization and memory occupancy are obtained by monitoring the system resource monitoring interface of the computing node where the tenant's running instance is located; the disk I / O anomaly ratio is obtained by counting the ratio of the number of timeouts in disk read and write operations to the total number of operations; the network anomaly ratio is obtained by monitoring the packet loss rate, delay jitter and other parameters during network transmission and calculating them comprehensively.

[0060] Furthermore, the health scoring model uses the key performance indicators collected above to calculate and generate a quantitative tenant health score H_t(i) using a weighted summation method. The specific health scoring model is as follows:

[0061] H_t(i)=α1·U_cpu+α2·U_mem+α3·I_io+α4·E_net

[0062] The value range of H_t(i) is 0-100%, and the lower the value, the closer the tenant is to the failure boundary. The weight parameters α1-α4 are dynamically configured according to the tenant's service level agreement (SLA) level. The following example shows:

[0063] VIP tenants (such as core e-commerce businesses): Configure α1 (CPU weight) to 0.4 and α2 (memory weight) to 0.3. Because their businesses require extremely high computing resource response speeds, the CPU metric accounts for the highest proportion.

[0064] For ordinary tenants (such as corporate websites), set α4 (network weight) to 0.3 and α3 (disk weight) to 0.2, focusing more on network connectivity and page loading speed when users access the site.

[0065] In addition, taking tenant A (VIP) as an example, if its U_cpu = 80%, U_mem = 60%, I_io = 5%, and E_net = 2%, the health score is: 0.4×80%+0.3×60%+0.2×5%+0.1×2%=32%+18%+1%+0.2%=51.2%. When this value exceeds the preset threshold (such as 50%), the system will trigger a sub-health warning.

[0066] The health score H_t(i) serves as a quantitative basis for tenant status and plays a dual role in decision support in the system:

[0067] Fault warning trigger: When the score is continuously below a threshold (e.g., 60%), the system determines that the tenant has entered a "sub-healthy" state and performs resource pre-allocation or migration preparations in advance to avoid service interruptions caused by sudden failures. For example, if the network packet loss rate of tenant B continues to increase, causing H_t(i) to drop to 45%, the system will automatically prepare a backup network link for it;

[0068] Migration priority decision-making: In resource competition scenarios, health scores can be used as a reference for migration order, prioritizing service stability for low-scoring (high-risk) tenants. For example, if data center DC1 is about to fail, the system sorts H_t(i) from smallest to largest, prioritizing the migration of tenants with the lowest health scores to the backup center.

[0069] As for step S3, this step ensures the accurate positioning of the fault impact range through real-time monitoring and mapping matching mechanism to avoid interference to unrelated tenants.

[0070] Specifically, the system monitors infrastructure in real time through monitoring probes deployed in each data center. Upon detecting any abnormal event, including node downtime, unreachable network links, and I / O queue timeouts, the system immediately locates the source of the failure and generates a set of faulty resources, F = {r_k}, where r_k represents the specific resource instance of the failure. For example, if server node S1 in data center DC1 fails, the system will include S1 as r_k in set F, forming F = {S1}, providing a clear identifier for the faulty resource for subsequent tenant association.

[0071] After obtaining the fault resource set F, the system traverses the constructed tenant resource mapping table M(i) and performs mathematical intersection operations to filter the affected tenants. The specific logic is: for each tenant i, if the intersection of its mapping table M(i) and the fault set F is not empty (i.e. ), the tenant is determined to be dependent on the faulty resource and is included in the affected tenant set A. For example, if tenant A's mapping table M(A) = {S1, V1, L1} is set to {S1}, then tenant A will be identified as affected because M(A) and F intersect {S1}. However, tenant B's mapping table M(B) = {S2, V2} does not intersect with F and is excluded from the processing. This precise matching based on mapping relationships avoids the disruption to unrelated tenants that can occur with traditional "broad-coverage" migrations.

[0072] In addition, the core advantage of this step is that it enables "tenant-level" fault impact domain identification. Through real-time matching of resource mapping tables, the system only performs subsequent processing (such as logical isolation and migration) on tenants who truly rely on the faulty resources, while the service operations of unrelated tenants are completely unaffected. This precise positioning significantly reduces unnecessary service interruptions and resource migration overhead. Compared with traditional cluster-level switching solutions, it can reduce interference with unrelated tenants while improving fault handling efficiency, providing technical support for refined disaster recovery in multi-tenant environments.

[0073] In step S4, once the system identifies the affected tenants, it performs multi-level logical isolation operations, including disconnecting the data path between the tenant instance and the shared component, freezing configuration write permissions, marking the tenant as isolated, and triggering disaster recovery snapshot backups.

[0074] Specifically, first, disconnect the data connection between the tenant instance and shared components such as distributed cache and middleware to prevent the fault from spreading to other tenants or platform core services through public components; second, freeze the tenant configuration write permission to prevent configuration changes under abnormal conditions from polluting system data and ensure the consistency of tenant status; at the same time, mark the tenant status as "isolated" in the resource scheduling platform to temporarily exit the regular scheduling cycle to avoid incorrect resource allocation or triggering invalid migration; if the system enables the disaster recovery snapshot mechanism, it will also trigger instant data backup to solidify the tenant's current service status, configuration parameters and data mirroring to prevent recovery failure due to status loss.

[0075] The isolation process follows the principle of non-destructive design. All operations do not directly interrupt the operation of tenant services. Instead, "soft isolation" is achieved by limiting their interaction with external components. For example, disconnecting the data connection only blocks the communication link between the tenant and shared resources, while the service process within the tenant remains running; freezing configuration writes does not affect the reading of existing configurations, but only prohibits new change operations. This design ensures that the tenant state remains available during the isolation period. When the subsequent migration or recovery process is started, the current state can be reused directly without reinitializing the service, which greatly shortens the service reconstruction time. At the same time, it avoids request loss or transaction anomalies caused by service interruption, providing state continuity guarantee for cross-data center migration.

[0076] For step S5, when it is detected that a tenant service instance cannot operate normally due to a resource failure (such as node downtime, network anomaly, etc.), the scheduling platform will start the target data center selection process to determine the optimal migration deployment location to ensure that the service is quickly restored in the new environment.

[0077] Specifically, the scheduling platform first collects the status of candidate data centers. Through the status collection module, the system obtains real-time operational status indicators for all candidate data centers. These indicators include at least access latency (L_j), which is the time loss in data transmission between different data centers; switching cost (C_j), including the time required for service migration and data synchronization; and resource availability (R_j), such as the percentage of remaining computing resource capacity. These key data provide a comprehensive basis for subsequent target center screening.

[0078] Next, the objective function DC* = argmin_j(ω1·L_j+ω2·C_j+ω3·R_j) is constructed using the multiple metrics collected above as factors. This function selects the optimal target data center by weighting the various metrics of each candidate data center. ω1 through ω3 are weight parameters that can be dynamically configured based on the tenant's SLA level. Specifically, argmin_j represents the index j that minimizes the objective function within the brackets, which is the optimal target data center number*.

[0079] Weight configuration is closely related to tenant priority, enabling differentiated service guarantees for resource allocation. When faced with tenants of varying importance, the system can allocate resources more specifically, ensuring that key tenants' services are prioritized in the optimal resource pool. This effectively improves the flexibility of resource scheduling and the ability to guarantee tenant services, effectively meeting the differentiated needs of different tenants in a multi-tenant environment. For example, for VIP tenants, to prioritize low latency and ensure service quality, ω1 is set to 0.5, giving it a higher weight in the network access latency indicator. The system calculates the comprehensive score of all candidate centers using this objective function, and ultimately selects the candidate data center with the lowest score as the optimal target data center at the tenant granularity.

[0080] As an example, Figure 3 As shown in Figure 1, assume that there are three candidate data centers, and the weight configuration of a VIP tenant in data center DC1 is ω1 = 0.5 (delay priority), ω2 = 0.3, and ω3 = 0.2. The indicators and scores of each center are as follows: Table 1:

[0081] Table 1

[0082] Data Center Delay L_j (ms) Switching cost C_j(s) Resource availability R_j (%) Weighted score <![CDATA[DC2]]> 70 60 55 35+18+11＝64 <![CDATA[DC3]]> 90 40 75 45+12+15＝72 <![CDATA[DC4]]> 50 30 85 25+9+17＝51

[0083] As shown in Table 1 above, the system calculated that DC4 had the lowest score and ultimately selected DC4 as the optimal target data center for migration, reflecting the priority protection of the low latency requirements of VIP tenants.

[0084] For step S6, when performing cross-data center migration of tenant services, the system first selects different migration modes such as hot migration, cold migration or active-active switching based on the capabilities of the target data center, and supports asynchronous or incremental replication to reduce switching time. Specifically, the system will first package the tenant instance status, configuration files, metadata and associated data paths, and then rebuild the operating environment in the target data center and start a new service instance. If the platform supports asynchronous or incremental replication technology, incremental synchronization will be used first, and only the data that has changed since the last backup will be copied, thereby significantly shortening the data transmission time. After the instance deployment is completed, the system will update the tenant routing table to direct subsequent requests to the newly deployed instance.

[0085] To ensure the controllability and service quality of the migration process, this step adopts a grayscale takeover mechanism combined with a health feedback confirmation strategy. The grayscale takeover mechanism requires the system not to switch all traffic to the new instance at once, but to gradually divert requests to the new instance in the target data center according to a preset ratio (such as 10%, 30%, 50% to 100%). The status of the new instance must be confirmed through health feedback at each diversion stage. The monitoring indicators include service reachability (such as the success rate of API interface response), performance response (such as whether the average latency is less than 200ms), error rate (such as whether the HTTP5xx error rate is less than 0.1%), and system resource usage (such as whether the CPU and memory usage are within a reasonable range). The system will compare these indicators with the preset thresholds, and only when all indicators meet the standards will it enter the next stage of traffic expansion.

[0086] This gradual traffic switching method allows the original instance to continue running during the migration process, so that when an abnormality is found in the new instance, the traffic can be rolled back in time to reduce the switching risk. For example, when tenant A's microservice is migrated from DC1 to DC2, the system will first divert 5% of the requests to the new instance in DC2, and continuously monitor its interface response time, system load and error code distribution. If the various indicators are stable, the diversion scale will be gradually expanded according to the predetermined proportion until 100% traffic takeover is completed. The entire process is automatically executed by the system without manual intervention, and supports interruption and rollback at any stage, effectively balancing switching efficiency and business continuity, and is particularly suitable for the differentiated needs of different SLA levels in multi-tenant environments.

[0087] In step S7, after the tenant service completes the cross-data center migration and restarts in the target data center, the system immediately initiates a multi-dimensional service recovery verification process to ensure that the service is fully available and consistent in the new environment. This service recovery verification includes: API service accessibility verification, configuration consistency verification, key log loading correctness verification, and 30-second response rate verification.

[0088] Specifically, first, a probe request is sent to the API endpoint to check whether the HTTP response status code is level 200 to verify the reachability of the API service; then the configuration file of the new instance is compared line by line with the master copy in the disaster recovery snapshot to ensure that there are no missing parameters or configuration conflicts; at the same time, key log files and status files are parsed to verify whether there are abnormal stack information or data loading errors during the service startup process; finally, within the 30-second verification window, the request response rate is calculated by injecting simulated traffic, and it is required to meet the preset SLA threshold (such as 99%).

[0089] If all check items pass, the system enters the state convergence phase: a cache clearing instruction is called to delete all state caches of the tenant in the old data center to prevent subsequent requests from being misrouted; a release instruction is sent to the resource management module to reclaim the computing, storage, and network resources of the failed node; and the tenant resource mapping table M(i) is synchronously updated, replacing the tenant's resource entry from the old resource identifier (e.g., DC1: S1) with the new resource identifier (e.g., DC4: S4) to ensure that the scheduling system can subsequently correctly address it. If any check item fails, the system immediately triggers the emergency plan: a rollback is first attempted based on the state retained during the grayscale takeover phase, switching traffic back to the original data center; if the rollback fails, the objective function calculation is reactivated, and the suboptimal data center is selected for a second switch.

[0090] In summary, the present invention constructs a resource mapping table with tenants as the minimum control unit, accurately identifies the affected tenants and performs logical isolation, thereby avoiding the problem of resource waste in the migration of all tenant services caused by node-level or cluster-level switching in the prior art, thereby minimizing the scope of the fault impact. The use of a multi-factor objective function combined with the tenant's SLA level to dynamically select the optimal data center can solve the defects of the existing scheduling strategy being opaque and unable to distinguish tenant priorities, and ensure that key tenant services are restored first. Finally, through the combination of the grayscale takeover mechanism and health feedback confirmation, as well as the multi-dimensional recovery verification after migration, the existing extensive recovery mechanism can be changed, effectively avoiding false service recovery and data inconsistency, improving the flexibility, strategy and controllability of disaster recovery switching, significantly enhancing the fault handling accuracy and resource scheduling efficiency of the data center in a multi-tenant environment, and ensuring business continuity.

[0091] Example 2

[0092] This embodiment provides a cross-data center fault isolation and switching system for a multi-tenant environment, including:

[0093] The resource mapping module is used to build and update the tenant resource mapping table; the health scoring module is used to collect tenant status and calculate health scores; the fault identification module is used to detect fault events and identify affected tenants; the isolation control module is used to perform tenant-level logical isolation operations; the scheduling decision module is used to select the optimal target data center based on a multi-factor objective function; the migration execution module is used to perform cross-data center service migration and implement grayscale takeover; the verification and convergence module is used for service recovery verification and resource status update.

[0094] Specifically, the system implements cross-data center fault isolation and switching in a multi-tenant environment through modular design, with each module working together to form a complete closed loop. The resource mapping module automatically generates a tenant resource mapping table during system initialization or tenant deployment, recording the binding relationship between tenants and computing, storage, and network resources, and supports dynamic updates during elastic scaling or operation and maintenance adjustments, providing a data basis for subsequent fault location. The health scoring module periodically collects key indicators such as CPU usage and memory usage of tenant instances, calculates the health score based on the SLA level configuration weight, and triggers an early warning when the score approaches the fault boundary, providing a quantitative basis for fault detection.

[0095] The fault identification module monitors events such as node downtime and network anomalies in real time, and accurately locates the affected tenant set by matching the resource mapping table to avoid interference from irrelevant tenants. The isolation control module performs logical isolation on the affected tenants, including disconnecting shared components, freezing configuration writes, marking isolation status, and triggering disaster recovery snapshots to prevent the spread of faults and maintain state consistency. The scheduling decision module includes a state collection unit and a policy scoring unit. The state collection unit obtains indicators such as network latency, switching cost, and resource availability of candidate data centers. The policy scoring unit calculates scores based on a multi-factor objective function and selects the optimal target center according to the SLA level configuration weight. For example, low-latency DC4 is given priority for VIP tenants.

[0096] The migration execution module selects hot migration and other modes according to the capabilities of the target center, packages the tenant status and rebuilds the environment, and implements grayscale takeover through the health feedback unit: gradually diverting traffic at a ratio of 10% to 100%, monitoring the response time, error rate and other indicators of the new instance in real time, and immediately rolling back traffic in case of anomalies. The verification and convergence module performs multi-dimensional verification such as API accessibility and configuration consistency after the service is restarted. If it passes, it clears the old center cache, releases resources and updates the mapping table. If it fails, it triggers a rollback or secondary switch. The system realizes automated control of the entire process from fault identification to service recovery through data interaction and process connection between modules, significantly improving disaster recovery accuracy and resource scheduling efficiency.

[0097] The above description is merely a preferred embodiment of the present invention and does not limit the present invention in any way. Any person skilled in the art who, without departing from the scope of the present invention, makes any equivalent substitution, modification, or other changes to the technical solution and technical content disclosed in the present invention shall be deemed to be within the scope of the present invention and still fall within the scope of protection of the present invention.

Claims

1. A cross-data center fault isolation and switching method in a multi-tenant environment, characterized in that: The following steps are involved: Build a tenant resource mapping table to record the relationship between tenants and their bound computing, storage, and network resources; Collect key performance indicators of tenant running instances and calculate health scores; Detecting a failure event, identifying an affected resource set, and determining an affected tenant set by matching the tenant resource mapping table; Performing a logical isolation operation on the affected tenant set; Based on multiple factors such as network access latency, switching cost, and resource availability, we construct an objective function, calculate the candidate data center scores, and select the optimal target data center. Perform cross-data center migration of tenant services, gradually redirect requests using a grayscale takeover mechanism, and confirm the status of the new instance through health feedback; Perform recovery verification on the migrated services. After verification, update the resource mapping table and release the old resources.

2. The method according to claim 1, characterized in that The health scoring model is as follows: the tenant health score is calculated by at least one indicator including CPU utilization, memory utilization, disk I / O anomaly ratio and network anomaly ratio. The weight of each indicator is configured according to the tenant's SLA level, where the lower the value, the closer to the failure boundary.

3. The method according to claim 1, characterized in that The objective function is: based on at least one factor including network access delay, switching cost and resource availability, the candidate data center is scored, and the candidate data center that minimizes the objective function value is selected as the optimal target data center. The weight of each factor is configured according to the tenant's SLA level.

4. The method according to claim 1, wherein The logical isolation operation includes: disconnecting the data path between the tenant instance and the shared component, freezing the configuration write permission, marking the tenant status as isolated, and triggering disaster recovery snapshot backup.

5. The method according to claim 1, wherein The grayscale takeover mechanism gradually directs requests to the new instance in the target data center according to a preset ratio, including 10%, 30%, 50% and up to 100%. At each stage, health feedback is used to confirm whether the service accessibility, performance response, error rate and system resource usage of the new instance meet the preset thresholds.

6. The method according to claim 1, characterized in that The service recovery verification includes: API service reachability verification, configuration consistency verification, key log loading correctness verification and response rate verification within 30 seconds.

7. The method according to claim 1, characterized in that The migration modes of cross-data center migration include hot migration, cold migration and active-active switching, and support asynchronous or incremental replication to reduce switching time.

8. A cross-data center fault isolation and switching system in a multi-tenant environment, characterized in that: include: Resource mapping module, used to build and update tenant resource mapping tables; Health score module, used to collect tenant status and calculate health score; Fault identification module, used to detect fault events and identify affected tenants; Isolation control module, used to perform tenant-level logical isolation operations; Scheduling decision module, used to select the optimal target data center based on a multi-factor objective function; Migration execution module, used to perform cross-data center service migration and implement grayscale takeover; and a verification and convergence module for service recovery verification and resource status update.

9. The system according to claim 8, characterized in that The scheduling decision module includes a state collection unit and a strategy scoring unit. The state collection unit obtains the network delay, switching cost and resource availability of the candidate data centers. The strategy scoring unit calculates the score based on the objective function and selects the optimal data center.

10. The system according to claim 8, wherein: The migration execution module includes a health feedback unit for real-time monitoring of the health status of the new instance and controlling the diversion ratio, supporting interruption switching and rollback of traffic in the event of an abnormality.

Images