Disaster recovery switching method and related equipment

Abstract

The invention provides a disaster recovery switching method and related equipment. The method is applied to a cloud system, and the cloud system comprises a cloud management platform and an infrastructure. The infrastructure at least comprises a first database node, a first storage node, a second database node and a second storage node. The first database node is used for processing the write request information of the tenant. The first storage node is used for sending the log data to the second storage node. And the second database node is used for realizing synchronous replication between the second storage node and the first storage node according to the log data under the condition that the first database node and / or the first storage node does not fail. Under the condition that a first database node and / or a first storage node fails, the method comprises the following steps: a cloud management platform indicates the first storage node to stop updating data according to the indication of the first database node; and the cloud management platform instructs the second database node to process the write request information of the tenant. The method can realize disaster recovery switching.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of cloud computing, and more particularly, to a disaster recovery switching method, a cloud system, a computing device cluster, a computer program product, and a computer readable storage medium. BACKGROUND

[0002] With the continuous development of database technology, tenants' demands for the availability and disaster recovery capability of databases are also increasing. Tenants not only hope to quickly recover from failures when database nodes fail, but also hope not to lose any data. In order to meet the needs of tenants, a master node and at least one backup node can be deployed for the database of a tenant, so that the operations of the tenant in the master node can be synchronized to the at least one backup node, so as to realize the recovery from failures by starting the backup node in the case of failure of the master node. Since a certain time is required for the backup node to synchronize data with the master node, the time required for the backup node to provide database services for the tenant under the condition of ensuring no data loss is long, resulting in poor experience of the tenant.

[0003] Therefore, how to quickly recover from failures when database nodes fail and avoid losing data as much as possible has become a problem to be solved. SUMMARY

[0004] The present application provides a disaster recovery switching method, a cloud system, a computing device cluster, a computer program product, and a computer readable storage medium, which can quickly recover from failures when database nodes fail and avoid losing data as much as possible.

[0005] In a first aspect, a disaster recovery switching method is provided. The method is applied to a cloud system, which includes a cloud management platform and an infrastructure. The cloud management platform is configured to manage the infrastructure, which includes a plurality of database nodes and a plurality of storage nodes. The plurality of database nodes includes a first database node and at least one backup database node of the first database node. The first database node and each of the at least one backup database node belong to different clusters. The first database node is configured to process write request information of a tenant, the write request information being used to request to update data. The plurality of storage nodes includes a first storage node corresponding to the first database node and a storage node corresponding to each of the at least one backup database node. The first storage node is configured to send log data to the storage node corresponding to each of the at least one backup database node. Each of the at least one backup database node is further configured to, in a case where the first database node and / or the first storage node does not fail, implement synchronous replication between the storage node corresponding to each of the at least one backup database node and the first storage node according to the log data. In a case where the first database node and / or the first storage node fails, the method includes: the cloud management platform sending first indication information to the first storage node, the first indication information being used to instruct the first storage node to stop updating data according to an instruction of the first database node; and the cloud management platform sending second indication information to a second database node, the second indication information being used to instruct the second database node to process the write request information of the tenant, the second database node belonging to the at least one backup database node.

[0006] In the embodiments of the present application, the first database node is a master node and is configured to process the write request information of the tenant. The first database node is further configured with the at least one backup database node. The first database node and the at least one backup database node are connected through the synchronous replication mode between the corresponding storage nodes, so as to synchronize the data and maintain the consistency of the data, thereby avoiding the loss of data in a case where the first database node fails. In other words, the at least one backup database node is in a normal working state rather than a shutdown state, so that the fast switching of the database node can be implemented in a case where the first database node and / or the first storage node fails, thereby providing the database service for the tenant by the second database node of the at least one backup database node and achieving the fast recovery from the failure.

[0007] In conjunction with the first aspect, in some implementations, in the event of a failure of the first database node and / or the first storage node, the method further includes: the cloud management platform sending a third instruction message to the first storage node, the third instruction message being used to instruct the first storage node to release the synchronous replication mode with the storage nodes corresponding to each backup database node; and / or, the cloud management platform sending a fourth instruction message to the second storage node, the second storage node being the storage node corresponding to the second database node, the fourth instruction message being used to instruct the second storage node to establish a synchronous replication mode with the first storage node and each storage node other than the second storage node among the storage nodes corresponding to each backup database node.

[0008] In this embodiment, after the first database node and / or the first storage node fails, the cloud management platform also needs to instruct the first storage node to release the synchronization replication mode with the storage node corresponding to the backup database node, so that the first storage node no longer sends log data to other storage nodes. The cloud management platform can also instruct the second storage node to establish a synchronization replication mode with the storage node corresponding to the backup database node, so that the second storage node sends log data to other storage nodes to achieve data synchronization.

[0009] In conjunction with the first aspect, in some implementations, the method further includes: if no status information of the first database node is received within each of the N preset time periods, the cloud management platform determines that the first database node has failed, where N is a positive integer; and / or, if no status information of the first storage node is received within each of the N preset time periods, the cloud management platform determines that the first storage node has failed, where N is a positive integer.

[0010] In this embodiment, each database node and / or storage node of the tenant needs to send status information to the cloud management platform to indicate that it has not failed. This allows the cloud management platform to determine that a node has failed if it does not receive status information at least once, and thus determine whether to perform disaster recovery switching.

[0011] In conjunction with the first aspect, in some implementations, when the second database node processes the tenant's write request information, the method further includes: if the data transmission delay between the second database node and the second storage node is greater than or equal to a first preset threshold, the cloud management platform sends a fifth instruction message to the second storage node, the fifth instruction message being used to instruct the second storage node to switch the synchronous replication mode between the second storage node and the first storage node, and the storage nodes corresponding to each backup database node (excluding the second storage node) to an asynchronous replication mode; and / or, if the data transmission delay between the second storage node and the third storage node is greater than or equal to the second preset threshold, the cloud management platform sends a sixth instruction message to the second storage node, the sixth instruction message being used to instruct the second storage node to switch the synchronous replication mode between the second storage node and the third storage node to an asynchronous replication mode, wherein the third storage node belongs to the storage nodes corresponding to the first storage node and each backup database node (excluding the second storage node); wherein the second storage node is the storage node corresponding to the second database node.

[0012] In this embodiment of the application, when the second database node is upgraded to the master node among the tenant's multiple database nodes, the cloud management platform can also switch the replication mode between storage nodes according to the data transmission latency between the second database node and the second storage node, and the data transmission latency between the second storage node and other storage nodes, thereby avoiding business blockage caused by network fluctuations, avoiding prolonging the time required for tenants to obtain response information, and thus improving the tenant experience.

[0013] In conjunction with the first aspect, in some implementations, if the first database node and / or the first storage node do not fail, the second database node is also used to process tenant read request information, which is used to request the reading of data.

[0014] In conjunction with the first aspect, in some implementations, when the second database node processes the tenant's write request information, the first database node and the backup database nodes other than the second database node in each backup database node are used to process the tenant's read request information, which is used to request to read data.

[0015] In this embodiment, the backup database node among the tenant's multiple database nodes can provide data read services to the tenant, thereby avoiding resource waste. Furthermore, keeping the backup database node operational reduces the switching time caused by restarting the backup database node and synchronizing data during disaster recovery failover, improving the tenant experience.

[0016] In conjunction with the first aspect, in some implementations, the method further includes: the cloud management platform receiving at least one of the following during each preset time period in at least one preset time period: status information of a first database node, status information of a first storage node, status information of each backup database node, and status information of the storage node corresponding to each backup database node, wherein the status information of the first database node is used to indicate whether the first database node has failed, the status information of the first storage node is used to indicate at least one of the following: whether the first storage node has failed, and whether the data transmission link between the first storage node and the storage node corresponding to each backup database node has failed, the status information of each backup database node is used to indicate whether each backup database node has failed, and the status information of the storage node corresponding to each backup database node is used to indicate at least one of the following: whether the storage node corresponding to each backup database node has failed, and whether the data transmission link between the storage node corresponding to each backup database node and the first storage node has failed.

[0017] In this embodiment, the cloud management platform can monitor the health status of each database node and each storage node, thereby promptly determining whether a database node and / or storage node has failed and performing timely disaster recovery switching. The cloud management platform can also monitor the data transmission links between storage nodes, thereby switching the replication mode promptly when data transmission latency between storage nodes is high, avoiding business blockage.

[0018] In conjunction with the first aspect, in some implementations, the cloud management platform includes a first arbitration component, a second arbitration component, and at least one third arbitration component. The first arbitration component belongs to a different region from the second arbitration component and the at least one third arbitration component. The second arbitration component belongs to the same region as the first database node. Each of the at least one third arbitration component belongs to the same region as one of the at least one backup database node. The first arbitration component is used to determine the second database node from the at least one backup database node in the event of a failure of the first database node and / or the first storage node. The second arbitration component is connected to the first database node and / or the first storage node. Each third arbitration component is connected to the backup database node corresponding to each third arbitration component.

[0019] In this embodiment, the cloud management platform can be configured with multiple arbitration components. By using arbitration components located in different regions from both the primary node (e.g., the first database node) and the backup node (e.g., the second database node), it can determine whether a backup node can be promoted to the primary node when the primary node fails. This achieves automatic failover, avoids the risks of manual operation, and reduces the time required for disaster recovery failover (e.g., reducing failover time from minutes to seconds). Setting up multiple arbitration components located in different regions can avoid the split-brain problem between multiple backup nodes, improve the reliability of the disaster recovery failover system, and prevent accidental failover.

[0020] Secondly, a cloud system is provided. This cloud system includes a cloud management platform and infrastructure. The cloud management platform manages the infrastructure, which includes multiple database nodes and multiple storage nodes. The multiple database nodes include a first database node and at least one backup database node of the first database node. The first database node and each of the at least one backup database node belong to a different cluster. The first database node processes write request information from tenants, which requests data updates. The multiple storage nodes include a first storage node corresponding to the first database node and a storage node corresponding to each backup database node. The first storage node sends log data to the storage node corresponding to each backup database node. Each backup database node is also used to synchronize and replicate data between the storage node corresponding to each backup database node and the first storage node, based on the log data, in the event that the first database node and / or the first storage node are not faulty. In the event of a failure of the first database node and / or the first storage node, the cloud management platform is configured to: send a first instruction message to the first storage node, the first instruction message being used to instruct the first storage node to stop updating data according to the instruction of the first database node; and send a second instruction message to the second database node, the second instruction message being used to instruct the second database node to process the tenant's write request information, the second database node being at least one backup database node.

[0021] In conjunction with the second aspect, in some implementations, in the event of a failure of the first database node and / or the first storage node, the cloud management platform is further configured to: send a third instruction to the first storage node, the third instruction being used to instruct the first storage node to release the synchronous replication mode with the storage nodes corresponding to each backup database node; and / or, send a fourth instruction to the second storage node, the second storage node being the storage node corresponding to the second database node, the fourth instruction being used to instruct the second storage node to establish a synchronous replication mode with the first storage node and each storage node other than the second storage node among the storage nodes corresponding to each backup database node.

[0022] In conjunction with the second aspect, in some implementations, the cloud management platform is also used to: determine that the first database node has failed if no status information of the first database node is received in each of the N preset time periods, where N is a positive integer; and / or determine that the first storage node has failed if no status information of the first storage node is received in each of the N preset time periods, where N is a positive integer.

[0023] In conjunction with the second aspect, in some implementations, when the second database node processes the tenant's write request information, the cloud management platform is further configured to: send a fifth instruction message to the second storage node when the data transmission delay between the second database node and the second storage node is greater than or equal to a first preset threshold. The fifth instruction message is used to instruct the second storage node to switch the synchronous replication mode between the second storage node and the first storage node, as well as the storage nodes corresponding to each backup database node (excluding the second storage node), to an asynchronous replication mode; and / or, when the data transmission delay between the second storage node and the third storage node is greater than or equal to the second preset threshold, send a sixth instruction message to the second storage node. The sixth instruction message is used to instruct the second storage node to switch the synchronous replication mode between the second storage node and the third storage node, where the third storage node belongs to the storage nodes corresponding to the first storage node and each backup database node (excluding the second storage node); wherein, the second storage node is the storage node corresponding to the second database node.

[0024] In conjunction with the second aspect, in some implementations, if the first database node and / or the first storage node do not fail, the second database node is also used to process tenant read request information, which is used to request the reading of data.

[0025] In conjunction with the second aspect, in some implementations, when the second database node processes the tenant's write request information, the first database node and the backup database nodes other than the second database node in each backup database node are used to process the tenant's read request information, which is used to request to read data.

[0026] In conjunction with the second aspect, in some implementations, the cloud management platform is also used to: receive at least one of the following within each preset time period in at least one preset time period: status information of a first database node, status information of a first storage node, status information of each backup database node, and status information of the storage node corresponding to each backup database node, wherein the status information of the first database node is used to indicate whether the first database node has failed, the status information of the first storage node is used to indicate at least one of the following: whether the first storage node has failed, and whether the data transmission link between the first storage node and the storage node corresponding to each backup database node has failed, the status information of each backup database node is used to indicate whether each backup database node has failed, and the status information of the storage node corresponding to each backup database node is used to indicate at least one of the following: whether the storage node corresponding to each backup database node has failed, and whether the data transmission link between the storage node corresponding to each backup database node and the first storage node has failed.

[0027] In conjunction with the second aspect, in some implementations, the cloud management platform includes a first arbitration component, a second arbitration component, and at least one third arbitration component. The first arbitration component belongs to a different region from the second arbitration component and the at least one third arbitration component. The second arbitration component belongs to the same region as the first database node. Each of the at least one third arbitration component belongs to the same region as one of the at least one backup database node. The first arbitration component is used to determine the second database node from the at least one backup database node in the event of a failure of the first database node and / or the first storage node. The second arbitration component is connected to the first database node and / or the first storage node. Each third arbitration component is connected to the backup database node corresponding to each third arbitration component.

[0028] Thirdly, this application provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method in the first aspect or any possible implementation of the first aspect.

[0029] Fourthly, this application provides a computer program product containing instructions that, when executed by a cluster of computer devices, cause the cluster of computer devices to perform the method described in the first aspect or any possible implementation thereof.

[0030] Fifthly, this application provides a computer-readable storage medium including computer program instructions that, when executed by a cluster of computing devices, perform the method described in the first aspect or any possible implementation thereof.

[0031] In a sixth aspect, a chip system is provided, the chip system including logic circuitry for coupling with an input / output interface, through which data is transmitted to perform the method described in the first aspect or any possible implementation thereof. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the cloud scenario.

[0033] Figure 2 This is a schematic structural diagram of a disaster recovery switching system according to an embodiment of this application.

[0034] Figure 3 This is a schematic flowchart of a disaster recovery switching method according to an embodiment of this application.

[0035] Figure 4 This is a schematic structural diagram of a cloud management platform according to an embodiment of this application.

[0036] Figure 5 This is a schematic structural diagram of a computing device according to an embodiment of this application.

[0037] Figure 6 This is a schematic structural diagram of a computing device cluster according to an embodiment of this application.

[0038] Figure 7 This is a schematic diagram illustrating the connection between computing devices 500A and 500B via a network according to an embodiment of this application. Detailed Implementation

[0039] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0040] This application will present various aspects, embodiments, or features relating to a system comprising multiple devices, components, modules, etc. It should be understood and appreciated that individual systems may include additional devices, components, modules, etc., and / or may not include all the devices, components, modules, etc. discussed in conjunction with the accompanying drawings. Furthermore, combinations of these approaches are also possible.

[0041] Furthermore, in the embodiments of this application, the words "exemplary," "for example," etc., are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" in the embodiments of this application should not be construed as being better or more advantageous than other embodiments or design schemes. Specifically, the use of the term "exemplary" is intended to present the concept in a concrete manner.

[0042] The business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0043] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including, but not limited to," unless otherwise specifically emphasized.

[0044] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects have an "OR" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0045] The technical solutions in this application embodiment can be applied to various cloud systems, including a cloud management platform and infrastructure. The cloud management platform is used to manage the infrastructure providing cloud services. The infrastructure includes at least one cloud data center, each cloud data center including at least one server, and each server deploying at least one database node and / or at least one storage node. The cloud management platform, as... Figure 1 As shown.

[0046] Figure 1 This is a schematic diagram of a cloud service scenario used in this application, such as... Figure 1 As shown, this cloud scenario may include: a cloud management platform 110, the Internet 120, and a client 130. For example... Figure 1 As shown, the cloud management platform 110 is used to manage the infrastructure that provides multiple cloud services. The infrastructure includes multiple cloud data centers, each containing at least one server, and each server containing cloud service resources to provide corresponding cloud services to tenants. For example, cloud service resources may be cloud databases or cloud storage resources.

[0047] The cloud management platform 110 can be located in a cloud data center. The cloud management platform 110 provides access interfaces (such as user interfaces or application program interfaces, APIs). Tenants can use client 130 to remotely access the access interface to register a cloud account and password on the cloud management platform 110 and log in. After successful authentication of the cloud account and password, the tenant can further select and purchase compute nodes (such as containers, virtual machines, servers, computing devices, etc.) with specific specifications (such as processors, memory, and disks) on the cloud management platform 110. After successful purchase, the cloud management platform 110 provides a remote login account and password for the purchased compute node. The tenant can remotely log in to the compute node through client 130 and install and run the tenant's applications on the compute node. Therefore, tenants can create, manage, log in to, and operate compute nodes in the cloud data center through the cloud management platform 110. The compute node can also be called a cloud server (Elastic Compute Service, ECS) or elastic instance (different cloud service providers may use different names).

[0048] It should be understood that cloud service tenants can be individuals, businesses, schools, hospitals, government agencies, etc.

[0049] The cloud management platform 110 includes, but is not limited to, a user console, compute management services, network management services, storage management services, authentication services, and image management services. The user console provides an interface or API for interaction with tenants. The compute management services manage servers running virtual machines and containers, as well as bare metal servers. The network management services manage network services (such as gateways and firewalls). The storage management services manage storage services (such as data bucket services). The authentication services manage tenant account passwords. The image management services manage virtual machine images. Tenants can log in to the cloud management platform 110 via client 130 and the internet 120 to manage their rented cloud services.

[0050] In some embodiments, the cloud management platform 110 includes multiple arbitration components, such as... Figure 2 The cloud management platform 210 is shown in the figure. Figure 2 This is a schematic structural diagram of a cloud system provided in an embodiment of this application. Figure 2 The cloud system 200 includes a cloud management platform 210 and infrastructure. The cloud management platform 210 includes multiple arbitration components, such as a first arbitration component 211, a second arbitration component 212, and a third arbitration component 213. The infrastructure includes multiple database nodes and multiple storage nodes, such as a first database node 220, a first storage node 230, a second database node 240, and a second storage node 250. Specifically, the first arbitration component 211 is deployed in a first region; the second arbitration component 212, the first database node 220, and the first storage node 230 are all deployed in a second region; and the third arbitration component 213, the first database node 240, and the first storage node 250 are all deployed in a third region. The first, second, and third regions are different regions.

[0051] For example, a first database node 220 is used to provide database services to a tenant. This first database node 220 can be considered as the master node among a tenant's multiple database nodes. The first database node 220 is used to receive and process write request information from the tenant, which requests data updates. The data to be updated by the write request information is stored in the first database node 220 and / or the first storage node 230. Processing the write request information by the first database node 220 includes writing data according to the tenant's write request information. The first database node 220 can also receive and process read request information from the tenant, which requests data read. The data to be read by the read request information is stored in the first database node 220 and / or the first storage node 230. Processing the read request information by the first database node 220 includes reading the data according to the tenant's read request information and providing it to the tenant.

[0052] For example, write request information may include insert statements, update statements, and delete statements. Read request information may include select statements.

[0053] For example, the first database node 220 sends status information to the second arbitration component 212 in the area where the first database node 220 is located at a preset time. This status information indicates whether the first database node 220 has failed. The preset time is determined according to a preset period (also known as a preset time interval). For example, assuming the preset period is 10 seconds, the first database node 220 can send a status message to the second arbitration component 212 every 10 seconds. Alternatively, if the first database node 220 fails, it will be unable to send status information to the second arbitration component 212.

[0054] For example, the first database node 220 is configured with at least one storage node. Each of the at least one storage node is used to provide storage services for the first database node 220. The following description takes the first storage node 230 as an example. It should be understood that the functions that each storage node corresponding to the first database node 220 can perform are similar to the functions that the first storage node 230 can perform, and for the sake of brevity, they will not be described again.

[0055] For example, the first storage node 230 is used to provide storage services to the first database node 220, that is, the first storage node 230 assists the first database node 220 in storing data. For example, when there is a large amount of tenant data, the storage capacity of the first database node 220 may be small and unable to store all the tenant data, so some data can be stored in the first storage node 230. The first storage node 230 is used to update and / or read data in the first storage node 230 according to the instructions of the first database node 220. When the first storage node 230 updates data according to the instructions of the first database node 220, it can also obtain log data. The log data is used to indicate at least one update operation performed on the data. The update operation includes at least one of the following: inserting data, modifying data, deleting data, etc.

[0056] For example, the first storage node 230 sends status information to the second arbitration component 212 in the region where the first storage node 230 is located at a preset time. This status information indicates at least one of the following: whether the first storage node 230 has failed, or whether the data transmission link between the first storage node 230 and the storage node corresponding to each backup database node has failed. This preset time is determined according to a preset period (also known as a preset time interval). Alternatively, if the first storage node 230 fails, the first storage node 230 is unable to send status information to the second arbitration component 212.

[0057] For example, the first database node 220 and the first storage node 230 belong to the same region, or the same availability zone (AZ), or the same data center. A region includes at least one AZ, and an AZ includes at least one data center.

[0058] For example, the first storage node generates log data on its own after receiving an instruction from the first database node 220, or the first storage node receives an instruction and log data from the first database node 220.

[0059] For example, the first database node 220 is configured with at least one backup database node. Each of these backup database nodes is used to provide database services to the tenant in the event of a failure of the first database node 220, thereby enabling rapid recovery from database failures and avoiding impact on the tenant's user experience. This at least one backup database node may include, for example, a second database node 240. In other words, when the first database node 220 is the primary node among the tenant's multiple database nodes, the second database node 240 is a backup node among the tenant's multiple database nodes.

[0060] For example, the first database node 220 and the second database node 240 belong to different database node clusters, or to different regions, or to different AZs, or to different data centers, which facilitates fault isolation and avoids the problem that the first database node and the second database node may fail at the same time, resulting in the inability to achieve rapid fault recovery.

[0061] For example, each region where the first database node 220 is located has a third arbitration component. Each third arbitration component is connected to the backup database node in that region, thereby receiving status information of the backup database node and storage node in that region. Each third arbitration component belongs to the cloud management platform 210.

[0062] For example, the second database node 240 sends status information to the third arbitration component 213 in the area where the second database node 240 is located at a preset time. This status information indicates whether the second database node 240 has failed. The preset time is determined according to a preset period (also known as a preset time interval). For example, assuming the preset period is 10 seconds, the second database node 240 can send a status message to the third arbitration component 213 every 10 seconds. Alternatively, if the second database node 240 fails, it will be unable to send status information to the third arbitration component 213.

[0063] For example, the second database node 240 is configured with at least one storage node, each of which provides storage services for the second database node 240. The following description uses the second storage node 250 as an example. It should be understood that the functions implemented by each storage node corresponding to the second database node 240 are similar to those implemented by the second storage node 250; for the sake of brevity, these will not be repeated.

[0064] For example, when the first database node 220 and / or the first storage node 230 are not faulty, the first storage node 230 and the second storage node 240 operate in a synchronous replication mode. This synchronous replication mode includes: when the first database node 220 sends an instruction to the first storage node 230 to write first data, the first storage node 230 writes the first data and sends first log data to the second storage node 250. This first log data is used to indicate the writing of the first data. Based on the first log data obtained by the second storage node 250, the second database node 240 instructs the second storage node 250 to write the first data. After the second storage node 250 writes the first data, it sends a response to the first storage node 230 indicating that the first data writing is complete. After receiving the response from the second storage node 250, the first storage node 230 sends a response to the first database node 220 indicating that the first data writing is complete. After receiving the response from the first storage node 230, the first database node 220 sends a response to the tenant indicating that the first data writing is complete. In other words, if the first database node 220 and / or the first storage node 230 do not fail, the second database node 240 is in a working state, and the second database node realizes synchronous replication between the second storage node and the first storage node based on log data from the first storage node.

[0065] For example, if the first database node 220 and / or the first storage node 230 do not fail, the second database node 240 is also used to provide data read services to the tenant. The second database node 240 is used to receive and process read request information from the tenant, which requests to read data. The data requested to be read is stored in the second database node 240 and / or the second storage node 250. The second database node 240 processes the read request information by: reading the data according to the tenant's read request information and providing it to the tenant.

[0066] For example, if the first database node 220 and / or the first storage node 230 do not fail, the second database node 240 is unable to receive and / or process tenant write request information.

[0067] For example, the second database node 240 and the second storage node 250 belong to the same region, or the same Availability Zone (AZ), or the same data center.

[0068] For example, the second storage node 250 sends status information to the third arbitration component 213 in the area where the second storage node 250 is located at a preset time. This status information indicates at least one of the following: whether the second storage node 250 has failed, or whether the data transmission link between the first storage node 230 and the second storage node 250 has failed. This preset time is determined according to a preset period (also known as a preset time interval). Alternatively, if the second storage node 250 fails, the second storage node 250 is unable to send status information to the third arbitration component 213.

[0069] For example, at least two of the preset times corresponding to the first database node 220, the first storage node 230, the second database node 240, and the second storage node 250 may be the same or different.

[0070] For example, if an arbitration component in a region (e.g., a second region or a third region) fails to receive status information from a database node within each of N preset time periods, or if the arbitration component in a region obtains status information indicating a database node failure, the arbitration component determines that the database node has failed. N is a positive integer. Each preset time period is determined based on a preset time corresponding to that preset time period. For example, the start time of each preset time period is the sum of the preset time corresponding to each preset time period and a first error value, and the end time of each preset time period is the sum of the preset time corresponding to each preset time period and a second error value. Similarly, if an arbitration component in a region (e.g., a second region or a third region) fails to receive status information from a storage node within each of N preset time periods, or if the arbitration component in a region obtains status information indicating a storage node failure, the arbitration component determines that the storage node has failed. N is a positive integer. Each preset time period is determined based on a preset time corresponding to that preset time period. For example, the start time of each preset time period is the sum of the preset time corresponding to each preset time period and a first error value, and the end time of each preset time period is the sum of the preset time corresponding to each preset time period and a second error value.

[0071] For example, assuming three preset times are 12:01:00, 12:01:15, and 12:01:30, with a first error value of -5 seconds and a second error value of 5 seconds, then the three preset time periods corresponding to these three preset times are 12:00:55-12:01:05, 12:01:10-12:01:20, and 12:01:25-12:01:35. If the second arbitration component 212 in the second region does not receive status information from the first database node 220 (or the first storage node 230) within the preset time period 12:01:25-12:01:35, then the second arbitration component 212 determines that the first database node 230 (or the first storage node 230) has failed. Alternatively, assuming that the second arbitration component 212 in the second region does not receive status information from the first database node 220 (or the first storage node 230) within the three preset time periods, the second arbitration component 212 determines that the first database node 220 (or the first storage node 230) has failed.

[0072] For example, in the event of a failure of the first database node 220 and / or the first storage node 230, the cloud management platform 210 performs... Figure 3 The method described in the article switches the primary node in the tenant's database nodes from the first database node 220 to the second database node 240, thereby achieving disaster recovery switching.

[0073] For example, if the first database node 220 and / or the first storage node 230 fails, and the first database node 220 is configured with multiple backup database nodes, all of which are in normal condition, the first arbitration component 211 determines from the multiple backup database nodes to make the second database node 240 the master node among the tenant's database nodes based on information sent by the arbitration component in the region where each backup database node is located.

[0074] For example, when the second database node 240 is the master node in a tenant's database nodes, the second database node 240 is used to receive and process write request information from the tenant, which requests to update data. The write request information requests the data to be updated to be stored in the second database node 240 and / or the second storage node 250. The second database node 240 processes the write request information by writing data according to the tenant's write request information. The second database node 240 can also receive and process read request information from the tenant, which requests to read data. The read request information requests the data to be read to be stored in the second database node 240 and / or the second storage node 250. The second database node 240 processes the read request information by reading the data according to the tenant's read request information and providing it to the tenant.

[0075] For example, when the second database node 240 is the master node among the tenant's database nodes, the second storage node 250 and the third storage node are in synchronous replication mode. The third storage node belongs to the storage nodes other than the second storage node 250 among the storage nodes corresponding to the first storage node 230 (which has recovered to a normal state) and each backup database node. This synchronous replication mode includes: when the second database node 240 sends an instruction to the second storage node 250 to write second data, the second storage node 250 writes the second data and sends second log data to the third storage node. This second log data is used to indicate the writing of the second data. The database node corresponding to the third storage node instructs the third storage node to write the second data based on the second log data obtained by the third storage node. After the third storage node writes the second data, the third storage node sends a response to the second storage node 250 to indicate that the second data writing is complete. After receiving the response from the third storage node, the second storage node 250 sends a response to the second database node 240 to indicate that the second data writing is complete. After receiving the response from the second storage node 250, the second database node 240 sends a response to the tenant to indicate that the second data writing is complete.

[0076] For example, when the second database node 240 is the master node among the tenant's database nodes, the third database node can also be used to provide data read services to the tenant. This third database node belongs to the first database node 220 that has recovered to a normal state, and to backup database nodes other than the second database node 240. The third database node is used to receive and process read request information from the tenant, which requests to read data. The data requested to be read is stored in the third database node and / or the corresponding storage node 250. The third database node's processing of the read request information includes: reading the data according to the tenant's read request information and providing it to the tenant.

[0077] In the cloud system provided in this application embodiment, a first database node is the master node, used to process tenant write request information. The first database node is also configured with at least one backup database node. Data synchronization between the first database node and the at least one backup database node is achieved through a synchronous replication mode between corresponding storage nodes, thereby maintaining data consistency and preventing data loss in the event of a failure of the first database node. When the first database node is the master node, the at least one backup database node is in a normal working state, not a shutdown state. Therefore, in the event of a failure of the first database node and / or the first storage node, a rapid switchover of database nodes can be achieved, allowing a second database node among the at least one backup database node to provide database services to the tenant, thus achieving rapid fault recovery.

[0078] Figure 3 This is a schematic flowchart of the disaster recovery switching method provided in the embodiments of this application. Figure 3 The methods described are applied to cloud systems, such as... Figure 1 , Figure 2 The cloud system in the middle. Figure 3 The method includes the following steps.

[0079] 310. In the event of a failure of the first database node and / or the first storage node, the cloud management platform sends a first instruction message to the first storage node.

[0080] In some embodiments, the cloud system includes a cloud management platform and infrastructure. The cloud management platform manages the infrastructure, which includes multiple database nodes and multiple storage nodes. The multiple database nodes include a first database node and at least one backup database node of the first database node. The first database node and each of the at least one backup database node belong to a different region. The first database node processes write request information from tenants, which requests data updates. The multiple storage nodes include a first storage node corresponding to the first database node and a storage node corresponding to each backup database node. The first storage node sends log data to the storage node corresponding to each backup database node. Each backup database node is also used to synchronize and replicate data between the storage node corresponding to each backup database node and the first storage node, based on the log data, in the event that the first database node and / or the first storage node are not faulty. For a detailed description of the first database node, the first storage node, the backup database node, and the storage node corresponding to the backup database node, see [link to relevant documentation]. Figure 2 The description in the text.

[0081] In some embodiments, in the event of a failure of the first database node and / or the first storage node, the cloud management platform sends a first instruction message to the first storage node. This first instruction message instructs the first storage node to stop updating data according to the instructions of the first database node. In other words, this first instruction message instructs the first storage node to freeze writes to the first database node.

[0082] For example, after receiving the first instruction information, the first storage node no longer updates data according to the instruction of the first database node, that is, it no longer accepts writes from the first database node. Since the first storage node and the storage nodes corresponding to each backup database node have a synchronous replication mode, the data in the first storage node and the storage nodes corresponding to each backup database node remain consistent when the first storage node no longer accepts writes from the first database node.

[0083] In some embodiments, the cloud management platform receives at least one of the following within each preset time period of at least one preset time period: status information of a first database node, status information of a first storage node, status information of each backup database node, and status information of the storage node corresponding to each backup database node. The status information of the first database node indicates whether the first database node has failed. The status information of the first storage node indicates whether the first storage node has failed and whether the data transmission link between the first storage node and the storage node corresponding to each backup database node has failed. The status information of each backup database node indicates whether each backup database node has failed. The status information of the storage node corresponding to each backup database node indicates whether the storage node corresponding to each backup database node has failed and whether the data transmission link between the storage node corresponding to each backup database node and the first storage node has failed. The method for determining the at least one preset time period is described in [reference needed]. Figure 2 The description in the text.

[0084] In some embodiments, if a first preset condition is met, the cloud management platform determines that a database node (e.g., a first database node or a backup database node) has failed. If a second preset condition is met, the cloud management platform determines that the database node (e.g., a first database node or a backup database node) has not failed. The first preset condition includes any one of the following: (1) The cloud management platform does not receive status information from the database node (e.g., a first database node or a backup database node) within each of N preset time periods, where N is a positive integer. (2) The status information of the database node (e.g., a first database node or a backup database node) is used to indicate that the database node (e.g., a first database node or a backup database node) has failed. The second preset condition includes any one of the following: (1) The cloud management platform receives status information from the database node (e.g., a first database node or a backup database node) within a preset time period. (2) The status information of the database node (e.g., a first database node or a backup database node) is used to indicate that the database node (e.g., a first database node or a backup database node) has not failed.

[0085] In some embodiments, if a third preset condition is met, the cloud management platform determines that the storage node (e.g., the storage node corresponding to the first storage node or the backup database node) has failed. If a fourth preset condition is met, the cloud management platform determines that the storage node (e.g., the storage node corresponding to the first storage node or the backup database node) has not failed. The third preset condition includes any one of the following: (1) The cloud management platform does not receive status information of the storage node (e.g., the storage node corresponding to the first storage node or the backup database node) in any of the N preset time periods, where N is a positive integer. (2) The status information of the storage node (e.g., the storage node corresponding to the first storage node or the backup database node) is used to indicate that the storage node (e.g., the storage node corresponding to the first storage node or the backup database node) has failed. The fourth preset condition includes any one of the following: (1) The cloud management platform receives status information of the storage node (e.g., the storage node corresponding to the first storage node or the backup database node) within a preset time period. (2) The status information of the storage node (e.g., the storage node corresponding to the first storage node or the backup database node) is used to indicate that the storage node (e.g., the storage node corresponding to the first storage node or the backup database node) has not failed.

[0086] In some embodiments, the cloud management platform can also predict potential failures and perform disaster recovery switching in advance based on a fault prediction model. This fault prediction model is an artificial intelligence (AI) model trained using historical fault data. When the fault prediction model predicts that the first database node and / or the first storage node is about to fail, the cloud management platform can execute... Figure 3 The methods described herein are used to achieve disaster recovery switching.

[0087] For example, "node not malfunctioning" and "node in normal state" have similar meanings and can be used interchangeably. "node malfunctioning" and "node in abnormal state" have similar meanings and can be used interchangeably.

[0088] In some embodiments, the cloud management platform determines whether to perform a disaster recovery switch based on the status information of each database node, the status information of each storage node, and a first correspondence. This first correspondence indicates the relationship between the status information of each database node, the status information of each storage node, and the operations performed by the cloud management platform.

[0089] For example, the first correspondence is used to indicate at least one of the following: (1) In the event of a failure of the primary node and / or the storage node corresponding to the primary node among multiple database nodes, and provided that at least one backup node among the multiple database nodes and the storage node corresponding to each of the at least one backup node are not failures, a disaster recovery switchover operation shall be performed. The disaster recovery switchover operation includes at least one of the following: instructing the storage node corresponding to the original primary node to stop writing new data; determining a new primary node; instructing the new primary node to enable the ability to process write request information; and instructing the storage node corresponding to the new primary node to establish a synchronous replication mode with the storage node corresponding to the backup node.

[0090] (2) In the event of a failure in the primary node and / or the storage node corresponding to the primary node among multiple database nodes, or in the event of a failure in each backup node and / or the storage node corresponding to each backup node among multiple database nodes, no disaster recovery switchover operation will be performed. The disaster recovery switchover operation is described above.

[0091] (3) In the event of a failure of the primary node and the storage node corresponding to the primary node in multiple database nodes, or in the event of a failure of a backup node and / or the storage node corresponding to the backup node in multiple database nodes, the synchronous replication mode between the storage node corresponding to the primary node and the storage node corresponding to the backup node shall be switched to the asynchronous replication mode.

[0092] (4) In the event of a failure of the primary node and the storage node corresponding to the primary node in multiple database nodes, and after a failure of one of the backup nodes and / or the storage node corresponding to the backup node in multiple database nodes is restored to normal, the asynchronous replication mode between the storage node corresponding to the primary node and the storage node corresponding to the backup node shall be switched to synchronous replication mode.

[0093] (5) If the data transmission delay between the master node and the storage node corresponding to the master node in multiple database nodes is greater than or equal to the first preset threshold, the synchronous replication mode between the storage node corresponding to the master node and the storage node corresponding to the backup node is switched to the asynchronous replication mode.

[0094] (6) When the data transmission delay between the master node and the storage node corresponding to the master node in multiple database nodes is less than the first preset threshold, the asynchronous replication mode between the storage node corresponding to the master node and the storage node corresponding to each backup node is switched to synchronous replication mode.

[0095] (7) If the delay of the data transmission link between the storage node corresponding to the master node and the storage node corresponding to the backup node in multiple database nodes is greater than or equal to the first preset threshold, the synchronous replication mode between the storage node corresponding to the master node and the storage node corresponding to the backup node is switched to the asynchronous replication mode.

[0096] (8) If the latency of the data transmission link between the storage node corresponding to the master node and the storage node corresponding to the backup node in multiple database nodes is less than the first preset threshold, the asynchronous replication mode between the storage node corresponding to the master node and the storage node corresponding to the backup node is switched to synchronous replication mode.

[0097] In some embodiments, data transmission between storage nodes and between database nodes and storage nodes uses different communication links to achieve network isolation between storage and database, thereby reducing interference between data transmission between storage nodes and database operations, and ensuring high-bandwidth and low-latency data synchronization as much as possible. For example, storage nodes replicate data through an independent high-speed plane, where data transmission speed is relatively fast.

[0098] For example, the storage node corresponding to the master node and the storage node corresponding to the backup node synchronize all data through direct connection during the first data synchronization, and synchronize through log data during subsequent data synchronization.

[0099] 320, The cloud management platform sends a second instruction message to the second database node.

[0100] After the cloud management platform sends the first instruction information to the first storage node, it sends a second instruction information to the second database node. This second instruction information instructs the second database node to process the tenant's write request information. This second database node is at least one backup database node of the first database node. (See also: [link to second database node description]). Figure 2 The description in the document states that, in the event of a failure of the first database node (i.e., the primary node) and / or the first storage node (i.e., the storage node corresponding to the primary node), the cloud management platform will switch the backup database node (i.e., the second database node) to the primary node.

[0101] In some embodiments, if the first database node has only one backup database node, and the cloud management platform determines that the backup database node and its corresponding storage node are not faulty, the cloud management platform determines that backup database node as the second database node. If the first database node has multiple backup database nodes, and the cloud management platform determines that the multiple backup database nodes and their corresponding storage nodes are not faulty, the cloud management platform determines the second database node from among the multiple backup database nodes.

[0102] In some embodiments, the cloud management platform has a first arbitration component, a second arbitration component, and at least one third arbitration component. The first arbitration component belongs to a different region from the second arbitration component and the at least one third arbitration component. The second arbitration component belongs to the same region as the first database node and the first storage node. Each of the at least one third arbitration component belongs to the same region as one of the at least one backup database node and the storage node corresponding to that backup database node. The first arbitration component is used to determine the second database node from the at least one backup database node in the event of a failure of the first database node and / or the first storage node. The second arbitration component is connected to the first database node and / or the first storage node. The second arbitration component is used to receive status information of the first database node and / or the first storage node. Each third arbitration component is connected to the backup database node and the storage node corresponding to that backup database node within the region where the third arbitration component is located. Each third arbitration component is used to receive status information of the database node and / or storage node within the region where the third arbitration component is located. See [link to documentation for first, second, and third arbitration components]. Figure 2 The description in the text.

[0103] In some embodiments, the first arbitration component has a persistent connection with the second arbitration component and each third arbitration component. The second arbitration component has a persistent connection with the first database node and the first storage node. Each third arbitration component has a persistent connection with the backup database node and storage node in the region where the third arbitration component is located.

[0104] For example, any one of the first arbitration component, the second arbitration component, and at least one third arbitration component can be deployed in a public cloud or / or a private cloud to achieve cross-cloud disaster recovery switching.

[0105] In some embodiments, where the first database node has only one backup database node, the cloud management platform includes a first arbitration component, a second arbitration component, and a third arbitration component. In the event of a failure in the first database node and / or the first storage node, the second arbitration component sends a message to the third arbitration component indicating that the first database node and / or the first storage node has failed. If the third arbitration component determines that the backup database node and its corresponding storage node are not failed, it identifies the backup database node as the second database node, and sends a second indication message to that backup database node. Alternatively, in the event of a failure in the first database node and / or the first storage node, the second arbitration component sends a message to the first arbitration component indicating that the first database node and / or the first storage node has failed. If it determines that the backup database node and its corresponding storage node are not failed, the first arbitration component sends a message to the third arbitration component, instructing the third arbitration component to send a second indication message to the backup database node.

[0106] In some embodiments, where the first database node has multiple backup database nodes, the cloud management platform includes a first arbitration component, a second arbitration component, and multiple third arbitration components. In the event of a failure in the first database node and / or the first storage node, the second arbitration component sends a message to the first arbitration component indicating that the first database node and / or the first storage node has failed. If the first arbitration component determines that the multiple backup database nodes and the storage node corresponding to each backup database node have not failed, the first arbitration component identifies a second database node from among the multiple backup database nodes and sends a message to the third arbitration component in the region where the second database node is located, instructing the third arbitration component to send a second instruction message to the backup database node. Since the first arbitration component is deployed outside the region where the first database node and each backup database node are located, in the event of a failure in the first database node, the first arbitration component can identify a second database node from among the multiple backup database nodes, thereby avoiding the split-brain problem caused by multiple backup database nodes simultaneously identifying themselves as the primary node among the tenant's database nodes.

[0107] For example, the first arbitration component determines the second database node itself from the one or more fault-free backup database nodes. Alternatively, the first arbitration component sends a message to each third arbitration component, instructing each backup database node to preempt the target lock, and designating the backup database node that preempts the target lock as the second database node. The target lock is set in either the first arbitration component or each third arbitration component. The target lock can only be held by one database node. In other words, once a backup database node preempts the target lock, other backup database nodes cannot hold the target lock until it is released.

[0108] In some embodiments, if the first database node and / or the first storage node are not faulty, the second database node is used to implement synchronous replication between the second storage node and the first storage node based on log data in the second storage node. The second storage node is the storage node corresponding to the second database node. The log data in the second storage node comes from the first storage node. See [link to documentation] for the second storage node and the synchronous replication mode between the second and first storage nodes. Figure 2 The description in the text.

[0109] In some embodiments, if the first database node and / or the first storage node do not fail, the second database node is used to provide database read services to the tenant. See [link to specific implementation details] Figure 2 The description in the document states that, in other words, when the second database node is a backup node, it only has the ability to process tenant read requests, but not tenant write requests.

[0110] In some embodiments, after the second database node receives the second instruction information, it is promoted to the master node among the tenant's database nodes. Therefore, the second database node's read-only restriction is lifted, granting it the ability to process tenant write requests. In the case of the second database node being promoted to master node, the first database node is demoted to a backup node among the tenant's database nodes.

[0111] For example, the primary node in a tenant's database node provides database services to the tenant, receiving and processing write requests from the tenant. The backup node in the tenant's database node synchronizes the data in the storage node corresponding to the backup node with the data in the storage node corresponding to the primary node, based on the log data in the storage node. The backup node can also provide data read services to the tenant, i.e., receive and process read requests from the tenant.

[0112] In some embodiments, the second instruction information is further used to instruct the second database node to terminate the read-only session. In other words, the second instruction information is also used to instruct the second database node to stop providing read data services to the tenant and instead begin providing write data services to the tenant.

[0113] In some embodiments, after receiving the second indication information, the second database node also sends a first response information to the cloud management platform. This first response information indicates that the second database node has the capability to process write request information.

[0114] In some embodiments, after sending the second instruction information to the second database node, the cloud management platform also sends a third instruction information to the first storage node. This third instruction information instructs the first storage node to release the synchronization replication mode with the storage nodes corresponding to each backup database node. In other words, in the event of a failure of the first database node and / or the first storage node, the cloud management platform can instruct the first storage node to stop synchronizing log data with the storage nodes corresponding to each backup database node.

[0115] In some embodiments, after sending the second instruction information to the second database node, the cloud management platform also sends a fourth instruction information to the second storage node. The fourth instruction information instructs the second storage node to establish a synchronous replication mode with the first storage node and each storage node other than the second storage node among the storage nodes corresponding to each backup database node. In other words, when the second database node is upgraded to the primary node, the second storage node needs to establish a synchronous replication mode with the storage nodes corresponding to each backup database node. This facilitates the second storage node sending log data to the storage nodes corresponding to each backup database node, thereby synchronizing the data stored in the second storage node and the storage nodes corresponding to each backup database node, and maintaining data consistency in the storage nodes.

[0116] In some embodiments, when the second database node is upgraded to the primary node, if the data transmission latency between the second database node and the second storage node is greater than or equal to a first preset threshold, the cloud management platform (e.g., a third arbitration component) sends a fifth instruction message to the second storage node. This fifth instruction message instructs the second storage node to switch the synchronous replication mode between itself and the first storage node, and among the storage nodes corresponding to each backup database node (excluding the second storage node itself), to an asynchronous replication mode. In other words, when the data transmission latency between the primary node and its corresponding storage node is high, the cloud management platform instructs the storage node to switch the synchronous replication mode with the storage nodes corresponding to the backup nodes to an asynchronous replication mode, thereby improving the speed of returning response information to tenants and avoiding impacting the tenant's user experience. Correspondingly, if the data transmission latency between the second database node and the second storage node is less than the first preset threshold, the cloud management platform (e.g., a third arbitration component) sends a seventh instruction message to the second storage node. This seventh instruction message instructs the second storage node to switch the asynchronous replication mode between itself and the first storage node, and among the storage nodes corresponding to each backup database node (excluding the second storage node itself), to a synchronous replication mode. In other words, when the data transmission latency between the primary node and its corresponding storage node is low, the cloud management platform instructs the storage node to switch from asynchronous replication mode to synchronous replication mode with the storage node corresponding to the backup node, thereby ensuring data consistency and avoiding data loss.

[0117] For example, the fifth indication information is also used to instruct the second storage node to record the breakpoint position so that, if the data transmission delay between the second database node and the second storage node is less than the first preset threshold, data synchronization between the second storage node and the storage node corresponding to the backup database node can be achieved according to the breakpoint position.

[0118] In some embodiments, when the second database node is upgraded to the primary node, if the data transmission latency between the second storage node and the third storage node is greater than or equal to a second preset threshold, the cloud management platform sends a sixth instruction message to the second storage node. This sixth instruction message instructs the second storage node to switch the synchronous replication mode between the second storage node and the third storage node to an asynchronous replication mode. The third storage node is one of the storage nodes other than the second storage node among the storage nodes corresponding to the first storage node and each backup database node. In other words, when the data transmission latency between the storage node corresponding to the primary node and the storage node corresponding to the backup node is large, the cloud management platform instructs the storage node corresponding to the primary node to switch the synchronous replication mode with the storage node corresponding to the backup node to an asynchronous replication mode, thereby improving the speed of returning response information to the tenant and avoiding impacting the tenant's user experience. Correspondingly, if the data transmission latency between the second storage node and the third storage node is less than the second preset threshold, the cloud management platform sends an eighth instruction message to the second storage node. This eighth instruction message instructs the second storage node to switch the asynchronous replication mode between the second storage node and the third storage node to a synchronous replication mode. In other words, when the data transmission latency between the storage node corresponding to the primary node and the storage node corresponding to the backup node is low, the cloud management platform instructs the storage node corresponding to the primary node to switch the asynchronous replication mode with the storage node corresponding to the backup node to synchronous replication mode, thereby ensuring data consistency and avoiding data loss.

[0119] For example, the sixth indication information is also used to instruct the second storage node to record the breakpoint position so that, if the data transmission delay between the second storage node and the third storage node is less than the second preset threshold, data synchronization between the second storage node and the third storage node can be achieved according to the breakpoint position.

[0120] In some embodiments, when the cluster containing the first database node includes a fourth database node, the cloud management platform may preferentially select the fourth database node as the new primary node in the event of a failure of the first database node and / or the first storage node. If every database node and / or the storage node corresponding to each database node in the cluster containing the first database node fails, the cloud management platform determines a second database node from backup database nodes in other clusters. These other clusters are clusters other than the cluster containing the first database node.

[0121] In this embodiment, the first database node is the master node, used to process write request information from tenants. The first database node is also configured with at least one backup database node. Data synchronization between the first database node and the at least one backup database node is achieved through a synchronous replication mode between corresponding storage nodes, thereby maintaining data consistency and preventing data loss in the event of a failure of the first database node. In other words, the at least one backup database node is in a normal working state, not a shutdown state. Therefore, in the event of a failure of the first database node and / or the first storage node, a rapid switchover of the database nodes can be achieved, allowing the second database node among the at least one backup database node to provide database services to tenants, thus enabling rapid fault recovery.

[0122] In this embodiment, the primary node and each backup node in the database belong to different clusters, thus avoiding the situation where all database nodes fail and achieving fault isolation. Furthermore, when database nodes need to be upgraded or downgraded, only some database nodes can be upgraded or downgraded, instead of all database nodes having to be upgraded or downgraded simultaneously.

[0123] In this embodiment, after configuring multiple database nodes, both the primary and backup nodes are in standby mode rather than downtime. Therefore, in the event of a failure in the primary node or its corresponding storage node, the cloud management platform can automatically issue an instruction to quickly switch the primary node without requiring a lengthy wait for the backup node to restart and synchronize data. This improves fault recovery speed and enhances user experience. Furthermore, the backup node can also provide data read services to the tenant, thereby improving resource utilization.

[0124] In this embodiment, the cloud management platform can be configured with multiple arbitration components. By using arbitration components located in different regions from both the primary and backup nodes, it can determine whether a backup node can be promoted to the primary node when the primary node fails. This achieves automatic failover, avoids the risks of manual operation, and reduces the time required for disaster recovery failover (e.g., reducing failover time from minutes to seconds). Setting up multiple arbitration components in different regions can prevent split-brain problems between multiple backup nodes, improve the reliability of the disaster recovery failover system, and avoid erroneous failovers.

[0125] In this embodiment, the storage node corresponding to the primary node and the storage node corresponding to the backup node achieve synchronous replication through log data, thereby ensuring strong data consistency as much as possible, such as achieving a recovery point objective (RPO) of 0, and thus minimizing data loss. The backup node can update the data in its storage node based on the log data in its corresponding storage node without needing a direct connection to the primary node.

[0126] In this embodiment of the application, when the data transmission delay between the database node and the storage node is large, and / or when the data transmission delay between the storage nodes is large, the synchronous replication mode is switched to the asynchronous replication mode, which can avoid business blockage caused by network fluctuations and enhance the universality of the solution.

[0127] Figure 4 This is a schematic structural diagram of the cloud management platform provided in the embodiments of this application. Figure 4 The cloud management platform 400 includes a transceiver module 410. Figure 4 The cloud management platform 400 can be used to execute Figure 3 The method in the middle. Figure 4 The cloud management platform 400, for example, is Figure 1 The cloud management platform 110 or Figure 2 The cloud management platform 210.

[0128] In some embodiments, the transceiver module 410 is used to perform Figure 3 Steps 310 and 320 in the process.

[0129] For example, in the event of a failure of the first database node and / or the first storage node, the transceiver module 410 is further configured to: send a third indication message to the first storage node; and / or, send a fourth indication message to the second storage node. The third and fourth indication messages are described in [reference needed]. Figure 3 The description in the text.

[0130] For example, the cloud management platform 400 may further include a processing module 420. This processing module 420 is further configured to: determine that the first database node has failed if no status information of the first database node is received within each of N preset time periods; and / or, determine that the first storage node has failed if no status information of the first storage node is received within each of N preset time periods, where N is a positive integer.

[0131] For example, when the second database node processes a tenant's write request information, the transceiver module 410 is further configured to: send a fifth indication message to the second storage node if the data transmission delay between the second database node and the second storage node is greater than or equal to a first preset threshold; and / or, send a sixth indication message to the second storage node if the data transmission delay between the second storage node and the third storage node is greater than or equal to the second preset threshold. The second storage node, third storage node, fifth indication message, and sixth indication message are described in [reference needed]. Figure 3 The description in the text.

[0132] For example, the transceiver module 410 is further configured to: receive at least one of the following during each preset time period in at least one preset time period: status information of the first database node, status information of the first storage node, status information of each backup database node, and status information of the storage node corresponding to each backup database node, see details. Figure 3 The description in the text.

[0133] Both the transceiver module 410 and the processing module 420 can be implemented in software or in hardware. For example, the implementation of the processing module 420 will be described below. Similarly, the implementation of the transceiver module 410 can be referenced from the implementation of the processing module 420.

[0134] As an example of a software functional unit, processing module 420 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, processing module 420 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0135] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same VPC or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0136] As an example of a hardware functional unit, the processing module 420 may include at least one computing device, such as a server. Alternatively, the processing module 420 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0137] The processing module 420 includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the processing module 420 can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the processing module 420 can be distributed within the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0138] Therefore, the modules of the various examples described in the embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0139] It should be noted that the cloud management platform provided in the above embodiments is only illustrated by the division of the above functional modules when executing the above methods. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the cloud management platform can be divided into different functional modules to complete all or part of the functions described above. For example, the transceiver module 410 can be used to execute any step in the above methods, and the processing module 420 can be used to execute any step in the above methods. The steps implemented by the transceiver module 410 and the processing module 420 can be specified as needed, and all the functions of the above cloud management platform can be realized by implementing different steps in the above methods through the transceiver module 410 and the processing module 420 respectively.

[0140] Furthermore, the cloud management platform and method embodiments provided above belong to the same concept, and their specific implementation process can be found in the method embodiments above, which will not be repeated here.

[0141] The method provided in this application can be executed by a computing device, which can also be referred to as a computer system. It includes a hardware layer, an operating system layer running on top of the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as processing units, memory, and memory control units; the functions and structure of this hardware will be described in detail later. The operating system can be any one or more computer operating systems that implement business processing through processes, such as Linux, Unix, Android, iOS, or Windows. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software. Optionally, the computer system can be a handheld device such as a smartphone, or a terminal device such as a personal computer; this application does not particularly limit this, as long as the method provided in this application can be used. The executing entity of the method provided in this application can be a computing device, or a functional module within the computing device capable of calling and executing programs.

[0142] Figure 5 This is a schematic structural block diagram of a computing device 500 provided in an embodiment of this application. The computing device 500 may be a server, a computer, or other device with computing capabilities. Figure 5 The computing device 500 shown includes at least one processor 510 and a memory 520.

[0143] It should be understood that this application does not limit the number of processors and memories in the computing device 500.

[0144] The processor 510 executes instructions in the memory 520, causing the computing device 500 to implement the method provided in this application. Alternatively, the processor 510 executes instructions in the memory 520, causing the computing device 500 to implement the various functional modules provided in this application, thereby implementing the method provided in this application.

[0145] Optionally, the computing device 500 also includes a communication interface 530. The communication interface 530 uses a transceiver module, such as, but not limited to, a network interface card or a transceiver, to enable communication between the computing device 500 and other devices or communication networks.

[0146] Optionally, the computing device 500 also includes a system bus 540, wherein the processor 510, memory 520, and communication interface 530 are respectively connected to the system bus 540. The processor 510 can access the memory 520 through the system bus 540; for example, the processor 510 can perform data read / write or code execution in the memory 520 through the system bus 540. The system bus 540 is a peripheral component interconnect express (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus 540 is divided into an address bus, a data bus, a control bus, etc. For ease of illustration, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0147] In one possible implementation, the processor 510 primarily functions to interpret the instructions (or code) of a computer program and process data within the computer software. The instructions of the computer program and the data within the computer software can be stored in the memory 520 or the cache of the processor 510.

[0148] Optionally, processor 510 may be an integrated circuit chip with signal processing capabilities. By way of example and not limitation, processor 510 may be a general-purpose processor, a digital signal processor (DSP), an ASIC, an FPGA, or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. A general-purpose processor may be a microprocessor, etc. For example, processor 510 may be a central processing unit (CPU).

[0149] The memory 520 provides runtime space for processes in the computing device 500. For example, the memory 520 stores the computer program (specifically, the program code) used to generate the process. After the computer program is run by the processor to generate a process, the processor allocates corresponding storage space for the process in the memory 520. Furthermore, the aforementioned storage space further includes text segments, initialized data segments, bit initialized data segments, stack segments, heap segments, etc. The memory 520 stores data generated during the process's execution, such as intermediate data or process data, in the aforementioned process-specific storage space.

[0150] Optionally, the memory, also known as RAM, is used to temporarily store the data processed by the processor 510, as well as data exchanged with external storage devices such as hard disks. As long as the computer is running, the processor 510 will load the data required for processing into RAM for computation, and then transfer the result back out after the computation is complete.

[0151] By way of example and not limitation, memory 520 may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile storage medium may be, for example, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory is random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus DRAM (DRDRAM). It should be noted that the memory 520 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0152] The structures of the computing device 500 listed above are merely illustrative and are not limited thereto. The computing device 500 in this application includes various hardware components in existing computer systems. For example, the computing device 500 also includes other memories besides the memory 520, such as disk storage. Those skilled in the art should understand that the computing device 500 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that the computing device 500 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the computing device 500 may only include the devices necessary for implementing the embodiments of this application, and may not necessarily include...Figure 5 All the devices shown.

[0153] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device may be a server. In some embodiments, the computing device may also be a desktop computer, a laptop computer, or a smartphone, or other terminal device.

[0154] like Figure 6 As shown, the computing device cluster includes at least one computing device 500. The memory 520 of one or more computing devices 500 in the computing device cluster may store the same instructions for performing the methods described above.

[0155] In some possible implementations, the memory 520 of one or more computing devices 500 in the computing device cluster may also each store a portion of the instructions for executing the above-described methods. In other words, a combination of one or more computing devices 500 can jointly execute the instructions of the above-described methods.

[0156] It should be noted that the memory 520 in different computing devices 500 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the aforementioned cloud management platform. That is, the instructions stored in the memory 520 of different computing devices 500 can implement the functions of one or more modules within the aforementioned cloud management platform.

[0157] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 7 One possible implementation is shown. For example... Figure 7 As shown, the two computing devices 500A and 500B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device.

[0158] It should be understood that Figure 7 The functions of computing device 500A shown can also be performed by multiple computing devices 500. Similarly, the functions of computing device 500B can also be performed by multiple computing devices 500.

[0159] This application embodiment also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions capable of running on a computing device cluster or stored on any available medium. When run by the computing device cluster, it causes the computing device cluster to perform the methods provided above, or causes the computing device cluster to implement the functions of the cloud management platform provided above.

[0160] This application embodiment also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., high-density digital video disc (DVD)), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method provided above.

[0161] In this embodiment of the application, a chip system is also provided. The chip system includes logic circuitry for coupling with an input / output interface to transmit data via the input / output interface, thereby executing the methods provided above.

[0162] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0163] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0164] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0165] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0166] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0167] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0168] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method of failover, the method comprising: The method is applied to a cloud system, the cloud system comprising a cloud management platform and an infrastructure, the cloud management platform being configured to manage the infrastructure, the infrastructure comprising a plurality of database nodes and a plurality of storage nodes, the plurality of database nodes comprising a first database node and at least one backup database node of the first database node, the first database node and each of the at least one backup database node belonging to different clusters, the first database node being configured to process write request information of a tenant, the write request information being configured to request to update data, the plurality of storage nodes comprising a first storage node corresponding to the first database node and a storage node corresponding to each of the backup database nodes, the first storage node being configured to send log data to the storage node corresponding to each of the backup database nodes, each of the backup database nodes being further configured to, in a case where the first database node and / or the first storage node does not fail, implement synchronous replication between the storage node corresponding to each of the backup database nodes and the first storage node according to the log data, in a case where the first database node and / or the first storage node fails, the method comprising: the cloud management platform sending first indication information to the first storage node, the first indication information being configured to instruct the first storage node to stop updating data according to an instruction of the first database node; the cloud management platform sending second indication information to a second database node, the second indication information being configured to instruct the second database node to process the write request information of the tenant, the second database node belonging to the at least one backup database node.

2. The method of claim 1, wherein, in a case where the first database node and / or the first storage node fails, the method further comprising: the cloud management platform sending third indication information to the first storage node, the third indication information being configured to instruct the first storage node to cancel a synchronous replication mode between the first storage node and the storage node corresponding to each of the backup database nodes; and / or, the cloud management platform sending fourth indication information to a second storage node, the second storage node being the storage node corresponding to the second database node, the fourth indication information being configured to instruct the second storage node to establish a synchronous replication mode between the second storage node and the first storage node and each of the storage nodes corresponding to each of the backup database nodes except the second storage node.

3. The method according to claim 1 or 2, characterized in that, the method further comprising: in a case where the cloud management platform does not receive state information of the first database node in each of N preset time periods, the cloud management platform determining that the first database node fails, N being a positive integer; and / or, in a case where the cloud management platform does not receive state information of the first storage node in each of N preset time periods, the cloud management platform determining that the first storage node fails, N being a positive integer.

4. The method according to any one of claims 1 to 3, characterized in that, in a case where the second database node processes the write request information of the tenant, the method further comprising: in a case where a data transmission delay between the second database node and a second storage node is greater than or equal to a first preset threshold, the cloud management platform sends fifth indication information to the second storage node, the fifth indication information being used to instruct the second storage node to switch a synchronous replication mode between the second storage node and the first storage node and storage nodes corresponding to each backup database node except the second storage node to an asynchronous replication mode; and / or in a case where a data transmission delay between the second storage node and a third storage node is greater than or equal to a second preset threshold, the cloud management platform sends sixth indication information to the second storage node, the sixth indication information being used to instruct the second storage node to switch a synchronous replication mode between the second storage node and the third storage node to an asynchronous replication mode, the third storage node belonging to the first storage node and the storage nodes corresponding to each backup database node except the second storage node; wherein the second storage node is a storage node corresponding to the second database node.

5. The method according to any one of claims 1 to 4, characterized in that, in a case where the first database node and / or the first storage node does not fail, the second database node is further used to process read request information of the tenant, the read request information being used to request reading data.

6. The method according to any one of claims 1 to 5, characterized in that, in a case where the second database node processes write request information of the tenant, the first database node and backup database nodes except the second database node among the backup database nodes are used to process read request information of the tenant, the read request information being used to request reading data.

7. The method according to any one of claims 1 to 6, characterized in that, The method further comprises: The cloud management platform receives at least one of the following in each of at least one preset time period: state information of the first database node, state information of the first storage node, state information of each backup database node, and state information of a storage node corresponding to each backup database node, wherein the state information of the first database node is used to indicate whether the first database node fails, the state information of the first storage node is used to indicate at least one of the following: whether the first storage node fails, and whether a data transmission link between the first storage node and the storage node corresponding to each backup database node fails, the state information of each backup database node is used to indicate whether each backup database node fails, and the state information of the storage node corresponding to each backup database node is used to indicate at least one of the following: whether the storage node corresponding to each backup database node fails, and whether a data transmission link between the storage node corresponding to each backup database node and the first storage node fails.

8. The method according to any one of claims 1 to 7, characterized in that, The cloud management platform comprises a first arbitration component, a second arbitration component and at least one third arbitration component, the first arbitration component and the second arbitration component and the at least one third arbitration component belong to different regions, the second arbitration component and the first database node belong to the same region, each of the at least one third arbitration component and one of the at least one backup database node belong to the same region, The first arbitration component is used to determine the second database node from the at least one backup database node in the case of failure of the first database node and / or the first storage node, the second arbitration component is connected with the first database node and / or the first storage node, and each third arbitration component is connected with the corresponding backup database node.

9. A cloud system, characterized by The cloud system comprises a cloud management platform and an infrastructure, the cloud management platform is used to manage the infrastructure, and the infrastructure comprises a plurality of database nodes and a plurality of storage nodes, The plurality of database nodes comprises a first database node and at least one backup database node of the first database node, the first database node and each of the at least one backup database node belong to different clusters, the first database node is used to process write request information of a tenant, and the write request information is used to request to update data, The plurality of storage nodes comprises a first storage node corresponding to the first database node and a storage node corresponding to each backup database node, the first storage node is used to send log data to the storage node corresponding to each backup database node, and each backup database node is further used to realize synchronous replication between the storage node corresponding to each backup database node and the first storage node according to the log data in the case of no failure of the first database node and / or the first storage node, In the case of failure of the first database node and / or the first storage node, the cloud management platform is used to: send first indication information to the first storage node, the first indication information is used to instruct the first storage node to stop updating data according to the instruction of the first database node; send second indication information to a second database node, the second indication information is used to instruct the second database node to process the write request information of the tenant, and the second database node belongs to the at least one backup database node.

10. The cloud system of claim 9, wherein, In the case of failure of the first database node and / or the first storage node, the cloud management platform is further used to: send third indication information to the first storage node, the third indication information is used to instruct the first storage node to cancel the synchronous replication mode between the first storage node and the storage node corresponding to each backup database node; and / or, sending fourth indication information to a second storage node, the second storage node being a storage node corresponding to the second database node, the fourth indication information being used to instruct the second storage node to establish a synchronous replication mode between the second storage node and each storage node other than the second storage node among the storage nodes corresponding to the first database node and the each backup database node.

11. The cloud system of claim 9 or 10, wherein, The cloud management platform is further configured to: determine that the first database node fails in a case where the state information of the first database node is not received in each of N preset time periods, N being a positive integer; and / or, determine that the first storage node fails in a case where the state information of the first storage node is not received in each of N preset time periods, N being a positive integer.

12. The cloud system of any one of claims 9 to 11, wherein, In a case where the second database node processes the write request information of the tenant, the cloud management platform is further configured to: in a case where a data transmission delay between the second database node and a second storage node is greater than or equal to a first preset threshold, send fifth indication information to the second storage node, the fifth indication information being used to instruct the second storage node to switch a synchronous replication mode between the second storage node and each storage node other than the second storage node among the storage nodes corresponding to the first database node and the each backup database node to an asynchronous replication mode; and / or, in a case where a data transmission delay between the second storage node and a third storage node is greater than or equal to a second preset threshold, send sixth indication information to the second storage node, the sixth indication information being used to instruct the second storage node to switch a synchronous replication mode between the second storage node and the third storage node to an asynchronous replication mode, the third storage node belonging to the storage nodes other than the second storage node among the storage nodes corresponding to the first database node and the each backup database node. The second storage node is a storage node corresponding to the second database node.

13. The cloud system of any of claims 9 to 12, wherein, In a case where the first database node and / or the first storage node does not fail, the second database node is further configured to process read request information of the tenant, the read request information being used to request reading data.

14. The cloud system of any one of claims 9 to 13, wherein, In a case where the second database node processes the write request information of the tenant, each backup database node other than the second database node among the first database node and the each backup database node is configured to process read request information of the tenant, the read request information being used to request reading data.

15. The cloud system of any one of claims 9 to 14, wherein, The cloud management platform is further configured to: receive at least one of the following in each of at least one preset time period: state information of the first database node, state information of the first storage node, state information of each backup database node, and state information of a storage node corresponding to each backup database node, wherein the state information of the first database node is used to indicate whether the first database node fails, the state information of the first storage node is used to indicate at least one of whether the first storage node fails and whether a data transmission link between the first storage node and the storage node corresponding to each backup database node fails, the state information of each backup database node is used to indicate whether each backup database node fails, and the state information of the storage node corresponding to each backup database node is used to indicate at least one of whether the storage node corresponding to each backup database node fails and whether a data transmission link between the storage node corresponding to each backup database node and the first storage node fails.

16. The cloud system of any one of claims 9 to 15, wherein, The cloud management platform comprises a first arbitration component, a second arbitration component, and at least one third arbitration component, the first arbitration component, the second arbitration component, and the at least one third arbitration component belong to different regions, the second arbitration component and the first database node belong to the same region, and each third arbitration component of the at least one third arbitration component and one backup database node of the at least one backup database node belong to the same region, The first arbitration component is configured to determine the second database node from the at least one backup database node in the case where the first database node and / or the first storage node fails, the second arbitration component is connected with the first database node and / or the first storage node, and each third arbitration component is connected with the backup database node corresponding to each third arbitration component.

17. A cluster of computing devices, characterized in that, The at least one computing device comprises a processor and a memory, The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 8.

18. A computer program product comprising instructions, characterized in that, The instructions, when executed by the computing device cluster, cause the computing device cluster to execute the method according to any one of claims 1 to 8.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium comprises computer program instructions, and when the computer program instructions are executed by the computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 8.

Images