Distributed service system, instance splitting method, device, medium and program product

Abstract

The embodiments of the present application provide a distributed service system, an instance splitting method, a device, a medium and a program product. In the embodiments of the present application, in response to an instance splitting event, the service instance is expanded, and a write-disable instruction is used to prevent new data from being written to the expanded service instance. The expanded service instance is then split into multiple service instances, thereby achieving the splitting of the service instance. In the expansion phase, the original service node of the service instance can still provide services, and data synchronization can be completed while ensuring service continuation. By preventing new data from being written to the expanded service instance, the multiple service instances after the split start to split from the same state, thereby ensuring data consistency of the service instances before and after the split.

Description

Technical Field

[0001] The present application relates to the field of cloud service technology, and in particular to a distributed service system, instance splitting method, device, medium and program product. Background Art

[0002] A distributed service system is a system composed of multiple services that collaborate with each other. These services are usually distributed on different hosts or servers and communicate through the network. In order to improve the reliability and disaster recovery capabilities of the distributed service system, multiple service nodes can be deployed for the same service instance to back up each other. Once a service node fails, other service nodes can continue to provide services.

[0003] Multiple service nodes of the same service instance can provide consistent coordination services (such as distributed lock services, etc.), and can select a master node from multiple service nodes based on the consensus protocol to provide services, and synchronize data to other service nodes to maintain the state consistency of multiple service nodes. In actual applications, the need for service instance splitting is real. Therefore, it is urgent to provide a service instance splitting solution to meet actual needs. Summary of the invention

[0004] Multiple aspects of the present application provide a distributed service system, an instance splitting method, a device, a medium, and a program product to implement the splitting of a service instance.

[0005] The embodiment of the present application provides a distributed service system, including: a service cluster corresponding to a first service instance; the first service instance is obtained by expanding the capacity of a second service instance in response to an instance split event, the service cluster includes multiple service nodes that independently provide consistency coordination services; data synchronization of the multiple service nodes is completed;

[0006] The master node among the multiple service nodes is used to: send a write-disable instruction to the slave nodes among the multiple service nodes to instruct the service cluster to no longer receive write requests; and, when the number of first target service nodes corresponding to each of the multiple third service instances to be split from the first service instance is greater than the corresponding set number, send a splitting instruction to the slave node; the first target service node is a service node among the multiple service nodes that confirms the write-disable instruction; wherein the set number corresponding to any third service instance is greater than or equal to half of the total number of service nodes corresponding to any third service instance, and less than the total number;

[0007] The master node and the slave node that receives the split instruction are used to: respond to the split instruction and split the first service instance into multiple third service instances.

[0008] The embodiment of the present application also provides an instance splitting method, which is applicable to any service node among multiple service nodes corresponding to a first service instance, wherein the first service instance is obtained by expanding the capacity of a second service instance in response to an instance splitting event, and the multiple service nodes independently provide consistency coordination services, and data synchronization is completed;

[0009] The method comprises:

[0010] In the case of acting as a master node, sending a write-disable instruction to a slave node among the multiple service nodes so that the service cluster no longer receives write requests;

[0011] When the number of first target service nodes corresponding to each of the plurality of third service instances to be split from the first service instance is greater than the corresponding set number, sending a split instruction to the slave node, so that the master node and the slave node receiving the split instruction respond to the split instruction to split the first service instance into the plurality of third service instances;

[0012] Among them, the first target service node is the service node among the multiple service nodes that confirms the write-disable instruction; the set number corresponding to any third service instance is greater than or equal to half of the total number of service nodes corresponding to any third service instance, and is less than the total number.

[0013] The embodiment of the present application also provides an electronic device, comprising: a memory, a processor, and a communication component; wherein the memory is used to store a computer program;

[0014] The processor is coupled to the memory and the communication component, and is configured to execute the computer program for performing the steps in the aforementioned example splitting method.

[0015] An embodiment of the present application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps in the aforementioned example splitting method.

[0016] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by one or more processors, causes the one or more processors to execute the steps in the aforementioned example splitting method.

[0017] In the embodiment of the present application, in response to the instance split event, the service instance is expanded, and the expanded service instance is prevented from having new data written to it by a write-disable instruction, and then the expanded service instance is split into multiple service instances, thereby realizing the splitting of the service instance. In the capacity expansion phase, the original service node of the service instance can still provide services, and data synchronization can be completed while ensuring service continuation. By preventing the expanded service instance from having new data written to it, the multiple service instances after the split start to split from the same state, and the data consistency of the service instances before and after the split can be guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 Schematic diagram of the partition splitting process provided for the traditional solution;

[0020] Figure 2 A schematic diagram of the structure of a distributed service system provided in an embodiment of the present application;

[0021] Figure 3 and Figure 4 A schematic diagram of an example splitting process provided in an embodiment of the present application;

[0022] Figure 5 A schematic diagram of a process in which data loss occurs due to some data synchronization methods provided in an embodiment of the present application;

[0023] Figure 6 A schematic diagram of the process of changing members through a joint consensus mechanism provided in an embodiment of the present application;

[0024] Figure 7 A schematic diagram comparing the format of a traditional voting request and the format of a voting request provided in an embodiment of the present application;

[0025] Figure 8 A schematic diagram of the temporal and spatial changes of the system during the example splitting process provided in the embodiment of the present application;

[0026] Fig. 9 A schematic diagram of a flow chart of an example splitting method provided in an embodiment of the present application;

[0027] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0029] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0030] The following is an explanation of the terms or concepts involved in this application.

[0031] Consensus protocol: In order to solve the problem of data consistency in distributed systems, a series of protocols have been proposed, such as Two Phase Commitment Protocol (2PC), Three Phase Commitment Protocol (3PC), Paxos consensus protocol and Raft protocol, etc.

[0032] Replicated state machine: In a distributed system, there is a cluster of multiple servers. Each server runs a deterministic finite state machine (DFSM), which means that for a given input, the state machine always produces the same output and state transition. The goal is to get these state machines to a consistent state so that it looks like only one state machine is working from the outside. If you can ensure that all state machines receive exactly the same sequence of commands and execute them in the same order, then these state machines will eventually reach the same final state. This means that no matter which state machine replica the client connects to, it will get the same result. Since each state machine runs the same logic independently, the system can tolerate the failure of some state machines without affecting the overall quality of service.

[0033] In the scenario of replicated state machines, each state machine maintains a log that records all commands received. The order of the log is very important because it determines the order in which the state machine executes the commands. A consensus protocol can be used to ensure that the logs on all state machines have a consistent sequence. The consensus protocol can decide which commands should be added to the log and in what order.

[0034] Assume that the replicated state machine system consists of three nodes, A, B, and C. These nodes run replicas of the same state machine. The client sends a command cmd1 to node A, and then sends a command cmd2. To ensure that all nodes execute the two commands in the same order, the system uses a consensus protocol to decide the content of the log. Specifically, node A receives cmd1 and tries to write it to the log. Through the consensus protocol, node A notifies nodes B and C that it wants to write cmd1 to the log. All nodes (A, B, and C) decide whether to agree to add cmd1 to the log according to the rules of the consensus protocol. Once cmd1 is accepted by the majority of nodes, it is written to the logs of all nodes. Next, node A sends cmd2 to B and C and repeats the above process. Through this mechanism, the logs on all nodes will contain the same command sequence cmd1 and cmd2, ensuring that all state machine replicas eventually reach the same final state.

[0035] Shared-nothing architecture: Shared-nothing architecture is a distributed system architecture in which each physical machine in the system works independently and does not share resources with other physical machines, such as memory, central processing unit (CPU) or disk storage resources, so that there is no single point of contention in the system.

[0036] In a distributed service system, how to achieve correct and efficient state management is the biggest challenge in system architecture design. Usually, computing distributed systems will be transferred to rely on a storage product (such as database, object storage, file storage, etc.) for state management, so the question becomes how the distributed storage system can achieve high reliability and high performance state management.

[0037] Distributed storage systems generally adopt a partition-based scheduling model. Distributed storage systems are divided into a control plane and a data plane. The data plane divides the user's storage space into several partitions according to certain rules. During runtime, a partition will be assigned to a server or a group of servers to provide services. A server or a group of servers can load multiple partitions at the same time. Partition-based distributed storage systems themselves also have shared storage (Shared-storage) architecture and shared-nothing (Shared-nothing) architecture.

[0038] In the shared storage architecture, partitions do not use the local file system to store persistent data, but store data in an underlying distributed file system. The control plane is responsible for partition scheduling, which is load balancing scheduling. On the other hand, when a server goes down, the control node is responsible for rescheduling the corresponding partition and quickly migrating it to other servers that are running normally.

[0039] In such a system architecture, the control plane supports state management. In order to achieve high availability (HA), the control plane usually needs to rely on a replicated state machine system based on the Shared-nothing architecture, or the control plane itself is a system based on a replicated state machine. In a replicated state machine system, a control instance includes multiple independent control nodes, and multiple control nodes are in a Shared-nothing architecture and do not share resources.

[0040] Multiple control nodes can provide consistent coordination services (such as distributed lock services, etc.). Based on the consensus protocol, a master node can be selected from multiple control nodes to provide control services, and the data can be synchronized to other control nodes to maintain the status consistency of multiple control nodes.

[0041] In addition to the control plane of the distributed storage system described above, any system using the shared-nothing architecture-based replication state machine has a real need for instance splitting, and the service has such a strong demand. The needs for instance splitting include but are not limited to: (1) Insufficient service capacity: As the usage of some tenants increases dramatically, the original system configuration may not be sufficient to support their needs. In this case, splitting these high-load tenants from the original instance and creating new instances to serve them can effectively improve the service quality and avoid affecting other tenants; (2) Inconsistent disaster recovery requirements: Different tenants may have different disaster recovery requirements. For example, some tenants with higher criticality may require higher-level disaster recovery measures, such as redundant deployment across availability zones (AZs). If all tenants are deployed on the same instance, it is difficult to meet such differentiated disaster recovery requirements; (3) Controlling the explosion radius: In order to reduce the impact range of a single point of failure, when an instance serves too many tenants, the risk can be dispersed by splitting the instance. In this way, even if an instance fails, the number of tenants affected will be reduced. The explosion radius refers to the impact range of the instance failure, which can be understood as the number of tenants affected.

[0042] In the shared-nothing architecture, the same solution as the shared-storage mentioned above can also be used for partitioned storage in the data plane. Each partition is a separate consensus protocol working group (such as Raft-Group). In order to allow a single node to be responsible for multiple Raft-Group copies, a multi-Raft (Multi-Raft) solution can be used. Based on the Multi-Raft partition system, there is naturally a strong demand for partition splitting and merging.

[0043] like Figure 1 As shown in the figure, partition splitting can be achieved based on Multi-Raft. That is, the data managed in the entire system is sliced ​​in a certain way, and each slice of data has its own copy. The data between these copies uses the Raft protocol to ensure data consistency. From a global perspective, there are multiple Raft-Groups in the entire system at the same time, and any node can be responsible for multiple Raft-Group copies. Figure 1 As shown in the figure, assuming that the data range of partition A (corresponding to a Raft-Group) is [1, 100), it needs to be split into two partitions, one of which has a data range of [1,50) and the other has a data range of [50, 100). The main steps to implement instance splitting based on such Multi-Raft are as follows: (1) First apply to the control component to create a new partition B with a service data range; (2) Modify the data range boundary of partition A to [1, 50); (3) Partition B starts to be enabled and officially serves requests for access with a data range of [50, 100). The partition splitting process is briefly explained here. This partition splitting method is an instance splitting mechanism similar to the shared storage replication state machine. The main reasons are:

[0044] a) During the split process, the Raft-Group corresponding to the original partition A continues to work by modifying the data boundary, and the newly generated partition B is on the same set of nodes as partition A. After the split is completed, the control component decides whether to migrate the newly generated partition B to other nodes based on the global load situation. Before partition B migrates to other nodes, it shares the resources of the same set of nodes with partition A.

[0045] b) The aforementioned partition splitting solution is implemented based on Multi-Raft. Therefore, the new and old partitions in the splitting process share a copy of persistent data, which has the advantages of data consistency and service availability under the shared-storage architecture.

[0046] Figure 1The partition splitting process of the data plane under the Shared-nothing architecture is shown, but the control plane under the Shared-nothing architecture is also a replicated state machine system based on the Shared-nothing architecture, and has the same control instance splitting requirements as the control plane of the Shared-storage architecture. However, the partition splitting scheme of the data plane under the Shared-storage architecture shown above is not applicable to the control instance splitting of the control plane. This is mainly because the control plane is a replicated state machine system based on the Shared-nothing architecture, and the concept of Raft-Group does not exist in this system. Therefore, how to realize the splitting of service instances (such as the aforementioned control instances) of the replicated state machine system based on the Shared-nothing architecture has become a technical problem that needs to be solved urgently.

[0047] In addition to the instance splitting requirements of the control plane of the distributed storage system described above, any distributed system that uses a replicated state machine system based on the Shared-nothing architecture has the service instance splitting requirements. For example, for a distributed lock service system, the service instance is a distributed lock instance, which provides distributed lock services to the outside world in the form of service pooling. The same distributed lock instance will serve multiple application clusters. It will also encounter the service capacity problems, disaster recovery requirements, and explosion radius problems described above, and also have the need for instance splitting.

[0048] In order to realize the service instance splitting, in some embodiments of the present application, in response to the instance splitting event, the service instance is expanded, and the expanded service instance is prevented from having new data written to it by a write-disable instruction, and then the expanded service instance is split into multiple service instances, thereby realizing the service instance splitting. In the capacity expansion phase, the original service node of the service instance can still provide services, and data synchronization can be completed while ensuring the continuation of services. By preventing the expanded service instance from having new data written to it, the multiple service instances after the split start to split from the same state, and the data consistency of the service instances before and after the split can be guaranteed.

[0049] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0050] It should be noted that the same reference numerals denote the same objects in the following drawings and embodiments, and therefore, once an object is defined in one drawing or embodiment, it does not need to be further discussed in the subsequent drawings and embodiments.

[0051] Figure 2 This is a schematic diagram of the structure of the distributed service system provided in the embodiment of the present application. Figure 2As shown, the distributed service system includes: multiple first service nodes 101 corresponding to the service instance C0 and an operation and maintenance node 20. The multiple first service nodes 101 can provide consistency coordination services and can reach consensus through a consensus protocol (such as the Raft protocol) to maintain the consistency of the multiple first service nodes 101.

[0052] Consensus protocols such as the Raft protocol and the Paxos protocol use the majority principle to implement master election and request processing. That is, any request requires the consent of more than half of the service nodes among multiple service nodes (that is, it is accepted by more than half of the service nodes) before the master node will process the request. Therefore, if more than half of the multiple service nodes are operating normally, the service supported by the service instance can be provided, and disaster tolerance can be performed for abnormalities of less than half of the service nodes. For example, if there are 5 or 6 service nodes, disaster tolerance can be performed for abnormalities of 2 service nodes. In the embodiment of the present application, multiple refers to at least 3, that is, more than 3. The number of service nodes can be an odd number or an even number. Preferably, the number of service nodes is an odd number, because the disaster tolerance capacity of an odd number of service nodes is the same as that of an even number of service nodes that is 1 greater than the odd number. For example, the disaster tolerance capacity of 5 and 6 service nodes is 2 service nodes.

[0053] In the embodiment of the present application, in addition to providing consistency coordination services, multiple first service nodes 101 may also provide other services supported by service instances, such as partition scheduling services, load balancing services, and data storage services provided by the management and control nodes of the distributed storage system shown in the above embodiments.

[0054] A service instance refers to a specific instance that provides a specific service. The service instance runs on multiple first service nodes 101. The service node is responsible for the resources required for the corresponding service. In the embodiment of the present application, the specific content of the service run by the service instance is not limited. For example, the service instance may be a management and control instance in a distributed storage system, providing partition scheduling services, load balancing services, and data storage services, etc. For another example, the service instance may be a distributed lock service instance in a distributed lock system, providing distributed lock services, etc.

[0055] In the embodiment of the present application, multiple first service nodes 101 are independent of each other, have independent resources, and do not share resources with each other. The resources of the first service node 101 include but are not limited to: processing resources such as processors, storage resources such as memory and disks, and input / output (IO) resources.

[0056] In an embodiment of the present application, the operation and maintenance node 20 can monitor the instance split event for the service instance C0, and when the instance split event for the service instance C0 is monitored, it responds to the instance split event and expands the service instance C0 to obtain the expanded service node.

[0057] In the embodiment of the present application, for the convenience of description and distinction, the original service instance C0 is defined as the second service instance C0, the service instance C0' after the second service instance C0 is expanded is defined as the first service instance C0', and the service instance to be split from the first service instance C0' is defined as the third service instance C0. 1-n . Wherein, n represents the nth third service instance. n=1, 2…, N. N is the total number of third service instances. Correspondingly, the original service node of the second service instance C0 is the first service node 101, and the service node expanded in response to the instance split event is defined as the second service node 102. Wherein, the number of third service instances C1 is multiple, and multiple means more than 2. For example, the first service instance C0' can be split into 2, 3, 4 or more third service instances C 1-n wait.

[0058] In the embodiments of the present application, the specific implementation form of the instance split event is not limited. In some embodiments, the instance split event can be triggered by an operation and maintenance personnel. When the operation and maintenance personnel determine that the second service instance C0 needs to be split according to the service performance of the second service instance C0, the instance split can be triggered through the human-computer interaction interface. Accordingly, when the operation and maintenance node 20 receives the instance split instruction provided by the human-computer interaction interface, it determines that an instance split event has occurred. The instance split event is implemented as receiving an instance split instruction.

[0059] In this embodiment, the operation and maintenance personnel may also specify instance splitting information. The instance splitting information is used to indicate how the second service instance C0 is specifically split, and may include: the third service instance C0 to be split into by the second service instance C0; 1-n The number of third service instances C 1-n The number and resource specifications of the corresponding service nodes. 1-n There is a service instance in which the corresponding service nodes are the aforementioned multiple first service nodes 101.

[0060] The resource specification of the service node is used to characterize the amount of hardware resources that the service node can provide, and can be measured using at least one of the following hardware resources: the number of physical cores of the processor and the size of the memory, etc. For example, if the resource specification of the second service node is 2 cores and 1G memory, then the second service node occupies the hardware resources of 2 processor cores and 1G memory.

[0061] The operation and maintenance node 20 may obtain the instance split information submitted through the human-computer interaction interface; then, the second service instance C0 may be expanded according to the instance split information to obtain multiple second service nodes 102. Specifically, the operation and maintenance node 20 may expand the capacity of the second service instance C0 according to the third service instance C0. 1-n The number K of service nodes and resource specifications corresponding to each other are selected, and K second service nodes that meet the resource specification requirements are selected, and the target service provided by the second service instance C0 is deployed on the second service node to achieve the expansion of the first service instance C0, and obtain the service instance after expansion, that is, the first service instance C0'. Among them, the target service provided by the second service instance C0 may include consistency coordination services; of course, it may also include the aforementioned data storage services, load balancing services, and partition scheduling services.

[0062] In some other embodiments, the operation and maintenance node 20 may autonomously monitor the instance splitting event.

[0063] Implementation method 1: When the capacity of a service instance is insufficient, tenants with high loads can be split from the original instance and new instances can be created to serve them specifically, which can effectively improve the service quality and avoid affecting other tenants. Accordingly, the operation and maintenance node 20 can monitor the load capacity of the second service instance C0. The load capacity of the second service instance C0 is used to characterize the idle resources that the second service instance C0 can process requests, and can be expressed as a percentage of the idle resources of the second service instance C0 to the total resources of the second service instance C0.

[0064] In implementation mode 1, a lower limit value of the load capacity can be set. If the load capacity of the second service instance C0 is less than the lower limit value, it means that the capacity, i.e., the resources, of the second service instance C0 are insufficient, and the second service instance C0 needs to be split. The lower limit value of the load capacity is the set load capacity. Accordingly, if the operation and maintenance node 20 detects that the load capacity of the service instance C0 is less than the set load capacity, it is determined that an instance splitting event has occurred. Furthermore, the operation and maintenance node 20 can also respond to the instance splitting event and determine the target user whose request volume is in a set position from the users of the second service instance C0. For example, the target user whose request volume is in the top M positions can be determined from the users of the second service instance C0. 1≤M is less than N and is an integer. N represents the total number of users of the second service instance C0. The target user is a high-load user.

[0065] Further, the operation and maintenance node 20 may determine the third service instance C to be split according to the resource demand information of the target user. 1-n The corresponding resource specifications of the second service nodes and the number K of the second service nodes.

[0066] Among them, the resource demand information of the target user may include: resource specifications and disaster recovery demand information of the service instance required by the target user. Among them, the resource specifications of the service instance are used to indicate the resource request amount of the target user, and are used to limit the resource specifications of the service node, and can be represented by the resource specifications of the service node. The disaster recovery demand information is used to indicate the disaster recovery capability of the service instance required by the target user, and can be represented by the number of service nodes that support disaster recovery, and can be used to limit the number of service nodes. Specifically, the number K of the second service nodes can be equal to: 2 times the number N of service nodes that support disaster recovery plus 1. That is, K=2N+1. For example, if the disaster recovery demand information is for disaster recovery of 2 service nodes, then the number of second service nodes is 5.

[0067] The operation and maintenance node 20 may expand the second service instance C0 according to the resource demand information of the target user to obtain multiple second service nodes 102. Specifically, the operation and maintenance node 20 may determine the third service instance C0 according to the resource specifications and disaster recovery demand information of the service instance required by the target user. 1-n The number K of service nodes and resource specifications corresponding to each service node; then, according to the third service instance C 1-n The number K of service nodes and resource specifications corresponding to each other are selected, K second service nodes that meet the resource specification requirements are selected, and the target service provided by the second service instance C0 is deployed on the second service nodes to achieve the expansion of the second service instance C0 and obtain the first service instance C0'.

[0068] In this implementation mode 1, a new third service instance can be split for a target user with a high load to specifically serve the target user, which can effectively improve the service quality and avoid affecting other users.

[0069] Implementation method 2: To reduce the impact of a single point of failure, when an instance serves too many tenants, you can disperse the risk by splitting the instance.

[0070] Accordingly, the operation and maintenance node 20 can monitor the number of users of the second service instance C0, and determine that an instance split event occurs when the number of users of the second service instance is greater than or equal to a set user threshold; according to the preset explosion radius, and the plurality of third service instances C0 to be split 1-n The second service instance C0 is expanded based on the resource demand information of the corresponding users to obtain multiple second service nodes 102. The explosion radius can be represented by the maximum number of users affected when the service instance is abnormal.

[0071] Specifically, the operation and maintenance node 20 may preset a blast radius to determine the third service instance C to be split 1-n The number N and the third service instance C 1-nThe number of users corresponding to each third service instance is less than or equal to the preset explosion radius.

[0072] Afterwards, multiple third service instances C 1-n The resource demand information of the corresponding users determines multiple service instances C 1-n The resource specifications of the corresponding second service nodes and the number of second service nodes K. Further, the resource specifications of the corresponding second service nodes and the number of second service nodes K may be 1-n The number K of service nodes and resource specifications corresponding to each other are selected, K second service nodes that meet the resource specification requirements are selected, and the target service provided by the service instance C0 is deployed on the second service nodes to achieve the expansion of the second service instance C0 and obtain the first service instance C0'.

[0073] In this implementation 2, when the number of users of the second service instance C0 is too large, the risk is dispersed by splitting the instance. In this way, even if a third service instance is abnormal, the number of users affected will be less than that of the first service instance C0.

[0074] Implementation method 3: Different users may have different disaster recovery requirements. For example, some users with higher criticality may require redundant deployment at the availability zone level, some users may require redundant deployment at the data center level, and some users may require redundant deployment at the service node level. If all users are deployed on the same service instance, it is difficult to meet such differentiated disaster recovery requirements.

[0075] Accordingly, the operation and maintenance node 20 can monitor the disaster recovery requirements of the users of the second service instance C0, and determine that an instance split event has occurred when the difference in disaster recovery requirements of the users of the second service instance C0 meets the set disaster recovery difference condition. For example, disaster recovery requirements can be divided using disaster recovery levels, and disaster recovery levels can be divided into: disaster recovery at the availability zone level, disaster recovery at the data center level, disaster recovery at the computer room level, disaster recovery at the rack level, and disaster recovery at the service node level, etc., according to the size of the disaster recovery range, but are not limited to this. Among them, the set disaster recovery difference condition can be implemented as a difference across disaster recovery levels, that is, when the container requirements of the users of the first service instance differ across disaster recovery levels, an instance split event is determined to have occurred.

[0076] Further, the operation and maintenance node 20 may classify users whose disaster recovery demand differences are within a set disaster recovery difference range to the same third service instance according to the disaster recovery demand of users of the second service instance C0, so as to determine the number of third service instances to be split. The disaster recovery difference range may be expressed as being at the same disaster recovery level. For example, users whose disaster recovery demands are at the same disaster recovery level may be classified into the same third service instance. For example, assuming that a data center includes three computer rooms, and one computer room includes 100 racks, the disaster recovery level of users whose disaster recovery demand is less than 100 racks may be classified into the rack level. Further, users at the rack level may be classified into the same third service instance.

[0077] Afterwards, the second service instance C0 may be expanded according to the resource demand information of the user corresponding to the third service instance to be split, so as to obtain multiple second service nodes 102 .

[0078] Specifically, according to a plurality of third service instances C 1-n The resource demand information of the corresponding users determines multiple third service instances C 1-n The resource specifications of the corresponding second service nodes and the number of second service nodes K. Further, the resource specifications of the corresponding second service nodes and the number of second service nodes K may be 1-n The number K of service nodes and resource specifications corresponding to each other are selected, K second service nodes that meet the resource specification requirements are selected, and the target service provided by the second service instance C0 is deployed on the second service nodes to achieve the expansion of the second service instance C0 and obtain the first service instance C0'.

[0079] In implementation mode 3, users at different disaster recovery levels use different service instances to provide services, which can meet the disaster recovery needs of users.

[0080] The aforementioned embodiments 1-3 only illustrate the implementation method of the instance split event and the implementation method of expanding the second service instance C0 in response to the instance split event, and do not constitute a limitation. The aforementioned embodiments 1-3 can be implemented individually, or any two implementation methods can be implemented in combination, or all three implementation methods can be implemented in combination. When two or three implementation methods are implemented in combination, the occurrence of an instance split event corresponding to any one of the implementation methods determines that an instance split event has occurred, and the corresponding service instance expansion method can be used to expand the service instance. If the instance split events corresponding to multiple implementation methods all occur when two or three implementation methods are implemented in combination, the service instance expansion methods corresponding to these methods can be combined to expand the service instance.

[0081] For example, if the instance splitting events corresponding to implementation modes 1 and 2 both occur, the target user can be separately divided into a third service instance, and the number of service nodes and resource specifications of the third service instance can be determined according to the resource demand information of the target user. Afterwards, a second service node that meets the resource specification requirements can be selected according to the number of service nodes and resource specifications corresponding to the third service instance corresponding to the target user, and the target service provided by the second service instance C0 can be deployed on the second service node.

[0082] For other users, the second service instance C0 may be expanded according to a preset explosion radius and resource demand information of users corresponding to other third service instances to be split, so as to obtain other second service nodes 102 .

[0083] For another example, if the instance splitting events corresponding to implementation modes 1 and 3 both occur, a target user whose request volume is in a set ranking can be determined from the users of the second service instance C0, and the target user can be separately assigned to a third service instance; then, according to the disaster recovery requirements of other users except the target user among the users of the second service instance C0, the users whose disaster recovery requirements are different from each other and fall within the set disaster recovery difference range can be assigned to the same third service instance. Then, according to the resource demand information of the users corresponding to each third service instance, the second service instance C0 can be expanded to obtain the first service instance C0'.

[0084] For another example, if the instance splitting events corresponding to implementation modes 2 and 3 both occur, then users whose disaster recovery requirements are different from the target user in the second service instance C0 can be divided into the same initial service instance according to the disaster recovery requirements of other users except the target user. Afterwards, if there is no case where the number of users in the initial service instance is greater than or equal to the set user number threshold, the initial service instance is the third service instance, and the second service instance C0 can be directly expanded according to the resource demand information of the users corresponding to each third service instance to obtain the first service instance C0'.

[0085] If the number of users in the initial service instance is greater than or equal to the set user threshold, then for the target service instance whose number of users in the initial service instance is greater than or equal to the set user threshold, the target service instance is split again according to the preset explosion radius to obtain multiple service instances after the target service instance is split. Among them, the third service instance includes: multiple service instances after the target service instance is split and the initial service instance whose number of users is less than the set user threshold. Afterwards, the second service instance C0 can be expanded according to the resource demand information of the users corresponding to each third service instance to obtain the first service instance C0'.

[0086] For another example, implementation modes 1-3 are implemented in combination, and the instance splitting events corresponding to implementation modes 1-3 all occur, and the target user whose request volume is in a set ranking can be determined from the users of the second service instance C0, and the target user can be separately assigned to a third service instance; then, based on the disaster recovery requirements of other users except the target user among the users of the second service instance C0, the users whose disaster recovery requirements are different from each other and fall within the set disaster recovery difference range are assigned to the same initial service instance. Then, if there is no situation in the initial service instance where the number of users is greater than or equal to the set user volume threshold, the third service instance includes the initial service instance and the third service instance corresponding to the aforementioned target user. Then, based on the resource demand information of the users corresponding to each third service instance, the second service instance C0 can be expanded to obtain the first service instance C0'.

[0087] If the number of users in the initial service instance is greater than or equal to the set user threshold, then for the target service instance whose number of users in the initial service instance is greater than or equal to the set user threshold, the target service instance is split again according to the preset explosion radius to obtain multiple third service instances after the target service instance is split. The third service instance includes: multiple service instances after the target service instance is split, the initial service instance whose number of users is less than the set user threshold, and the third service instance corresponding to the aforementioned target user. Afterwards, the second service instance C0 can be expanded according to the resource demand information of the users corresponding to each third service instance to obtain the first service instance C0'.

[0088] Combination Figure 2-Figure 4 , in response to the instance split event, the second service instance C0 is expanded to obtain the expanded service instance, namely the first service instance C0'. The service cluster corresponding to the first service instance C0' is the expanded service cluster. The service cluster includes multiple service nodes 10. The multiple service nodes 10 may include: the original first service node 101 and the second service node 102 expanded in response to the instance split event. For the sake of ease of description, the original first service node 101 and the second service node 102 expanded in response to the instance split event are collectively referred to as service nodes 10. The service nodes 10 are independent of each other, have independent resources, and do not share resources with each other. Multiple service nodes 10 can independently provide consistency coordination services.

[0089] After expanding the capacity of the second service instance C0 in response to the instance split event, the operation and maintenance node 20 may also instruct the first service node 101 to synchronize data to the second service node 102. Specifically, the operation and maintenance node 20 may send a data synchronization instruction to any first service node 101. The data synchronization instruction may include an identifier of the second service node 102. In an embodiment of the present application, the identifier of the service node may uniquely represent a service node, and may be an Internet Protocol (IP) address of the service node, a unique number or an identity (ID) in a distributed service system, etc.

[0090] If the first service node that receives the data synchronization instruction is a slave node among the multiple first service nodes, the data synchronization instruction is forwarded to the master node among the multiple first service nodes. Further, the master node responds to the data synchronization instruction and can send the stored data to the multiple second service nodes 102, thereby synchronizing the data to the multiple second service nodes 102, and realizing data synchronization of the multiple service nodes 10. If the first service node that receives the data synchronization instruction is a master node, the data synchronization instruction can be sent to the multiple second service nodes 102 in response to the data synchronization instruction, thereby synchronizing the data to the multiple second service nodes 102, and realizing data synchronization of the multiple service nodes 10.

[0091] Specifically, a master node among the plurality of first service nodes 101 may send data to the plurality of second service nodes 102 corresponding to the identifiers of the plurality of second service nodes 102 according to the identifiers of the plurality of second service nodes 102 .

[0092] The second service node 102 may store data sent by the master node in the first service node 101. The second service node 102 may persistently store data sent by the master node in the first service node 101, thereby achieving data synchronization of multiple service nodes 10.

[0093] The multiple service nodes 10 include: a leader and followers. In the embodiment of the present application, for ease of description and distinction, the leader among the multiple service nodes 10 is defined as the first leader; the leader among the multiple first service nodes 101 is defined as the second leader. Generally, in the first service instance expansion stage and the multiple service node association stage, the first leader and the second leader are the same service node, that is, the multiple first service nodes 101 include the first leader.

[0094] After the second service node 102 stores the data sent by the first master node, a write success message may be returned to the first master node. The operation and maintenance node 20 may also send the correspondence between the multiple third service instances to be split from the first service instance and the identifier of the service node to any first service node 101. Optionally, the operation and maintenance node 20 may encapsulate the correspondence between the multiple third service instances to be split and the identifier of the service node into the aforementioned data synchronization instruction, and send the data synchronization instruction to any first service node 101. If the first service node 101 is the first master node, the correspondence between the multiple third service instances to be split and the identifier of the service node may be stored. If the first service node 101 is a slave node, the correspondence between the multiple third service instances to be split and the identifier of the service node is forwarded to the first master node. The first master node stores the correspondence between the multiple third service instances to be split and the identifier of the service node.

[0095] Based on the correspondence between the multiple third service instances to be split and the identifiers of the service nodes, the first service node may count the number of service nodes that return write success messages among the service nodes corresponding to the multiple third service instances to be split.

[0096] In some embodiments, the first master node may determine that data synchronization of multiple service nodes is completed when the number of services receiving write success messages is equal to the total number of slave nodes.

[0097] In other embodiments, since the consensus protocol adopts the majority principle for master election and request processing, in order to ensure the high availability of the distributed service system, when the master node is abnormal, a new master node can still be re-elected so that the status of the entire distributed service system can continue to move forward. The majority principle is also continued in the data synchronization stage. That is, as long as more than half of the service nodes of the instance are running normally before and after the instance splits, the services supported by the service instance can be provided. Based on this, the first master node can determine that data synchronization is completed when the number of service nodes that return a successful write message is greater than or equal to the set number R. Among them, the set number R is greater than or equal to half of the total number of multiple service nodes 10 corresponding to the first service instance C0', and less than the total number of multiple service nodes 10.

[0098] However, this data synchronization method may cause inconsistent data before and after the instance split. Figure 5 An ellipse in the figure represents a service node. Figure 5As shown, assuming that service node {1, 2, 3} is the original first service node of the second service instance C0; service node {4, 5, 6} is the second service node that is expanded, the aforementioned data synchronization method may result in the data synchronization of service node {1, 2, 3, 4} being completed, while the data synchronization of service node {5, 6} fails. In this way, when the instance is split, service node {1, 2, 3} is split into one service instance, and service node {4, 5, 6} is split into another service instance, but only one service node {4} of the service instance corresponding to service node {4, 5, 6} has the latest data. If service node {4} in the service instance corresponding to service node {4, 5, 6} is abnormal, the service instance corresponding to service node {4, 5, 6} will cause data loss.

[0099] In order to solve this problem, in some embodiments of the present application, for the multiple third service instances C to be split 1-n The corresponding set quantity M can be set. Among them, one third service instance corresponds to one set quantity M. The set quantity M corresponding to any third service instance is greater than or equal to half of the total number P of service nodes corresponding to the third service instance, and less than the total number P of service nodes corresponding to the service instance. That is, ≤M<P. Generally, M= For example, if P=5, then M can be 2.5. Multiple third service instances C 1-n The corresponding set numbers M may be the same or different, and are specifically determined by the number of service nodes corresponding to each third service instance.

[0100] In the multiple third service instances C to be split 1-n When the number of service nodes that return a successful write message among the corresponding service nodes is greater than the corresponding set number M, it is determined that the data synchronization of multiple service nodes 10 of the first service instance C0' is completed. In this way, each third service instance after the instance split can have data of more than half of the service nodes, which is consistent with the main node before the split. Even if a service node that has completed data synchronization has an abnormality, other service nodes that have completed data synchronization can provide complete data, which can reduce the risk of data loss before and after the instance split.

[0101] In other embodiments, a joint consensus mechanism may be introduced to resolve the risk of data loss caused by the aforementioned instance expansion. The following first describes the joint consensus mechanism required for the instance splitting solution provided in the embodiment of the present application. The joint consensus mechanism is a common method for implementing member changes.

[0102] Among them, service nodes sometimes fail during the operation of the distributed service system, and it is necessary to support the dynamic addition and deletion of service nodes. Member change is to change the set of nodes running the consistency coordination protocol (such as the Raft protocol) during the operation of the service cluster, such as adding, deleting, replacing, etc. The member change process cannot affect the availability of the distributed service system. Member change is also a consistency issue, that is, all service nodes reach a consensus on the new member. During the member change process, the members participating in the vote will change.

[0103] The Joint Consensus mechanism allows the service cluster to configure C from the old members (i.e. the original service nodes) first. old Switch to a transitional member configuration, called the Joint Consensus member configuration, which is the old member configuration C old And the new member (ie, new service node) configuration C new Combination C old,new Once the joint consensus members configure C old,new Submit, and then switch the service instance to the new member configuration C new Among them, the joint consensus member configuration C old,new Commit refers to the old member configuration C old The majority and new member configuration C new The majority in the other service nodes confirmed that the old member configuration C old The majority and new member configuration C new The majority in the group confirms that the other service nodes belong to the same member group as itself.

[0104] Combine the following Figure 6 The working principle of the joint consensus mechanism is explained. Figure 6 As shown, assuming that the set of original service nodes corresponding to a service instance is C old = {1, 2, 3}. The original service node can provide consistency coordination services. As long as any two of the three service nodes accept any request, the request reaches a consensus. The service node accepts the request means that the service node writes the request to the log, that is, the request is persistently stored. In each embodiment of the present application, the service node can log the requests in the order in which the requests are accepted. The log number can also be called the log sequence number, which is generally an increasing natural number.

[0105] If the service node of the service instance needs to be changed to service node {4, 5, 6}, the transition phase of the joint consensus (JointConsensus), that is, the original service node set C old = {1, 2, 3} and the new service node set C new={4, 5, 6} work together to elect a leader. In the transition phase of the joint consensus, for any request received by the service instance, only the original service node set C old The majority of the group accepts the request, and the new set of service nodes C new The master node will process the request only after the majority of the parties accept the request. After the transition phase of the joint consensus, the service instance will officially switch to the new service node set C. new , complete the member change. During the whole process, it can be ensured that the original service node and the new service node will not be used at the same time, so there will be no risk of "brain split". Among them, the service node is in the transition stage of joint consensus, that is, the service node is in the joint consensus state. That is, in the joint consensus state, only the original service node set C old The majority of the group accepts the request, and the new set of service nodes C new The master node will process the request only if the majority of the nodes accept the request.

[0106] The aforementioned joint consensus mechanism is used to implement member switching of service instances, not to implement the splitting of service instances. The embodiment of the present application can implement a solution for splitting service instances. The following is an exemplary description of the implementation method of the joint consensus mechanism applied to the service instance splitting provided in the embodiment of the present application.

[0107] Specifically, after determining that data synchronization of multiple service nodes 10 is completed, the first master node may modify the stored member group identifier to the identifiers of multiple service nodes under the condition of data synchronization of multiple service nodes. Among them, service nodes of the same member group correspond to the same service instance. The member group identifier stored by the service node is used to record the identifier of the service node corresponding to the same service instance as the service node. The first master node modifies the stored member group identifier to the identifiers of multiple service nodes, that is, the first master node confirms that the multiple service nodes belong to the same member group, that is, correspond to the same service instance. Among the identifiers of the multiple service nodes, the identifier of the first service node is originally stored by the first master node, and the identifier of the second service node is provided by the operation and maintenance node. The first master node may also store multiple third service instances C to be split. 1-n The correspondence between the ID of the service node.

[0108] Furthermore, the first master node may also send a joint instruction to the slave nodes in the multiple service nodes. The joint instruction may include: the identifiers of the multiple service nodes and the correspondence between the identifiers of the multiple third service instances to be split and the service nodes. Specifically, the joint instruction is used to instruct the slave node to modify the member group identifier to the identifiers of the multiple service nodes. The slave node refers to all the slave nodes that serve as the first master node in the multiple service nodes. The slave node may respond to the joint instruction, modify the stored member group identifier to the identifiers of the multiple service nodes, and store the correspondence between the multiple third service instances to be split and the identifiers of the service nodes, so that the multiple service nodes enter a joint consensus state (corresponding to Figure 4 in the “joint consensus” of Figure 4 The first service instance in the joint consensus state uses C joint express.

[0109] The service node that modifies the stored member group identifier to the identifiers of multiple service nodes may return a successful response message to the first master node. The first master node may determine whether the slave node has successfully modified the stored member group identifier to the identifiers of multiple service nodes based on whether the successful response message is received within a set time period. If the first master node receives a successful response message of a joint instruction returned by a slave node within a set time period, it is determined that the slave node has successfully modified the stored member group identifier to the identifiers of multiple service nodes.

[0110] In the embodiment of the present application, for the convenience of description and distinction, the service node whose stored member group identifier is successfully modified to the identifier of multiple service nodes is defined as the second target service node. Among them, the second target service node includes the first master node. Since the consensus protocol adopts the majority principle for master election and request processing, in order to ensure the high availability of the distributed service system, when the master node is abnormal, a new master node can still be re-elected so that the status of the entire distributed service system can continue to move forward. The majority principle is also continued in the joint consensus stage. That is, as long as more than half of the service nodes of the instance are running normally before and after the instance is split, the services supported by the service instance can be provided. Therefore, in the joint consensus stage, each third service instance C to be split 1-n As long as more than half of the service nodes are in the joint consensus state, it can be considered that the first service instance C0' before the split is in the joint consensus state.

[0111] Based on this, for the multiple third service instances C to be split 1-n , a set number M corresponding to each third service instance can be set. For the description of the set number M, please refer to the relevant content of the above embodiment, which will not be repeated here. Accordingly, the first master node can also count the number of second target service nodes in the service nodes corresponding to each third service instance, and in the multiple third service instances C to be split 1-nWhen the number of corresponding second target service nodes is greater than the corresponding set number M, it is determined that the multiple service nodes are in a joint consensus state. Further, the first master node may also send a first identifier corresponding to the joint consensus state to the slave node. The slave node may determine that the multiple service nodes are in a joint consensus state based on the first identifier. The role of the joint consensus state will be described in the following embodiments and will not be repeated here.

[0112] When multiple service nodes 10 are in a joint consensus state (such as Figure 6 C joint The first master node may forward the target request to the slave node for any request received in the joint consensus state (for ease of description, defined as the target request). The slave node that receives the target request may store the target request, such as persistently storing the target request, for example, writing the target request to a log. After the slave node stores the target request, it may return a storage success message to the first master node. If the first master node receives a storage success message for the target request returned by the slave node within the set time, it is determined that the slave node has stored the target request, and the slave node may be used as a target service node (defined as the third target service node).

[0113] Since the consensus protocol uses the majority principle to elect the master and process requests, in order to ensure the high availability of the distributed service system, when the master node is abnormal, a new master node can still be re-elected so that the status of the entire distributed service system can continue to move forward. The majority principle is also followed in the joint consensus state. Therefore, in the joint consensus state, each third service instance C 1-n As long as more than half of the service nodes store the target request, the first service instance is split into multiple third service instances C 1-n Afterwards, each third service instance C 1-n The corresponding service nodes are more than half of the service nodes, and the status of the master node before the split is consistent, so that the new service instance to be generated is the third service instance C. 1-n Compared with the first service instance C0', the data on it is not lost, which ensures the data consistency before and after the service instance split.

[0114] Based on this, for the multiple third service instances to be split, a set number M corresponding to each third service instance can be set. For the description of the set number M, please refer to the relevant content of the aforementioned embodiment, which will not be repeated here. Accordingly, the first master node can also count the number of third target service nodes in the service nodes corresponding to each third service instance, and when the number of third target service nodes corresponding to multiple third service instances is greater than the set number M corresponding to each of them, it is determined that the multiple service nodes have reached a consensus on the target request and can process the target request. In this way, in the joint consensus state, each third service instance C to be split 1-n As long as more than half of the service nodes store the target request, the service instance C 1-n Afterwards, each third service instance C 1-n More than half of the service nodes are consistent with the state of the master node before the split, so that the new service instance to be generated is the third service instance C. 1-n Compared with the first service instance C0', the data on it is not lost, which ensures the data consistency before and after the service instance split.

[0115] Among them, the slave node that stores the target request mentioned above can also process the stored requests in the order of first-in-first-out. That is, when the target request is traversed, the target request can be processed to reach consistency with the state of the first master node to maintain data consistency.

[0116] Under the joint consensus mechanism, even if less than half of the service nodes corresponding to one or some third service instances in the aforementioned service instance expansion phase achieve data synchronization, when a new target request (generally a write request) arrives at a service node that is not synchronized with the first master node, the complete data can be obtained based on the target request forwarded by the first master node. Specifically, for a service node that is not synchronized with the first master node, the first master node can be requested for data with a log sequence number less than the log sequence number of the target request based on the log sequence number of the target request, thereby obtaining complete data, which can prevent the data of the third service instance after the split from being inconsistent with the service instance before the split.

[0117] For example, assume that service node {1, 2, 3} is the original first service node of the second service instance C0; service node {4, 5, 6} is the expanded second service node, where service node {1, 2, 3, 4} completes data synchronization, while service node {5, 6} fails to synchronize data. When service nodes {1, 2, 3, 4, 5, 6} are in a joint consensus state, when a target request arrives, the first master node can forward the target request to service node {5, 6}. Service node {5, 6} can request data with a log sequence number smaller than the log sequence number of the target request from the first master node based on the log sequence number of the target request, thereby obtaining complete data. In this way, the third service instance C after the split 1-n The corresponding service node is consistent with the state of the primary node before the split, so that the new service instance to be generated, namely the third service instance C 1-n Compared with the first service instance C0', the data on it is not lost, which ensures the data consistency before and after the service instance split.

[0118] In order to prevent data inconsistency among multiple service nodes, the first master node sends a seal instruction to the slave node when multiple service nodes are in a joint consensus state. The seal instruction is used to instruct multiple service nodes to no longer receive write requests, that is, no longer respond to write requests, and no new data will be written to the first service instance. The slave node can confirm the seal instruction. Specifically, the slave node that receives the seal request can store the seal instruction, such as persistently storing the seal instruction, for example, writing the seal instruction to a log. After the slave node stores the seal instruction, it can return a confirmation success message to the first master node. If the first master node receives a confirmation success message for the seal instruction returned by the slave node within the set time, it is determined that the slave node confirms the seal instruction, and the slave node can be used as a target service node (defined as the first target service node). The above process corresponds to Figure 3 and Figure 4 "Prohibit writing" in .

[0119] Since the consensus protocol uses the majority principle to elect the master and process requests, in order to ensure the high availability of the distributed service system, when the master node is abnormal, a new master node can still be re-elected so that the status of the entire distributed service system can continue to advance. Therefore, in the joint consensus state, each third service instance C 1-n As long as more than half of the service nodes confirm the write-forbidden instruction, the server will be split into multiple third service instances C. 1-n Afterwards, each third service instance C 1-n The state of the determined master node is consistent with the state of the master node before the split, so that the new service instance to be generated, namely the third service instance C 1-nCompared with the first service instance C0', the data on it is not lost, which ensures the data consistency before and after the service instance split.

[0120] Based on this, for the multiple third service instances to be split, a set number M corresponding to each third service instance can be set. For the description of the set number M, please refer to the relevant content of the aforementioned embodiment, which will not be repeated here. Based on the pre-set number M, the first master node can also count the number of first target service nodes in the service nodes corresponding to each third service instance, and when the number of first target service nodes corresponding to multiple third service instances is greater than the corresponding set number M, it is determined that multiple service nodes have confirmed the write-disable instruction. In this way, when the service instance C is split into multiple third service instances, the first master node can confirm the write-disable instruction. 1-n Afterwards, each third service instance C 1-n The corresponding service node has the majority of service nodes in the master election, which is consistent with the state of the master node before the split, so that the new service instance to be generated, namely the third service instance C 1-n Compared with the first service instance C0', the data on it is not lost, which ensures the data consistency before and after the service instance split.

[0121] Further, when the number of first target service nodes corresponding to the plurality of third service instances is greater than the set number M corresponding to each, the first master node may send a split instruction to the slave node. The split instruction is used to instruct the plurality of service nodes to split the first service instance C0' into a plurality of third service instances C 1-n .

[0122] Specifically, when the number of first target service nodes corresponding to multiple third service instances is greater than the set number M corresponding to each, the first master node can obtain the correspondence between the multiple third service instances and the identifiers of the service nodes. The correspondence may be provided to the first master node by the aforementioned operation and maintenance node 20. Further, the first master node may generate a split instruction based on the correspondence between the multiple third service instances and the identifiers of the service nodes. For example, the correspondence between the multiple third service instances and the identifiers of the service nodes may be encapsulated into the split instruction. Further, the first master node may send the split instruction to the slave node.

[0123] Accordingly, the service node that obtains the split instruction can respond to the split instruction and convert the first service instance C in the joint consensus state joint Split into multiple third service instances C 1-n The service nodes that obtain the splitting instruction include the first master node itself and the slave nodes that receive the splitting instruction.

[0124] Specifically, the first master node and any service node A among the slave nodes that receive the split instruction can obtain the identifier of the service node belonging to the same third service instance as the service node A according to the correspondence between the multiple third service instances and the identifiers of the service nodes carried in the split instruction; and modify the stored member group identifier to the identifier of the service node corresponding to the same third service instance as itself, thereby splitting the first service instance into multiple third service instances. Figure 3 and Figure 4 As shown, the first service instance can be split into two third service instances C with service nodes {1,2,3} and {4,5,6} respectively. 1-1 and C 1-2 Etc. The above process corresponds to Figure 3 and Figure 4 The "split" in.

[0125] In this embodiment, in response to the instance split event, the service instance is expanded, and the expanded service instance is prevented from being written with a write-disable instruction, and then the expanded service instance is split into multiple service instances, thereby realizing the splitting of the service instance. In the capacity expansion phase, the original service node of the service instance can still provide services, and data synchronization can be completed while ensuring service continuation. By preventing the expanded service instance from being written with new data, the multiple service instances after the split start to split from the same state, and the data consistency of the service instances before and after the split can be guaranteed.

[0126] In addition, the entire process of the instance splitting solution provided in the embodiment of the present application does not require the introduction of additional management and control dependencies, thereby simplifying the system architecture.

[0127] In some embodiments of the present application, during the period from the completion of the service instance expansion to the split of the service instance, the write-disabled instruction makes the distributed service system service unavailable, and the service unavailable time window is relatively long. In order to reduce the service unavailable window, a joint consensus mechanism can be introduced in some embodiments, such as Figure 4 As shown, when the first service instance C0' is in joint consensus, the first master node can use a write-disable instruction to prevent new data from being written to the expanded service instance, and then use a split instruction to split the first service instance into multiple third service instances, thereby achieving the splitting of the service instance.

[0128] Among them, regarding the specific implementation method of uniting multiple service nodes corresponding to the first service instance to make them enter the joint consensus state and the working process of multiple service nodes under the joint consensus state, as well as using the write-disable instruction to prevent new data from being written to the expanded service instance, and then splitting the first service instance into multiple third service instances through the split instruction, please refer to the relevant content of the aforementioned embodiments, which will not be repeated here.

[0129] In this implementation, the original service nodes of the service instance can still provide services during the expansion phase, and data synchronization can be completed while ensuring service continuity. In the time period between the completion of service instance expansion and the splitting of the service instance, the joint consensus mechanism is used to enable multiple service nodes to be in a joint consensus state, and the joint consensus mechanism can still be used to process new requests, which can maintain data consistency while ensuring service continuity and reduce the unavailable time window.

[0130] For the first master node, the number of service nodes that confirm the aforementioned splitting instruction in the service nodes corresponding to each third service instance can also be counted. For the convenience of description and distinction, the service node that confirms the splitting instruction is defined as the fourth target service node. Specifically, if the first master node receives a confirmation success message for the splitting instruction returned by the slave node within the set time length, it is determined that the slave node confirms the splitting instruction, and the slave node can be used as a fourth target service node. Regarding the specific implementation method of the slave node confirming the splitting instruction, please refer to the relevant content of the aforementioned slave node confirming the write-disable instruction, which will not be repeated here.

[0131] In the joint consensus state, each third service instance C to be split 1-n As long as more than half of the service nodes confirm the split instruction, the split into multiple third service instances C 1-n Afterwards, each third service instance C 1-n The service nodes with the majority provide the service.

[0132] Based on this, for the multiple third service instances to be split, a set number M corresponding to each third service instance can be set. For the description of the set number M, please refer to the relevant content of the aforementioned embodiment, which will not be repeated here. Based on the pre-set set number M, the first master node can also count the number of fourth target service nodes in the service nodes corresponding to each third service instance, and when the number of fourth target service nodes corresponding to multiple third service instances is greater than the corresponding set number M, it is determined that multiple service nodes have confirmed the split instruction, then the first master node can modify the stored member group identifier to the identifier of the service node corresponding to the same third service instance as the first master node, so that the first master node exits the joint consensus state. In this way, each third service instance C to be split 1-n As long as more than half of the service nodes confirm the split instruction, the split into multiple third service instances C 1-n Afterwards, each third service instance C 1-n The majority of service nodes provide services, which can ensure the continuity of services.

[0133] However, if an exception occurs in the first master node during the service instance splitting phase, that is, an exception occurs in the first master node during the process of sending the splitting instruction. This may cause some slave nodes to fail to receive the splitting instruction. If more than half of the service nodes corresponding to one or some third service instances do not receive the splitting instruction, the splitting of the third service instance will fail. The service node that has not received the splitting instruction also needs to learn a clear global state from "somewhere", that is, the information that the instance split has occurred globally.

[0134] In order to enable the slave nodes that have not received the split instruction to learn the information that the global instance split has occurred, in some embodiments of the present application, the voting request used by the service node to seize the master can be improved. The voting request used in the traditional master election process is as follows: Figure 7 As shown in the left figure, it may include a triple consisting of a term number (epoch), a log ID (logid) and a server ID (serverid). Among them, the term number is an integer used to identify a logical time period. Every time the service cluster undergoes a potential master node change, the value of the term number will increase. The term number ensures that all service nodes in the service cluster are aware of the current election cycle in progress. If a service node receives a voting request with a higher term number, it will know that any previous voting commitments have expired and can reconsider new voting requests.

[0135] A log ID is a reference to a specific entry in the election log. Each log entry has a unique index value, which is the log ID. The log ID is used to prove that the candidate's log is at least as new (or more recent) as the recipient of the vote request. This is because only the service node with the latest log can become the primary node. Therefore, the candidate needs to provide the log ID of the last entry in its log to prove its eligibility.

[0136] The server ID is an identifier that uniquely identifies each service node in the service cluster. It is used to clarify which service node is requesting a vote, which helps the recipient determine the specific object of the vote.

[0137] In this embodiment, if Figure 7 As shown in the right figure, a target field is added to the traditional voting request. The target field is used to record whether the service cluster is in a joint consensus state. Among them, the first identifier can be used to indicate that the service cluster is in a joint consensus state, and the second identifier can be used to indicate that the service cluster is not in a joint consensus state. The first identifier and the second identifier are different identifiers, and the specific contents of the two are not limited. For example, the first identifier can be "1", the second identifier can be "0", etc.

[0138] In this embodiment, when the first master node determines that the service cluster enters the joint consensus state, it writes the first identifier into the target field, stores the target field, and may also send the first identifier to the slave node. The slave node may write the first identifier into the target field and store the target field.

[0139] After executing the split instruction, the first master node and the slave node that receives the split instruction can modify the value of the target field to the second identifier to indicate that they have exited the joint consensus state.

[0140] Based on the target field, the slave node can use the improved voting request to re-elect the master when the first master node is abnormal. That is, the slave node can send a voting request to other service nodes belonging to the same member group as the slave node to elect the master according to the currently stored member group identifier. The voting request includes a target field, and the voting request carries the value of the target field.

[0141] Specifically, the slave node can encapsulate the value of the target field stored in itself into the voting request, and send a voting request to other service nodes belonging to the same member group to elect a master according to the currently stored member group identifier. For a slave node that has executed a split instruction, its stored member group identifier is the identifier of the service node corresponding to the same third service instance as the slave node, and the value of the target field is a second identifier representing that it is not in a joint consensus state. Even if there are still service nodes in the member group of the slave node that have not received the split instruction, it can be determined that the service node that sent the voting request corresponds to the same third service instance according to the second identifier corresponding to the target field carried in the received voting request, and then vote on the voting request.

[0142] For the slave nodes that have not received the split instruction, the member group identifiers stored therein are the identifiers of multiple service nodes, and the value of the target field is the first identifier that represents that it is not in a joint consensus state. The slave node sends a voting request to the service node corresponding to the member group identifier recorded therein, and naturally also sends the voting request to the fourth target service node that has received the split instruction.

[0143] If the fourth target service node that has received the split instruction receives the voting request, it can obtain the value of the target field from the voting request. If the value of the target field is the first identifier, it is determined that the slave node that sent the voting request is the target slave node, and the target slave node is still in a joint consensus state, then the slave node that sent the voting request can be informed of the fact that the first service instance C0' has been split. Specifically, the correspondence between the multiple third service instances to be split and the identifiers of the service nodes can be returned to the target slave node that sent the voting request. In this way, the target slave node can know the fact that the first service instance C0' has been split based on the correspondence between the multiple third service instances to be split and the identifiers of the service nodes, and can determine the identifier of the service node corresponding to the same third service instance as the target slave node based on the correspondence between the multiple third service instances to be split and the identifiers of the service nodes; thereafter, the stored member group identifier can be modified to the identifier of the service node corresponding to the same third service instance as the target slave node to achieve the splitting of the service instance.

[0144] In this embodiment, by strengthening the information of the voting request, it is possible to support the service nodes that have not received the split instruction to learn the global split information of the service instance from the service nodes that have received the split instruction, guide the service nodes that have not received the split instruction to enter the split stage, and realize the split of the service instance.

[0145] In the distributed service system based on joint consensus provided in the embodiment of the present application, the term number and log ID mark the time and space before and after the service instance split in the distributed service system. Among them, time refers to logical time and can be represented by the term number. Specifically, Figure 8 As shown in the figure, the service instance starts from the second service instance C0, then expands to the first service instance C0', and even to the service instance C in the joint consensus state. joint , the time and space of the entire system are continuous, that is, from the original service instance C0 to the service instance C in the joint consensus state before the split joint , the log sequence on each service node in the distributed service system is completely consistent, and the term number and log ID are synchronously and monotonically increasing. 1-n ( Figure 8 Only the third service instance is C 1-1 and C 1-2 Multiple third service instances C 1-n Time and space bifurcation, starting from the same starting point, independently promotes subsequent state transitions according to their own time and space, processes their own requests, and realizes the split of the services provided by the service instance.

[0146] In addition to the distributed service system provided in the aforementioned embodiment, the embodiment of the present application also provides an instance splitting method. The instance splitting method provided in the embodiment of the present application is exemplarily described below.

[0147] Fig. 9 A flowchart of an instance splitting method provided in an embodiment of the present application. The instance splitting method is applicable to any service node among multiple service nodes corresponding to a first service instance. The first service instance is obtained by expanding the capacity of a second service instance in response to an instance splitting event. Multiple service nodes independently provide consistency coordination services, and data synchronization is completed. Fig. 9 As shown, the instance splitting method mainly includes:

[0148] 901. When serving as a master node, send a write-disable instruction to slave nodes among multiple service nodes, so that the service cluster no longer receives write requests.

[0149] 902. When the number of first target service nodes in the service nodes corresponding to the multiple third service instances to be split is greater than the corresponding set number, a split instruction is sent to the slave node, so that the master node and the slave node that receives the split instruction respond to the split instruction and split the first service instance into multiple third service instances.

[0150] Among them, the first target service node is the service node that confirms the write-disable instruction; the set number corresponding to any third service instance is greater than or equal to half of the total number of service nodes corresponding to the third service instance, and less than the total number of service nodes corresponding to the third service instance.

[0151] In the embodiment of the present application, the operation and maintenance node can monitor the instance split event for the second service instance C0, and in the case of monitoring the instance split event for the second service instance C0, respond to the instance split event and expand the second service instance C0 to obtain the expanded first service instance C0' and the expanded second service node. The specific implementation method of the operation and maintenance node monitoring the instance split event can be found in the relevant content of the aforementioned system embodiment, which will not be repeated here.

[0152] After scaling up the second service instance C0 in response to the instance split event, the operation and maintenance node may also instruct the first service node to synchronize data to the second service node. Specifically, the operation and maintenance node may send a data synchronization instruction to any first service node. The data synchronization instruction may include an identifier of the second service node.

[0153] If the first service node that receives the data synchronization instruction is a slave node among multiple first service nodes, the data synchronization instruction is forwarded to the master node among the multiple first service nodes. Further, the master node responds to the data synchronization instruction and can send the stored data to multiple second service nodes, thereby synchronizing the data to the multiple second service nodes, and realizing data synchronization of multiple service nodes. If the first service node that receives the data synchronization instruction is a master node, the stored data can be sent to multiple second service nodes in response to the data synchronization instruction, thereby synchronizing the data to the multiple second service nodes, and realizing data synchronization of multiple service nodes.

[0154] Specifically, a master node among the plurality of first service nodes may send data to the plurality of second service nodes corresponding to the identifiers of the plurality of second service nodes according to the identifiers of the plurality of second service nodes. The second service node stores the data sent by the master node among the first service nodes. The second service node may persistently store the data sent by the master node among the first service nodes to achieve data synchronization of the plurality of service nodes.

[0155] The multiple service nodes include: a master node and a slave node. In the embodiment of the present application, for the convenience of description and distinction, the master node among the multiple service nodes is defined as the first master node; the master node among the multiple first service nodes is defined as the second master node. Generally, in the first service instance expansion stage and the multiple service node joint stage, the first master node and the second master node are the same service node, that is, the multiple first service nodes include the first master node.

[0156] After the second service node stores the data sent by the first master node, a write success message may be returned to the first master node. The operation and maintenance node may also send the correspondence between the multiple third service instances to be split into by the first service instance and the identifiers of the service nodes to any first service node. Optionally, the operation and maintenance node may encapsulate the correspondence between the multiple third service instances and the identifiers of the service nodes into the aforementioned data synchronization instruction, and send the data synchronization instruction to any first service node. If the first service node is the first master node, the correspondence between the multiple third service instances and the identifiers of the service nodes may be stored. If the first service node is a slave node, the correspondence between the multiple third service instances and the identifiers of the service nodes is forwarded to the first master node. The first master node stores the correspondence between the multiple third service instances and the identifiers of the service nodes.

[0157] Based on the correspondence between the identifiers of the multiple third service instances and the service nodes, the first service node may count the number of service nodes that return a write success message among the service nodes corresponding to the multiple third service instances to be split.

[0158] In some embodiments, the first master node may determine that data synchronization of multiple service nodes is completed when the number of services receiving write success messages is equal to the total number of slave nodes.

[0159] In other embodiments, the first master node may determine that data synchronization is completed when the number of service nodes that return a write success message is greater than or equal to a set number R. The set number R is greater than or equal to half of the total number of the multiple service nodes corresponding to the first service instance C0' and less than the total number of the multiple service nodes.

[0160] However, this data synchronization method may lead to inconsistent data before and after the instance split. In order to solve this problem, in some embodiments of the present application, for multiple third service instances C to be split, 1-n , a set number M corresponding to each third service instance can be set. Among them, one third service instance corresponds to one set number M. The set number M corresponding to any third service instance is greater than or equal to half of the total number P of service nodes corresponding to the third service instance, and less than the total number P of service nodes corresponding to the service instance. That is, ≤M<P. Multiple third service instances C 1-n The corresponding set numbers M may be the same or different, and are specifically determined by the number of service nodes corresponding to each third service instance.

[0161] In the multiple third service instances C to be split 1-n When the number of service nodes that return a successful write message among the corresponding service nodes is greater than the corresponding set number M, it is determined that the data synchronization of multiple service nodes of the first service instance C0' is completed. In this way, each third service instance after the instance split can have data of more than half of the service nodes, which is consistent with the main node before the split. Even if a service node that has completed data synchronization has an abnormality, other service nodes that have completed data synchronization can provide complete data, which can reduce the risk of data loss before and after the instance split.

[0162] In other embodiments, a joint consensus mechanism may be introduced to address the risk of data loss caused by the aforementioned instance expansion. Specifically, after determining that data synchronization of multiple service nodes is complete, the first master node may modify the stored member group identifier to the identifiers of multiple service nodes while the data of multiple service nodes are synchronized. Among them, service nodes of the same member group correspond to the same service instance. The member group identifier stored by the service node is used to record the identifiers of the service nodes corresponding to the same service instance as itself. The first master node modifies the stored member group identifier to the identifiers of multiple service nodes, that is, the first master node confirms that the multiple service nodes belong to the same member group, that is, the same service instance. Among the identifiers of the multiple service nodes, the identifier of the first service node is originally stored by the first master node, and the identifier of the second service node is provided by the operation and maintenance node. The first master node may also store multiple third service instances C 1-n The correspondence between the ID of the service node.

[0163] Furthermore, the first master node may also send a joint instruction to the slave nodes among the multiple service nodes. The joint instruction may include: the identifiers of the multiple service nodes and the correspondence between the identifiers of the multiple third service instances and the service nodes. Specifically, the joint instruction is used to instruct the slave node to modify the member group identifier to the identifiers of the multiple service nodes. The slave node refers to all the slave nodes that serve as the first master node among the multiple service nodes. The slave node may respond to the joint instruction, modify the stored member group identifier to the identifiers of the multiple service nodes, and store the correspondence between the multiple third service instances and the identifiers of the service nodes, so that the multiple service nodes enter a joint consensus state.

[0164] The service node that modifies the stored member group identifier to the identifiers of multiple service nodes may return a successful response message to the first master node. The first master node may determine whether the slave node has successfully modified the stored member group identifier to the identifiers of multiple service nodes based on whether the successful response message is received within a set time period. If the first master node receives a successful response message of a joint instruction returned by a slave node within a set time period, it is determined that the slave node has successfully modified the stored member group identifier to the identifiers of multiple service nodes.

[0165] In the embodiment of the present application, for the convenience of description and distinction, the service node that successfully modifies the stored member group identifier to the identifiers of multiple service nodes is defined as the second target service node. Among them, the second target service node includes the first master node. Since the consensus protocol adopts the majority principle for master election and request processing, in order to ensure the high availability of the distributed service system, for the multiple third service instances C to be split 1-n, a set number M corresponding to each third service instance can be set. For the description of the set number M, please refer to the relevant content of the above embodiment, which will not be repeated here. Accordingly, the first master node can also count the number of second target service nodes in the service nodes corresponding to each third service instance, and 1-n When the number of corresponding second target service nodes is greater than the corresponding set number M, it is determined that the multiple service nodes are in a joint consensus state. Further, the first master node may also send a first identifier corresponding to the joint consensus state to the slave node. The slave node may determine that the multiple service nodes are in a joint consensus state based on the first identifier.

[0166] When multiple service nodes are in a joint consensus state, for any request received in the joint consensus state (defined as a target request for ease of description), the first master node may forward the target request to the slave node. The slave node that receives the target request may store the target request, such as persistently storing the target request, for example, writing the target request to a log, etc. After the slave node stores the target request, it may return a storage success message to the first master node. If the first master node receives a storage success message for the target request returned by the slave node within the set time, it is determined that the slave node has stored the target request, and the slave node may be used as a target service node (defined as the third target service node).

[0167] Since the consensus protocol adopts the majority principle for master election and request processing, in order to ensure the high availability of the distributed service system, when the master node is abnormal, a new master node can still be re-elected so that the status of the entire distributed service system can continue to move forward. Therefore, a corresponding set number M can be set for each of the multiple third service instances to be split. For the description of the set number M, please refer to the relevant content of the aforementioned embodiment, which will not be repeated here. Accordingly, the first master node can also count the number of third target service nodes in the service nodes corresponding to each third service instance, and when the number of third target service nodes corresponding to multiple third service instances is greater than the corresponding set number M, it is determined that multiple service nodes have reached a consensus on the target request and can process the target request. In this way, in the joint consensus state, each third service instance C to be split 1-n As long as more than half of the service nodes store the target request, the service instance C 1-n Afterwards, each third service instance C 1-n More than half of the service nodes are consistent with the state of the master node before the split, so that the new service instance to be generated is the third service instance C. 1-n Compared with the first service instance C0', the data on it is not lost, which ensures the data consistency before and after the service instance split.

[0168] Among them, the slave node that stores the target request mentioned above can also process the stored requests in the order of first-in-first-out. That is, when the target request is traversed, the target request can be processed to reach consistency with the state of the first master node to maintain data consistency.

[0169] Under the joint consensus mechanism, even if less than half of the service nodes corresponding to one or some third service instances in the aforementioned service instance expansion phase achieve data synchronization, when a new target request (generally a write request) arrives at a service node that is not synchronized with the first master node, the complete data can be obtained based on the target request forwarded by the first master node. Specifically, for a service node that is not synchronized with the first master node, the first master node can be requested for data with a log sequence number less than the log sequence number of the target request based on the log sequence number of the target request, thereby obtaining complete data, which can prevent the data of the third service instance after the split from being inconsistent with the service instance before the split.

[0170] In order to prevent data inconsistency among multiple service nodes, the first master node sends a seal instruction to the slave node when multiple service nodes are in a joint consensus state. The seal instruction is used to instruct multiple service nodes to no longer receive write requests, that is, no longer respond to write requests, and no new data will be written to the first service instance. The slave node can confirm the seal instruction. Specifically, the slave node that receives the seal request can store the seal instruction, such as persistently storing the seal instruction, for example, writing the seal instruction into a log, etc. After the slave node stores the seal instruction, it can return a confirmation success message to the first master node. If the first master node receives a confirmation success message for the seal instruction returned by the slave node within the set time, it is determined that the slave node confirms the seal instruction, and the slave node can be used as a target service node (defined as the first target service node).

[0171] Since the consensus protocol adopts the majority principle for master election and request processing, in order to ensure the high availability of the distributed service system, when the master node is abnormal, a new master node can still be re-elected so that the status of the entire distributed service system can continue to advance. Therefore, for the multiple third service instances to be split, a corresponding set number M can be set. For the description of the set number M, please refer to the relevant content of the aforementioned embodiment, which will not be repeated here. Based on the pre-set set number M, the first master node can also count the number of first target service nodes in the service nodes corresponding to each third service instance, and when the number of first target service nodes corresponding to multiple third service instances is greater than the corresponding set number M, it is determined that multiple service nodes have confirmed the write-disable instruction. In this way, when splitting into multiple third service instances C 1-n Afterwards, each third service instance C 1-nThe corresponding service node has the majority of service nodes in the master election, which is consistent with the state of the master node before the split, so that the new service instance to be generated, namely the third service instance C 1-n Compared with the first service instance C0', the data on it is not lost, which ensures the data consistency before and after the service instance split.

[0172] Further, when the number of first target service nodes corresponding to the plurality of third service instances is greater than the set number M corresponding to each, the first master node may send a split instruction to the slave node. The split instruction is used to instruct the plurality of service nodes to split the first service instance C0' into a plurality of third service instances C 1-n .

[0173] Specifically, when the number of first target service nodes corresponding to multiple third service instances is greater than the set number M corresponding to each, the first master node can obtain the correspondence between the multiple third service instances and the identifiers of the service nodes. The correspondence may be provided to the first master node by the aforementioned operation and maintenance node. Further, the first master node may generate a split instruction based on the correspondence between the multiple third service instances and the identifiers of the service nodes. For example, the correspondence between the multiple third service instances and the identifiers of the service nodes may be encapsulated into the split instruction. Further, the first master node may send the split instruction to the slave node.

[0174] Accordingly, the first master node and the slave node that receives the split instruction can respond to the split instruction and convert the first service instance C in the joint consensus state joint Split into multiple third service instances C 1-n .

[0175] Specifically, the first master node and each service node in the slave node that receives the split instruction can obtain the identifier of the service node corresponding to the same third service instance as itself based on the correspondence between the multiple third service instances carried by the split instruction and the identifiers of the service nodes; and modify the stored member group identifier to the identifier of the service node corresponding to the same third service instance as itself, thereby splitting the first service instance into multiple third service instances.

[0176] In this embodiment, in response to the instance split event, the service instance is expanded, and the expanded service instance is prevented from being written with a write-disable instruction, and then the expanded service instance is split into multiple service instances, thereby realizing the splitting of the service instance. In the capacity expansion phase, the original service node of the service instance can still provide services, and data synchronization can be completed while ensuring service continuation. By preventing the expanded service instance from being written with new data, the multiple service instances after the split start to split from the same state, and the data consistency of the service instances before and after the split can be guaranteed.

[0177] In addition, the entire process of the instance splitting solution provided in the embodiment of the present application does not require the introduction of additional management and control dependencies, thereby simplifying the system architecture.

[0178] In some embodiments of the present application, during the time period from the completion of the service instance expansion to the splitting of the service instance, the write disable instruction makes the distributed service system service unavailable, and the time window of service unavailability is relatively long. In order to reduce the service unavailability window, a joint consensus mechanism may be introduced in some embodiments. When the first service instance C0' is in a joint consensus, the first master node may, through the write disable instruction, prevent new data from being written to the expanded service instance, and then split the first service instance into multiple third service instances through the split instruction, thereby realizing the splitting of the service instance.

[0179] Among them, regarding the specific implementation method of uniting multiple service nodes corresponding to the first service instance to make them enter the joint consensus state and the working process of multiple service nodes under the joint consensus state, as well as using the write-disable instruction to prevent new data from being written to the expanded service instance, and then splitting the first service instance into multiple third service instances through the split instruction, please refer to the relevant content of the aforementioned embodiments, which will not be repeated here.

[0180] In this implementation, the original service nodes of the service instance can still provide services during the expansion phase, and data synchronization can be completed while ensuring service continuity. In the time period between the completion of service instance expansion and the splitting of the service instance, the joint consensus mechanism is used to enable multiple service nodes to be in a joint consensus state, and the joint consensus mechanism can still be used to process new requests, which can maintain data consistency while ensuring service continuity and reduce the unavailable time window.

[0181] For the first master node, the number of service nodes that confirm the aforementioned splitting instruction in the service nodes corresponding to each third service instance can also be counted. For the convenience of description and distinction, the service node that confirms the splitting instruction is defined as the fourth target service node. Specifically, if the first master node receives a confirmation success message for the splitting instruction returned by the slave node within the set time length, it is determined that the slave node confirms the splitting instruction, and the slave node can be used as a fourth target service node. Regarding the specific implementation method of the slave node confirming the splitting instruction, please refer to the relevant content of the aforementioned slave node confirming the write-disable instruction, which will not be repeated here.

[0182] In the joint consensus state, each third service instance C to be split 1-n As long as more than half of the service nodes confirm the split instruction, the split into multiple third service instances C 1-n Afterwards, each third service instance C 1-nThe service nodes with the majority provide the service.

[0183] Based on this, for the multiple third service instances to be split, a set number M corresponding to each third service instance can be set. For the description of the set number M, please refer to the relevant content of the aforementioned embodiment, which will not be repeated here. Based on the pre-set number M, the first master node can also count the number of fourth target service nodes in the service nodes corresponding to each third service instance, and when the number of fourth target service nodes corresponding to multiple third service instances is greater than the corresponding set number M, it is determined that multiple service nodes have confirmed the split instruction, then the first master node can modify the stored member group identifier to the identifier of the service node corresponding to the same third service instance as itself, so that the first master node exits the joint consensus state. In this way, each third service instance C to be split 1-n As long as more than half of the service nodes confirm the split instruction, the split into multiple third service instances C 1-n Afterwards, each third service instance C 1-n The majority of service nodes provide services, which can ensure the continuity of services.

[0184] However, if an exception occurs in the first master node during the service instance splitting phase, that is, an exception occurs in the first master node during the process of sending the splitting instruction. This may cause some slave nodes to fail to receive the splitting instruction. If more than half of the service nodes corresponding to one or some third service instances do not receive the splitting instruction, the splitting of the third service instance will fail. The service node that has not received the splitting instruction also needs to learn a clear global state from "somewhere", that is, the information that the instance split has occurred globally.

[0185] In order to enable the slave nodes that have not received the split instruction to learn the information that the instance split has occurred globally, in some embodiments of the present application, the voting request for the service node to seize the master can be improved. A target field is added on the basis of the traditional voting request. The target field is used to record whether the service cluster is in a joint consensus state. Among them, the first identifier can be used to represent that the service cluster is in a joint consensus state, and the second identifier can be used to represent that the service cluster is not in a joint consensus state. The first identifier and the second identifier are different identifiers, and the specific contents of the two are not limited. For example, the first identifier can be "1", the second identifier can be "0", and so on.

[0186] In this embodiment, when the first master node determines that the service cluster has entered the joint consensus state, it writes the first identifier into the target field, stores the target field, and may also send the first identifier to the slave node. The slave node may write the first identifier into the target field and store the target field. After executing the split instruction, the first master node and the slave node that receives the split instruction may modify the value of the target field to the second identifier to indicate that they have exited the joint consensus state.

[0187] Based on the target field, the slave node can use the improved voting request to re-elect the master when the first master node is abnormal. That is, the slave node can send a voting request to other service nodes belonging to the same member group to elect the master according to the currently stored member group identifier. The voting request includes a target field, and the voting request carries the value of the target field.

[0188] Specifically, the slave node can encapsulate the value of the target field stored in itself into the voting request, and send a voting request to other service nodes belonging to the same member group to elect a master according to the currently stored member group identifier. For a slave node that has executed a split instruction, its stored member group identifier is the identifier of the service node corresponding to the same third service instance as itself, and the value of the target field is a second identifier representing that it is not in a joint consensus state. Even if there are still service nodes in the member group of the slave node that have not received the split instruction, it can determine that the service node that sent the voting request corresponds to the same third service instance according to the second identifier corresponding to the target field carried in the received voting request, and then vote on the voting request.

[0189] For the slave nodes that have not received the split instruction, the member group identifier stored by them is the identifier of multiple service nodes, and the value of the target field is the first identifier that represents that it is not in a joint consensus state. The slave node sends a voting request to the service node corresponding to the member group identifier recorded by it, and naturally also sends the voting request to the fourth target service node that received the split request.

[0190] If the fourth target service node that receives the split request receives the voting request, it can obtain the value of the target field from the voting request. If the value of the target field is the first identifier, it is determined that the slave node that sends the voting request is the target slave node, and the target slave node is still in a joint consensus state, then the sending target slave node can be informed of the fact that the first service instance C0' has been split. Specifically, the correspondence between multiple third service instances and the identifiers of the service nodes can be returned to the target slave node. In this way, the target slave node can know the fact that the first service instance C0' has been split based on the correspondence between the multiple third service instances and the identifiers of the service nodes, and can determine the identifier of the service node corresponding to the same third service instance as the target slave node based on the correspondence between the multiple third service instances and the identifiers of the service nodes; thereafter, the stored member group identifier can be modified to the identifier of the service node corresponding to the same third service instance as the target slave node to achieve the splitting of the service instance.

[0191] In this embodiment, by strengthening the information of the voting request, it is possible to support the service nodes that have not received the split instruction to learn the global split information of the service instance from the service nodes that have received the split instruction, guide the service nodes that have not received the split instruction to enter the split stage, and realize the split of the service instance.

[0192] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 901 and 902 can be device A; for another example, the execution subject of step 901 can be device A, and the execution subject of step 902 can be device B; and so on.

[0193] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations appearing in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this document or may be executed in parallel, and the sequence numbers of the operations, such as 901, 902, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel.

[0194] Accordingly, an embodiment of the present application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps in the above-mentioned example splitting method.

[0195] The embodiment of the present application also provides a computer program product, including a computer program, which, when executed by one or more processors, causes the one or more processors to execute the steps in the above-mentioned example splitting method. In the embodiment of the present application, the specific implementation form of the computer program product is not limited. In some embodiments, the computer program product can be implemented as an application (Application, APP), a mini-program, a computer-side client, a program module, a plug-in, an installation package, a software development kit (Software Development Kit, SDK), an image file of a CD (such as an ISO file), a plug-in or software in the form of software as a service (Software as a Service, SaaS), etc., but is not limited to this.

[0196] Fig.10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig.10 As shown, the electronic device includes: a memory 100a, a processor 100b and a communication component 100c. The memory 100a is used to store computer programs.

[0197] The processor 100b is coupled to the memory 100a and the communication component 100c, and is used to execute a computer program to execute the steps in the example splitting method provided in the above embodiments. The specific implementation of each step can be found in the relevant description of the above embodiments, which will not be repeated here.

[0198] In some optional embodiments, such as Fig.10 As shown, the electronic device may also include optional components such as a communication component 100c, a power component 100d, a display component 100e and an audio component 100f. Fig.10 Some components are shown schematically, and it does not mean that electronic devices must include Fig.10 The components shown do not necessarily mean that the electronic device can only include Fig.10 Components shown.

[0199] in addition, Fig.10 The components in the dashed box are optional components, not mandatory components, and may depend on the product form of the electronic device. The electronic device of this embodiment may be implemented as a terminal device such as a desktop computer, a laptop computer, a mobile phone, or an IoT device; or it may be various server devices such as a traditional server, a cloud server, or a server cluster.

[0200] In an embodiment of the present application, the memory is used to store a computer program and can be configured to store various other data to support operations on the device where it is located. Among them, the processor can execute the computer program stored in the memory to implement the corresponding control logic. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random-Access Memory, SRAM), electrically erasable programmable read only memory (Electrically Erasable Programmable Read Only Memory, EEPROM), erasable programmable read only memory (Electrical Programmable Read Only Memory, EPROM), programmable read only memory (Programmable Read Only Memory, PROM), read only memory (Read Only Memory, ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0201] In the embodiment of the present application, the processor may be any hardware processing device that can execute the logic of the above method. Optionally, the processor may be a central processing unit (CPU), a graphics processing unit (GPU) or a microcontroller unit (MCU); it may also be a field programmable gate array (FPGA), a programmable array logic device (PAL), a general array logic device (GAL), a complex programmable logic device (CPLD) or other programmable devices; or it may be an advanced reduced instruction set (RISC) processor (Advanced RISC Machines, ARM) or a system on chip (System on Chip, SoC), etc., but not limited thereto.

[0202] In an embodiment of the present application, the communication component is configured to facilitate wired or wireless communication between the device in which it is located and other devices. The device in which the communication component is located can access a wireless network based on a communication standard, such as Wireless Fidelity (WiFi), 2G or 3G, 4G, 5G or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component can also be implemented based on Near Field Communication (NFC) technology, Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology or other technologies.

[0203] In an embodiment of the present application, the display component may include a liquid crystal display (LCD) and a touch panel (TP). If the display component includes a touch panel, the display component may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0204] In an embodiment of the present application, a power supply component is configured to provide power to various components of the device in which it is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply component is located.

[0205] In an embodiment of the present application, the audio component may be configured to output and / or input audio signals. For example, the audio component includes a microphone (Microphone, MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal may be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal. For example, for a device with a language interaction function, voice interaction with a user may be achieved through an audio component.

[0206] It should be noted that the descriptions such as “first” and “second” in this article are used to distinguish different messages, devices, modules, etc., and do not represent the order of precedence, nor do they limit “first” and “second” to different types.

[0207] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, compact disc read-only memory (CD-ROM), optical storage, etc.) containing computer-usable program codes.

[0208] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0209] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0210] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0211] In a typical configuration, a computing device includes one or more processors (CPU, etc.), input / output interfaces, network interfaces, and memory.

[0212] Memory may include non-permanent storage in a computer-readable medium, in the form of random-access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0213] Computer storage media is readable storage media, also known as readable media. Readable storage media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0214] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the above elements.

[0215] The above contents are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A distributed service system, characterized in that: include: A service cluster corresponding to the first service instance; the first service instance is obtained by expanding the capacity of the second service instance in response to an instance split event, the service cluster includes multiple service nodes that independently provide consistency coordination services; data synchronization of the multiple service nodes is completed; The master node among the multiple service nodes is used to: send a write-disable instruction to the slave nodes among the multiple service nodes to instruct the service cluster to no longer receive write requests; And, when the number of first target service nodes corresponding to each of the multiple third service instances to be split from the first service instance is greater than the corresponding set number, sending a split instruction to the slave node; the first target service node is a service node among the multiple service nodes that confirms the write-disable instruction; wherein the set number corresponding to any third service instance is greater than or equal to half of the total number of service nodes corresponding to any third service instance, and less than the total number; The master node and the slave node that receives the split instruction are used to: respond to the split instruction and split the first service instance into multiple third service instances.

2. The system according to claim 1, characterized in that The master node is also used for: When the data synchronization of the multiple service nodes is completed, modifying the stored member group identifiers to the identifiers of the multiple service nodes, and storing the correspondence between the multiple third service instances to be split and the identifiers of the service nodes; and, sending a joint instruction to the slave node; The slave node is used to respond to the joint instruction, modify the stored member group identifier to the identifiers of the multiple service nodes, and store the corresponding relationship so that the multiple service nodes enter a joint consensus state.

3. The system according to claim 2, characterized in that The master node is also used for: When the number of second target service nodes corresponding to each of the plurality of third service instances to be split is greater than the corresponding set number, determining that the plurality of service nodes enter the joint consensus state; The second target service node is: a service node among the multiple service nodes that successfully modifies the stored member group identifier into identifiers of multiple service nodes; For a target request received in the joint consensus state, forwarding the target request to the slave node; When the number of third target service nodes corresponding to each of the plurality of third service instances to be split is greater than the corresponding set number, determining that the plurality of service nodes reach a consensus on the target request and processing the target request; The third target service node is a service node storing the target request among the multiple service nodes.

4. The system according to claim 3, characterized in that The master node is also used for: When the number of fourth target service nodes corresponding to each of the plurality of third service instances to be split is greater than the corresponding set number, modifying the stored member group identifier to the identifier of the service node corresponding to the same third service instance as the master node, so that the master node exits the joint consensus state; The fourth target service node is a service node among the multiple service nodes that confirms the split instruction.

5. The system according to claim 4, characterized in that The master node is further configured to: when it is determined that the multiple service nodes enter a joint consensus state, send a first identifier representing that the multiple service nodes are in a joint consensus state to the slave node; The slave node is used to send a voting request to other service nodes belonging to the same member group as the slave node to elect a master according to the currently stored member group identifier and the value of the stored target field when the master node is abnormal; The voting request carries the value of the target field, where the target field is used to record whether the multiple service nodes are in a joint consensus state; The fourth target service node is configured to return the corresponding relationship to a target slave node when receiving a voting request in which the value of the target field is the first identifier, the target slave node being the slave node that sends the voting request in which the value of the target field is the first identifier; The target slave node is used to modify the stored member group identifier to the identifier of the service node corresponding to the same third service instance as the target slave node according to the corresponding relationship.

6. The system according to any one of claims 1 to 5, characterized in that: The distributed service system further includes an operation and maintenance node; the multiple service nodes include: multiple first service nodes corresponding to the first service instance before being expanded, and multiple second service nodes expanded in response to the instance split event; the multiple first service nodes include the master node; The operation and maintenance node is used to expand the first service instance in response to the instance split event to obtain the multiple second service nodes; and instruct the multiple first service nodes to synchronize data to the multiple second service nodes; The master node is used to send the stored data to the multiple second service nodes to synchronize the data to the multiple second service nodes.

7. An instance splitting method, applicable to any service node among a plurality of service nodes corresponding to a first service instance, characterized in that: The first service instance is obtained by expanding the capacity of the second service instance in response to the instance split event, the multiple service nodes independently provide consistency coordination services, and data synchronization is completed; The method comprises: In the case of acting as a master node, sending a write-disable instruction to slave nodes among the multiple service nodes to instruct the service cluster to which the multiple service nodes belong to no longer receive write requests; When the number of first target service nodes corresponding to each of the plurality of third service instances to be split from the first service instance is greater than the corresponding set number, sending a split instruction to the slave node, so that the master node and the slave node receiving the split instruction respond to the split instruction to split the first service instance into the plurality of third service instances; Among them, the first target service node is the service node among the multiple service nodes that confirms the write-disable instruction; the set number corresponding to any third service instance is greater than or equal to half of the total number of service nodes corresponding to any third service instance, and is less than the total number.

8. The method according to claim 7, characterized in that The method further comprises: Acquire a correspondence between the multiple third service instances to be split and identifiers of the service nodes; The split instruction is generated according to the corresponding relationship.

9. The method according to claim 8, characterized in that The master node responds to the split instruction to split the first service instance into multiple third service instances, including: Obtaining the corresponding relationship from the split instruction; According to the corresponding relationship, obtaining an identifier of a service node corresponding to the same third service instance as the primary node; Modify the stored member group identifier to the identifier of the service node corresponding to the same third service instance as the master node, so as to split the third service instance corresponding to the member group to which the master node belongs from the first service instance; The service nodes belonging to the same member group correspond to the same third service instance.

10. The method according to claim 9, characterized in that The method further comprises: When the data synchronization of the plurality of service nodes is completed, modifying the stored member group identifiers to identifiers of the plurality of service nodes, and storing the corresponding relationship; A joint instruction is sent to the slave node so that the slave node responds to the joint instruction, the stored member group identifier is modified to the identifiers of the multiple service nodes and the corresponding relationship is stored, so that the multiple service nodes enter a joint consensus state.

11. The method according to claim 10, characterized in that The method further comprises: When the number of second target service nodes corresponding to each of the plurality of third service instances to be split is greater than the corresponding set number, determining that the plurality of service nodes enter the joint consensus state; the second target service node is: a service node among the plurality of service nodes that successfully modifies the stored member group identifier to the identifiers of the plurality of service nodes; For a target request received in the joint consensus state, forwarding the target request to the slave node; When the number of third target service nodes corresponding to each of the multiple third service instances to be split is greater than the corresponding set number, it is determined that the multiple service nodes have reached a consensus on the target request and process the target request; the third target service node is the service node that stores the target request among the multiple service nodes.

12. The method according to claim 11, characterized in that The method further comprises: When the number of fourth target service nodes corresponding to each of the plurality of third service instances to be split is greater than the corresponding set number, modifying the stored member group identifier to the identifier of the service node corresponding to the same third service instance as the master node, so that the master node exits the joint consensus state; The fourth target service node is a service node among the multiple service nodes that confirms the split instruction.

13. The method according to claim 12, characterized in that The method further comprises: In the case of determining that the multiple service nodes enter the joint consensus state, sending a first identifier representing that the multiple service nodes are in the joint consensus state to the slave node, so that the slave node writes the first identifier into a stored target field; the target field is used to record whether the multiple service nodes are in the joint consensus state; Wherein, when the master node is abnormal, the slave node sends a voting request to other service nodes belonging to the same member group to elect a master according to the currently stored member group identifier and the stored value of the target field; the voting request carries the value of the target field, so that a fourth target service node among the other service nodes returns the corresponding relationship to a target slave node when receiving a voting request whose value of the target field is the first identifier; the target slave node is a slave node that sends a voting request whose value of the target field is the first identifier; The target slave node modifies the stored member group identifier into the identifier of the service node corresponding to the same third service instance as the slave node according to the corresponding relationship.

14. An electronic device, characterized in that: include: Memory, processor and communication components; wherein the memory is used to store computer programs; The processor is coupled to the memory and the communication component, and is configured to execute the computer program to perform the steps in the method according to any one of claims 7 to 13.

15. A computer-readable storage medium storing computer instructions, characterized in that: When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps in the method according to any one of claims 7 to 13.

16. A computer program product, characterized in that The method comprises a computer program which, when executed by one or more processors, causes the one or more processors to execute the steps of the method according to any one of claims 7 to 13.

Images