Distributed system cluster online sharding expansion method and device

Abstract

A method and device for online sharding and capacity expansion of a distributed system cluster can be used in the financial field or other fields. The method includes: cluster numbering of newly added clusters, setting a sequence number segment interval of the newly added cluster in the cluster routing configuration rule; the sequence numbers in the sequence number segment interval are continuous and monotonically increasing numbers; receiving an external data request, and determining the hash value corresponding to the business field in the external data request; if the number of cluster data in the original cluster is less than the number of sequence numbers in its sequence number segment interval, then determining the cluster number of the corresponding original cluster according to the hash value, and routing the cluster data to the corresponding original cluster; if the number of cluster data in the original cluster is not less than the number of sequence numbers in its sequence number segment interval, then determining the cluster number of the corresponding newly added cluster according to the hash value, and routing the cluster data to the corresponding newly added cluster. The present invention avoids large-scale data migration for cluster expansion, requires almost no change time, reduces the impact on the business system, and saves resource costs.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed systems, and in particular to a method and device for online sharding and capacity expansion of a distributed system cluster. Background Art

[0002] Currently, when a new business system is first launched, the volume is relatively low. To save resources and costs, a minimum cluster deployment is generally adopted. For systems using a modular or microservices architecture, the database and application containers are deployed together. Database sharding typically determines the cluster where data resides by hashing the business field and then taking the modulus of the initial cluster (where the business field is the hash element). The formula is: mod(hash(business field), number of clusters n). Initially, the number of clusters n is relatively small. In a same-city active-active deployment architecture, n is typically an even number, with two campuses deployed in equal proportions, for example, n = 2. As business volume increases, the number of clusters needs to be expanded. For 24 / 7 services, to ensure business continuity, expansion downtime is completed within normal production hours. A common practice is to double the capacity, meaning the number of clusters after expansion is n × 2. The formula for calculating the sharding of business data in the new cluster after expansion is: mod(hash(business field), number of clusters n * 2).

[0003] With a double capacity expansion approach, data originally in Cluster 0 will inevitably be moved to Cluster 0 and Cluster 2 after expansion, while data originally in Cluster 1 will be moved to Cluster 1 and Cluster 3 after expansion. Using database semi-synchronous replication technology, online data expansion is possible. However, in real-world production, data distribution may be uneven. Some clusters have more active services and faster data growth, which can quickly lead to insufficient database performance and capacity. Double capacity expansion every time not only wastes resources but also requires full data migration, resulting in a wide impact and a significant workload. Summary of the Invention

[0004] In response to the problems existing in the prior art, the main purpose of the embodiments of the present invention is to provide a method and device for online sharding expansion of a distributed system cluster, so as to realize online sharding expansion of the cluster, ensure business continuity, reduce the workload of data replication and migration, etc.

[0005] To achieve the above objectives, an embodiment of the present invention provides a method for online sharding capacity expansion of a distributed system cluster, the method comprising:

[0006] Assign a cluster number to the newly added cluster, and set a sequence number segment corresponding to the newly added cluster in the cluster routing configuration rule of the distributed system; wherein the cluster routing configuration rule includes the cluster number and sequence number segment of the original cluster, and the sequence numbers in the sequence number segment of the original cluster and the newly added cluster are continuous and monotonically increasing numbers;

[0007] Receive an external data request and use a hash algorithm to determine a hash value corresponding to a business field in the external data request;

[0008] If the number of cluster data in the original cluster is less than the number of sequence numbers in its sequence number segment, the cluster number of the original cluster corresponding to the business field is determined according to the hash value, and the cluster data corresponding to the business field is routed to the corresponding original cluster;

[0009] If the number of cluster data in the original cluster is not less than the number of sequence numbers in its sequence number segment, the cluster number of the newly added cluster corresponding to the business field is determined according to the hash value, and the cluster data corresponding to the business field is routed to the corresponding newly added cluster.

[0010] Optionally, in an embodiment of the present invention, the cluster numbering of the newly added clusters includes:

[0011] The newly added cluster corresponding to the newly built cluster device in the distributed system is assigned a cluster number, and the service switch of the newly added cluster is turned off.

[0012] Optionally, in an embodiment of the present invention, setting the sequence number segment interval corresponding to the newly added cluster in the cluster routing configuration rule of the distributed system includes:

[0013] According to the cluster number of the newly added cluster and the sequence number segment of the original cluster, the sequence number segment corresponding to the newly added cluster is added to the cluster information list of the cluster routing configuration rule, and the service switch of the newly added cluster is turned on.

[0014] Optionally, in an embodiment of the present invention, determining the cluster number of the original cluster corresponding to the service field according to the hash value includes:

[0015] The hash value is compared with the cluster number of the original cluster to determine the cluster number of the original cluster corresponding to the service field.

[0016] An embodiment of the present invention further provides a distributed system cluster online sharding expansion device, the device comprising:

[0017] A new cluster module is used to assign cluster numbers to newly added clusters and set the sequence number segment interval corresponding to the newly added cluster in the cluster routing configuration rules of the distributed system; wherein the cluster routing configuration rules include the cluster number and sequence number segment interval of the original cluster, and the sequence numbers in the sequence number segment intervals of the original cluster and the newly added cluster are continuous and monotonically increasing numbers;

[0018] A hash value module is used to receive an external data request and determine a hash value corresponding to a business field in the external data request using a hash algorithm;

[0019] A first routing module is configured to determine the cluster number of the original cluster corresponding to the service field according to the hash value if the amount of cluster data in the original cluster is less than the number of sequence numbers in its sequence number segment, and route the cluster data corresponding to the service field to the corresponding original cluster;

[0020] The second routing module is used to determine the cluster number of the newly added cluster corresponding to the business field according to the hash value if the amount of cluster data in the original cluster is not less than the number of sequence numbers in its sequence number segment, and route the cluster data corresponding to the business field to the corresponding newly added cluster.

[0021] Optionally, in an embodiment of the present invention, the newly added cluster module is further configured to assign a cluster number to the newly added cluster corresponding to the newly built cluster device in the distributed system, and to turn off a service switch of the newly added cluster.

[0022] Optionally, in one embodiment of the present invention, the newly added cluster module is also used to add the serial number segment interval corresponding to the newly added cluster to the cluster information list of the cluster routing configuration rule according to the cluster number of the newly added cluster and the serial number segment interval of the original cluster, and turn on the service switch of the newly added cluster.

[0023] Optionally, in an embodiment of the present invention, the first routing module is further configured to compare the hash value with the cluster number of the original cluster to determine the cluster number of the original cluster corresponding to the service field.

[0024] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method when executing the program.

[0025] The present invention also provides a computer-readable storage medium storing a computer program for executing the above method.

[0026] The present invention realizes shard expansion in a distributed system by gradually increasing clusters based on monotonically increasing hash elements, avoiding large-scale data migration during cluster expansion, requiring almost no change time, reducing the impact on the business system, and saving resource costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0028] Figure 1 Schematic diagram of data sharding for doubling cluster capacity;

[0029] Figure 2 This is a flow chart of a distributed system cluster online sharding expansion method according to an embodiment of the present invention;

[0030] Figure 3 This is a schematic diagram of online sharding expansion in an embodiment of the present invention;

[0031] Figure 4 A schematic diagram of an online sharding and capacity expansion architecture for a distributed system cluster in a specific embodiment of the present invention;

[0032] Figure 5 This is a flowchart of online sharding expansion in a specific embodiment of the present invention;

[0033] Figure 6 This is a schematic diagram of the structure of a distributed system cluster online sharding expansion device according to an embodiment of the present invention;

[0034] Figure 7 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The embodiments of the present invention provide a method and device for online sharding capacity expansion of a distributed system cluster, which can be used in the financial field or other fields. It should be noted that the method and device for online sharding capacity expansion of a distributed system cluster of the present invention can be used in the financial field, and can also be used in any field other than the financial field. The application field of the method and device for online sharding capacity expansion of a distributed system cluster of the present invention is not limited.

[0036] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0037] The commonly used approach at present is to adopt the double expansion method. After the data is expanded, the distribution is as follows: Figure 1 The figure shows a schematic diagram of data sharding for double cluster expansion. The steps for expansion are as follows:

[0038] 1. Add a cross-park semi-synchronous backup database to each original cluster database and wait for data synchronization to complete.

[0039] 2. Modify the routing configuration, expand the number of clusters to twice the original number of clusters, use the new routing algorithm: mod(hash(business field), number of clusters n*2), and suspend external services for the newly expanded business.

[0040] 3. Change the semi-synchronous standby database to the primary database, and add a new cluster to connect to the newly added primary database. Figure 1 , business cluster 1 uses the original backup database of cluster 1, and business cluster 2 uses the original backup database of cluster 0.

[0041] 4. The newly expanded cluster begins to take over services.

[0042] 5. Delete redundant data in each cluster.

[0043] With a double capacity expansion approach, data originally in Cluster 0 will inevitably be moved to Cluster 0 and Cluster 2 after expansion, while data originally in Cluster 1 will be moved to Cluster 1 and Cluster 3 after expansion. Using database semi-synchronous replication technology, online data expansion is possible. However, in real-world production, data distribution may be uneven. Some clusters have more active services and faster data growth, which can quickly lead to insufficient database performance and capacity. Double capacity expansion every time not only wastes resources but also requires full data migration, resulting in a wide impact and a significant workload.

[0044] like Figure 2 The flowchart of a distributed system cluster online sharding expansion method according to an embodiment of the present invention is shown. The execution subject of the distributed system cluster online sharding expansion method provided by the embodiment of the present invention includes but is not limited to a computer. The method shown in the figure includes:

[0045] Step S1, cluster number the newly added cluster, and set the serial number segment interval corresponding to the newly added cluster in the cluster routing configuration rule of the distributed system; wherein, the cluster routing configuration rule includes the cluster number and serial number segment interval of the original cluster, and the serial numbers in the serial number segment interval of the original cluster and the newly added cluster are continuous and monotonically increasing numbers.

[0046] Among them, such as Figure 3 The figure shows a schematic diagram of online sharding expansion in an embodiment of the present invention. Specifically, in a distributed system, for example, the service field A used for sharding, i.e., the hash element, is monotonically increasing, such as a gradually increasing number starting from 0 to 999999999. The service volume carried by each cluster is determined based on the service development situation, for example, every two clusters carry the service volume corresponding to 999999. In the initial launch, to save costs, there are only two clusters. The cluster information list is {[0-999999]:(0,1)}, representing clusters 0 and 1. [0-999999] is the sequence number range carried by clusters 0 and 1.

[0047] Furthermore, when the 999999 serial numbers are exhausted, a new number segment will be activated. Specifically, a new cluster device is built in the distributed system, and the newly added cluster is configured in the cluster routing configuration rules of the distributed system.

[0048] Specifically, for example, in the original cluster routing configuration rule, the cluster information list is {[0-999999]:(0,1)}. Since the 999999 numbers are about to be exhausted, the new clusters are numbered 2 and 3. The cluster information list is changed to {[0-999999]:(0,1), [1000000-1999999]:(2,3)}, indicating the addition of two new subclusters, 2 and 3, while the original clusters 0 and 1 remain unchanged.

[0049] Furthermore, the serial numbers in the serial number segment intervals of the original cluster and the newly added cluster are continuous and monotonically increasing numbers. For example, the original clusters are cluster 0 and cluster 1, and the corresponding serial number segment intervals are [0-999999]. Then the serial number segment interval corresponding to the newly added cluster should be continuous with the serial number segment interval of the original cluster, which can be [1000000-1999999].

[0050] Step S2: receiving an external data request and using a hash algorithm to determine a hash value corresponding to a business field in the external data request.

[0051] Wherein, an external data request is received, and according to the business field in the external data request, for example, the business field is A. The business field A is used as the cluster sharding field, and the hash value hash(A) of the business field is calculated.

[0052] Step S3: If the number of cluster data in the original cluster is less than the number of sequence numbers in its sequence number segment, the cluster number of the original cluster corresponding to the business field is determined according to the hash value, and the cluster data corresponding to the business field is routed to the corresponding original cluster.

[0053] Among them, if the number of cluster data in the original cluster is less than the number of sequence numbers in its sequence number segment, it means that the sequence numbers in the sequence number segment of the original cluster have not been exhausted, then calculate the remainder value of the hash value and the number of original clusters, that is, mod(hash(A),2).

[0054] Furthermore, the remainder value is compared with the cluster information list in the cluster routing configuration rule. For example, if the remainder value is 0, the cluster data corresponding to the external data request is routed to cluster 0.

[0055] Step S4: If the number of cluster data in the original cluster is not less than the number of sequence numbers in its sequence number segment, the cluster number of the newly added cluster corresponding to the business field is determined according to the hash value, and the cluster data corresponding to the business field is routed to the corresponding newly added cluster.

[0056] If the number of cluster data in the original cluster is not less than the number of sequence numbers in its sequence number range, the sequence numbers in the original cluster's sequence number range have been exhausted. For example, when the original cluster's sequence numbers 0-999999 are exhausted, the new cluster's sequence numbers are activated. The routing calculation method for external data requests for the newly added cluster is similar to the above method, calculating the remainder of the hash value and the number of newly added clusters, that is, using mod(hash(A),2) to calculate the remainder.

[0057] Furthermore, the remainder value is compared with the cluster information list in the cluster routing configuration rule. For example, if the remainder value is 0, then sequence number 0 corresponds to cluster 2, and the cluster data corresponding to the external data request is routed to cluster 2.

[0058] As an embodiment of the present invention, cluster numbering the newly added cluster includes: cluster numbering the newly added cluster corresponding to the newly built cluster device in the distributed system, and turning off the service switch of the newly added cluster.

[0059] In this case, a new cluster device is built in the distributed system, and the newly added cluster is set in the cluster routing configuration rules of the distributed system. Specifically, if the original cluster numbers are 0 and 1, the newly added cluster numbers can be set to 2 and 3.

[0060] Furthermore, since the routes corresponding to the newly added cluster have not been modified, it will not receive any services. Therefore, the service switch of the newly added cluster is turned off.

[0061] As an embodiment of the present invention, setting the serial number segment interval corresponding to the newly added cluster in the cluster routing configuration rules of the distributed system includes: adding the serial number segment interval corresponding to the newly added cluster to the cluster information list of the cluster routing configuration rules according to the cluster number of the newly added cluster and the serial number segment interval of the original cluster, and turning on the service switch of the newly added cluster.

[0062] Among them, the serial numbers in the serial number segment interval of the original cluster and the newly added cluster are continuous and monotonically increasing numbers. For example, the original clusters are cluster 0 and cluster 1, and the corresponding serial number segment interval is [0-999999]. Then the serial number segment interval corresponding to the newly added cluster should be continuous with the serial number segment interval of the original cluster, which can be [1000000-1999999].

[0063] Furthermore, after the routing configuration rules corresponding to the newly added cluster are set up, the service switch of the newly added cluster can be turned on so that it can receive external data requests.

[0064] As an embodiment of the present invention, determining the cluster number of the original cluster corresponding to the service field according to the hash value includes: comparing the hash value with the cluster number of the original cluster to determine the cluster number of the original cluster corresponding to the service field.

[0065] The remainder of the hash value and the original number of clusters is calculated, that is, mod(hash(A),2). The remainder is compared with the cluster information list in the cluster routing configuration rule. For example, if the remainder is 0, the cluster data corresponding to the external data request is routed to cluster 0.

[0066] As a specific embodiment of the present invention, Figure 4 The following diagram shows the architecture for online sharding and scaling distributed system clusters. It consists of three main components: the configuration center, the access gateway layer, and the business processing layer. The configuration center stores routing configuration information and cluster traffic control switches. Both the access gateway layer and the business processing layer access the configuration center to obtain relevant configurations.

[0067] Among them, the present invention utilizes the monotonically increasing characteristics of the hash element field and the cluster data is carried out around this element, specifies the serial number segment of each cluster hash element, and gradually increases the number of clusters. For example, assuming that the business is carried out according to the customer dimension, and the customer code is a 9-digit monotonically increasing number, the system can agree on the number of customer codes supported by each cluster, such as clusters 0 and 1 support the [0-999999] serial number segment. When the usage of cluster 0 reaches 80%, clusters 2 and 3 are enabled, and the serial number segment is [1000000-1999999]. It is only necessary to synchronize the relevant parameter data to the new cluster, which can realize the online upgrade without interruption of business, save resources compared to the double expansion method, and the system expansion is not noticeable to customers.

[0068] In a distributed system, the business field A used for sharding, i.e., the hash element, is monotonically increasing, for example, starting from 0 and gradually increasing to 999999999. The business volume carried by each cluster is determined based on business development, for example, every two clusters carry the business volume corresponding to 999999. In the initial launch, to save costs, there are only two clusters. The configuration center stores the cluster information list {[0-999999]:(0,1)}, representing clusters 0 and 1. The calculation rule for sharding is:

[0069] (1) Determine which serial number range A is in, assuming it is between 0 and 999999.

[0070] (2) Use mod(hash(A),2) to compare the remainder with the sequence number of the configuration center's information list (0,1). For example, if it is 0, it is routed to cluster 0.

[0071] in, Figure 5This is a flowchart of the online sharding expansion method. Assuming that the business volume continues to increase and the number range 0-999999 is about to be exhausted, a new number range is enabled. The general steps are as follows:

[0072] In step S101, new cluster related equipment is built and put online. The cluster numbers are 2 and 3. Since the routes have not been modified, no services will be received.

[0073] Step S102: Modify the cluster routing configuration of the configuration center to {[0-999999]:(0,1), [1000000-1999999]:(2,3)}, add two new subclusters 2 and 3, and keep the original clusters 0 and 1 unchanged to continue receiving requests.

[0074] In step S103, the new cluster begins receiving requests. When the sequence numbers 0-999999 are exhausted, the new cluster sequence number is activated. The routing calculation method for the new cluster request is similar, using mod(hash(A),2) to calculate the remainder. The remainder is the sequence number of the information list (2,3) in the configuration center. For example, if it is 0, it is routed to cluster 2.

[0075] Further, such as Figure 3 The figure below shows a schematic diagram of shard expansion based on monotonically increasing hash elements. As business volume grows, the number of clusters gradually increases. If the existing pre-allocated clusters are not under much pressure, you can add sequence number segments to the original clusters. When calculating shards, you can match multiple sequence number segments, such as {([0-999999], [2000000-2999999]): (0, 1), [1000000-1999999]: (2, 3)}. This allows for dynamic adjustment of cluster data processing.

[0076] This invention leverages the monotonically increasing nature of business fields in sharded tables and databases to gradually add clusters, enabling online cluster expansion as business grows. This ensures business continuity, reduces data duplication and migration workload, and ensures that the upgrade process is seamless to all customers.

[0077] The present invention realizes shard expansion in a distributed system by gradually increasing clusters based on monotonically increasing hash elements, avoiding large-scale data migration during cluster expansion, requiring almost no change time, reducing the impact on the business system, and saving resource costs.

[0078] like Figure 6 FIG2 is a schematic diagram of the structure of a distributed system cluster online sharding expansion device according to an embodiment of the present invention. The device shown in the figure includes:

[0079] A new cluster module 10 is used to cluster number the newly added cluster and set the serial number segment interval corresponding to the newly added cluster in the cluster routing configuration rule of the distributed system; wherein, the cluster routing configuration rule includes the cluster number and serial number segment interval of the original cluster, and the serial numbers in the serial number segment interval of the original cluster and the newly added cluster are continuous and monotonically increasing numbers.

[0080] In a distributed system, for example, the business field A used for sharding, or the hash element, is monotonically increasing, such as a gradually increasing number starting from 0 to 999999999. The business volume carried by each cluster is determined based on business development, for example, every two clusters carry the business volume corresponding to 999999. To save costs in the initial launch, there are only two clusters. The cluster information list is {[0-999999]:(0,1)}, representing clusters 0 and 1. [0-999999] is the sequence number range carried by clusters 0 and 1.

[0081] Furthermore, when the 999999 serial numbers are exhausted, a new number segment will be activated. Specifically, a new cluster device is built in the distributed system, and the newly added cluster is configured in the cluster routing configuration rules of the distributed system.

[0082] Specifically, for example, in the original cluster routing configuration rule, the cluster information list is {[0-999999]:(0,1)}. Since the 999999 numbers are about to be exhausted, the new clusters are numbered 2 and 3. The cluster information list is changed to {[0-999999]:(0,1), [1000000-1999999]:(2,3)}, indicating the addition of two new subclusters, 2 and 3, while the original clusters 0 and 1 remain unchanged.

[0083] Furthermore, the serial numbers in the serial number segment intervals of the original cluster and the newly added cluster are continuous and monotonically increasing numbers. For example, the original clusters are cluster 0 and cluster 1, and the corresponding serial number segment intervals are [0-999999]. Then the serial number segment interval corresponding to the newly added cluster should be continuous with the serial number segment interval of the original cluster, which can be [1000000-1999999].

[0084] The hash value module 20 is configured to receive an external data request and determine a hash value corresponding to a business field in the external data request using a hash algorithm.

[0085] Wherein, an external data request is received, and according to the business field in the external data request, for example, the business field is A. The business field A is used as the cluster sharding field, and the hash value hash(A) of the business field is calculated.

[0086] The first routing module 30 is configured to determine the cluster number of the original cluster corresponding to the service field according to the hash value if the amount of cluster data in the original cluster is less than the number of sequence numbers in its sequence number segment, and route the cluster data corresponding to the service field to the corresponding original cluster.

[0087] Among them, if the number of cluster data in the original cluster is less than the number of sequence numbers in its sequence number segment, it means that the sequence numbers in the sequence number segment of the original cluster have not been exhausted, then calculate the remainder value of the hash value and the number of original clusters, that is, mod(hash(A),2).

[0088] Furthermore, the remainder value is compared with the cluster information list in the cluster routing configuration rule. For example, if the remainder value is 0, the cluster data corresponding to the external data request is routed to cluster 0.

[0089] The second routing module 40 is configured to determine the cluster number of the newly added cluster corresponding to the business field according to the hash value if the amount of cluster data in the original cluster is not less than the number of sequence numbers in its sequence number segment, and route the cluster data corresponding to the business field to the corresponding newly added cluster.

[0090] If the number of cluster data in the original cluster is not less than the number of sequence numbers in its sequence number range, the sequence numbers in the original cluster's sequence number range have been exhausted. For example, when the original cluster's sequence numbers 0-999999 are exhausted, the new cluster's sequence numbers are activated. The routing calculation method for external data requests for the newly added cluster is similar to the above method, calculating the remainder of the hash value and the number of newly added clusters, that is, using mod(hash(A),2) to calculate the remainder.

[0091] Furthermore, the remainder value is compared with the cluster information list in the cluster routing configuration rule. For example, if the remainder value is 0, then sequence number 0 corresponds to cluster 2, and the cluster data corresponding to the external data request is routed to cluster 2.

[0092] As an embodiment of the present invention, the newly added cluster module is further configured to assign a cluster number to the newly added cluster corresponding to the newly built cluster device in the distributed system, and to turn off the service switch of the newly added cluster.

[0093] As an embodiment of the present invention, the newly added cluster module is further used to add the serial number segment interval corresponding to the newly added cluster to the cluster information list of the cluster routing configuration rule according to the cluster number of the newly added cluster and the serial number segment interval of the original cluster, and turn on the service switch of the newly added cluster.

[0094] As an embodiment of the present invention, the first routing module is further configured to compare the hash value with the cluster number of the original cluster to determine the cluster number of the original cluster corresponding to the service field.

[0095] Based on the same application concept as the aforementioned distributed system cluster online sharding expansion method, the present invention also provides the aforementioned distributed system cluster online sharding expansion device. Since the principles of solving the problem of the distributed system cluster online sharding expansion device are similar to those of the distributed system cluster online sharding expansion method, the implementation of the distributed system cluster online sharding expansion device can be referenced to the implementation of the distributed system cluster online sharding expansion method, and the repeated parts will not be repeated here.

[0096] The present invention realizes shard expansion in a distributed system by gradually increasing clusters based on monotonically increasing hash elements, avoiding large-scale data migration during cluster expansion, requiring almost no change time, reducing the impact on the business system, and saving resource costs.

[0097] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method when executing the program.

[0098] The present invention also provides a computer-readable storage medium storing a computer program for executing the above method.

[0099] like Figure 7 As shown, the electronic device 600 may further include: a communication module 110, an input unit 120, an audio processing unit 130, a display 160, and a power supply 170. It is worth noting that the electronic device 600 does not necessarily have to include Figure 7 In addition, the electronic device 600 may also include all components shown in Figure 7 For components not shown, reference may be made to the prior art.

[0100] like Figure 7 As shown, the central processing unit 100 is sometimes also referred to as a controller or an operation control unit, and may include a microprocessor or other processor device and / or logic device. The central processing unit 100 receives inputs and controls the operations of various components of the electronic device 600 .

[0101] Memory 140 may be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It may store the aforementioned failure-related information and may also store programs that execute the relevant information. The CPU 100 may execute the programs stored in memory 140 to implement information storage or processing.

[0102] The input unit 120 provides input to the CPU 100. The input unit 120 may be, for example, a keypad or touch input device. The power supply 170 is used to provide power to the electronic device 600. The display 160 is used to display objects such as images and text. The display may be, for example, an LCD display, but is not limited thereto.

[0103] The memory 140 may be a solid-state memory, such as a read-only memory (ROM), a random access memory (RAM), or a SIM card. Alternatively, it may be a memory that retains information even when power is off, can be selectively erased, and is provided with more data. Examples of such memory are sometimes referred to as EPROMs. The memory 140 may also be some other type of device. The memory 140 includes a buffer memory 141 (sometimes referred to as a buffer). The memory 140 may include an application / function storage unit 142 for storing application programs and function programs or processes for executing the operations of the electronic device 600 via the central processing unit 100.

[0104] The memory 140 may also include a data storage unit 143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 144 of the memory 140 may include various driver programs for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0105] The communication module 110 is a transmitter / receiver 110 that transmits and receives signals via an antenna 111. The communication module (transmitter / receiver) 110 is coupled to the central processor 100 to provide input signals and receive output signals, which may be the same as in a conventional mobile communication terminal.

[0106] Based on different communication technologies, multiple communication modules 110 may be provided in the same electronic device, such as a cellular network module, a Bluetooth module, and / or a wireless local area network module. The communication module (transmitter / receiver) 110 is also coupled to a speaker 131 and a microphone 132 via an audio processor 130 to provide audio output via the speaker 131 and receive audio input from the microphone 132, thereby implementing common telecommunication functions. The audio processor 130 may include any suitable buffer, decoder, amplifier, etc. Furthermore, the audio processor 130 is also coupled to the central processing unit 100, enabling local recording via the microphone 132 and playback of stored audio via the speaker 131.

[0107] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0108] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0109] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0110] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0111] Specific embodiments are used in the present invention to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A distributed system cluster online sharding expansion method, characterized in that: The method comprises: Assign a cluster number to the newly added cluster, and set a sequence number segment corresponding to the newly added cluster in the cluster routing configuration rule of the distributed system; wherein the cluster routing configuration rule includes the cluster number and sequence number segment of the original cluster, and the sequence numbers in the sequence number segment of the original cluster and the newly added cluster are continuous and monotonically increasing numbers; Receive an external data request and use a hash algorithm to determine a hash value corresponding to a business field in the external data request; If the number of cluster data in the original cluster is less than the number of sequence numbers in its sequence number segment, the cluster number of the original cluster corresponding to the business field is determined according to the hash value, and the cluster data corresponding to the business field is routed to the corresponding original cluster; If the number of cluster data in the original cluster is not less than the number of sequence numbers in its sequence number segment, the cluster number of the newly added cluster corresponding to the business field is determined according to the hash value, and the cluster data corresponding to the business field is routed to the corresponding newly added cluster; Determining the cluster number of the original cluster corresponding to the service field according to the hash value includes: Comparing the hash value with the cluster number of the original cluster to determine the cluster number of the original cluster corresponding to the service field; The remainder of the hash value and the original number of clusters is calculated and compared with the cluster information list in the cluster routing configuration rule. The hash element field has a monotonically increasing characteristic, and cluster data is developed around this element. The sequence number segment of each cluster hash element is specified, and the number of clusters is gradually increased. If the original pre-allocated cluster is not under much pressure, add sequence number segments to the original cluster so that multiple sequence number segments can be matched when calculating shards, and the cluster's data processing can be dynamically adjusted.

2. The method according to claim 1, characterized in that The cluster numbering of the newly added clusters includes: The newly added cluster corresponding to the newly built cluster device in the distributed system is assigned a cluster number, and the service switch of the newly added cluster is turned off.

3. The method according to claim 2, characterized in that The step of setting the sequence number segment corresponding to the newly added cluster in the cluster routing configuration rule of the distributed system includes: According to the cluster number of the newly added cluster and the sequence number segment of the original cluster, the sequence number segment corresponding to the newly added cluster is added to the cluster information list of the cluster routing configuration rule, and the service switch of the newly added cluster is turned on.

4. A distributed system cluster online sharding expansion device, characterized in that: The device comprises: A new cluster module is used to assign cluster numbers to newly added clusters and set the sequence number segment interval corresponding to the newly added cluster in the cluster routing configuration rules of the distributed system; wherein the cluster routing configuration rules include the cluster number and sequence number segment interval of the original cluster, and the sequence numbers in the sequence number segment intervals of the original cluster and the newly added cluster are continuous and monotonically increasing numbers; A hash value module is used to receive an external data request and determine a hash value corresponding to a business field in the external data request using a hash algorithm; A first routing module is configured to determine the cluster number of the original cluster corresponding to the service field according to the hash value if the amount of cluster data in the original cluster is less than the number of sequence numbers in its sequence number segment, and route the cluster data corresponding to the service field to the corresponding original cluster; A second routing module is configured to determine the cluster number of the newly added cluster corresponding to the business field according to the hash value if the amount of cluster data in the original cluster is not less than the number of sequence numbers in its sequence number segment, and route the cluster data corresponding to the business field to the corresponding newly added cluster; The first routing module is further configured to compare the hash value with the cluster number of the original cluster to determine the cluster number of the original cluster corresponding to the service field; The remainder of the hash value and the original number of clusters is calculated and compared with the cluster information list in the cluster routing configuration rule. The hash element field has a monotonically increasing characteristic, and cluster data is developed around this element. The sequence number segment of each cluster hash element is specified, and the number of clusters is gradually increased. If the original pre-allocated cluster is not under much pressure, add sequence number segments to the original cluster so that multiple sequence number segments can be matched when calculating shards, and the cluster's data processing can be dynamically adjusted.

5. The device according to claim 4, characterized in that The newly added cluster module is further configured to assign cluster numbers to newly added clusters corresponding to the newly built cluster devices in the distributed system, and to turn off the service switches of the newly added clusters.

6. The device according to claim 5, characterized in that The new cluster module is further configured to add the sequence number segment corresponding to the new cluster to the cluster information list of the cluster routing configuration rule according to the cluster number of the new cluster and the sequence number segment of the original cluster, and turn on the service switch of the new cluster.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 3 is implemented.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program for executing the method according to any one of claims 1 to 3.

Images