Flexible data partition routing method for complex service scenario

Through the shard hash algorithm and dynamic adjustment of data slices, the problems of uneven data and high relocation overhead in complex business scenarios are solved, data balance and scalability are achieved, and data relocation costs are reduced.

WO2024239357A1PCT designated stage expired Publication Date: 2024-11-28SHENZHEN INST OF COMPUTING SCI

Patent Information

Application Number
PCT/CN2023/097129
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-19
Filing Date
2023-05-30
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

The existing data partition routing methods have problems of data inequality and high relocation overhead in complex business scenarios, which are difficult to effectively solve the needs of data balance and scalability.

Method used

The shard hash algorithm is used to generate data slices to generate data slices, and dynamically adjust the number and allocation of data slices according to the data expansion expectations and load size, and flexibly allocate them to the storage nodes to avoid clockwise handover of shards, and support the data slices during expansion. move.

Benefits of technology

It effectively solves the problems of data inequality and high relocation overhead, improves data balance and scalability, and reduces the cost of data relocation in complex business scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2023097129_28112024_PF_FP_ABST
    Figure CN2023097129_28112024_PF_FP_ABST
Patent Text Reader

Abstract

Provided in the present application is a flexible data partition routing method for a complex service scenario. The method is applied to a distributed relational database, wherein the distributed relational database comprises several server nodes, and the server nodes are connected to each other. The method comprises: when data is acquired, generating data slices from data shards by means of a sharding Hash algorithm, wherein the number of data slices is determined according to an expansion expectation with regard to the data; allocating the data slices to corresponding data nodes according to a data space where the data is located; and when capacity expansion occurs, determining the number of relocated data slices according to a data volume and a load size. By means of sharding data into data slices, the limitation in terms of data balance in a complex service scenario is solved, thereby effectively avoiding the problem of data being unbalanced.
Need to check novelty before this filing date? Find Prior Art

Description

A flexible data partition routing method for complex business scenarios Technical Field

[0001] The present application relates to the technical field of database data routing, and in particular to a flexible data partition routing method for complex business scenarios. Background Art

[0002] As business complexity increases, the demand for database scalability is also increasing. Distributed databases achieve horizontal expansion by distributing data across different storage nodes, but how to distribute data in a balanced manner while minimizing the cost of relocation during expansion becomes a challenge.

[0003] Many distributed database systems implement data routing by partitioning data based on the value range of specific fields. However, this can lead to uneven data distribution, necessitating further partitioning. Some systems also use a consistent hashing algorithm (DHT) to generate data routes to minimize data migration, but this requires the use of virtual nodes to avoid data skew. Consequently, existing data partitioning and routing methods suffer from high migration overhead and data imbalance.

[0004] Summary of the Invention

[0005] In view of the above problems, the present application is proposed to provide a flexible data partition routing method for complex business scenarios that overcomes the above problems or at least partially solves the above problems, including:

[0006] A flexible data partition routing method for complex business scenarios, the method being applied to a distributed relational database; wherein the distributed relational database includes a plurality of server nodes, the server nodes being interconnected; the method comprising:

[0007] When data is acquired, the data is fragmented into data slices using a fragmentation hash algorithm; wherein the number of data slices is determined based on the expected expansion of the data;

[0008] Allocating the data slices to corresponding data nodes according to the data space where the data is located;

[0009] When capacity expansion occurs, the number of data slices to be relocated is determined based on the amount of data and the load size.

[0010] Furthermore, the step of generating data slices by sharding the data using a sharding hash algorithm includes:

[0011] Convert the data into a hash ring and divide it into data slices of the same size;

[0012] determining a distribution key for the data;

[0013] Mapping the value of the distribution key to a hash value through a sharding hash algorithm;

[0014] The shard ID is obtained by taking the modulus of the number of the data slices in the data space by the hash value.

[0015] The step of allocating the data slices to corresponding data nodes according to the data space where the data is located includes:

[0016] Determine the data node corresponding to the data slice according to the data space where the data is located;

[0017] Allocate the data slices to storage nodes within the data nodes.

[0018] Furthermore, the step of allocating the data slices to storage nodes within the data nodes includes:

[0019] Generate a storage node and obtain the ID of the storage node;

[0020] The data slices are allocated according to the order of the ids.

[0021] Furthermore, when capacity expansion occurs, the step of determining the number of data slices to be relocated based on the data volume and load size includes:

[0022] The data slices are allocated to the storage nodes according to the number of the data slices and the number of the storage nodes.

[0023] Furthermore, the method further comprises:

[0024] When an external query is intercepted, the communication node generates an access plan and determines participating storage nodes.

[0025] Furthermore, the method further comprises:

[0026] Manage data routing through mobile nodes.

[0027] A flexible data partition routing device for complex business scenarios, comprising:

[0028] A data sharding module is used to, when acquiring data, shard the data into data slices using a sharding hash algorithm; wherein the number of the data slices is determined according to the attributes of the data;

[0029] A data allocation module, configured to allocate the data slices to corresponding data nodes according to the data space where the data is located;

[0030] The capacity expansion partition module is used to determine the number of data slices to be relocated according to the data volume and load size when capacity expansion occurs.

[0031] A computer device comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the steps of the data partition routing method described above when executed by the processor.

[0032] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the data partition routing method described above.

[0033] This application has the following advantages:

[0034] In the embodiments of the present application, in response to the problem of data imbalance in the prior art, the present application provides a solution for static allocation of data partitions and dynamic adjustment of capacity expansion, specifically: when data is acquired, the data is partitioned into data slices through a sharding hash algorithm; wherein, the number of data slices is determined according to the expected expansion of the data; the data slices are allocated to corresponding data nodes according to the data space where the data is located; when capacity expansion occurs, the number of data slices to be relocated is determined based on the amount of data and the load size. By sharding the data into data slices, when a node exits, it does not need to hand over the slice it is responsible for clockwise to the subsequent node, and the entire slice can be handed over to any node more flexibly, solving the limitations of data balance in complex business scenarios and effectively avoiding the problem of data imbalance. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for the description of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0036] FIG1 is a flowchart of a flexible data partition routing method for complex business scenarios provided by an embodiment of the present application;

[0037] FIG2 is a flowchart of data allocation steps when the number of data slices is 10 according to an embodiment of the present application;

[0038] FIG3 is a flowchart of the steps for expanding a partition when the number of data slices provided by an embodiment of the present application is 10;

[0039] FIG4 is a flowchart of the steps for expanding a partition when the number of data slices provided by an embodiment of the present application is 100;

[0040] FIG5 is a flowchart of a flexible data partition routing network for complex business scenarios provided by an embodiment of the present application;

[0041] FIG6 is a structural block diagram of a flexible data partition routing device for complex business scenarios provided by an embodiment of the present application;

[0042] FIG7 is a schematic structural diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] To make the objectives, features, and advantages of this application more readily apparent, the present application is further described below in conjunction with the accompanying drawings and specific embodiments. It is apparent that the embodiments described are only a portion of the embodiments of this application, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments in this application without inventive effort are also within the scope of protection of this application.

[0044] Through analysis of existing technologies, the inventors discovered that many distributed database systems implement data routing by partitioning based on the value range of specific fields. However, when actual business changes occur, this can lead to uneven data distribution, necessitating further partitioning. Some systems also use the consistent hashing algorithm (DHT) to generate data routes to minimize data migration, but this requires the use of virtual nodes to avoid data skew. Therefore, existing data partitioning routing methods suffer from high migration overhead and data imbalance, presenting limitations in achieving data balance and mitigating data overhead in complex business scenarios.

[0045] 1 , a flexible data partition routing method for complex business scenarios provided by an embodiment of the present application is shown. The method is applied to a distributed relational database; wherein the distributed relational database includes a plurality of server nodes, and the server nodes are interconnected. The method includes:

[0046] S110: When data is acquired, the data is fragmented into data slices using a fragmentation hash algorithm; wherein the number of the data slices is determined based on the expected expansion of the data;

[0047] S120, allocating the data slice to a corresponding data node according to the data space where the data is located;

[0048] S130: When capacity expansion occurs, the number of data slices to be relocated is determined based on the data volume and load size.

[0049] In the embodiments of the present application, in response to the problem of data imbalance in the prior art, the present application provides a solution for static allocation of data partitions and dynamic adjustment of capacity expansion, specifically: when data is acquired, the data is partitioned into data slices through a sharding hash algorithm; wherein, the number of data slices is determined according to the expected expansion of the data; the data slices are allocated to corresponding data nodes according to the data space where the data is located; when capacity expansion occurs, the number of data slices to be relocated is determined based on the amount of data and the load size. By sharding the data into data slices, when a node exits, it does not need to hand over the slice it is responsible for clockwise to the subsequent node, and the entire slice can be handed over to any node more flexibly, solving the limitations of data balance in complex business scenarios and effectively avoiding the problem of data imbalance.

[0050] Next, a flexible data partition routing method for complex business scenarios in this exemplary embodiment will be further described.

[0051] It should be noted that this application addresses the problem of high migration overhead in the existing technology. By sharding the data into data slices, when a node exits, it does not need to hand over the slice it is responsible for clockwise to the subsequent node. The entire slice can be handed over to any node more flexibly. It also solves the limitations of data migration overhead in complex business scenarios and effectively reduces the problem of migration overhead.

[0052] It should be noted that the present application is applied to a distributed relational database, which is used to provide transparent data services to clients; wherein the distributed relational database includes a number of server nodes, which are interconnected.

[0053] As described in the above step S110, when the data is acquired, the data is fragmented into data slices using a fragmentation hash algorithm.

[0054] In one embodiment of the present invention, the specific process of "sharding the data to generate data slices by using a shard hash algorithm" in step S110 can be further explained in combination with the following description.

[0055] As described in the following steps, the data is converted into a hash ring and divided into data slices of equal size;

[0056] Determining a distribution key for the data as described in the following steps;

[0057] As described in the following steps, the value of the distribution key is mapped to a hash value through a shard hash algorithm;

[0058] As described in the following steps, the number of the data slices in the data space is modulo the hash value to obtain the slice ID.

[0059] As an example, the data is converted into a Hash ring through a hash algorithm, and the Hash ring is divided into several data slices of the same size, wherein the number of data slices can be determined based on the expected expansion of the data. The distribution key of the data is determined. Specifically, the distribution key is usually manually specified, and generally takes the column with the most dispersed data, such as the order number of e-commerce data, so that the data can be distributed as evenly as possible on all data slices. The hash value corresponding to the value of the above distribution key is calculated by the hash algorithm. Since the range of the hash value is much larger than the number of data slices, it is also necessary to modulo the hash value with the number of data slices. The calculation result is the data slice id ([0, number of data slices - 1]), thereby determining a unique slice for each piece of data.

[0060] It should be noted that in this application, a data slice is the smallest unit for data migration, fault recovery, and replica replication. The granularity of the data slice sharding determines query efficiency and management costs, so the number of data slices should be determined based on the user's business expansion needs. At the same time, the number of data slice shards also determines the number of storage nodes supported by the database.

[0061] In a specific implementation, when the number of data slices is set to 10 in this embodiment, the database supports a maximum of 10 storage nodes.

[0062] In another specific implementation, when the number of data slices is set to 100 in this embodiment, the database supports a maximum of 100 storage nodes.

[0063] As described in the above step S120, the data slices are allocated to corresponding data nodes according to the data space where the data is located.

[0064] In an embodiment of the present invention, the specific process of "allocating the data slices to corresponding data nodes according to the data space where the data is located" in step S120 can be further explained in combination with the following description.

[0065] As described in the following steps, the data node corresponding to the data slice is determined according to the data space where the data is located;

[0066] As described in the following steps, the data slices are allocated to storage nodes within the data nodes.

[0067] As an example, obtain the data space where the data is located and the data slice generated by the data, determine the corresponding data node based on the data space, obtain several storage nodes within the data node, and allocate the data slice to the storage node; wherein, the number of the storage nodes is no less than 2.

[0068] In one embodiment of the present invention, the specific process of "determining the data node corresponding to the data slice according to the data space where the data is located" can be further explained in combination with the following description.

[0069] As described in the following steps, a storage node is generated and the ID of the storage node is obtained;

[0070] As described in the following steps, the data slices are allocated according to the order of the IDs.

[0071] As an example, determine the number of data slices, generate several DN Groups as storage nodes, and obtain the ids of the storage nodes, wherein the number of storage nodes can be determined according to the size of the data to be stored, wherein the storage node needs to store at least one data slice, and therefore the number of storage nodes cannot exceed the number of data slices. Divide the number of data slices by the number of storage nodes and round down to find the number of data slices that must be allocated to each storage node at least. Divide the number of data slices by the number of storage nodes and take the remainder to determine the number of storage nodes that need to be allocated an additional data slice. First, allocate the data slice to each storage node according to the number of data slices that must be allocated to each storage node at least, and then allocate an additional data slice to the storage nodes in ascending order of the ids of the storage nodes until all the data slices are allocated.

[0072] In a specific implementation, referring to Figure 2 , this embodiment generates three DN groups as storage nodes and obtains the IDs of the storage nodes. In this embodiment, the number of data slices is 10. Calculation shows that each storage node must be allocated at least two data slices, and one storage node requires an additional data slice. First, two data slices are allocated to each storage node. Then, in ascending order of the storage node IDs, the additional data slices are allocated to the storage node with the smallest ID.

[0073] As described in the above step S130, when capacity expansion occurs, the number of data slices to be relocated is determined based on the data volume and load size.

[0074] In one embodiment of the present invention, the specific process of "determining the number of data slices to be relocated according to the data volume and load size when capacity expansion occurs" in step S130 can be further explained in combination with the following description.

[0075] As described in the following steps, the data slices are allocated to the storage nodes according to the number of the data slices and the number of the storage nodes.

[0076] As an example, when expansion occurs, the number of data slices and the number of storage nodes before expansion are obtained, and the number of storage nodes to be allocated is calculated based on the number of storage nodes before expansion and the number of nodes after expansion. The number of storage nodes to be allocated cannot exceed the number of data slices. Divide the number of data slices by the number of storage nodes and round down to obtain the number of data slices to be allocated to each storage node at least. Divide the number of data slices by the number of storage nodes and take the remainder to determine how many storage nodes need to be allocated an additional data slice. First, allocate the data slice to each storage node according to the number of data slices to be allocated to each storage node at least, and then allocate additional data slices to the storage node.

[0077] It should be noted that when allocating additional data slices, the greater the number of data slices originally stored by the storage node, the higher the priority. However, each storage node can only be allocated one additional data slice at most. For storage nodes that originally stored the same number of data slices, additional data slices are allocated in ascending order of storage node IDs.

[0078] In a specific implementation, referring to FIG3 , the number of data slices in this embodiment is 10, the number of the storage nodes before expansion is 3, and the number of nodes after expansion is 1. It can be calculated that each storage node must be allocated at least 2 data slices, and 2 storage nodes need to be allocated an additional data slice. Among them, 1 storage node originally stored 4 data slices (Class A), 2 storage nodes originally stored 3 data slices (Class B), and 1 newly added storage node originally did not store any data slices (Class C). Then, the storage nodes of Class A will give priority to allocating 1 additional data slice, and the 2 storage nodes of Class B will allocate the remaining 1 additional data slice to the storage node with a smaller ID according to the priority order of the storage node ID from small to large.

[0079] In a specific implementation, referring to FIG4 , the number of data slices in this embodiment is 100, the number of storage nodes before expansion is 1, and the number of nodes after expansion is 9. It can be calculated that at least 10 data slices must be allocated to each storage node. After expansion is completed, each storage node stores 10 data slices. Expansion is performed again. At this time, the number of storage nodes before expansion is 10, and the number of nodes after expansion is 10. It can be calculated that at least 5 data slices must be allocated to each storage node.

[0080] In one embodiment of the present invention, the method further includes:

[0081] As described in the following steps, when an external query is intercepted, the communication node generates an access plan and determines participating storage nodes.

[0082] As an example, when an external query is monitored, the data space to which the queried data table belongs is determined based on the metadata, wherein each data space maintains a routing table, which contains the correspondence between the data slices and data nodes under the corresponding data space (the data slices must exist on a data node). If the query does not come with a filtering condition, all data nodes under the data space are queried. If the query specifies a distribution key value condition, the optimizer will calculate the corresponding data slice id based on the specified distribution key value, and then query the specific data node based on the data slice id, thereby achieving distributed pruning.

[0083] In one embodiment of the present invention, the method further includes:

[0084] Data routing is managed through the mobile node as described in the following steps.

[0085] As an example, this embodiment manages data routing through mobile nodes. Specifically, the data slice where the data is located is first located through Hash, and then the storage node information corresponding to the data slice is found.

[0086] 5 , a flexible data partition routing network structure for complex business scenarios provided by an embodiment of the present application is shown, including: a coordination group, a DTF (Digital Transmission Facility, digital transmission equipment) and several computing groups;

[0087] The coordination group includes a communication node and a plurality of listening nodes; the computing group includes a plurality of storage nodes, and the storage nodes are composed of a plurality of processor nodes.

[0088] As an example, the listening node is used to listen to external queries.

[0089] As an example, one computing group corresponds to one data space.

[0090] As an example, the storage node implements load sharing through several internal processor nodes.

[0091] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0092] 6 , a flexible data partition routing device for complex business scenarios is shown in accordance with an embodiment of the present application;

[0093] Specifically include:

[0094] The data sharding module 610 is configured to, when acquiring data, shard the data into data slices using a sharding hash algorithm; wherein the number of the data slices is determined according to the attributes of the data;

[0095] A data allocation module 620 is configured to allocate the data slices to corresponding data nodes according to the data space where the data is located;

[0096] The capacity expansion partition module 630 is used to determine the number of data slices to be relocated according to the data volume and load size when capacity expansion occurs.

[0097] In one embodiment of the present invention, the data sharding module 610 includes:

[0098] A ring cutting submodule, used to convert the data into a hash ring and cut it into data slices of the same size;

[0099] a distribution key determination module, configured to determine a distribution key for the data;

[0100] A hash mapping submodule, configured to map the value of the distribution key to a hash value using a sharding hash algorithm;

[0101] The shard ID acquisition submodule is used to obtain the shard ID by taking the modulus of the number of the data slices in the data space through the hash value.

[0102] In one embodiment of the present invention, the data distribution module 620 includes:

[0103] A node determination submodule, configured to determine a data node corresponding to the data slice according to the data space where the data is located;

[0104] The storage submodule is configured to allocate the data slices to storage nodes within the data nodes.

[0105] In one embodiment of the present invention, the storage submodule includes:

[0106] A node generation unit, configured to generate a storage node and obtain an ID of the storage node;

[0107] A data sharding unit is used to allocate the data slices according to the order of the IDs.

[0108] In one embodiment of the present invention, the capacity expansion partition module 630 includes:

[0109] The expansion submodule is used to allocate the data slices to the storage nodes according to the number of the data slices and the number of the storage nodes.

[0110] In one embodiment of the present invention, the apparatus further comprises:

[0111] a communication module, configured to generate an access plan and determine participating storage nodes when an external query is monitored;

[0112] The routing module is used to manage data routing through mobile nodes.

[0113] 7 , a computer device of a flexible data partition routing method and apparatus for complex business scenarios of the present invention is shown, which may specifically include the following:

[0114] The computer device 12 is a general-purpose computing device. The components of the computer device 12 may include but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 connecting different system components (including the system memory 28 and the processing unit 16).

[0115] The bus 18 represents one or more of several types of bus 18 structures, including a memory bus 18 or memory controller, a peripheral bus 18, an accelerated graphics port, a processor, or a local bus 18 that utilizes any of a variety of bus 18 architectures. Examples of such architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus 18, a Micro Channel Architecture (MAC) bus 18, an Enhanced ISA bus 18, an Audio Video Electronics Standards Association (VESA) local bus 18, and a Peripheral Component Interconnect (PCI) bus 18.

[0116] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0117] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be configured to read and write to non-removable, non-volatile magnetic media (commonly referred to as a "hard drive"). Although not shown in FIG. 7 , a magnetic disk drive may be provided for reading and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), as well as an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media). In these cases, each drive may be connected to bus 18 via one or more data media interfaces. The memory may include at least one program product having a set (e.g., at least one) of program modules 42 configured to perform the functions of various embodiments of the present invention.

[0118] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in a memory. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules 42, and program data, each of which, or some combination thereof, may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0119] The computer device 12 can also communicate with one or more external devices 14 (e.g., a keyboard, pointing device, display 24, camera, etc.), one or more devices that enable an operator to interact with the computer device 12, and / or any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., a network card, modem, etc.). Such communication can occur via an input / output (I / O) interface 22. Furthermore, the computer device 12 can communicate with one or more networks (e.g., a local area network (LAN)), a wide area network (WAN), and / or a public network (e.g., the Internet) via a network adapter 20. As shown in FIG7 , the network adapter 20 communicates with the other modules of the computer device 12 via a bus 18. It should be understood that, although not shown in FIG7 , other hardware and / or software modules can be used in conjunction with the computer device 12, including, but not limited to, microcode, device drivers, redundant processing units 16, external disk drive arrays, RAID systems, tape drives, and data backup storage systems 34.

[0120] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing a flexible data partition routing method for complex business scenarios provided by an embodiment of the present invention.

[0121] That is, when the processing unit 16 executes the above program, it realizes: when data is obtained, the data is sharded into data slices through the shard hash algorithm; wherein, the number of the data slices is determined according to the expected expansion of the data; the data slices are allocated to the corresponding data nodes according to the data space where the data is located; when expansion occurs, the number of the data slices to be relocated is determined according to the amount of data and the load size.

[0122] In an embodiment of the present invention, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a flexible data partition routing method for complex business scenarios as provided in all embodiments of the present application:

[0123] That is, when the program is executed by the processor, it is implemented as follows: when data is obtained, the data is fragmented into data slices through the fragmentation hash algorithm; wherein, the number of the data slices is determined according to the expected expansion of the data; the data slices are allocated to the corresponding data nodes according to the data space where the data is located; when expansion occurs, the number of the data slices to be relocated is determined according to the amount of data and the load size.

[0124] Any combination of one or more computer-readable media may be employed. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0125] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0126] The computer program code for performing the operations of the present invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the operator's computer, partially on the operator's computer, as a separate software package, partially on the operator's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the operator's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to connect through the Internet). The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referenced to each other.

[0127] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the underlying inventive concepts. Therefore, the appended claims are intended to include the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0128] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0129] The above is a detailed introduction to a flexible data partition routing method for complex business scenarios provided by this application. This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method of this application and its core idea; at the same time, for general technical personnel in this field, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A flexible data partition routing method for complex business scenarios, the method is applied to a distributed relational database; wherein: The distributed relational database includes a plurality of server nodes, and the server nodes are connected to each other; the method includes: When the data is acquired, the data is sharded into data slices by using a sharding hash algorithm; wherein the number of the data slices is determined according to the expected expansion of the data; Allocating the data slices to corresponding data nodes according to the data space where the data is located; When capacity expansion occurs, the number of data slices to be relocated is determined based on the amount of data and the load size.

2. The method according to claim 1, characterized in that: The step of generating data slices by sharding the data using a sharding hash algorithm comprises: Convert the data into a hash ring and divide it into data pieces of the same size; determining a distribution key for the data; Mapping the value of the distribution key to a hash value through a sharding hash algorithm; The shard ID is obtained by taking the modulus of the number of the data slices in the data space through the hash value.

3. The method according to claim 1, characterized in that The step of allocating the data slices to corresponding data nodes according to the data space where the data is located includes: Determine the data node corresponding to the data slice according to the data space where the data is located; The data slices are allocated to storage nodes within the data nodes.

4. The method according to claim 3, characterized in that The step of allocating the data slices to storage nodes within the data nodes comprises: Generate a storage node and obtain the ID of the storage node; The data slices are allocated according to the order of the ids.

5. The method according to claim 1, characterized in that The step of determining the number of data slices to be relocated according to the amount of data and the load size when capacity expansion occurs includes: The data slices are allocated to the storage nodes according to the number of the data slices and the number of the storage nodes.

6. The method according to claim 1, characterized in that The method further comprises: When an external query is intercepted, the communication node generates an access plan and determines the participating storage nodes. point.

7. The method according to claim 1, characterized in that The method further comprises: Manage data routing through mobile nodes.

8. A flexible data partition routing device for complex business scenarios, characterized in that: The device comprises: A data sharding module, used for sharding the data into data slices by using a sharding hash algorithm when acquiring the data; wherein the number of the data slices is determined according to the attributes of the data; A data allocation module, used to allocate the data slices to corresponding data nodes according to the data space where the data is located; The capacity expansion partition module is used to determine the number of data slices to be relocated according to the amount of data and the load size when capacity expansion occurs.

9. A computer device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the method according to any one of claims 1 to 7 when executed by the processor.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Expanding and shrinking method for distributed cluster with nodes supporting multiple data fragments

    CN104615657A

  • Capacity expansion method and device

    CN106843745A

  • Data migration method and device of distributed database cluster

    CN106844510A

  • Data writing and expansion method and device of distributed caching system

    CN108810041A

  • Data migration method, device and system and computer readable storage medium

    CN113761288A

Cited By

  • Cloud cluster automatic expansion method and system, and storage medium

    CN121644573A