Data fragment conversion method, device and equipment for distributed database

Abstract

The invention provides a data fragment conversion method, device and equipment for a distributed database, which can be applied to the technical field of databases. The method comprises the following steps: in response to a fragmented transaction request, respectively creating target business data empty tables on I data nodes included in a distributed database; distributing data in the initial business data table to a plurality of threads in a balanced manner to obtain target data corresponding to the plurality of threads; determining a to-be-stored data node corresponding to the target data based on a fragmentation rule of the target business data empty table; based on the first mapping relation and the to-be-stored data node, storing the target data on the thread of the ith data node into a target storage pool, corresponding to the to-be-stored data node, on the ith data node; and under the condition that the data volume in the target storage pool is equal to a preset quantity threshold value, packaging the target data in the target storage pool and storing the target data in a target business data empty table on the to-be-stored data node.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a data sharding conversion method, device and equipment for a distributed database. Background Art

[0002] In a distributed transaction database, the data distribution of list sharding is relatively intuitive, and the sharding rules can be flexibly customized according to the business, which is convenient for querying. However, since the sharding is customized, it may cause data skew. Hash sharding distributes data to different data nodes, which can evenly distribute data between data nodes, and then directly use the corresponding threads on the data nodes to pull data. It is impossible to ensure the data load balance between multiple threads on the same data node while ensuring the uniform distribution of data nodes. The problem of load imbalance between multiple threads on the same data node reduces data transmission efficiency and high concurrency performance. Summary of the invention

[0003] In view of the above problems, the present invention provides a method, device and equipment for data shard conversion in a distributed database.

[0004] According to a first aspect of the present invention, a data sharding conversion method for a distributed database is provided, comprising: in response to a sharding transaction request for an initial business data table, creating a target business data empty table on I data nodes included in the distributed database, respectively, wherein I is a positive integer; evenly distributing the data in the initial business data table to multiple threads, and obtaining target data corresponding to each of the multiple threads, wherein each data node comprises I storage pools and multiple threads, and there is a first mapping relationship between the i-th storage pool and the i-th data node, I≥i>1, and i is an integer; based on the sharding rule of the target business data empty table, determining the data node to be stored corresponding to the target data on each thread; based on the first mapping relationship and the data node to be stored corresponding to the target data on each thread, storing the target data on the thread of the i-th data node in the target storage pool corresponding to the data node to be stored on the i-th data node; when the amount of data in the target storage pool is equal to a preset quantity threshold, encapsulating the target data in the target storage pool and saving it to the target business data empty table on the data node to be stored.

[0005] Optionally, when the amount of data in the target storage pool is equal to a preset quantity threshold, encapsulating the target data in the target storage pool and saving it in the target business data empty table on the data node to be stored includes: when the amount of data in the target storage pool is equal to the preset quantity threshold, encapsulating the target data in the target storage pool to obtain an encapsulated data packet; storing the encapsulated data packet in a message queue corresponding to the target storage pool; and sending the encapsulated data packet in the message queue to the data node to be stored, wherein the encapsulated data packet is stored in the target business data empty table on the data node to be stored.

[0006] Optionally, based on the sharding rule of the target business data empty table, determining the data node to be stored corresponding to the target data on each thread includes: when the sharding rule of the target business data empty table is a list sharding rule, determining the data node to be stored corresponding to the target data on each thread according to the first data mapping relationship table in the list sharding rule, wherein the first data mapping relationship table represents the correspondence between the target key value corresponding to the target field in the initial business data table and the data node to be stored, and the target data is associated with the target key value.

[0007] Optionally, based on the sharding rule of the empty table of target business data, determining the data nodes to be stored corresponding to the target data on each thread also includes: when the sharding rule of the empty table of target business data is a hash sharding rule, using a hash algorithm to process multiple key values ​​of the target field to obtain hash values ​​corresponding to each of the multiple key values; taking the modulus of the hash values ​​corresponding to each of the multiple key values ​​according to the number of data nodes to determine the modulus corresponding to each of the multiple key values; determining the data nodes to be stored corresponding to the target data on each thread according to the modulus corresponding to each of the multiple key values ​​and the association relationship, wherein there is an association relationship between the target data and the modulus of the key value.

[0008] Optionally, evenly distributing the data in the initial business data table to multiple threads to obtain target data corresponding to each of the multiple threads includes: distributing the data in the initial business data table to multiple threads based on data characteristics of the first field in the initial business data table and the number of threads to obtain initial data corresponding to each of the multiple threads; and adjusting the initial data corresponding to each of the multiple threads based on a balancing strategy to obtain target data corresponding to each of the multiple threads.

[0009] Optionally, based on a balancing strategy, initial data corresponding to each of the multiple threads are adjusted to obtain target data corresponding to each of the multiple threads, including: detecting the performance of each of the multiple threads corresponding to the i-th data node; when the performance of each of the multiple threads on the i-th data node is uneven, using a hash algorithm to process the associated key value of the second field in the initial business data table to obtain hash values ​​corresponding to each of the multiple associated key values, wherein a second mapping relationship exists between the associated key values ​​and the initial data on the multiple threads; based on the multiple hash values ​​and the second mapping relationship, distributing the initial data to the multiple threads on the i-th data node to obtain target data corresponding to each of the multiple threads.

[0010] A second aspect of the present invention provides a data shard conversion device for a distributed database, comprising:

[0011] A response module, configured to respond to a sharding transaction request for an initial business data table and to create a target business data empty table on I data nodes included in the distributed database, where I is a positive integer;

[0012] An allocation module is used to evenly allocate the data in the initial business data table to multiple threads to obtain target data corresponding to each of the multiple threads, wherein each data node includes I storage pools and multiple threads, and there is a first mapping relationship between the i-th storage pool and the i-th data node, I≥i>1, and i is an integer;

[0013] A determination module, used to determine the data node to be stored corresponding to the target data on each thread based on the sharding rule of the target business data empty table;

[0014] a conversion module, configured to store the target data on the thread of the i-th data node into a target storage pool corresponding to the data node to be stored on the i-th data node based on the first mapping relationship and the data node to be stored corresponding to the target data on each thread;

[0015] The sending module is used to encapsulate the target data in the target storage pool and save it to the target business data empty table on the data node to be stored when the amount of data in the target storage pool is equal to a preset quantity threshold.

[0016] The third aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors execute the above-mentioned data sharding conversion method for a distributed database.

[0017] The fourth aspect of the present invention also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above-mentioned data sharding conversion method for a distributed database.

[0018] The fifth aspect of the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned data sharding conversion method for a distributed database.

[0019] According to the data sharding conversion method, device and equipment for distributed database provided by the present invention, the target data corresponding to each of the multiple threads is obtained by evenly distributing the data in the initial business data table to multiple threads; based on the first mapping relationship and the data node to be stored, the target data on the thread of the i-th data node is stored in the target storage pool corresponding to the data node to be stored on the i-th data node, and finally the target data is encapsulated and saved in the target business data empty table on the data node to be stored. Because the data in the initial business data table is evenly distributed to multiple threads before the target data is pulled to the data node to be stored, the workload of each thread is guaranteed to be balanced, and the problem of unbalanced load on each thread corresponding to the data node to be stored directly based on the sharding rules of the target business data empty table is solved; the target data in the target storage pool on the i-th data node is encapsulated and sent to the data node to be stored at one time, which reduces the number of data transmissions, improves the efficiency of data pulling and the flexibility of distributed storage, and realizes high-concurrency changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0021] Figure 1 The application scenario of the data shard conversion method and device for a distributed database according to an embodiment of the present invention is shown.

[0022] Figure 2 A flow chart of a data shard conversion method for a distributed database according to an embodiment of the present invention is shown.

[0023] Figure 3 A flow chart of saving target data into a target business data empty table according to an embodiment of the present invention is shown.

[0024] Figure 4 A structural block diagram of a data shard conversion device for a distributed database according to an embodiment of the present invention is shown.

[0025] Figure 5A block diagram of an electronic device suitable for implementing a data shard conversion method for a distributed database according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0026] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0027] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0028] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0029] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0030] In the process of implementing the present invention, it is found that in the distributed transaction database, the data distribution of list sharding is relatively intuitive, and the sharding rules can be flexibly customized according to the business, which is convenient for querying. However, since the sharding is customized, it may cause data skew. Hash sharding distributes data to different data nodes, which can make the data nodes evenly distributed with data, and then directly use the corresponding threads on the data nodes to pull data. It is impossible to ensure the data load balance between multiple threads on the same data node under the premise of ensuring the uniform distribution of data nodes. The problem of load imbalance between multiple threads on the same data node reduces the data transmission efficiency and high concurrency performance.

[0031] In view of this, an embodiment of the present invention provides a data sharding conversion method, device and equipment for a distributed database. The method includes: in response to a sharding transaction request, creating a target business data empty table on I data nodes included in the distributed database respectively; evenly distributing the data in the initial business data table to multiple threads to obtain target data corresponding to each of the multiple threads; determining the data node to be stored corresponding to the target data based on the sharding rule of the target business data empty table; based on the first mapping relationship and the data node to be stored, storing the target data on the thread of the i-th data node in the target storage pool corresponding to the data node to be stored on the i-th data node; when the amount of data in the target storage pool is equal to a preset quantity threshold, encapsulating the target data in the target storage pool and saving it to the target business data empty table on the data node to be stored.

[0032] In the technical solution of the present invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0033] Figure 1 The application scenario of the data shard conversion method and device for a distributed database according to an embodiment of the present invention is shown.

[0034] like Figure 1 As shown, the business system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a cluster server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the cluster server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0035] The user may use at least one of the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the cluster server 105 through the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0036] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0037] The cluster server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The cluster distributed database is deployed on the cluster server 105, and the cluster server 105 includes multiple server nodes that can analyze and process the received user request and other data, and feed back the processing results (such as web pages, information, or data obtained or generated according to the user request) to the terminal device.

[0038] It should be noted that the data shard conversion method for a distributed database provided in the embodiment of the present invention can generally be executed by the cluster server 105. Accordingly, the data shard conversion device for a distributed database provided in the embodiment of the present invention can generally be set in the cluster server 105. The data shard conversion method for a distributed database provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the cluster server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the cluster server 105. Correspondingly, the data shard conversion device for a distributed database provided in the embodiment of the present invention can also be set in a server or server cluster that is different from the cluster server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the cluster server 105.

[0039] Alternatively, the data shard conversion method for a distributed database provided in the embodiment of the present invention may also be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103, or may also be executed by other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103. Accordingly, the data shard conversion apparatus for a distributed database provided in the embodiment of the present invention may also be arranged in the first terminal device 101, the second terminal device 102, or the third terminal device 103, or may be arranged in other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103.

[0040] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0041] It should be noted that the sequence numbers of the operations in the following method are only used as representations of the operations for the purpose of description, and should not be regarded as representing the execution order of the operations. Unless explicitly stated, the method does not need to be executed completely in the order shown.

[0042] Figure 2 A flow chart of a data shard conversion method for a distributed database according to an embodiment of the present invention is shown.

[0043] like Figure 2 As shown, the method 200 includes operations S210 to S250.

[0044] In operation S210, in response to a sharding transaction request for an initial business data table, empty target business data tables are created on I data nodes included in the distributed database.

[0045] Optionally, the initial business data table includes business data generated based on business transactions. The distributed database includes a control node and I data nodes, where I is a positive integer. The sharding transaction request represents an instruction to shard the data in the initial business data table on I data nodes.

[0046] Optionally, the control node adds a read lock to the initial business data table in response to the received shard transaction request for the initial business data table, and then creates empty target business data tables on I data nodes respectively. The read lock level allows the current shard transaction request to perform a read operation on the initial business data.

[0047] Optionally, the target business data empty table is an empty data table, and the table structure of the target business data empty table is the same as that of the initial business data table.

[0048] In operation S220, the data in the initial business data table is evenly distributed to the multiple threads to obtain target data corresponding to each of the multiple threads.

[0049] Optionally, based on a load balancing algorithm, the data in the initial business data table can be evenly distributed to multiple threads to obtain target data corresponding to each of the multiple threads. A series of threads (such as fetcher threads) are started on each data node, and the multiple threads are responsible for pulling the target data corresponding to each of the multiple threads from the initial business data table in parallel.

[0050] Optionally, each data node includes multiple threads, and the fetcher thread creates a corresponding storage pool (Batch) on each data node according to the number of data nodes. The number of data nodes is I, and each data node includes I storage pools. There is a first mapping relationship between the i-th storage pool and the i-th data node, I≥i>1, and i is an integer.

[0051] For example, the distributed database includes data node A1, data node B2, and data node C3. Data node A1 includes three storage pools, namely storage pool 001A, storage pool 002A, and storage pool 003A. The first storage pool 001A has a first mapping relationship with the first data node A1; the second storage pool 002A has a first mapping relationship with the second data node B2; and the third storage pool 003A has a first mapping relationship with the third data node C3.

[0052] For example, taking the above data node A1, data node B2, and data node C3 as an example, data node B2 includes storage pool 001B, storage pool 002B, and storage pool 003B. The first storage pool 001B has a first mapping relationship with the first data node A1; the second storage pool 002B has a first mapping relationship with the second data node B2; and the third storage pool 003B has a first mapping relationship with the third data node C3. Data node C3 includes storage pool 001C, storage pool 002C, and storage pool 003C. The first storage pool 001C has a first mapping relationship with the first data node A1; the second storage pool 002C has a first mapping relationship with the second data node B2; and the third storage pool 003C has a first mapping relationship with the third data node C3.

[0053] In operation S230, based on the sharding rule of the target business data empty table, the data node to be stored corresponding to the target data on each thread is determined.

[0054] Optionally, the data node to be stored represents the data node to which the target data is to be sent. The data node to be stored corresponding to the target data on each thread is calculated based on the sharding rule of the target business data empty table.

[0055] For example, the sharding rule of the empty table according to the target business data may be a hash sharding rule, a list sharding rule, etc.

[0056] In operation S240, based on the first mapping relationship and the data nodes to be stored corresponding to the target data on each thread, the target data on the thread of the i-th data node is stored in the target storage pool corresponding to the data node to be stored on the i-th data node.

[0057] For example, the target data on a thread of the first data node A1 is "abc", and the data node to be stored corresponding to the target data "abc" is the second data node B2. Based on the first mapping relationship, the target storage pool corresponding to the second data node B2 is determined to be 002A on the first data node A1. The thread pulls the target data "abc" from the initial business data table and stores it in the target storage pool 002A on the first data node A1.

[0058] In operation S250, when the amount of data in the target storage pool is equal to a preset quantity threshold, the target data in the target storage pool is encapsulated and saved in an empty table of target business data on the data node to be stored.

[0059] Optionally, a preset quantity threshold represents the maximum amount of data that the storage pool can accommodate, and the thread continuously stores the target data in the corresponding target storage pool. When the amount of data in the target storage pool is equal to the preset quantity threshold, the target data in the target storage pool is encapsulated and sent to the data node to be stored, and finally saved in the target business data empty table on the data node to be stored.

[0060] Optionally, before pulling the target data to the data node to be stored, the data in the initial business data table is evenly distributed to multiple threads, thereby ensuring a balanced workload for each thread and solving the load imbalance problem of directly distributing data to the threads corresponding to the data node to be stored based on the sharding rules of the target business data empty table; the target data in the target storage pool on the i-th data node is encapsulated and sent to the data node to be stored at one time, thereby reducing the number of data transmissions, improving the efficiency of data pulling and the flexibility of distributed storage, and realizing high-concurrency changes.

[0061] Figure 3 A flow chart of saving target data into a target business data empty table according to an embodiment of the present invention is shown.

[0062] like Figure 3 As shown, encapsulating the target data in the target storage pool and saving it to the target business data empty table on the data node to be stored includes operations S310 to S330.

[0063] In operation S310, when the amount of data in the target storage pool is equal to a preset amount threshold, target data in the target storage pool is encapsulated to obtain an encapsulated data packet.

[0064] In operation S320, the encapsulated data packet is stored in a message queue corresponding to the target storage pool.

[0065] In operation S330, the encapsulated data packet in the message queue is sent to the data node to be stored.

[0066] Optionally, each storage pool of the data node continuously pulls in new target data, and when the amount of data in the target storage pool is equal to a preset quantity threshold, the target data in the target storage pool is encapsulated and packaged to obtain an encapsulated data packet. When the amount of data in the target storage pool is not equal to the preset quantity threshold, the new target data continues to be pulled into the target storage pool.

[0067] Optionally, each data node includes a message queue corresponding to one storage pool, and the message queue is used to temporarily store the encapsulated data packet. After the encapsulated data packet is stored in the corresponding message queue, a new target storage pool is created to continue to store subsequent target data.

[0068] For example, when the amount of data in the target storage pool 002A on the first data node A1 is equal to the preset quantity threshold, the target data in the target storage pool 002A is encapsulated and packaged to obtain an encapsulated data packet. After the encapsulated data packet is saved in the corresponding message queue, a new target storage pool 002A is created on the first data node A1, and then the thread on the first data node A1 continues to pull the target data to be stored in the data node B2 into the new target storage pool 002A.

[0069] Optionally, there are a series of consumer threads on each data node. The consumer threads on the data nodes to be stored pull encapsulated data packets from the message queues corresponding to the data nodes to be stored on each data node, and store the encapsulated data packets in the target business data empty table on the data nodes to be stored.

[0070] For example, the data node to be stored is the second data node B2, and a series of consumer threads of the second data node B2 pull encapsulated data packets from the message queues corresponding to the data node B2 on the data nodes A1, B2, and C3 within the cluster.

[0071] Optionally, after the consumer thread successfully pulls and saves to the target business data empty table, the consumer thread sends a message to ensure the reliability of the data; if an error occurs during the pulling process, the consumer thread should record the error information and retry or alarm according to the preset strategy.

[0072] Optionally, after the pull is successful, the physical file names of the initial business data table and the target business data empty table in the system table are exchanged first, and then the target business data empty table is deleted.

[0073] Optionally, after all data has been successfully migrated and confirmed, the read lock is released and the transaction is committed. The completion of the data shard conversion ensures that all data changes are permanently saved.

[0074] Optionally, for each target data pulled by the fetcher thread, the data node to be stored is determined, the target data is directly appended to the end of the target storage pool, and finally encapsulated and sent to the data node to be stored, thereby reducing the number of data transmissions and the consumption of computing resources.

[0075] Optionally, based on the sharding rule of the target business data empty table, determining the data node to be stored corresponding to the target data on each thread includes: when the sharding rule of the target business data empty table is a list sharding rule, determining the data node to be stored corresponding to the target data on each thread according to the first data mapping relationship table in the list sharding rule, wherein the first data mapping relationship table represents the correspondence between the target key value corresponding to the target field in the initial business data table and the data node to be stored, and the target data is associated with the target key value.

[0076] Optionally, the target field is a key field. During the process of the fetcher thread pulling the target data, the system can directly use the first data mapping relationship table to query and directly determine the data node to be stored corresponding to the target data.

[0077] For example, the target field may be a city, and the target key values ​​corresponding to the target field may include north x, sky x, top x, etc. The first data mapping relationship table defines the corresponding relationship between north x and data node A1, the corresponding relationship between sky x and data node B2, and the corresponding relationship between top x and data node C3.

[0078] Optionally, the target data is the data in the data row where the target key value is located. For example, when the target key value of the target field is Northx, the data in the data row where Northx is located is determined as the target data, and the data node to be stored corresponding to the target data is data node A1.

[0079] Optionally, the data range of the target field is divided, and the list sharding rule may further define the correspondence between the target key values ​​in different data ranges and the data nodes to be stored.

[0080] Optionally, based on the sharding rule of the empty table of target business data, determining the data nodes to be stored corresponding to the target data on each thread also includes: when the sharding rule of the empty table of target business data is a hash sharding rule, using a hash algorithm to process multiple key values ​​of the target field to obtain hash values ​​corresponding to each of the multiple key values; taking the modulus of the hash values ​​corresponding to each of the multiple key values ​​according to the number of data nodes to determine the modulus corresponding to each of the multiple key values; determining the data nodes to be stored corresponding to the target data on each thread according to the modulus corresponding to each of the multiple key values ​​and the association relationship, wherein there is an association relationship between the target data and the modulus of the key value.

[0081] Optionally, the hash algorithm may be determined based on a hash function (hash), and the hash algorithm is used to process multiple key values ​​of the target field to obtain hash values ​​corresponding to each of the multiple key values.

[0082] Optionally, the hash values ​​corresponding to the multiple key values ​​are uniformly modulo according to the number of data nodes. For example, the number of data nodes is 3, and the hash values ​​corresponding to the multiple key values ​​are uniformly modulo, and the modulus can be 0, 1, or 2.

[0083] Optionally, the modulus corresponding to the key value is 0, indicating that the data node to be stored corresponding to the key value is the first data node A1; the modulus corresponding to the key value is 1, indicating that the data node to be stored corresponding to the key value is the second data node B2; the modulus corresponding to the key value is 2, indicating that the data node to be stored corresponding to the key value is the third data node C3.

[0084] Optionally, the association relationship between the target data and the modulus of the key value represents that the modulus corresponding to the key value is the modulus corresponding to the target data associated with the key value.

[0085] Optionally, the module corresponding to the target data is determined according to the modules and association relationships corresponding to each of the multiple key values, thereby determining the data node to be stored corresponding to the target data on each thread.

[0086] Optionally, evenly distributing the data in the initial business data table to multiple threads to obtain target data corresponding to each of the multiple threads includes: distributing the data in the initial business data table to multiple threads based on data characteristics of the first field in the initial business data table and the number of threads to obtain initial data corresponding to each of the multiple threads; and adjusting the initial data corresponding to each of the multiple threads based on a balancing strategy to obtain target data corresponding to each of the multiple threads.

[0087] Optionally, the data characteristic of the first field can be the value range of the first field. For example, the value range of the first field is (0, 100), the number of threads is 10, and the value range of the first field is divided into 10 parts to obtain 10 sub-value ranges. The data of each of the 10 sub-value ranges are allocated to 10 threads to obtain the initial data corresponding to each thread.

[0088] Optionally, the sub-value range may also be determined based on the number of threads and the processing capability of each thread.

[0089] Optionally, when the first field is divided into ranges, the number of key values ​​may be very large. The amount of data corresponding to different sub-value ranges may be different, causing great pressure on a certain thread. Therefore, it is necessary to adjust the initial data corresponding to each of the multiple threads based on a balancing strategy to obtain the target data corresponding to each of the multiple threads.

[0090] Optionally, after the adjustment, multiple threads on the same data node are load balanced, and the target data represents the data after the initial data is redistributed.

[0091] Optionally, during the thread pulling process, it may be detected that the processing speed of some threads is significantly lower than that of other threads, so the sub-value range or the number of threads can be dynamically adjusted to optimize performance. Multi-threaded load balancing can avoid some threads being overloaded while other threads are idle, thereby improving the overall data pulling efficiency.

[0092] Optionally, based on a balancing strategy, initial data corresponding to each of the multiple threads are adjusted to obtain target data corresponding to each of the multiple threads, including: detecting the performance of each of the multiple threads corresponding to the i-th data node; when the performance of each of the multiple threads on the i-th data node is uneven, using a hash algorithm to process the associated key value of the second field in the initial business data table to obtain hash values ​​corresponding to each of the multiple associated key values, wherein a second mapping relationship exists between the associated key values ​​and the initial data on the multiple threads; based on the multiple hash values ​​and the second mapping relationship, distributing the initial data to the multiple threads on the i-th data node to obtain target data corresponding to each of the multiple threads.

[0093] Optionally, the control node detects the performance of each of the multiple threads corresponding to each data node in the system. When it is detected that the processing speed of some threads on the i-th data node is significantly lower than that of other threads or the performance of the multiple threads is uneven, the initial data corresponding to the i-th data node is reallocated.

[0094] Optionally, the second field is different from the first field. The associated key value represents the key value of the second field in the initial business data table.

[0095] Optionally, the initial data includes an associated key value, and in each data row in the initial business data table, the data row where the associated key value is located is the initial data corresponding to the associated data, and the second mapping relationship represents the correspondence between the associated key value and the initial data.

[0096] For example, it is monitored that the performance of multiple threads of the first data node A1 is uneven. The first data node A1 corresponds to three threads, the first thread is assigned four pieces of initial data, the second thread is assigned one piece of initial data, and the third thread is assigned one piece of initial data. The second field is determined to be the age field, and the associated key values ​​of the second field in the six pieces of initial data are processed using a hash algorithm to obtain hash values ​​corresponding to the six associated key values.

[0097] For example, taking the age field as the second field, the modulus of the six hash values ​​is calculated according to the number of threads, and the modulus corresponding to each of the six associated key values ​​is 0 for two associated key values, 1 for two associated key values, and 1 for two associated key values. A modulus of 0 indicates that the data is to be allocated to the first thread, a modulus of 1 indicates that the data is to be allocated to the second thread, and a modulus of 3 indicates that the data is to be allocated to the third thread.

[0098] Optionally, according to the initial data corresponding to the associated key value, the six initial data are distributed to the three threads on the first data node A1 to obtain target data corresponding to each of the three threads. Each thread corresponds to two target data.

[0099] Based on the above-mentioned data shard conversion method for distributed database, the present invention also provides a data shard conversion device for distributed database. Figure 4 The device is described in detail.

[0100] Figure 4 A structural block diagram of a data shard conversion device for a distributed database according to an embodiment of the present invention is shown.

[0101] like Figure 4 As shown, the data shard conversion device 400 for a distributed database of this embodiment includes a response module 410 , an allocation module 420 , a determination module 430 , a conversion module 440 and a sending module 450 .

[0102] The response module 410 is used to respond to the sharding transaction request for the initial business data table and create a target business data empty table on I data nodes included in the distributed database, where I is a positive integer. In one embodiment, the response module 410 can be used to perform the operation S210 described above, which will not be repeated here.

[0103] The allocation module 420 is used to evenly allocate the data in the initial business data table to multiple threads to obtain target data corresponding to each of the multiple threads, wherein each data node includes I storage pools and multiple threads, and there is a first mapping relationship between the i-th storage pool and the i-th data node, I≥i>1, and i is an integer. In one embodiment, the allocation module 420 can be used to perform the operation S220 described above, which will not be repeated here.

[0104] The determination module 430 is used to determine the data node to be stored corresponding to the target data on each thread based on the sharding rule of the target business data empty table. In one embodiment, the determination module 430 can be used to perform the operation S230 described above, which will not be repeated here.

[0105] The conversion module 440 is used to store the target data on the thread of the i-th data node in the target storage pool corresponding to the data node to be stored on the i-th data node based on the first mapping relationship and the data node to be stored corresponding to the target data on each thread. In one embodiment, the conversion module 440 can be used to perform the operation S240 described above, which will not be repeated here.

[0106] The sending module 450 is used to encapsulate the target data in the target storage pool and save it to the target business data empty table on the data node to be stored when the amount of data in the target storage pool is equal to the preset amount threshold. In one embodiment, the sending module 450 can be used to perform the operation S250 described above, which will not be repeated here.

[0107] Optionally, the sending module 450 includes a first sending submodule, a second sending submodule and a third sending submodule.

[0108] The first sending submodule is used to encapsulate the target data in the target storage pool to obtain an encapsulated data packet when the amount of data in the target storage pool is equal to a preset amount threshold.

[0109] The second sending submodule is used to store the encapsulated data packet in a message queue corresponding to the target storage pool.

[0110] The third sending submodule is used to send the encapsulated data packet in the message queue to the data node to be stored, wherein the encapsulated data packet is stored in the target business data empty table on the data node to be stored.

[0111] Optionally, the determination module 430 includes a first determination submodule.

[0112] The first determination submodule is used to determine the data node to be stored corresponding to the target data on each thread according to the first data mapping relationship table in the list sharding rule when the sharding rule of the target business data empty table is a list sharding rule, wherein the first data mapping relationship table represents the correspondence between the target key value corresponding to the target field in the initial business data table and the data node to be stored, and the target data is associated with the target key value.

[0113] Optionally, the determination module 430 further includes a second determination submodule, a third determination submodule and a fourth determination submodule.

[0114] The second determination submodule is used to process multiple key values ​​of the target field using a hash algorithm when the sharding rule of the target business data empty table is a hash sharding rule to obtain hash values ​​corresponding to each of the multiple key values.

[0115] The third determination submodule is used to perform modulo calculation on the hash values ​​corresponding to each of the multiple key values ​​according to the number of data nodes to determine the modulo corresponding to each of the multiple key values.

[0116] The fourth determination submodule is used to determine the data node to be stored corresponding to the target data on each thread according to the moduli and association relationships corresponding to the multiple key values, wherein there is an association relationship between the target data and the moduli of the key value.

[0117] Optionally, the allocation module 420 includes a first allocation submodule and a second allocation submodule.

[0118] The first allocation submodule is used to allocate the data in the initial business data table to multiple threads according to the data characteristics of the first field in the initial business data table and the number of threads, so as to obtain initial data corresponding to each of the multiple threads.

[0119] The second allocation submodule is used to adjust the initial data corresponding to each of the multiple threads based on the balancing strategy to obtain the target data corresponding to each of the multiple threads.

[0120] Optionally, the second allocation submodule includes a first allocation unit, a second allocation unit and a third allocation unit.

[0121] The first allocation unit is used to detect the performance of each of the multiple threads corresponding to the i-th data node.

[0122] The second allocation unit is used to process the associated key value of the second field in the initial business data table by using a hash algorithm when the performance of multiple threads on the i-th data node is uneven, so as to obtain hash values ​​corresponding to each of the multiple associated key values, wherein a second mapping relationship exists between the associated key value and the initial data on the multiple threads.

[0123] The third allocation unit is used to allocate the initial data to multiple threads on the i-th data node based on the multiple hash values ​​and the second mapping relationship to obtain target data corresponding to each of the multiple threads.

[0124] Optionally, any multiple modules among the response module 410, the allocation module 420, the determination module 430, the conversion module 440 and the sending module 450 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. Optionally, at least one of the response module 410, the allocation module 420, the determination module 430, the conversion module 440 and the sending module 450 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in an appropriate combination of any of them. Alternatively, at least one of the response module 410 , the allocation module 420 , the determination module 430 , the conversion module 440 , and the sending module 450 may be at least partially implemented as a computer program module, and when the computer program module is executed, a corresponding function may be performed.

[0125] Figure 5 A block diagram of an electronic device suitable for implementing a data shard conversion method for a distributed database according to an embodiment of the present invention is shown.

[0126] Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0127] like Figure 5 As shown, the computer electronic device 500 according to an embodiment of the present invention includes a processor 501, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 502 or the program loaded from the storage part 508 to the random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include an on-board memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0128] In RAM 503, various programs and data required for the operation of electronic device 500 are stored. Processor 501, ROM 502 and RAM 503 are connected to each other via bus 504. Processor 501 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 502 and / or RAM 503. It should be noted that the program can also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in one or more memories.

[0129] Optionally, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to the bus 504. The electronic device 500 may further include one or more of the following components connected to the input / output (I / O) interface 505: an input portion 506 including a keyboard, a mouse, etc.; an output portion 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 508 including a hard disk, etc.; and a communication portion 509 including a network interface card such as a LAN card, a modem, etc. The communication portion 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage portion 508 as needed.

[0130] Optionally, the method flow according to the embodiment of the present invention can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the processor 501, the above-mentioned functions defined in the system of the embodiment of the present invention are executed. Optionally, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0131] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the data sharding conversion method for a distributed database according to an embodiment of the present invention is implemented.

[0132] Optionally, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include, but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.

[0133] For example, optionally, the computer-readable storage medium may include the ROM 502 and / or the RAM 503 described above and / or one or more memories other than the ROM 502 and the RAM 503 .

[0134] An embodiment of the present invention also includes a computer program product, which includes a computer program, which contains program code for executing the method provided by the embodiment of the present invention. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the data sharding conversion method for a distributed database provided by the embodiment of the present invention.

[0135] When the computer program is executed by the processor 501, the above functions defined in the system / device of the embodiment of the present invention are performed. Optionally, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0136] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 509, and / or installed from the removable medium 511. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0137] Optionally, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages, specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, such as Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0138] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments of the present invention can be combined and / or combined in various ways, even if such a combination or combination is not explicitly recorded in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features recorded in the various embodiments of the present invention can be combined and / or combined in various ways. All these combinations and / or combinations fall within the scope of the present invention.

[0139] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A data sharding conversion method for a distributed database, characterized in that: The method comprises: In response to a sharding transaction request for an initial business data table, a target business data empty table is created on I data nodes included in the distributed database, where I is a positive integer; Evenly distribute the data in the initial business data table to multiple threads to obtain target data corresponding to each of the multiple threads, wherein each of the data nodes includes I storage pools and multiple threads, and there is a first mapping relationship between the i-th storage pool and the i-th data node, I≥i>1, and i is an integer; Based on the sharding rule of the target business data empty table, determine the data node to be stored corresponding to the target data on each of the threads; Based on the first mapping relationship and the data node to be stored corresponding to the target data on each of the threads, storing the target data on the thread of the i-th data node into a target storage pool on the i-th data node corresponding to the data node to be stored; When the amount of data in the target storage pool is equal to a preset quantity threshold, the target data in the target storage pool is encapsulated and saved in an empty table of target business data on the data node to be stored.

2. The method according to claim 1, characterized in that Each of the data nodes includes a message queue corresponding to one storage pool; Wherein, when the amount of data in the target storage pool is equal to a preset quantity threshold, encapsulating the target data in the target storage pool and saving it to the target business data empty table on the data node to be stored includes: When the amount of data in the target storage pool is equal to a preset amount threshold, encapsulating the target data in the target storage pool to obtain an encapsulated data packet; Storing the encapsulated data packet in the message queue corresponding to the target storage pool; The encapsulated data packet in the message queue is sent to the data node to be stored, wherein the encapsulated data packet is stored in the target business data empty table on the data node to be stored.

3. The method according to claim 1, characterized in that The determining, based on the sharding rule of the empty table of target business data, the data node to be stored corresponding to the target data on each thread comprises: In the case where the sharding rule of the target business data empty table is a list sharding rule, the data node to be stored corresponding to the target data on each thread is determined according to the first data mapping relationship table in the list sharding rule, wherein the first data mapping relationship table represents the correspondence between the target key value corresponding to the target field in the initial business data table and the data node to be stored, and the target data is associated with the target key value.

4. The method according to claim 1, characterized in that: The determining, based on the sharding rule of the empty table of target business data, the data node to be stored corresponding to the target data on each thread further comprises: In the case where the sharding rule of the target business data empty table is a hash sharding rule, a plurality of key values ​​of the target field are processed by using a hash algorithm to obtain hash values ​​corresponding to each of the plurality of key values; Taking a modulus of the hash value corresponding to each of the plurality of key values ​​according to the number of the data nodes, to obtain a modulus corresponding to each of the plurality of key values; According to the moduli and association relationships corresponding to the multiple key values ​​respectively, the data node to be stored corresponding to the target data on each of the threads is determined, wherein the association relationship exists between the target data and the moduli of the key value.

5. The method according to claim 1, characterized in that The step of evenly distributing the data in the initial business data table to multiple threads to obtain target data corresponding to each of the multiple threads comprises: According to the data characteristics of the first field in the initial business data table and the number of threads, the data in the initial business data table is distributed to the plurality of threads to obtain initial data corresponding to each of the plurality of threads; Based on the balancing strategy, the initial data corresponding to each of the plurality of threads are adjusted to obtain the target data corresponding to each of the plurality of threads.

6. The method according to claim 5, characterized in that The step of adjusting the initial data corresponding to each of the plurality of threads based on the balancing strategy to obtain the target data corresponding to each of the plurality of threads includes: Detecting the performance of each of the multiple threads corresponding to the i-th data node; In the case where the performance of the plurality of threads on the i-th data node is uneven, the associated key value of the second field in the initial business data table is processed by a hash algorithm to obtain hash values ​​corresponding to the plurality of associated key values, wherein a second mapping relationship exists between the associated key value and the initial data on the plurality of threads; Based on the multiple hash values ​​and the second mapping relationship, the initial data is distributed to the multiple threads on the i-th data node to obtain target data corresponding to each of the multiple threads.

7. A data shard conversion device for a distributed database, characterized in that: The device comprises: A response module, configured to, in response to a sharding transaction request for an initial business data table, create a target business data empty table on I data nodes included in the distributed database, respectively, where I is a positive integer; an allocation module, configured to evenly allocate the data in the initial business data table to the multiple threads, and obtain target data corresponding to each of the multiple threads, wherein each of the data nodes includes I storage pools and the multiple threads, and a first mapping relationship exists between the i-th storage pool and the i-th data node, I≥i>1, and i is an integer; A determination module, configured to determine, based on a sharding rule of the target business data empty table, a data node to be stored corresponding to the target data on each of the threads; a conversion module, configured to store the target data on the thread of the i-th data node into a target storage pool corresponding to the data node to be stored on the i-th data node based on the first mapping relationship and the data node to be stored corresponding to the target data on each of the threads; The sending module is used to encapsulate the target data in the target storage pool and save it to the target business data empty table on the data node to be stored when the amount of data in the target storage pool is equal to a preset quantity threshold.

8. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more instructions, Wherein, when the one or more instructions are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: Executable instructions are stored thereon, and when the instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images