Data transmission method, device, equipment and medium between distributed databases

Abstract

The embodiment of the present invention provides a method, device, equipment and medium for data transmission between distributed databases, which relates to the field of data processing technology, wherein the method includes: for each data table to be transmitted, a mapping relationship between a first data shard of a source database and a second data shard of a target database is established; the coordinator of the target database decomposes the data insertion task into multiple data insertion subtasks and sends them to each participant respectively; in each participant, in response to the data insertion subtask, activates the respective transmission tool, and for each second data shard in each participant, determines the first data shard corresponding to the mapping of the second data shard through the transmission tool based on the mapping relationship, and reads the data of the data table to be transmitted from the first data shard corresponding to the mapping and stores it on the second data shard. The process of reading data between data shards in this scheme does not rely on the coordinator, which is conducive to avoiding the problem of single-point performance bottleneck.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data transmission method, device, equipment and medium between distributed databases. Background Art

[0002] Due to factors such as business needs, system architecture changes, or technology updates, data needs to be migrated from one distributed database to another.

[0003] When faced with the problem of data migration between different distributed database clusters, whether the stored data can be quickly transferred from one distributed database to another plays a decisive role in the operation of the business that relies on the data. Finding an efficient method for data transmission between distributed databases is the core of solving the data migration problem.

[0004] A distributed database system is usually a unified whole logically; physically, it is composed of different physical nodes; data is stored in the form of multiple data shards on these physical nodes. At the same time, there is usually a data shard used to store the metadata of the data shard (data sharding method and strategy, etc.). The physical node that usually stores this data shard is also responsible for scheduling, coordinating and controlling the data operations of other nodes, and is called a coordinator; while the remaining physical nodes are responsible for independently storing data and processing tasks from the coordinator, and are called participants.

[0005] Obviously, in the scenario of data transmission, which involves massive data processing, the coordinator can easily become a single point performance bottleneck, limiting the speed of data transmission. The current distributed cluster data transmission methods either use file migration or introduce other intermediate storage media. These methods not only fail to overcome the single point performance bottleneck of the coordinator, but may also introduce additional storage costs or increase system complexity. Summary of the invention

[0006] In view of this, an embodiment of the present invention provides a method for data transmission between distributed databases to solve the technical problem in the prior art that the coordinator becomes a single point performance bottleneck affecting the data transmission speed. The method includes:

[0007] For each data table to be transmitted, a mapping relationship between a first data shard of a source distributed database and a second data shard of a target distributed database is established;

[0008] When the coordinator of the target distributed database receives the data insertion task, the data insertion task is decomposed into a plurality of data insertion subtasks, and the plurality of data insertion subtasks are respectively sent to each participant, wherein one distributed node among the distributed nodes of the target distributed database is the coordinator, and the other distributed nodes are participants;

[0009] In each of the participants, in response to the insert data subtask, the respective transmission tools are activated, and for each of the second data shards in each of the participants, the first data shard corresponding to the mapping of the second data shard is determined by the transmission tool based on the mapping relationship, and the data of the data table to be transmitted is read from the first data shard corresponding to the mapping and stored on the second data shard, wherein the transmission tool is stored on each distributed node of the target distributed database, and the mapping relationship is stored on each distributed node of the target distributed database.

[0010] The embodiment of the present invention also provides a data transmission device between distributed databases to solve the technical problem that the coordinator becomes a single point performance bottleneck affecting the data transmission speed in the prior art. The device includes:

[0011] A mapping module, used to establish a mapping relationship between a first data shard of a source distributed database and a second data shard of a target distributed database for each data table to be transmitted;

[0012] A task distribution module, configured to decompose the data insertion task into a plurality of data insertion subtasks when the coordinator of the target distributed database receives the data insertion task, and send the plurality of data insertion subtasks to each participant respectively, wherein one distributed node among the distributed nodes of the target distributed database is the coordinator, and the other distributed nodes are participants;

[0013] A data transmission module is used to activate the respective transmission tools in each of the participants in response to the insert data subtask, and for each of the second data shards in each of the participants, determine the first data shard corresponding to the mapping of the second data shard through the transmission tool based on the mapping relationship, and read the data of the data table to be transmitted from the first data shard corresponding to the mapping and store it on the second data shard, wherein the transmission tool is stored on each distributed node of the target distributed database, and the mapping relationship is stored on each distributed node of the target distributed database.

[0014] An embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-mentioned methods for data transmission between distributed databases to solve the technical problem in the prior art that the coordinator becomes a single point performance bottleneck affecting the data transmission speed.

[0015] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program for executing any of the above-mentioned data transmission methods between distributed databases to solve the technical problem in the prior art that the coordinator becomes a single point performance bottleneck affecting the data transmission speed.

[0016] Compared with the prior art, the beneficial effects that can be achieved by at least one of the above-mentioned technical solutions adopted in the embodiments of this specification include at least: proposing, for each data table to be transmitted, determining the mapping relationship between the first data shard of the source distributed database and the second data shard of the target distributed database, and storing the transmission tool and the mapping relationship on each distributed node of the target distributed database, so that when the coordinator of the target distributed database receives the insert data task, the insert data task is decomposed into multiple insert data subtasks, and the multiple insert data subtasks are sent to each participant respectively. In each of the participants, in response to the insert data subtask, the respective transmission tools are activated, and for each second data shard in each of the participants, the first data shard corresponding to the mapping of the second data shard is determined by the transmission tool based on the mapping relationship, and the corresponding data shard is obtained from the mapping. The data of the data table to be transmitted read by the first data shard is stored on the second data shard, so that after each participant receives the insert data subtask, a connection between the corresponding second data shard and the first data shard is established through a mapping relationship based on their own transmission tools, and the table data stored in the first data shard is read and stored on the second data shard, so that the process of the second data shard of each participant reading data from the first data shard does not depend on the coordinator, which is conducive to avoiding the problem of single-point performance bottleneck of the coordinator, and further conducive to making full use of physical resources including CPU, memory and network bandwidth, improving transmission speed and realizing efficient data transmission; at the same time, the above-mentioned data transmission method between distributed databases is based on each data table to be transmitted, that is, concurrent management of multiple data tables is used to carry out data transmission at the same time, further making full use of the advantages of the distributed structure and realizing efficient data transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 is a flow chart of a method for data transmission between distributed databases provided by an embodiment of the present invention;

[0019] Figure 2The present invention provides a schematic diagram of a mapping relationship between a first data shard of a source distributed database and a second data shard of a target distributed database. Figure 1 ;

[0020] Figure 3 The present invention provides a schematic diagram of a mapping relationship between a first data shard of a source distributed database and a second data shard of a target distributed database. Figure 2 ;

[0021] Figure 4 The present invention provides a schematic diagram of a mapping relationship between a first data shard of a source distributed database and a second data shard of a target distributed database. Figure 3 ;

[0022] Figure 5 The present invention provides a schematic diagram of a mapping relationship between a first data shard of a source distributed database and a second data shard of a target distributed database. Figure 4 ;

[0023] Figure 6 The present invention provides a schematic diagram of a mapping relationship between a first data shard of a source distributed database and a second data shard of a target distributed database. Figure 5 ;

[0024] Figure 7 The present invention provides a schematic diagram of a mapping relationship between a first data shard of a source distributed database and a second data shard of a target distributed database. Figure 6 ;

[0025] Figure 8 The present invention provides a schematic diagram of a mapping relationship between a first data shard of a source distributed database and a second data shard of a target distributed database. Figure 7 ;

[0026] Fig. 9 It is a schematic diagram of a data flow between a first data shard of a source distributed database and a second data shard of a target distributed database provided by an embodiment of the present invention;

[0027] Fig.10 is a structural block diagram of a computer device provided by an embodiment of the present invention;

[0028] Fig.11 It is a structural block diagram of a data transmission device between distributed databases provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0030] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0031] In an embodiment of the present invention, a method for data transmission between distributed databases is provided. Figure 1 As shown, the method includes:

[0032] Step S101: for each data table to be transmitted, a mapping relationship between a first data shard of a source distributed database and a second data shard of a target distributed database is established;

[0033] Step S102: When the coordinator of the target distributed database receives the data insertion task, the data insertion task is decomposed into a plurality of data insertion subtasks, and the plurality of data insertion subtasks are sent to each participant respectively, wherein one distributed node among the distributed nodes of the target distributed database is the coordinator, and the other distributed nodes are participants;

[0034] Step S103: In each of the participants, in response to the insert data subtask, the respective transmission tools are activated, and for each of the second data shards in each of the participants, the first data shard corresponding to the mapping of the second data shard is determined by the transmission tool based on the mapping relationship, and the data of the data table to be transmitted is read from the first data shard corresponding to the mapping and stored on the second data shard, wherein the transmission tool is stored on each distributed node of the target distributed database, and the mapping relationship is stored on each distributed node of the target distributed database.

[0035] Depend on Figure 1It can be seen from the process shown that in the embodiment of the present invention, after each participant receives the insert data subtask, a connection is established between the corresponding second data shard and the first data shard through a mapping relationship based on their own transmission tools, and the table data stored in the first data shard is read and stored on the second data shard, so that the process of the second data shard of each participant reading data from the first data shard does not depend on the coordinator, which is conducive to avoiding the problem of single-point performance bottleneck of the coordinator, and further conducive to making full use of physical resources including CPU, memory and network bandwidth, improving transmission speed and realizing efficient data transmission; at the same time, the above-mentioned data transmission method between distributed databases is based on each data table to be transmitted, that is, concurrent management of multiple data tables is used to carry out data transmission at the same time, further making full use of the advantages of the distributed structure and realizing efficient data transmission.

[0036] In specific implementation, the source distributed database and the target distributed database are the starting point and the end point of data transmission. The direction of data transmission is from the source distributed database to the target distributed database.

[0037] In specific implementation, the above-mentioned data insertion task may include relevant information of the data table to be transmitted, so that the second data shards of each participant of the target distributed database read the data corresponding to the data table to be transmitted from the first data shard of the source distributed database based on the above-mentioned mapping relationship.

[0038] In specific implementation, in order to realize the transmission of data between distributed databases from a certain data shard of the source distributed database to a data shard of the target distributed database, a method for establishing a mapping relationship between the first data shard of the source distributed database and the second data shard of the target distributed database is proposed (realizing the establishment of a mapping relationship between all the first data shards of the source distributed database and the second data shard of the target distributed database), for example,

[0039] For the source distributed database and the target distributed database, respectively count the number of distributed nodes, the total number of data shards, and the number of data shards on each distributed node;

[0040] According to the different situations of the number of distributed nodes, the total number of data shards and the number of data shards on each distributed node between the source distributed database and the target distributed database, a mapping relationship between the first data shard of the source distributed database and the second data shard of the target distributed database is determined, in which each of the first data shards in the mapping relationship maps to one second data shard to avoid data loss or duplication.

[0041] In specific implementation, according to the number of distributed nodes between the source distributed database and the target distributed database, the total number of data shards, and the number of data shards on each distributed node, the mapping relationship between the first data shard of the source distributed database and the second data shard of the target distributed database is determined in the following ways:

[0042] First way:

[0043] If the total number of data shards between the source distributed database and the target distributed database is the same, a one-to-one mapping relationship between the first data shard and the second data shard is established in ascending order of the data shard numbers in the source distributed database and the target distributed database.

[0044] like Figure 2 As shown, the data shards of the source distributed database (i.e., the source database) and the target distributed database (i.e., the target database) are sorted in ascending order according to the numbers, and then based on the sorted ranking, a one-to-one mapping relationship is established between the first data shards and the second data shards of the same ranking, such as a mapping relationship is established between the second data shard numbered 0 of the target distributed database and the first data shard numbered 0 of the source distributed database, a mapping relationship is established between the second data shard numbered 1 of the target distributed database and the first data shard numbered 1 of the source distributed database, and so on.

[0045] Second way:

[0046] If the number of distributed nodes between the source distributed database and the target distributed database is the same, and the number of data shards of each distributed node in the source distributed database and the target distributed database is the same, the distributed nodes of the source distributed database are sorted according to the host name to obtain a first sort, and the distributed nodes of the target distributed database are sorted according to the host name to obtain a second sort;

[0047] A mapping relationship is established between the first data shards and the second data shards of the distributed nodes with the same ranking in the first sorting and the second sorting.

[0048] Specifically, firstly, the distributed nodes of the source distributed database and the target distributed database are sorted by host name, respectively, to obtain the first sort and the second sort, and then a mapping relationship is established between the first data shard and the second data shard of the distributed nodes with the same ranking in the first sort and the second sort, that is, the second data shard in a distributed node with a certain ranking in the target distributed database is only mapped to the first data shard in the distributed node with the same ranking in the source distributed database. Figure 3 ( Figure 3 Each distributed node of the target distributed database has 1 second data shard), Figure 4 ( Figure 4 As shown in the figure, each distributed node of the target distributed database has 3 second data shards), the second data shard in the first-ranked distributed node of the target distributed database is only mapped to the first data shard in the first-ranked distributed node of the source distributed database, and the second data shard in the second-ranked distributed node of the target distributed database is only mapped to the first data shard in the second-ranked distributed node of the source distributed database, and so on. The mapping relationship of the data shards is determined by the number of data shards in the distributed nodes.

[0049] In the second method, when the number of data shards of distributed nodes with the same ranking (such as the i-th ranked distributed nodes of the source distributed database and the target distributed database) is different, the first data shard and the second data shard in the distributed nodes with the same ranking of the source distributed database and the target distributed database are sorted from small to large according to the numbers, and then mapped in turn based on the number sorting. For example, for a certain data table, if the number of first data shards in the distributed nodes with the same ranking is greater than the number of second data shards, there will be a situation where one second data shard corresponds to multiple first data shards, such as Figure 3 The mapping relationship between the first data shard and the second data shard in the distributed node. If the number of first data shards in the distributed nodes of the same ranking is less than the number of second data shards, that is, the number of second data shards is greater, some (some) second data shards will not participate in the mapping, such as Figure 4 The mapping relationship between the first data shard and the second data shard in the distributed node.

[0050] The third way:

[0051] If the number of distributed nodes between the source distributed database and the target distributed database is different and the total number of the data shards is also different, the first data shards of the source distributed database are sorted by number to obtain a third sort, and the second data shards of the target distributed database are sorted by number to obtain a fourth sort;

[0052] A mapping relationship is established between the first data slice and the second data slice in turn according to the third sorting and the fourth sorting.

[0053] The process of establishing a mapping relationship between the first data slice and the second data slice in turn according to the third sorting and the fourth sorting includes:

[0054] If the total amount of the first data slices is greater than the total amount of the second data slices, after the mapping relationship is established for all the second data slices, the second data slices are repeatedly used to establish the mapping relationship with the remaining first data slices according to the fourth sorting.

[0055] The process of establishing a mapping relationship between the first data slice and the second data slice in turn according to the third sorting and the fourth sorting includes:

[0056] If the total amount of the first data slices is less than the total amount of the second data slices, for the first data table to be transmitted, after the first data slices and the second data slices are mapped in turn according to the third sorting and the fourth sorting, if there are still second data slices for which no mapping relationship has been established, then for the second data table to be transmitted, the second data slices ranked after the second data slices for which a mapping relationship has been established are mapped to the first data slices according to the third sorting and the fourth sorting, such as Figure 7 , Figure 8 shown.

[0057] Specifically, the data shards of the source distributed database and the target distributed database are sorted from small to large according to the numbers to obtain the third sort and the fourth sort, and then the first data shard and the second data shard are mapped in turn based on the third sort and the fourth sort. For a certain data table, if the number of first data shards is greater than the number of second data shards, that is, the number of first data shards is greater, there will be a situation where one second data shard corresponds to multiple first data shards, such as Figure 6 As shown, the second data slices numbered 0 and 1 repeatedly establish a mapping relationship; if the number of first data slices is less than the number of second data slices, that is, the number of second data slices is greater, some (some) second data slices will not participate in the mapping, such as Figure 5 As shown, the first data slices numbered 0-3 are mapped to the second data slices one by one, and the second data slices numbered 4 and 5 are not involved in the mapping. Figure 5 The structure of the source distributed database and the target distributed database shown in the figure has three mapping relationships generated by the round-robin method, as follows: Figure 5 , Figure 7 and Figure 8 As shown, Figure 5 The figure shows that the number of first data shards is less than the number of second data shards. For the first data table, after a one-to-one mapping relationship is established between the first data shards and the second data shards, the second data shards numbered 4 and 5 do not participate in the mapping. Figure 7 It is a continuation Figure 5In the case of, in the process of establishing a mapping relationship for the second data table, the mapping relationship between the first data slice and the second data slice is established starting from the second data slices numbered 4 and 5 according to the numbering sequence of the second data slices, and the mapping relationship is repeatedly established for the second data slices numbered 0 and 1. Figure 8 It is a continuation Figure 7 In the case of , in the process of establishing the mapping relationship for the third data table, the mapping relationship between the first data shard and the second data shard is established starting from the second data shards numbered 2 and 3 according to the numbering of the second data shards. In the process of transmitting all data tables stored in the meta-distributed database, each data table uses one of the above three mapping relationships, which can ensure the load balancing of each distributed node to a certain extent.

[0058] In specific implementation, in the process of establishing a mapping relationship, a corresponding mapping relationship is established for each data table to be transmitted, and the mapping relationship includes the mapping of all first data shards and second data shards of the source distributed database (not necessarily including all second data shards of the target distributed database). For example, for a data table to be transmitted, a first mapping relationship between all first data shards and second data shards is established, and for another data table to be transmitted, a second mapping relationship between all first data shards and second data shards is also established. Between the first mapping relationship and the second mapping relationship, all first data shards are the same, and the second data shards may be the same or different (the specific situation varies depending on the above three methods of establishing the mapping relationship).

[0059] In a specific implementation, the transmission tool may be an executable file tool, which, when activated, sends a select request to the designated first data slice according to a specific mapping relationship to read data. The executable file tool may be stored on each distributed node through a distributed table.

[0060] In specific implementation, after sending an insert request (i.e., the above-mentioned data insertion task) to the coordinator of the target database, the coordinator sends the data insertion subtask to all participants. When the participants receive the task and start to execute it, they will activate the above-mentioned transmission tools stored in their respective data in the process. The transmission tool establishes a connection with the first data shard according to the specific mapping relationship and reads the table data stored in the first data shard, forming a data flow from the first data shard to the second data shard, such as Fig. 9 As shown, however, the data flow does not pass through the coordinator, realizing data transmission from the first data shard to the second data shard, thus overcoming the single-point performance bottleneck of the coordinator.

[0061] In the specific implementation, the following only takes a data table as an example to illustrate the process of the above-mentioned data transmission method between distributed databases:

[0062] 1. Store the executable file tool on each distributed node of the target distributed database through a distributed table.

[0063] 2. Use the round-robin mapping method to obtain the mapping relationship between the second data shard and the first data shard of the data table.

[0064] 3. Send an insert command (i.e. the insert data task mentioned above) to the coordinator of the target distributed database. After receiving the command, the coordinator breaks down the task into subtasks and sends them to the participants for independent execution.

[0065] 4. After receiving the insert task (i.e. the insert data subtask) distributed by the coordinator, the participant activates the local executable file tool. For each second data shard, if the second data shard does not have the corresponding mapping relationship after checking, that is, the second data shard is idle in this round, the task is terminated directly; otherwise, if the second data shard has the corresponding mapping relationship after checking, a select request is initiated to the first data shard of the corresponding source distributed database, and the task is terminated after receiving all the data about the data table stored in the first data shard.

[0066] 5. After all participants in the target distributed database have completed their subtasks, the coordinator triggers the data movement mechanism between the distributed nodes and moves the data to the correct second data shard storage according to the distribution rules of the data table.

[0067] 6. At this point, the data in this data table has been transferred from the source distributed database to the target distributed database.

[0068] In specific implementation, the data movement mechanism is a mechanism used to move data between different distributed nodes in a distributed database cluster, and is one of the core functions of a distributed database.

[0069] On the one hand, this mechanism ensures that data is correctly distributed to different data shards according to the specified distribution rules to achieve distributed storage; on the other hand, it also uses parallel processing to ensure that data can be efficiently transmitted between data shards, speeding up data access; not only that, in the process of data movement between distributed nodes, this mechanism can also ensure data integrity and consistency to avoid data loss or duplicate insertion.

[0070] The above-mentioned data transmission method between distributed databases, in conjunction with concurrent management, can realize data transmission of multiple data tables at the same time, and can fully utilize the advantages of the distributed structure, at least fully utilize one physical machine resource of the CPU, memory or network bandwidth, overcome the single-point performance bottleneck of the coordinator, greatly improve the data transmission speed, and achieve efficient data transmission.

[0071] In this embodiment, a computer device is provided, such as Fig.10 As shown, it includes a memory 1001, a processor 1002, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any of the above-mentioned methods for transmitting data between distributed databases is implemented.

[0072] Specifically, the computer device may be a computer terminal, a server or a similar computing device.

[0073] In this embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program for executing any of the above-mentioned methods for transmitting data between distributed databases.

[0074] Specifically, computer-readable storage media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable storage media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0075] Based on the same inventive concept, an embodiment of the present invention also provides a data transmission device between distributed databases, as described in the following embodiments. Since the principle of solving the problem by the data transmission device between distributed databases is similar to the data transmission method between distributed databases, the implementation of the data transmission device between distributed databases can refer to the implementation of the data transmission method between distributed databases, and the repeated parts will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements predetermined functions. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0076] Fig.11 is a structural block diagram of a data transmission device between distributed databases according to an embodiment of the present invention. Fig.11As shown, the device comprises:

[0077] A mapping module 1101 is used to establish a mapping relationship between a first data shard of a source distributed database and a second data shard of a target distributed database for each data table to be transmitted;

[0078] The task distribution module 1102 is used for decomposing the insert data task into a plurality of insert data subtasks when the coordinator of the target distributed database receives the insert data task, and sending the plurality of insert data subtasks to each participant respectively, wherein one distributed node among the distributed nodes of the target distributed database is the coordinator, and the other distributed nodes are participants;

[0079] The data transmission module 1103 is used to activate the respective transmission tools in each of the participants in response to the insert data subtask, and for each of the second data shards in each of the participants, determine the first data shard corresponding to the mapping of the second data shard through the transmission tool based on the mapping relationship, and read the data of the data table to be transmitted from the first data shard corresponding to the mapping and store it on the second data shard, wherein the transmission tool is stored on each distributed node of the target distributed database, and the mapping relationship is stored on each distributed node of the target distributed database.

[0080] In one embodiment, a mapping module is used to count the number of distributed nodes, the total number of data shards, and the number of data shards on each distributed node for the source distributed database and the target distributed database, respectively; and determine a mapping relationship between a first data shard of the source distributed database and a second data shard of the target distributed database according to different situations of the number of distributed nodes, the total number of data shards, and the number of data shards on each distributed node between the source distributed database and the target distributed database, wherein each of the first data shards in the mapping relationship maps to one second data shard.

[0081] In one embodiment, the mapping module is used to establish a one-to-one mapping relationship between the first data shard and the second data shard in the order of the data shard numbers in the source distributed database and the target distributed database if the total number of data shards between the source distributed database and the target distributed database is the same.

[0082] In one embodiment, a mapping module is used to sort the distributed nodes of the source distributed database according to the host name to obtain a first sorting if the number of distributed nodes between the source distributed database and the target distributed database is the same, and the number of data shards of each distributed node in the source distributed database and the target distributed database is the same, and sort the distributed nodes of the target distributed database according to the host name to obtain a second sorting; establish a mapping relationship between the first data shard and the second data shard of the distributed nodes with the same ranking in the first sorting and the second sorting.

[0083] In one embodiment, a mapping module is used to sort the first data shards of the source distributed database by number to obtain a third sort, and sort the second data shards of the target distributed database by number to obtain a fourth sort if the number of distributed nodes between the source distributed database and the target distributed database is different and the total number of the data shards is also different; and establish a mapping relationship between the first data shard and the second data shard in turn according to the third sort and the fourth sort.

[0084] In one embodiment, the mapping module is used to repeatedly use the second data slices to establish a mapping relationship with the remaining first data slices according to the fourth sorting after the mapping relationship is established for all the second data slices if the total amount of the first data slices is greater than the total amount of the second data slices.

[0085] In one embodiment, a mapping module is used for, if the total amount of the first data slices is less than the total amount of the second data slices, after establishing a mapping relationship between the first data slices and the second data slices in turn according to the third sorting and the fourth sorting for the first data table to be transmitted, if there are still second data slices for which a mapping relationship has not been established, then, for the second data table to be transmitted, establishing a mapping relationship between the second data slices ranked after the second data slices for which a mapping relationship has been established and the first data slices according to the third sorting and the fourth sorting.

[0086] The embodiment of the present invention achieves the following technical effects: after each participant receives the insert data subtask, a connection is established between the corresponding second data shard and the first data shard through a mapping relationship based on their own transmission tools, and the table data stored in the first data shard is read and stored on the second data shard, so that the process of the second data shard of each participant reading data from the first data shard does not depend on the coordinator, which is conducive to avoiding the problem of single-point performance bottleneck of the coordinator, and further conducive to making full use of physical resources including CPU, memory and network bandwidth, improving transmission speed and realizing efficient data transmission; at the same time, the above-mentioned data transmission method between distributed databases is based on each data table to be transmitted, that is, concurrent management of multiple data tables is used to carry out data transmission at the same time, further making full use of the advantages of the distributed structure and realizing efficient data transmission.

[0087] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

[0088] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for data transmission between distributed databases, characterized in that: include: For each data table to be transmitted, a mapping relationship between a first data shard of a source distributed database and a second data shard of a target distributed database is established; When the coordinator of the target distributed database receives the data insertion task, the data insertion task is decomposed into a plurality of data insertion subtasks, and the plurality of data insertion subtasks are respectively sent to each participant, wherein one distributed node among the distributed nodes of the target distributed database is the coordinator, and the other distributed nodes are participants; In each of the participants, in response to the insert data subtask, a respective transmission tool is activated, and for each of the second data shards in each of the participants, the first data shard corresponding to the mapping of the second data shard is determined by the transmission tool based on the mapping relationship, and the data of the data table to be transmitted is read from the first data shard corresponding to the mapping and stored on the second data shard, wherein the transmission tool is stored on each distributed node of the target distributed database, and the mapping relationship is stored on each distributed node of the target distributed database; Determining a mapping relationship between a first data shard of a source distributed database and a second data shard of a target distributed database includes: For the source distributed database and the target distributed database, respectively count the number of distributed nodes, the total number of data shards, and the number of data shards on each distributed node; According to the different situations of the number of distributed nodes, the total number of data shards and the number of data shards on each distributed node between the source distributed database and the target distributed database, a mapping relationship between the first data shard of the source distributed database and the second data shard of the target distributed database is determined, wherein each of the first data shards in the mapping relationship maps to one second data shard.

2. The method for data transmission between distributed databases according to claim 1, characterized in that: Determining a mapping relationship between a first data shard of the source distributed database and a second data shard of the target distributed database according to different conditions of the number of distributed nodes, the total number of data shards, and the number of data shards on each distributed node between the source distributed database and the target distributed database includes: If the total number of data shards between the source distributed database and the target distributed database is the same, a one-to-one mapping relationship between the first data shard and the second data shard is established in ascending order of the data shard numbers in the source distributed database and the target distributed database.

3. The data transmission method between distributed databases according to claim 1, characterized in that: Determining a mapping relationship between a first data shard of the source distributed database and a second data shard of the target distributed database according to different conditions of the number of distributed nodes, the total number of data shards, and the number of data shards on each distributed node between the source distributed database and the target distributed database includes: If the number of distributed nodes between the source distributed database and the target distributed database is the same, and the number of data shards of each distributed node in the source distributed database and the target distributed database is the same, the distributed nodes of the source distributed database are sorted according to the host name to obtain a first sort, and the distributed nodes of the target distributed database are sorted according to the host name to obtain a second sort; A mapping relationship is established between the first data shards and the second data shards of the distributed nodes with the same ranking in the first sorting and the second sorting.

4. The data transmission method between distributed databases according to claim 1, characterized in that: Determining a mapping relationship between a first data shard of the source distributed database and a second data shard of the target distributed database according to different conditions of the number of distributed nodes, the total number of data shards, and the number of data shards on each distributed node between the source distributed database and the target distributed database includes: If the number of distributed nodes between the source distributed database and the target distributed database is different and the total number of the data shards is also different, the first data shards of the source distributed database are sorted by number to obtain a third sort, and the second data shards of the target distributed database are sorted by number to obtain a fourth sort; A mapping relationship is established between the first data slice and the second data slice in turn according to the third sorting and the fourth sorting.

5. The method for data transmission between distributed databases as claimed in claim 4, characterized in that: Establishing a mapping relationship between the first data slice and the second data slice in turn according to the third sorting and the fourth sorting includes: If the total amount of the first data slices is greater than the total amount of the second data slices, after the mapping relationship is established for all the second data slices, the second data slices are repeatedly used to establish the mapping relationship with the remaining first data slices according to the fourth sorting.

6. The method for data transmission between distributed databases as claimed in claim 4, characterized in that: Establishing a mapping relationship between the first data slice and the second data slice in turn according to the third sorting and the fourth sorting includes: If the total amount of the first data slices is less than the total amount of the second data slices, for the first data table to be transmitted, after establishing a mapping relationship between the first data slices and the second data slices in turn according to the third sorting and the fourth sorting, if there are still second data slices for which a mapping relationship has not been established, then for the second data table to be transmitted, a mapping relationship is established between the second data slices ranked after the second data slices for which a mapping relationship has been established and the first data slices according to the third sorting and the fourth sorting.

7. A data transmission device between distributed databases, characterized in that: include: A mapping module, used to establish a mapping relationship between a first data shard of a source distributed database and a second data shard of a target distributed database for each data table to be transmitted; A task distribution module, configured to decompose the data insertion task into a plurality of data insertion subtasks when the coordinator of the target distributed database receives the data insertion task, and send the plurality of data insertion subtasks to each participant respectively, wherein one distributed node among the distributed nodes of the target distributed database is the coordinator, and the other distributed nodes are participants; A data transmission module, configured to activate a respective transmission tool in each of the participants in response to the insert data subtask, determine, for each of the second data shards in each of the participants, the first data shard corresponding to the mapping of the second data shard through the transmission tool based on the mapping relationship, and read the data of the data table to be transmitted from the first data shard corresponding to the mapping and store it on the second data shard, wherein the transmission tool is stored on each distributed node of the target distributed database, and the mapping relationship is stored on each distributed node of the target distributed database; The mapping module is used to count the number of distributed nodes, the total number of data shards, and the number of data shards on each distributed node for the source distributed database and the target distributed database, respectively; according to the different situations of the number of distributed nodes, the total number of data shards, and the number of data shards on each distributed node between the source distributed database and the target distributed database, determine the mapping relationship between the first data shard of the source distributed database and the second data shard of the target distributed database, in which each of the first data shards in the mapping relationship maps to one second data shard.

8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for transmitting data between distributed databases according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the data transmission method between distributed databases according to any one of claims 1 to 6.

Images