Data transmission method and apparatus

Abstract

The embodiment of the application discloses a data transmission method and device, which is applied to a subscription end device in a data transmission system, the data transmission system further comprises a source end device, a consumer end device and a target end device, the method comprises the following steps: obtaining a to-be-synchronized data set from the source end device; transmitting the to-be-synchronized data set to the consumer end device in a two-phase commit mode, and instructing the consumer end device to hash the to-be-synchronized data set into a plurality of files of a WAL, and transmitting the plurality of files to the target end device, so that the to-be-synchronized data set is transmitted to the consumer end device in the two-phase commit mode, thereby ensuring that the to-be-synchronized data set is successfully transmitted, and the to-be-synchronized data set is transmitted to the plurality of files of the WAL, thereby improving the data synchronization transmission efficiency.

Description

Technical Field

[0001] This application relates to the field of electronic technology, specifically to a data transmission method and apparatus. Background Technology

[0002] Existing data synchronization technologies rely on storage services, such as the structured query language (MySQL) itself, to ensure the orderliness of data during synchronization. MySQL is a relational database management system; relational databases store data in different tables. The structured query language (SQL) used by MySQL is the most commonly used standardized language for accessing databases. During synchronization, a globally unique ordering marker is assigned to ensure the orderliness of the data.

[0003] However, due to its over-reliance on the characteristics of the storage service itself, the data transmission process relies on a globally unique ordered tag, resulting in single-threaded synchronous input and output of data, which affects the efficiency of the data transmission process. Summary of the Invention

[0004] This application provides a data transmission method and apparatus that can ensure no data loss and improve data transmission efficiency during data synchronization.

[0005] In a first aspect, embodiments of this application provide a data transmission method applied to a subscription device in a data transmission system, wherein the data transmission system further includes a source device, a consumer device, and a target device, and the method includes:

[0006] Obtain the set of data to be synchronized from the source device;

[0007] The data set to be synchronized is transmitted to the consumer device through a two-phase commit method, and the consumer device is instructed to hash the data set to be synchronized to multiple files in WAL, and then transmit the multiple files to the target device.

[0008] Secondly, embodiments of this application provide a data transmission method applied to a consumer device in a data transmission system, wherein the data transmission system further includes a source device, a subscriber device, and a target device, and the method includes:

[0009] The subscription device receives a set of data to be synchronized via a two-phase commit method, wherein the set of data to be synchronized is data obtained by the subscription device from the source device.

[0010] The set of data to be synchronized is hashed into multiple files in the WAL;

[0011] The multiple files are transferred to the target device.

[0012] Thirdly, embodiments of this application provide a data transmission apparatus applied to a subscription end device in a data transmission system. The data transmission system further includes a source end device, a consumer end device, and a target end device. The data transmission apparatus includes:

[0013] The first acquisition unit is used to acquire a set of data to be synchronized from the source device;

[0014] The first transmission unit is used to transmit the set of data to be synchronized to the consumer device through a two-phase commit method, and instruct the consumer device to hash the set of data to be synchronized to multiple files in WAL, and transmit the multiple files to the target device.

[0015] Fourthly, embodiments of this application provide a data transmission apparatus applied to a consumer device in a data transmission system. The data transmission system further includes a source device, a subscriber device, and a target device. The data transmission apparatus includes:

[0016] The receiving unit is configured to receive a set of data to be synchronized transmitted by the subscribing device in a two-phase commit manner, wherein the set of data to be synchronized is data obtained by the subscribing device from the source device;

[0017] A hash unit is used to hash the set of data to be synchronized to multiple files in the WAL;

[0018] The second transmission unit is used to transmit the plurality of files to the target terminal device.

[0019] Fifthly, embodiments of this application provide a subscription terminal device, including a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the programs include instructions for performing the steps in the first aspect of embodiments of this application.

[0020] In a sixth aspect, embodiments of this application provide a consumer device, including a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the programs include instructions for performing the steps in the first aspect of embodiments of this application.

[0021] In a seventh aspect, embodiments of this application provide a computer-readable storage medium storing a computer program for electronic data interchange, wherein the computer program causes a computer to perform some or all of the steps described in the first aspect of embodiments of this application.

[0022] Eighthly, embodiments of this application provide a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps described in the first aspect of embodiments of this application. The computer program product may be a software installation package. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1A This is a schematic diagram of the architecture of a data transmission system for implementing a data transmission method, provided in an embodiment of this application.

[0025] Figure 1B This is a flowchart illustrating a data transmission method disclosed in an embodiment of this application;

[0026] Figure 2 This is a flowchart illustrating another data transmission method disclosed in an embodiment of this application;

[0027] Figure 3 This is a flowchart illustrating another data transmission method disclosed in an embodiment of this application;

[0028] Figure 4A This is a schematic diagram of another subscription terminal device disclosed in an embodiment of this application;

[0029] Figure 4B This is a schematic diagram of the structure of another consumer device disclosed in an embodiment of this application;

[0030] Figure 5 This is a schematic diagram of the structure of a data transmission device disclosed in an embodiment of this application;

[0031] Figure 6A This is a schematic diagram of another data transmission device disclosed in an embodiment of this application;

[0032] Figure 6B This is disclosed in the embodiments of this application. Figure 6A The data transmission device shown is a modified structure. Detailed Implementation

[0033] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0034] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0035] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0036] To facilitate a better understanding of the technical solutions described in this invention, the technical terms involved in the embodiments of this invention are explained below:

[0037] MySQL is a relational database management system. Relational databases store data in different tables.

[0038] Two-phase commit (2PC): An algorithm designed to ensure consistency among all nodes in a distributed system architecture when committing transactions.

[0039] Write-Ahead Logging (WAL): A highly efficient logging algorithm. For non-in-memory databases, disk I / O operations are a major bottleneck to database efficiency. With the same amount of data, a database system using WAL performs only about half the disk write operations of a traditional rollback log system during transaction commits, significantly improving the efficiency of database disk I / O operations and thus enhancing database performance.

[0040] The embodiments of this application will be described in detail below.

[0041] The data transmission method provided in this application is applicable to scenarios involving cross-data center data synchronization. Please refer to... Figure 1A This is a schematic diagram of the architecture of a data transmission system for implementing a data transmission method according to an embodiment of the present invention. The data transmission system may include a source device 1001, a subscriber device 1002, a consumer device 1003, and a target device 1004. The source device is the device of the data publisher, the subscriber device is the device of the data subscriber, the consumer device is the device of the data consumer, and the target device is the device of the data synchronization and storage provider. The source device is connected to the subscriber device, the subscriber device is also connected to the consumer device, and the consumer device is also connected to the target device. The source device 1001, subscriber device 1002, consumer device 1003, and target device 1004 correspond to different data centers. When the source device has a set of data to be synchronized, the remote device can transmit the set of data to be synchronized to the subscriber device. The subscriber device then transmits the set of data to be synchronized to the consumer device via a two-phase commit. The consumer device then hashes the set of data to be synchronized into multiple files in the WAL (Write-Ahead Allocation) and transmits these multiple files to the target device. This enables cross-data center synchronous data transmission, ensuring data synchronization between different data centers in multiple locations. The set of data to be synchronized may include at least one piece of data to be synchronized.

[0042] Please see Figure 1B , Figure 1B This is a flowchart illustrating a data transmission method disclosed in an embodiment of this application, applied to a subscription device in a data transmission system. The data transmission system further includes a source device, a consumer device, and a target device. The data transmission method includes the following steps:

[0043] 101. Obtain the set of data to be synchronized from the source device.

[0044] In this embodiment of the application, when there is data to be synchronized in the remote device that needs to be transmitted synchronously, the remote device can transmit the set of data to be synchronized, including at least one set of data to be synchronized, to the subscription device, so that the subscription device can obtain the set of data to be synchronized from the source device.

[0045] Optionally, in step 101 above, obtaining the set of data to be synchronized from the source device may include the following steps:

[0046] The set of data to be synchronized transmitted by the source device is obtained through a single thread.

[0047] During data synchronization, each piece of data in the set to be synchronized carries a sequence identifier representing the transmission order. Therefore, each piece of data to be synchronized transmitted from the source device can be retrieved sequentially through a single thread, according to its corresponding sequence identifier. Specifically, by retrieving the set of data to be synchronized transmitted from the source device through a single thread, the data can be retrieved according to the time order represented by the sequence identifier.

[0048] 102. The set of data to be synchronized is transmitted to the consumer device through a two-phase commit method, and the consumer device is instructed to hash the set of data to be synchronized to multiple files in WAL, and transmit the multiple files to the target device.

[0049] In this system, both the subscriber and consumer devices can determine the success of their own operations, but they cannot determine the success of operations performed by other devices. Using a two-phase commit approach, the subscriber device sends an inquiry message to the consumer device in the first phase, asking whether it can send the data set to be synchronized. If the subscriber device receives feedback from the consumer device within a preset time, it proceeds to the second phase, where it transmits the data set to be synchronized to the consumer device. If the subscriber device does not receive feedback from the consumer device within the preset time, it sends a rollback message to the consumer device, thus ensuring successful transmission of the data set to be synchronized to the consumer device. The subscriber device can instruct the consumer device to hash the data set to be synchronized into multiple files within the WAL (Write-Ahead Allocation) and transmit these multiple files to the target device. This method of transmitting the data set to be synchronized across multiple files within the WAL improves the efficiency of synchronization data transmission.

[0050] As can be seen, the data transmission method described in this application embodiment obtains the data set to be synchronized from the source device; transmits the data set to be synchronized to the consumer device through a two-phase commit, and instructs the consumer device to hash the data set to be synchronized to multiple files in the WAL, and transmits the multiple files to the target device. In this way, the data set to be synchronized is transmitted to the consumer device through a two-phase commit, thereby ensuring the successful transmission of the data set to be synchronized. In addition, the data synchronization transmission efficiency is improved by transmitting the data set to be synchronized to multiple files in the WAL.

[0051] Consistent with the above, Figure 2 This is a flowchart illustrating a data transmission method disclosed in an embodiment of this application. The data transmission method is applied to a consumer device in a data transmission system, which further includes a source device, a subscriber device, and a target device. The data transmission method includes the following steps:

[0052] 201. Receive the set of data to be synchronized transmitted by the subscriber device through a two-phase commit method, wherein the set of data to be synchronized is data obtained by the subscriber device from the source device.

[0053] In this process, both the subscriber and consumer devices can determine the success of their own operations, but they cannot determine the success of other devices' operations. Through a two-phase commit method, the subscriber device can send an inquiry message to the consumer device in the first phase, asking whether it can send the set of data to be synchronized. If the subscriber device receives a feedback message from the consumer device within a preset time, it enters the second phase. In the second phase, the subscriber device transmits the set of data to be synchronized to the consumer device. If the subscriber device does not receive a feedback message from the consumer device within the preset time, it sends a rollback message to the consumer device, thereby ensuring that the consumer device successfully receives the set of data to be synchronized.

[0054] 202. Hash the set of data to be synchronized into multiple files of WAL.

[0055] WAL (Write-Ahead Allocation) can significantly improve efficiency for non-memory data while ensuring data integrity. Therefore, by using WAL to store the data to be synchronized in a set, distributing the set across multiple files in the WAL, and then transferring these files to the target device, the efficiency of synchronous data transmission can be improved. Specifically, data in the set to be synchronized can be written to multiple files through multiple data transmission channels, thereby increasing disk I / O utilization and improving file write speed.

[0056] Optionally, the set of data to be synchronized includes multiple sets of data to be synchronized, each set of data to be synchronized corresponding to a routing information. In step 202 above, hashing the set of data to be synchronized to multiple files in the WAL may include the following steps:

[0057] 21. Divide the multiple data to be synchronized into multiple groups according to the multiple routing information corresponding to the multiple data to be synchronized, and the data to be synchronized in each group corresponds to the same routing information;

[0058] 22. Write the data to be synchronized for each of the multiple groups into the file corresponding to that group in the WAL, thus obtaining multiple files.

[0059] The aforementioned routing information can be the device identifier of the routing device corresponding to the data to be synchronized. Based on the routing information of the data to be synchronized, data with the same routing information can be grouped into a single group. Then, the data to be synchronized in each group is written to a file corresponding to that group. Thus, the data to be synchronized in multiple groups can be written to multiple files, with each group corresponding to a specific file. In this way, the number of files can be dynamically adjusted based on the routing information. Different numbers of files result in different transmission rates for the data to be synchronized, thereby allowing for dynamic adjustment of the transmission rate of the data set to be synchronized.

[0060] Optionally, in step 22 above, writing the data to be synchronized for each of the multiple groups into the file corresponding to that group in the WAL to obtain multiple files may include the following steps:

[0061] 2201. If the number of data types of all data to be synchronized in group i is greater than a preset value, the data to be synchronized in group i is divided into multiple subgroups, and the data to be synchronized in each subgroup corresponds to the same data type, wherein group i is any one of the multiple groups;

[0062] 2202. Write the data to be synchronized of each subgroup in the multiple subgroups of the group i into the file corresponding to the subgroup in the WAL, to obtain multiple files.

[0063] Specifically, for any group i among multiple groups, if the number of data types of all data to be synchronized in group i is greater than a preset value, it indicates that there are many data types in group i and a large amount of data in group i. Therefore, all data to be synchronized in group i can be divided into different subgroups based on data type. Specifically, data to be synchronized in group i with the same data type is grouped into the same subgroup. Then, the data to be synchronized in each subgroup is written to the file corresponding to that subgroup, thus allowing data to be synchronized in the same subgroup to be written to a single file. In this way, the number of files can be dynamically adjusted according to data type. When there are many data types in group i, the number of corresponding files increases, and the transmission rate of the data to be synchronized also varies, thus allowing for dynamic adjustment of the transmission rate of the data set to be synchronized.

[0064] Optionally, the group i includes m data to be synchronized. In this embodiment, the following steps may also be included:

[0065] Obtain the type identifiers of the m data to be synchronized in group i, thus obtaining m type identifiers;

[0066] By counting the m type identifiers, the number n of all data types to be synchronized in group i is obtained, where n is less than or equal to m.

[0067] Before step 2201, the number n of data types of all data to be synchronized in group i can be determined. Specifically, the type identifiers of m data to be synchronized in group i can be obtained, resulting in m type identifiers. Each data to be synchronized corresponds to a type identifier. Different data to be synchronized may correspond to the same type identifier or different type identifiers. Thus, the number n of type identifiers can be counted.

[0068] Optionally, in step 2202 above, writing the data to be synchronized from each of the multiple sub-groups of group i into the file corresponding to that sub-group in the WAL to obtain multiple files may include the following steps:

[0069] By writing files sequentially, the data to be synchronized from each of the multiple subgroups of group i is written to a file in the WAL, resulting in multiple files.

[0070] In this embodiment of the application, when writing the data to be synchronized to a file, writing the file in sequence can avoid a large amount of disk seek time, thereby improving the file writing speed.

[0071] 203. Transfer the plurality of files to the target terminal device.

[0072] Consumer devices can transfer multiple files to target devices. Through two-phase commit, the data set to be synchronized at the data source end is consistent with the data set to be transmitted to the target device, thereby ensuring that the data set to be synchronized is not lost during the transmission process.

[0073] Optionally, in step 203 above, transferring the plurality of files to the target terminal device may include the following steps:

[0074] The multiple files are transferred to the target device asynchronously using multiple threads.

[0075] By using asynchronous multiple threads to transfer multiple files to the target device, the CPU utilization of the consumer device and the transfer rate of multiple files can be improved.

[0076] As can be seen, the data transmission method described in this application embodiment receives the set of data to be synchronized transmitted by the subscriber device through a two-phase commit, wherein the set of data to be synchronized is data obtained by the subscriber device from the source device; the set of data to be synchronized is hashed into multiple files of WAL; and the multiple files are transmitted to the target device. In this way, the set of data to be synchronized is transmitted to the consumer device through a two-phase commit, thereby ensuring the successful transmission of the set of data to be synchronized, and the data synchronization transmission efficiency is improved by transmitting the set of data to be synchronized into multiple files of WAL.

[0077] Consistent with the above, Figure 3 This is a flowchart illustrating a data transmission method disclosed in an embodiment of this application. Applied to... Figure 1A The data transmission system shown may include a source device, a subscriber device, a consumer device, and a target device. The data transmission method includes the following steps:

[0078] 301. The source device sends a set of data to be synchronized to the subscriber device through a single thread. The set of data to be synchronized includes multiple sets of data to be synchronized, and each set of data to be synchronized corresponds to a routing information.

[0079] 302. The subscription device transmits the set of data to be synchronized to the consumer device through a two-phase commit.

[0080] 303. The consumer device divides the multiple data to be synchronized into multiple groups according to the multiple routing information corresponding to the multiple data to be synchronized, and the data to be synchronized in each group corresponds to the same routing information.

[0081] 304. The consumer device writes the data to be synchronized for each of the multiple groups into the file corresponding to that group in the WAL in a sequential manner, thereby obtaining multiple files.

[0082] 305. The consumer device transmits the multiple files to the target device asynchronously through multiple threads.

[0083] The specific descriptions of steps 301-305 above can be found in [reference needed]. Figure 1B and Figure 2 The corresponding description of the data transmission method is omitted here.

[0084] As can be seen, the data transmission method described in this application involves the source device sending a set of data to be synchronized to the subscriber device via a single thread. The set of data to be synchronized includes multiple sets of data, each corresponding to a routing information. The subscriber device transmits the set of data to be synchronized to the consumer device using a two-phase commit. The consumer device divides the multiple sets of data to be synchronized into multiple groups based on the multiple routing information corresponding to the multiple sets of data to be synchronized. The data to be synchronized in each group corresponds to the same routing information. The consumer device writes the data to be synchronized in each group to the file corresponding to that group in the WAL in a sequential file writing manner, resulting in multiple files. The consumer device transmits the multiple files to the target device asynchronously via multiple threads. In this way, the set of data to be synchronized is transmitted to the consumer device through a two-phase commit, thereby ensuring successful transmission of the set of data to be synchronized. Furthermore, by transmitting the set of data to be synchronized to multiple files in the WAL, the data synchronization transmission efficiency is improved.

[0085] Please see Figure 4A , Figure 4A This is a schematic diagram of the structure of a subscription terminal device disclosed in an embodiment of this application. As shown in the figure, the subscription terminal device includes a processor, a memory, a communication interface, and one or more programs. The one or more programs are stored in the memory and configured to be executed by the processor. The programs include instructions for performing the following steps:

[0086] Obtain the set of data to be synchronized from the source device;

[0087] The set of data to be synchronized is transmitted to the consumer device through a two-phase commit method, and the consumer device is instructed to hash the set of data to be synchronized to multiple files in the Write-Ahead Log (WAL), and then transmit the multiple files to the target device.

[0088] In one possible example, regarding the acquisition of the set of data to be synchronized from the source device, the above procedure specifically includes instructions to perform the following steps:

[0089] The set of data to be synchronized transmitted by the source device is obtained through a single thread.

[0090] Please see Figure 4B , Figure 4B This is a schematic diagram of the structure of a consumer device disclosed in an embodiment of this application. As shown in the figure, the consumer device includes a processor, a memory, a communication interface, and one or more programs. The one or more programs are stored in the memory and configured to be executed by the processor. The programs include instructions for performing the following steps:

[0091] The subscription device receives a set of data to be synchronized via a two-phase commit method, wherein the set of data to be synchronized is data obtained by the subscription device from the source device.

[0092] The set of data to be synchronized is hashed into multiple files in the WAL;

[0093] The multiple files are transferred to the target device.

[0094] In one possible example, the set of data to be synchronized includes multiple sets of data to be synchronized, each set of data corresponding to a routing information. Regarding the hashing of the set of data to be synchronized to multiple files in the WAL, the above procedure includes instructions for performing the following steps:

[0095] The multiple data to be synchronized are divided into multiple groups according to the multiple routing information corresponding to the multiple data to be synchronized, and the data to be synchronized in each group corresponds to the same routing information.

[0096] The data to be synchronized for each of the multiple groups is written into the file corresponding to that group in the Write-Ahead Log (WAL), resulting in multiple files.

[0097] In one possible example, regarding the step of writing the data to be synchronized for each of the plurality of groups into the file corresponding to that group in the write-ahead log (WAL), resulting in multiple files, the above procedure includes instructions for performing the following steps:

[0098] If the number of data types of all data to be synchronized in group i is greater than a preset value, the data to be synchronized in group i is divided into multiple subgroups, and the data to be synchronized in each subgroup corresponds to the same data type, wherein group i is any one of the multiple groups;

[0099] The data to be synchronized for each of the multiple subgroups of group i is written into the file corresponding to that subgroup in the WAL, resulting in multiple files.

[0100] In one possible example, the group i includes m data items to be synchronized, and the above program also includes instructions for performing the following steps:

[0101] Obtain the type identifiers of the m data to be synchronized in group i, thus obtaining m type identifiers;

[0102] By counting the m type identifiers, the number n of all data types to be synchronized in group i is obtained, where n is less than or equal to m.

[0103] In one possible example, regarding the process of writing the data to be synchronized from each of the multiple sub-groups of group i into a file in the WAL to obtain multiple files, the above procedure includes instructions for performing the following steps:

[0104] By writing files sequentially, the data to be synchronized from each of the multiple subgroups of group i is written to a file in the WAL, resulting in multiple files.

[0105] The above primarily describes the solutions of the embodiments of this application from the perspective of the method execution process. It is understood that, in order to achieve the above functions, the electronic device includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments provided herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0106] This application embodiment can divide the electronic device into functional units according to the above method example. For example, each function can be divided into a separate functional unit, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0107] Please see Figure 5 , Figure 5 This is a schematic diagram of a data transmission device disclosed in an embodiment of this application, applied to a subscription end device in a data transmission system. The data transmission system further includes a source end device, a consumer end device, and a target end device. The data transmission device includes a first acquisition unit 501 and a first transmission unit 502, wherein...

[0108] The first acquisition unit 501 is used to acquire a set of data to be synchronized from the source device;

[0109] The first transmission unit 502 is used to transmit the set of data to be synchronized to the consumer device in a two-phase commit manner, and instruct the consumer device to hash the set of data to be synchronized to multiple files in WAL, and transmit the multiple files to the target device.

[0110] Optionally, in terms of obtaining the set of data to be synchronized from the source device, the first acquisition unit 501 is specifically used for:

[0111] The set of data to be synchronized transmitted by the source device is obtained through a single thread.

[0112] As can be seen, the data transmission device described in this application obtains the data set to be synchronized from the source device; transmits the data set to be synchronized to the consumer device through a two-phase commit method; instructs the consumer device to hash the data set to be synchronized to multiple files in the WAL; and transmits the multiple files to the target device. In this way, the data set to be synchronized is transmitted to the consumer device through a two-phase commit method, thereby ensuring the successful transmission of the data set to be synchronized. Furthermore, by transmitting the data set to be synchronized to multiple files in the WAL, the efficiency of data synchronization transmission is improved.

[0113] It should be noted that the subscription device described in the embodiments of this application is presented in the form of functional units. The term "unit" as used herein should be understood in the broadest possible sense, and the object used to implement the functions described in each "unit" may be, for example, an integrated circuit ASIC, a single circuit, a processor (shared, dedicated, or chipset) and memory for executing one or more software or firmware programs, combinational logic circuits, and / or other suitable components that provide the above functions.

[0114] The first acquisition unit 501 and the first transmission unit 502 may be control circuits or processors.

[0115] Please see Figure 6A , Figure 6A This is a schematic diagram of a data transmission device disclosed in an embodiment of this application, applied to a consumer device in a data transmission system. The data transmission system further includes a source device, a subscriber device, and a target device. The data transmission device includes a receiving unit 601, a hashing unit 602, and a second transmission unit 603.

[0116] The receiving unit 601 is used to receive the set of data to be synchronized transmitted by the subscription device in a two-phase commit manner, wherein the set of data to be synchronized is data obtained by the subscription device from the source device;

[0117] The hashing unit 602 is used to hash the set of data to be synchronized to multiple files in the WAL;

[0118] The second transmission unit 603 is used to transmit the plurality of files to the target terminal device.

[0119] Optionally, the set of data to be synchronized includes multiple sets of data to be synchronized, each set of data to be synchronized corresponding to a routing information. In the process of hashing the set of data to be synchronized to multiple files in the WAL, the hashing unit 602 is specifically used for:

[0120] The multiple data to be synchronized are divided into multiple groups according to the multiple routing information corresponding to the multiple data to be synchronized, and the data to be synchronized in each group corresponds to the same routing information.

[0121] The data to be synchronized for each of the multiple groups is written into the file corresponding to that group in the Write-Ahead Log (WAL), resulting in multiple files.

[0122] Optionally, in the step of writing the data to be synchronized from each of the plurality of groups into the file corresponding to that group in the Write-Ahead Log (WAL) to obtain multiple files, the hash unit is specifically used for:

[0123] If the number of data types of all data to be synchronized in group i is greater than a preset value, the data to be synchronized in group i is divided into multiple subgroups, and the data to be synchronized in each subgroup corresponds to the same data type, wherein group i is any one of the multiple groups;

[0124] The data to be synchronized for each of the multiple subgroups of group i is written into the file corresponding to that subgroup in the WAL, resulting in multiple files.

[0125] Optionally, the group i includes m data items to be synchronized, such as... Figure 6B , Figure 6B for Figure 6A The variant structure of the data transmission device shown is similar to... Figure 6A In comparison, it may also include: a second acquisition unit 604 and a statistics unit 605, wherein,

[0126] The second acquisition unit 604 is used to acquire the type identifiers of the m data to be synchronized in the group i, and obtain m type identifiers;

[0127] The statistical unit 605 is used to count the m type identifiers to obtain the number n of data types of all data to be synchronized in group i, where n is less than or equal to m.

[0128] Optionally, in the step of writing the data to be synchronized from each of the multiple sub-groups of the group i into a file in the WAL to obtain multiple files, the hash unit is specifically used for:

[0129] By writing files sequentially, the data to be synchronized from each of the multiple subgroups of group i is written to a file in the WAL, resulting in multiple files.

[0130] Optionally, in the process of transferring the plurality of files to the target terminal device, the second transmission unit is specifically used for:

[0131] The multiple files are transferred to the target device asynchronously using multiple threads.

[0132] As can be seen, the data transmission device described in this application embodiment receives a set of data to be synchronized transmitted by the subscriber device through a two-phase commit method. The set of data to be synchronized is data obtained by the subscriber device from the source device. The set of data to be synchronized is hashed into multiple files of WAL. The multiple files are then transmitted to the target device. In this way, the set of data to be synchronized is transmitted to the consumer device through a two-phase commit method, thereby ensuring the successful transmission of the set of data to be synchronized. Furthermore, by transmitting the set of data to be synchronized into multiple files of WAL, the data synchronization transmission efficiency is improved.

[0133] It should be noted that the consumer devices described in the embodiments of this application are presented in the form of functional units. The term "unit" as used herein should be understood in the broadest possible sense, and the object used to implement the functions described in each "unit" may be, for example, an integrated circuit ASIC, a single circuit, a processor (shared, dedicated, or chipset) and memory for executing one or more software or firmware programs, combinational logic circuits, and / or other suitable components that provide the above functions.

[0134] The receiving unit 601, hashing unit 602, second transmission unit 603, second acquisition unit 604, and statistics unit 605 may be control circuits or processors.

[0135] This application also provides a computer storage medium storing a computer program for electronic data interchange, which causes a computer to perform some or all of the steps of any of the data transmission methods described in the above method embodiments.

[0136] This application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of any of the data transmission methods described in the above method embodiments.

[0137] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0138] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0139] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0140] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0141] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software program module.

[0142] If the integrated unit is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0143] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage device, which may include a flash drive, ROM, RAM, disk, or optical disk, etc.

[0144] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data transmission method, characterized in that, A subscription device is applied in a data transmission system, the data transmission system further comprising a source device, a consumer device, and a target device, wherein the source device, the subscription device, the consumer device, and the target device correspond to different data centers, and the method includes: The source device obtains a set of data to be synchronized, which includes multiple data to be synchronized, and each data to be synchronized corresponds to a routing information. The set of data to be synchronized is transmitted to the consumer device through a two-phase commit method, and the consumer device is instructed to hash the set of data to be synchronized to multiple files in the Write-Ahead Log (WAL), and then transmit the multiple files to the target device. The consumer device is used to: divide the multiple data to be synchronized into multiple groups according to the multiple routing information corresponding to the multiple data to be synchronized, wherein the data to be synchronized in each group corresponds to the same routing information; and write the data to be synchronized in each of the multiple groups into the file corresponding to the group in the WAL to obtain the multiple files.

2. The method according to claim 1, characterized in that, The step of obtaining the set of data to be synchronized from the source device includes: The set of data to be synchronized transmitted by the source device is obtained through a single thread.

3. A data transmission method, characterized in that, A consumer-end device applied in a data transmission system, the data transmission system further comprising a source-end device, a subscriber-end device, and a target-end device, wherein the source-end device, the subscriber-end device, the consumer-end device, and the target-end device correspond to different data centers, the method comprising: The subscription device receives a set of data to be synchronized via a two-phase commit method, wherein the set of data to be synchronized is data obtained by the subscription device from the source device. The set of data to be synchronized is hashed into multiple files in the WAL (Write-Ahead Allocation) system. The set of data to be synchronized includes multiple pieces of data to be synchronized, and each piece of data to be synchronized corresponds to a routing information. The multiple pieces of data to be synchronized are divided into multiple groups according to the multiple routing information corresponding to the multiple pieces of data to be synchronized. The data to be synchronized in each of the multiple groups is written into the file corresponding to that group in the WAL system, resulting in multiple files. The multiple files are transferred to the target device.

4. The method according to claim 3, characterized in that, The step involves writing the data to be synchronized from each of the multiple groups into the file corresponding to that group in the WAL, resulting in multiple files, including: If the number of data types of all data to be synchronized in group i is greater than a preset value, the data to be synchronized in group i is divided into multiple subgroups, and the data to be synchronized in each subgroup corresponds to the same data type, wherein group i is any one of the multiple groups; The data to be synchronized for each of the multiple subgroups of group i is written into the file corresponding to that subgroup in the WAL, resulting in multiple files.

5. The method according to claim 4, characterized in that, The group i includes m data items to be synchronized, and the method further includes: Obtain the type identifiers of the m data to be synchronized in group i, thus obtaining m type identifiers; By counting the m type identifiers, the number n of all data types to be synchronized in group i is obtained, where n is less than or equal to m.

6. The method according to claim 4 or 5, characterized in that, The step of writing the data to be synchronized from each of the multiple sub-groups of group i into a file in the WAL results in multiple files, including: By writing files sequentially, the data to be synchronized from each of the multiple subgroups of group i is written to a file in the WAL, resulting in multiple files.

7. The method according to claim 6, characterized in that, The step of transferring the plurality of files to the target terminal device includes: The multiple files are transferred to the target device asynchronously using multiple threads.

8. A data transmission device, characterized in that, A subscription device is used in a data transmission system, the data transmission system further including a source device, a consumer device, and a target device, wherein the source device, the subscription device, the consumer device, and the target device correspond to different data centers, and the data transmission device includes: The first acquisition unit is used to acquire a set of data to be synchronized from the source device. The set of data to be synchronized includes multiple data to be synchronized, and each data to be synchronized corresponds to a routing information. The first transmission unit is used to transmit the set of data to be synchronized to the consumer device through a two-phase commit method, and instruct the consumer device to hash the set of data to be synchronized to multiple files in WAL, and transmit the multiple files to the target device. The consumer device is used to: divide the multiple data to be synchronized into multiple groups according to the multiple routing information corresponding to the multiple data to be synchronized, wherein the data to be synchronized in each group corresponds to the same routing information; and write the data to be synchronized in each of the multiple groups into the file corresponding to the group in the WAL to obtain the multiple files.

9. The data transmission device according to claim 8, characterized in that, In the process of obtaining the set of data to be synchronized from the source device, the first obtaining unit is specifically used for: The set of data to be synchronized transmitted by the source device is obtained through a single thread.

10. A data transmission device, characterized in that, A consumer-end device applied in a data transmission system, the data transmission system further comprising a source-end device, a subscriber-end device, and a target-end device, wherein the source-end device, the subscriber-end device, the consumer-end device, and the target-end device correspond to different data centers, the data transmission device comprising: The receiving unit is configured to receive a set of data to be synchronized transmitted by the subscribing device in a two-phase commit manner, wherein the set of data to be synchronized is data obtained by the subscribing device from the source device; A hashing unit is used to hash the set of data to be synchronized to multiple files in the WAL (Write-Ahead Allocation) system. The set of data to be synchronized includes multiple pieces of data to be synchronized, each piece of data to be synchronized corresponds to a routing information. The multiple pieces of data to be synchronized are divided into multiple groups according to the multiple routing information corresponding to the multiple pieces of data to be synchronized. The data to be synchronized in each of the multiple groups is written into the file corresponding to that group in the WAL system, resulting in multiple files. The second transmission unit is used to transmit the plurality of files to the target terminal device.

11. The data transmission apparatus according to claim 10, characterized in that, In the process of writing the data to be synchronized from each of the multiple groups into the file corresponding to that group in the Write-Ahead Log (WAL) to obtain multiple files, the hash unit is specifically used for: If the number of data types of all data to be synchronized in group i is greater than a preset value, the data to be synchronized in group i is divided into multiple subgroups, and the data to be synchronized in each subgroup corresponds to the same data type, wherein group i is any one of the multiple groups; The data to be synchronized for each of the multiple subgroups of group i is written into the file corresponding to that subgroup in the WAL, resulting in multiple files.

12. The data transmission apparatus according to claim 11, characterized in that, The group i includes m data items to be synchronized. The device further includes a second acquisition unit and a statistics unit, wherein... The second acquisition unit is used to acquire the type identifiers of the m data to be synchronized in the group i, and obtain m type identifiers; The statistical unit is used to count the m type identifiers to obtain the number n of data types of all data to be synchronized in group i, where n is less than or equal to m.

13. The data transmission apparatus according to claim 11 or 12, characterized in that, In the process of writing the data to be synchronized from each of the multiple sub-groups of group i into a file in the WAL to obtain multiple files, the hash unit is specifically used for: By writing files sequentially, the data to be synchronized from each of the multiple subgroups of group i is written to a file in the WAL, resulting in multiple files.

14. The data transmission apparatus according to claim 13, characterized in that, In the process of transferring the plurality of files to the target terminal device, the second transmission unit is specifically used for: The multiple files are transferred to the target device asynchronously using multiple threads.

15. A subscription terminal device, characterized in that, The device includes a processor, a memory, a communication interface, and one or more programs, said programs being stored in the memory and configured to be executed by the processor, said programs including instructions for performing the steps of the method as described in claim 1 or 2.

16. A consumer-grade device, characterized in that, The device includes a processor, a memory, a communication interface, and one or more programs, said programs being stored in the memory and configured to be executed by the processor, said programs including instructions for performing the steps of the method as described in any one of claims 3-7.

17. A computer-readable storage medium, characterized in that, A computer program for storing electronic data interchange, wherein the computer program causes a computer to perform the method as described in any one of claims 1-7.

18. A computer program product, characterized in that, The computer program product includes a non-transitory computer-readable storage medium storing a computer program that causes a computer to perform the method as described in any one of claims 1-7.

Images