Data synchronization distribution method and device and storage medium

Abstract

The application discloses a data synchronization distribution method and device and a storage medium. The method comprises the following steps: using an open source framework to synchronize and take out to-be-processed data from a relational database, wherein the open source framework is a Canal open source framework developed by using a programming language java; performing grouping processing on the to-be-processed data to obtain a to-be-processed data group; using multi-threading to perform merging processing on the to-be-processed data in the to-be-processed data group, to obtain a merged data group; and performing synchronous distribution processing on the merged data group. The application solves the technical problem that the data distribution method in the prior art cannot guarantee data sequence in a high concurrency case.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically, to a data synchronization and distribution method, apparatus, and storage medium. Background Technology

[0002] With the advancement of science and technology and the continuous development of smart devices, the types and quantities of data have increased significantly. Therefore, all enterprises and institutions need to process data quickly and accurately.

[0003] Existing database synchronization methods cannot guarantee that business data will be executed in the predetermined order and with high concurrency; if a single thread is used to ensure the orderliness of data, the data synchronization rate will be limited.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This invention provides a data synchronization and distribution method, apparatus, and storage medium to at least solve the technical problem that existing data distribution methods cannot guarantee data order under high concurrency.

[0006] According to one aspect of the present invention, a data synchronization and distribution method is provided, comprising: synchronously retrieving data to be processed from a relational database using an open-source framework; grouping the data to be processed into data groups; merging the data to be processed in the data groups using multiple threads to obtain merged data groups; and synchronously distributing the merged data groups.

[0007] In an exemplary embodiment, the above-mentioned grouping of the data to be processed to obtain a data group includes: grouping the data to be processed according to the primary key of the data to be processed and the retrieval order of the data to be processed to obtain a data group, wherein the primary key is used to identify a piece of data to be processed, and the retrieval order is used to determine the order in which the data to be processed is retrieved from the relational database.

[0008] In an exemplary embodiment, after grouping the data to be processed to obtain a data group to be processed, the method further includes: configuring a storage order according to the weight values ​​of the data to be processed; and storing the data group to be processed in a key-value database based on the storage order.

[0009] In an exemplary embodiment, before merging the data to be processed in the data group to be processed using multiple threads to obtain a merged data group, the method further includes: asynchronously retrieving the data group to be processed from the key-value database using the multiple threads.

[0010] In an exemplary embodiment, the above-mentioned use of multiple threads to merge the data to be processed in the data to be processed group to obtain a merged data group includes: merging the data to be processed that are deleted to obtain first merged data; merging the data to be processed that are updated to obtain second merged data; and constructing the merged data group based on the first merged data and the second merged data.

[0011] In one exemplary embodiment, the weight value of the first merged data is determined based on the weight value of the data to be processed whose operation is deletion in the first merged data; the weight value of the second merged data is determined based on the weight value of the data to be processed whose operation is updating in the second merged data; the merged data group is constructed according to the weight values ​​of the first merged data and the weight values ​​of the second merged data, and the weight value of the merged data group is determined.

[0012] In one exemplary embodiment, the above-described synchronous distribution process of the merged data groups includes: synchronous distribution according to the weight values ​​of the multiple merged data groups; if there are merged data groups with the same weight value, then the merged data groups with the same weight value are processed concurrently.

[0013] According to another aspect of the present invention, a data synchronization and distribution apparatus is also provided, comprising: an acquisition module, configured to synchronously retrieve data to be processed from a relational database using an open-source framework; a grouping module, configured to group the data to be processed into groups to obtain a data group to be processed; a merging module, configured to merge the data to be processed in the data group to be processed using multiple threads to obtain a merged data group; and a distribution module, configured to synchronously distribute the merged data group.

[0014] According to another aspect of the present invention, a non-volatile storage medium is also provided, wherein the non-volatile storage medium stores a plurality of instructions, the instructions being adapted to be loaded by a processor and executed any one of the above-described data synchronization and distribution methods.

[0015] According to another aspect of the present invention, a processor is also provided, which is used to run a program, wherein the program is configured to execute any of the above-described data synchronization and distribution methods at runtime.

[0016] According to another aspect of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform any of the above-described data synchronization and distribution methods.

[0017] In this embodiment of the invention, data to be processed is retrieved synchronously from a relational database using an open-source framework; the data to be processed is grouped to obtain a data group to be processed; multi-threading is used to merge the data to be processed in the data group to obtain a merged data group; the merged data group is then synchronously distributed. This achieves the goal of greatly reducing the amount of data to be synchronized by using the idea of ​​sorting within the group and the concept of data merging. This achieves the technical effect of ensuring data order while distributing data under high concurrency, thereby solving the technical problem that existing data distribution methods cannot guarantee data order under high concurrency. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0019] Figure 1 This is a hardware structure block diagram of a computer terminal for a data synchronization and distribution method according to an embodiment of this application;

[0020] Figure 2 This is a flowchart of a data synchronization and distribution method according to an embodiment of this application;

[0021] Figure 3 This is a schematic diagram of a data synchronization and distribution process according to an embodiment of this application;

[0022] Figure 4 This is a structural block diagram of a data synchronization and distribution device according to an embodiment of this application. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] The methods and embodiments provided in this application can be executed on a computer terminal, a computer terminal, or a similar computing module. Taking running on a computer terminal as an example, Figure 1 This is a hardware structure block diagram of a computer terminal for a data synchronization and distribution method according to an embodiment of this application. Figure 1 As shown, a computer terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a processing module such as a microprocessor (MCU) or a programmable logic device (FPGA)) and a memory 104 for storing data are also shown. In one exemplary embodiment, the computer terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the computer terminal described above. For example, the computer terminal may also include components that are more complex than those described above. Figure 1 The more or fewer components shown, or having the same Figure 1 Equivalent functions or ratios shown Figure 1 The functions shown have more different configurations.

[0026] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the data synchronization and distribution method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage modules, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0027] The transmission module 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the computer terminal's communication provider. In one example, the transmission module 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission module 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0028] This embodiment provides a data synchronization and distribution method. Figure 2 This is a flowchart of a data synchronization and distribution method according to an embodiment of this application, the process including the following steps:

[0029] Step S202: Use an open-source framework to synchronously retrieve the data to be processed from the relational database;

[0030] Step S204: The above data to be processed is grouped to obtain data groups to be processed;

[0031] Step S206: Use multi-threading to merge the data to be processed in the above-mentioned data group to obtain a merged data group;

[0032] Step S208: Perform synchronous distribution processing on the above-mentioned merged data group.

[0033] Through the above steps, the data to be processed is retrieved synchronously from the relational database using an open-source framework, namely the Canal open-source framework developed using the Java programming language; the data to be processed is grouped to obtain a data group to be processed; multi-threading is used to merge the data to be processed in the data group to obtain a merged data group; and the merged data group is synchronously distributed.

[0034] It should be noted that the data to be processed retrieved from the relational database is ordered.

[0035] It should also be noted that the open-source framework Canal is developed in Java and supports MySQL binlog parsing. It provides middleware for incremental data subscription and / or consumption based on incremental database log parsing. MySQL is a relational database management system and one of the most commonly used relational database management systems.

[0036] In the embodiments of this application, such as Figure 3The diagram illustrates the data synchronization and distribution process. The open-source framework Canal retrieves data from MySQL and groups it according to the primary key and order of the data. The data within each group is also ordered. Multi-threading is used to merge each group of data. The processed data is then distributed according to weight, business requirements, etc., and the distributed data is saved to a relational database or a non-relational database.

[0037] Through the embodiments of this application, data ordering is guaranteed to a certain extent under high concurrency. Using Redis as a data cache can handle large concurrent programs. Through the idea of ​​group sorting and data merging, the amount of data to be synchronized can be greatly reduced, while ensuring the orderliness of data synchronization.

[0038] In one exemplary embodiment, the above-described grouping of the data to be processed to obtain a data group to be processed includes:

[0039] Step S302: Group the data to be processed according to the primary key and the retrieval order of the data to be processed to obtain a data group to be processed. The primary key is used to identify a piece of data to be processed, and the retrieval order is used to determine the order in which the data to be processed is retrieved from the relational database.

[0040] In this embodiment of the application, the data grouping program groups the data according to the primary key of the data and the order in which the data is retrieved. After grouping, the data within each group is also ordered.

[0041] It should be noted that the primary key mentioned above is used to identify a piece of data to be processed, and the order mentioned above is used to determine the retrieval order of the data to be processed.

[0042] In one exemplary embodiment, after grouping the data to be processed to obtain the data groups, the method further includes:

[0043] Step S402: Configure the storage order according to the weight values ​​of the data to be processed.

[0044] Step S404: Based on the above storage order, store the above data group to be processed into a key-value database.

[0045] In this embodiment of the application, the grouped data to be processed is stored in the Redis database, and the data to be processed is stored in the key-value database according to the weight value of the data to be processed.

[0046] As an optional implementation, users can also customize the sorting according to the table name of the data. For example, if table A is to be executed before table B, the weight value of the data group in table A can be defined as larger, and the weight value of the data group in table B can be defined as smaller.

[0047] In an exemplary embodiment, before merging the data to be processed in the data group to be processed using multiple threads to obtain the merged data group, the method further includes:

[0048] Step S502: The above-mentioned multi-threaded method is used to asynchronously retrieve the above-mentioned data group to be processed from the key-value database.

[0049] In this embodiment, a multi-threaded approach is used to asynchronously retrieve the stored data groups to be processed from the key-value database Redis. Each thread can retrieve one or more data groups to be processed.

[0050] In one exemplary embodiment, the above-mentioned multi-threaded merging process of the data to be processed in the data group to be processed to obtain a merged data group includes:

[0051] Step S602: Merge the above-mentioned data to be processed that has been deleted to obtain the first merged data;

[0052] Step S604: Merge the above-mentioned data to be processed that has been updated to obtain the second merged data;

[0053] Step S606: Construct the above-mentioned merged data group based on the first merged data and the second merged data.

[0054] In this embodiment of the application, data in the data group to be processed that ultimately perform the same operation are merged, and the merged data group to be processed is the aforementioned merged data group.

[0055] Optionally, multiple threads can be used to merge each group of data. For example, if the last operation on some data is delete, then that part of the data can be merged; if the last operation on some data is update, then that part of the data can be merged. The merged data can also be used for business processing.

[0056] As an optional embodiment, the weight value of the first merged data is determined based on the weight value of the data to be processed that is deleted in the first merged data; the weight value of the second merged data is determined based on the weight value of the data to be processed that is updated in the second merged data; the merged data group is constructed according to the weight values ​​of the first merged data and the weight values ​​of the second merged data, and the weight value of the merged data group is determined.

[0057] It should be noted that during the merging phase, the data is also ordered and can be processed with high concurrency.

[0058] In one exemplary embodiment, the above-described synchronous distribution process of the merged data group includes:

[0059] Step S702: Perform synchronous distribution processing according to the weight values ​​of the multiple merged data groups;

[0060] Step S704: If there are merged data groups with the same weight value, then the merged data groups with the same weight value are processed concurrently.

[0061] In this embodiment of the application, the processed data is distributed according to the weight value and business requirements (if the weight values ​​are the same, they can be distributed concurrently; if the weight values ​​are different, the data is distributed in order according to the weight). The distributed data can be saved to a relational database or a non-relational database.

[0062] By following the steps above, we can ensure data order to a certain extent under high concurrency, and handle large concurrent programs. By using the idea of ​​sorting within groups and the concept of data merging, we can greatly reduce the amount of data to be synchronized.

[0063] Figure 4 This is a structural block diagram of a data synchronization and distribution device according to an embodiment of this application; as shown below. Figure 4 As shown, it includes: an acquisition module 40, a grouping module 42, a merging module 44, and a distribution module 46, wherein:

[0064] Module 40 is used to synchronously retrieve data to be processed from a relational database using an open-source framework.

[0065] Grouping module 42 is used to group the above-mentioned data to be processed to obtain data groups to be processed;

[0066] The merging module 44 is used to merge the data to be processed in the above-mentioned data to be processed group using multiple threads to obtain a merged data group.

[0067] Distribution module 46 is used to synchronously distribute the above-mentioned merged data group.

[0068] It should be noted that in this application Figure 4 The specific structure of the data synchronization and distribution device shown is merely illustrative. In practical applications, the data synchronization and distribution module in this application can be more advanced than... Figure 4 The data synchronization and distribution devices shown have more or less structure.

[0069] It should be noted that the data synchronization and distribution device described above corresponds to steps S202 to S208 in the embodiments. The instances and application scenarios implemented by the modules and corresponding steps are the same, but are not limited to the content disclosed in the above embodiments. It should also be noted that the above units, as part of a module, can run on a computer terminal.

[0070] Furthermore, it should be noted that the optional or preferred implementation methods of this embodiment can be described in detail in the relevant description of the specific implementation methods, which will not be repeated here.

[0071] The aforementioned data synchronization and distribution module may also include a processor and a memory. The aforementioned data synchronization and distribution module, etc., are all stored in the memory as program units, and the processor executes the aforementioned program units stored in the memory to realize the corresponding functions.

[0072] The processor contains a core that retrieves corresponding program units from memory. One or more cores may be configured. Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory includes at least one memory chip.

[0073] According to an embodiment of this application, an embodiment of a non-volatile storage medium is also provided. Optionally, in this embodiment, the non-volatile storage medium includes a stored program, wherein, when the program runs, it controls the device where the non-volatile storage medium is located to execute any of the above-mentioned data synchronization and distribution methods.

[0074] Optionally, in this embodiment, the non-volatile storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals, and the non-volatile storage medium includes stored programs.

[0075] Optionally, during program execution, the device containing the non-volatile storage medium may be controlled to perform the following functions: synchronously retrieve data to be processed from a relational database using an open-source framework; group the data to be processed into data groups; merge the data to be processed in the data groups using multi-threading to obtain merged data groups; and synchronously distribute the merged data groups.

[0076] Optionally, during program execution, the device containing the non-volatile storage medium is controlled to perform the following function: grouping the data to be processed according to the primary key and the retrieval order of the data to be processed to obtain a group of data to be processed, wherein the primary key is used to identify a piece of data to be processed, and the retrieval order is used to determine the order in which the data to be processed is retrieved from the relational database.

[0077] Optionally, during program execution, the device containing the non-volatile storage medium may be controlled to perform the following functions: configure the storage order according to the weight values ​​of the data to be processed; and store the data group to be processed into a key-value database based on the storage order.

[0078] Optionally, during program execution, the device containing the non-volatile storage medium may be controlled to perform the following function: asynchronously retrieve the aforementioned data group to be processed from the aforementioned key-value database using the aforementioned multi-threading.

[0079] Optionally, during program execution, the device containing the non-volatile storage medium is controlled to perform the following functions: merge the above-mentioned data to be processed that is operated to be deleted to obtain first merged data; merge the above-mentioned data to be processed that is operated to be updated to obtain second merged data; and construct the above-mentioned merged data group based on the above-mentioned first merged data and the above-mentioned second merged data.

[0080] Optionally, during program execution, the device containing the non-volatile storage medium performs the following functions: determining the weight value of the first merged data based on the weight value of the data to be processed that is deleted in the first merged data; determining the weight value of the second merged data based on the weight value of the data to be processed that is updated in the second merged data; constructing the merged data group according to the weight values ​​of the first merged data and the second merged data, and determining the weight value of the merged data group.

[0081] Optionally, during program execution, the device containing the non-volatile storage medium may be controlled to perform the following functions: synchronously distribute data according to the weight values ​​of the multiple merged data groups; if there are merged data groups with the same weight value, then the merged data groups with the same weight value shall be processed concurrently.

[0082] According to an embodiment of this application, an embodiment of a processor is also provided. Optionally, in this embodiment, the processor is used to run a program, wherein the program executes any of the above-described data synchronization and distribution methods during runtime.

[0083] According to an embodiment of this application, an embodiment of an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform any of the above-described data synchronization and distribution methods.

[0084] According to an embodiment of this application, an embodiment of a computer program product is also provided, which, when executed on a data processing device, is adapted to execute a program that initializes any of the above-described data synchronization and distribution method steps.

[0085] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0086] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0087] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The module embodiments described above are merely illustrative; for example, the division of the units described above can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0088] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0089] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0090] If the aforementioned integrated units are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable non-volatile storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a non-volatile storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned non-volatile storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0091] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A data synchronization and distribution method, characterized in that, include: An open-source framework is used to synchronously retrieve the data to be processed from a relational database in the order in which the data is retrieved. The data to be processed is grouped to obtain data groups to be processed. The data to be processed that were deleted are merged to obtain the first merged data; The data to be processed that is updated is merged to obtain the second merged data; The weight value of the first merged data is determined based on the weight value of the data to be processed whose operation in the first merged data is deletion. The weight value of the second merged data is determined based on the weight value of the data to be processed that is updated in the second merged data. The merged data group is constructed according to the weight values ​​of the first merged data and the second merged data, and the weight value of the merged data group is determined. The data is synchronously distributed according to the weight values ​​of the multiple merged data groups. If there are merged data groups with the same weight value, then the merged data groups with the same weight value will be processed concurrently; The step of grouping the data to be processed to obtain a data group includes: grouping the data to be processed according to the primary key and the retrieval order to obtain a data group, wherein the primary key is used to identify a piece of data to be processed, and the retrieval order is used to determine the order in which the data to be processed is retrieved from the relational database; After grouping the data to be processed to obtain data groups, the method further includes: Configure the storage order according to the weight values ​​of the data to be processed; Based on the storage order, the data group to be processed is stored in a key-value database.

2. The method according to claim 1, characterized in that, Before merging the data to be processed in the data group to be processed using multi-threading to obtain a merged data group, the method further includes: The multi-threaded method asynchronously retrieves the stored data group to be processed from the key-value database.

3. A data synchronization and distribution device, characterized in that, include: The acquisition module is used to synchronously retrieve the data to be processed from the relational database according to the retrieval order of the data to be processed using an open-source framework. The grouping module is used to group the data to be processed to obtain data groups to be processed; The merging module merges the data to be processed that has been deleted to obtain the first merged data. The data to be processed that is updated is merged to obtain the second merged data; The merged data group is constructed based on the first merged data and the second merged data; And, the weight value of the first merged data is determined based on the weight value of the data to be processed whose operation is deletion in the first merged data; The weight value of the second merged data is determined based on the weight value of the data to be processed that is updated in the second merged data. The merged data group is constructed according to the weight values ​​of the first merged data and the second merged data, and the weight value of the merged data group is determined. The distribution module is used to synchronously distribute the data according to the weight values ​​of the multiple merged data groups; if there are merged data groups with the same weight value, the merged data groups with the same weight value are processed concurrently. The grouping module is further configured to perform grouping processing based on the primary key of the data to be processed and the retrieval order of the data to be processed, to obtain a data group to be processed, wherein the primary key is used to identify a piece of data to be processed, and the retrieval order is used to determine the order in which the data to be processed is retrieved from the relational database; Furthermore, after grouping the data to be processed to obtain the data to be processed groups, the storage order is configured according to the weight values ​​of the data to be processed. Based on the storage order, the data group to be processed is stored in a key-value database.

4. A non-volatile storage medium, characterized in that, The non-volatile storage medium stores multiple instructions, which are adapted to be loaded by a processor and executed by the data synchronization and distribution method according to any one of claims 1 to 2.

5. A processor, characterized in that, The processor is used to run a program, wherein the program is configured to execute the data synchronization and distribution method according to any one of claims 1 to 2 at runtime.

6. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the data synchronization and distribution method according to any one of claims 1 to 2.

Images