Data synchronization method, data synchronization device, and computer storage medium

Abstract

The application discloses a data synchronization method, a data synchronization device and a computer storage medium. The data synchronization method is applied to a database. The database comprises a management node and a data node. The data synchronization method comprises the following steps: in response to a data synchronization request, the management node sends the data synchronization request to the data node; the data node analyzes the data synchronization request and obtains an execution plan; and the data node performs data filtering on data in the database according to the execution plan, and outputs the data filtered and to be synchronized to a storage medium and / or another database. Through the method, the data synchronization request can be analyzed by using the data node in the database to obtain an execution plan, and the data to be synchronized can be synchronized to the storage medium and / or another database according to the execution plan, so that the data node can perform the data synchronization operation in parallel, and the data synchronization rate is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, in particular to a data synchronization method, a data synchronization device and a computer storage medium. BACKGROUND

[0002] With the rapid development of the Internet, people in various fields of social production and social life have generated various business-related data. Because the amount of data is too large, databases are generally used to store and manage them, and single-node databases have been difficult to meet the requirements of mass data storage and high-concurrency data access rapid response. Therefore, distributed databases have become the mainstream choice in the big data industry by adopting multiple nodes to split data and data tables, relieving the way of mass data storage.

[0003] In an application scenario, when data is synchronized between different database clusters in a distributed database, it is limited by the cache of the management node, and a large amount of data cannot be synchronized at a time, resulting in low data synchronization efficiency between different databases. SUMMARY

[0004] The technical problem solved by the present application is how to improve the efficiency of data synchronization between different databases. To this end, the present application provides a data synchronization method, a data synchronization device and a computer storage medium.

[0005] To solve the above technical problems, one technical solution adopted by the present application is to provide a data synchronization method, which is applied to a database including a management node and a data node. The data synchronization method comprises: in response to a data synchronization request, the management node sends the data synchronization request to the data node; the data node analyzes the data synchronization request and obtains an execution plan; the data node performs data filtering on the data in the database according to the execution plan, and outputs the filtered data to be synchronized to a storage medium and / or other database.

[0006] After outputting the filtered data to be synchronized to the storage medium and / or other database, the data node sends the data synchronization result to the management node; if all data synchronization results returned by the data nodes are successful, the management node sends a synchronization success signal to the client; if there is a data synchronization result returned by one data node that is a failure, the management node sends a synchronization failure signal to the client.

[0007] The data synchronization method further comprises: the data node records the time and / or data number of outputting the filtered data to be synchronized; when there is a data synchronization result returned by one data node that is a failure, the data node continues to output the data to be synchronized based on the time and / or data number.

[0008] To solve the above technical problems, another technical solution adopted by the present application is to provide a data synchronization method, which is applied to a data synchronization system, the data synchronization system comprising a first database, a business processing module, and a second database; the data synchronization method comprising: in response to a data synchronization request, the business processing module sending a data synchronization instruction to the first database; the first database outputting to-be-synchronized data according to the data synchronization instruction, wherein the manner in which the first database outputs the to-be-synchronized data is as described above; and the business processing module transmitting the to-be-synchronized data to the second database so as to synchronize the to-be-synchronized data in the second database.

[0009] The data synchronization system further comprises a storage medium; the first database outputting to-be-synchronized data according to the data synchronization instruction comprises: the first database outputting to-be-synchronized data into the storage medium according to the data synchronization instruction; and the business processing module transmitting the to-be-synchronized data to the second database so as to synchronize the to-be-synchronized data in the second database comprises: the business processing module transmitting the to-be-synchronized data from the storage medium to the second database so as to synchronize the to-be-synchronized data in the second database.

[0010] The data synchronization system further comprises a storage medium; the first database outputting to-be-synchronized data according to the data synchronization instruction comprises: the first database outputting to-be-synchronized data into the storage medium according to the data synchronization instruction; and the business processing module transmitting the to-be-synchronized data to the second database so as to synchronize the to-be-synchronized data in the second database comprises: the business processing module transmitting the to-be-synchronized data from the storage medium to the second database so as to synchronize the to-be-synchronized data in the second database.

[0011] The data synchronization system further comprises a storage medium; the first database outputting to-be-synchronized data according to the data synchronization instruction comprises: the first database outputting to-be-synchronized data into the storage medium according to the data synchronization instruction; and the business processing module transmitting the to-be-synchronized data to the second database so as to synchronize the to-be-synchronized data in the second database comprises: the business processing module transmitting the to-be-synchronized data from the storage medium to the second database so as to synchronize the to-be-synchronized data in the second database.

[0012] The data synchronization system further comprises a storage medium; the first database outputting to-be-synchronized data according to the data synchronization instruction comprises: the first database outputting to-be-synchronized data into the storage medium according to the data synchronization instruction; and the business processing module transmitting the to-be-synchronized data to the second database so as to synchronize the to-be-synchronized data in the second database comprises: the business processing module transmitting the to-be-synchronized data from the storage medium to the second database so as to synchronize the to-be-synchronized data in the second database.

[0013] The data synchronization system further comprises a storage medium; the first database outputting to-be-synchronized data according to the data synchronization instruction comprises: the first database outputting to-be-synchronized data into the storage medium according to the data synchronization instruction; and the business processing module transmitting the to-be-synchronized data to the second database so as to synchronize the to-be-synchronized data in the second database comprises: the business processing module transmitting the to-be-synchronized data from the storage medium to the second database so as to synchronize the to-be-synchronized data in the second database.

[0014] To solve the above technical problems, another technical solution adopted by the present application is to provide a data synchronization apparatus, which comprises a processor and a memory, the memory being coupled to the processor, the memory storing program data, and the processor being configured to execute the program data to implement the above-described data synchronization method.

[0015] To solve the above technical problems, another technical solution adopted by the present application is to provide a computer-readable storage medium, which stores program data, the program data being configured to implement the above-described data synchronization method when executed.

[0016] The beneficial effects of the present application are: different from the prior art, the data synchronization method provided by the present application is applied to a database, the database includes a management node and a data node; in response to a data synchronization request, the management node sends the data synchronization request to the data node; the data node analyzes the data synchronization request to obtain an execution plan; the data node performs data filtering on the data in the database according to the execution plan, and outputs the data filtered after the data filtering to a storage medium and / or other database. Through the above manner, compared with the conventional data synchronization method, the present application uses the data node in the database to analyze the data synchronization request and obtain a specific execution plan, and then the data node filters the to-be-synchronized data according to the execution plan and synchronizes the to-be-synchronized data to the storage medium and / or other database. The execution plan can be pushed down to the data node, and the data node itself can filter and synchronize the to-be-synchronized data to the storage medium and / or other database according to the execution plan, a large amount of data can be synchronized at one time, and the problem that the management node becomes a bottleneck and affects the normal operation of the business is avoided. Compared with the conventional data synchronization method in which the management node issues instructions to each data node for data synchronization according to the execution plan, the data synchronization method provided by the present application is not limited by the cache of the management node, so as to consume a large amount of time. By pushing the execution plan to the data node, each data node can perform data synchronization operation in parallel, thereby improving the rate and reliability of data synchronization. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0018] Among them:

[0019] Figure 1 is a flowchart of the first embodiment of the data synchronization method provided by the present application;

[0020] Figure 2 is a flowchart of the data synchronization method provided by the present application applied to a database;

[0021] Figure 3 is a flowchart of the second embodiment of the data synchronization method provided by the present application;

[0022] Figure 4 is a structural diagram of an embodiment of the data synchronization system provided by the present application;

[0023] Figure 5is a structural schematic diagram of a first embodiment of a data synchronization device provided in the present application;

[0024] Figure 6 is a structural schematic diagram of a second embodiment of a data synchronization device provided in the present application;

[0025] Figure 7 is a structural schematic diagram of an embodiment of a computer readable storage medium provided in the present application. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present application will be clearly and completely described in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0027] The term "and / or" in the present document is only used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. In addition, the character " / " in the present document generally represents an "or" relationship between the associated objects. In addition, "multiple" in the present document means two or more than two. In addition, the term "at least one" in the present document means any one of multiple or any combination of at least two of multiple, for example, including at least one of A, B and C can mean including any one or more elements selected from the set consisting of A, B and C.

[0028] With the rapid development of computers and information technology, the scale of industry application systems expands rapidly, and the data generated by industry applications grows explosively, which has far exceeded the processing capacity of traditional computing technology and information systems. The centralized database gradually shows its limitations in the face of large-scale data processing. Therefore, people hope to find a method that can quickly process data and respond to user access in time, and also hope to centrally analyze, manage and maintain data. The distributed database emerges as the times require.

[0029] The data synchronization method provided in the present application is applied in a distributed database. The distributed database refers to a logically unified database composed of multiple physically dispersed data storage units connected by a high-speed computer network. The basic idea is to store the data in the original centralized database on multiple data storage nodes connected by a network to obtain greater storage capacity and higher concurrent access.

[0030] Since the distributed database is connected by multiple data storage units, data synchronization needs to be used to keep the consistency of the distributed database. Data synchronization is to synchronize the data in one data storage unit to another data storage unit in the distributed database.

[0031] The execution subject of the data synchronization method can be a data synchronization device. For example, the data synchronization method can be executed by a terminal device or a server or other processing device. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementation manners, the data synchronization method can be implemented by a processor invoking computer readable instructions stored in a memory.

[0032] The data synchronization device, the data synchronization system, and the database can all be the execution subject of the data synchronization method provided in the embodiments of the present application. The data synchronization system and the database can be carried on the data synchronization device as the core part.

[0033] Referring to Figure 1 and Figure 2 , Figure 1 is a flowchart of a first embodiment of the data synchronization method provided in the present application; Figure 2 is a flowchart of the data synchronization method provided in the present application applied to a database.

[0034] In an embodiment of the present application, the data synchronization method is applied to a database. The database includes a management node and a data node.

[0035] Specifically, the management node and the data node are two nodes in a distributed database. In a conventional distributed database design, the management node is responsible for parsing an incoming service request and issuing the service request to the data node, and collecting and summarizing the statistical results from the data node and returning the statistical results to the calling party. The data node is responsible for specific business computing.

[0036] Step 11: In response to the data synchronization request, the management node sends the data synchronization request to the data node.

[0037] Specifically, the user can select the database to be synchronized and the specific data to be synchronized on the display screen of the data synchronization device by checking or filtering in the list to form a data synchronization request. The database in the data synchronization device can send the data synchronization request directly to each data node by the management node according to the obtained data synchronization request of the user, without the management node analyzing the data synchronization request. By pushing down the step of analyzing the data synchronization request in the conventional data synchronization method to the data node, the problem of blocking in the synchronization process caused by the limitation of the cache space size of the management node is avoided.

[0038] Step 12: The data node analyzes the data synchronization request and obtains an execution plan.

[0039] Specifically, the execution plan can include the specific data to be synchronized and the location of the data to be synchronized. After the data node analyzes the data synchronization request, a synchronization statement corresponding to the business request, i.e., the execution plan, is generated.

[0040] Step 13: The data node performs data filtering on the data in the database according to the execution plan, and outputs the data to be synchronized after data filtering to the storage medium and / or other database.

[0041] Specifically, the data node filters all data in the database according to the execution plan obtained in step 12 to obtain the data to be synchronized corresponding to the data synchronization request. The data node then synchronizes the filtered data to be synchronized to the storage medium and / or other database according to the execution plan.

[0042] Specifically, after the data synchronization device outputs the data to be synchronized after data filtering to the storage medium and / or other database, the data node in the data synchronization device sends the data synchronization result to the management node; if the data synchronization result returned by all data nodes is successful, the management node sends a synchronization success signal to the client; if the data synchronization result returned by one data node is failed, the management node sends a synchronization failure signal to the client.

[0043] Specifically, the data node also records the time and / or data number of outputting each data to be synchronized after data filtering. For example, when the data in a data node is the first input, and the input time is 0:10, the data node records the data number of the data as 1 and the input time as 0:10. The data synchronization device can choose to synchronize the data with data numbers 1-10 at the same time in batches to realize parallel synchronization of the data.

[0044] Optionally, when there is one data node returning a data synchronization result as failure, the data node can continue to output the to-be-synchronized data according to the data number of the to-be-synchronized data after locating the position of the to-be-synchronized data with errors based on time and / or data number.

[0045] In an embodiment of the present application, the data synchronization method can also be applied to a data synchronization system. Referring to Figure 3 and Figure 4 , Figure 3 is a flowchart of a second embodiment of the data synchronization method provided by the present application; Figure 4 is a structural diagram of an embodiment of the data synchronization system provided by the present application.

[0046] Specifically, the data synchronization system comprises a first database, a business processing module, and a second database. The first database is a source database of to-be-synchronized data, supports multiple different database products, including but not limited to greenplum, hive, hbase, etc. The second database is a destination database of to-be-synchronized data, supports multiple different types of databases, including greenplum, hive, hbase, etc. The first database and the second database can be converted to each other according to user needs, which is not limited here.

[0047] Specifically, the business processing module is built-in with connection engines between multiple different databases and a preset mapping table including data field type definitions between commonly used databases, which can write the to-be-synchronized data on each data node from the source distributed database to the storage medium specified by the user after obtaining the ip address of each data node.

[0048] Optionally, the data synchronization system further comprises a storage medium, which can be stored in a local file system, and also supports being configured to specify a remote distributed file system such as hdfs, etc.

[0049] Step 31: In response to the data synchronization request, the business processing module sends a data synchronization instruction to the first database.

[0050] Specifically, after the business processing module sends the data synchronization instruction to the first database, it further comprises: the business processing module acquires the structure of the target table in the first database where the to-be-synchronized data is located, and establishes a corresponding table in the second database for storing the to-be-synchronized data according to the structure of the target table and the preset mapping table.

[0051] Step 32: The first database outputs the to-be-synchronized data according to the data synchronization instruction.

[0052] Specifically, the first database outputs the to-be-synchronized data to the storage medium according to the data synchronization instruction.

[0053] Specifically, the first database outputs the to-be-synchronized data in the manner of the data synchronization method of steps 11-13.

[0054] Specifically, the business processing module can trigger the writing of the to-be-synchronized data on each data node from the first database to the storage medium in batches according to the output time and / or data number corresponding to the to-be-synchronized data, wherein each data node includes a data file.

[0055] Step 33: The business processing module transmits the to-be-synchronized data to the second database to synchronize the to-be-synchronized data in the second database.

[0056] Specifically, the business processing module transmits the to-be-synchronized data from the storage medium to the second database to synchronize the to-be-synchronized data in the second database. By adding the storage medium, the problem of data incompatibility between the first database and the second database can be solved.

[0057] Specifically, after the to-be-synchronized data is transmitted to the second database in batches according to the output time and / or data number corresponding to the to-be-synchronized data, the data file corresponding to each batch stored in the storage medium is deleted to release the memory space.

[0058] Specifically, the business processing module transmits the to-be-synchronized data to the corresponding table in the second database.

[0059] Optionally, if the synchronization of a batch of to-be-synchronized data fails, the business processing module records the output time and / or data number corresponding to the batch of to-be-synchronized data, and rolls back the data synchronization method to delete the data file corresponding to the batch of to-be-synchronized data from the storage medium. The business processing module also deletes the corresponding synchronized data in the second database according to the data number corresponding to the range of the batch, and returns the synchronization failure information to the user. After the business processing module receives the user's positioning and repair of the data synchronization failure reason, the data synchronization can be continued from the failed batch of to-be-synchronized data according to the output time and / or data number corresponding to the batch of to-be-synchronized data, realizing the breakpoint continuation of data synchronization.

[0060] Taking the synchronization of data from the Greenplum database to the Hive database as an example:

[0061] The business processing module can automatically obtain the structure of the table where the data is located from the Greenplum database according to the specified database type of the specified data destination, i.e., the second database, and the user authentication information, and create a corresponding table in the Hive database according to the field type mapping relationship. The business processing module selects the storage medium hdfs, and exports the table data from the Greenplum database to the storage medium hdfs in batches, and further imports the data files in the storage medium hdfs into the hive in order.

[0062] If a batch of data synchronization fails, the business processing module records the data number and / or output time of the data to obtain the starting record of the batch of data, and rolls back the entire data synchronization process, deletes the batch of data from the storage medium hdfs, and deletes the synchronized data in the hive database according to the starting record of the batch of data. The business processing module returns the data synchronization failure information to the user, and after the user confirms and repairs the data synchronization failure reason, the business processing module can continue to synchronize the remaining data from the Greenplum database to the Hive database from the starting record of the failed batch according to the data number and / or output time of the batch of data.

[0063] Unlike the prior art, the data synchronization method provided by the application is applied to a database, the database includes a management node and a data node; in response to a data synchronization request, the management node sends the data synchronization request to the data node; the data node analyzes the data synchronization request to obtain an execution plan; the data node filters data in the database according to the execution plan, and outputs the filtered data to be synchronized to a storage medium and / or other database. By the above-mentioned manner, compared with the conventional data synchronization method, the application uses the data node in the database to analyze the data synchronization request and obtain a specific execution plan, and then the data node filters the data to be synchronized according to the execution plan and synchronizes it to the storage medium and / or other database. The execution plan can be pushed down to the data node, and the data node itself can filter and synchronize the data to be synchronized according to the execution plan and into the storage medium and / or other database, which can synchronize a large amount of data at a time, avoiding the problem that the management node becomes a bottleneck and affects the normal operation of the business. Compared with the conventional data synchronization method in which the management node issues instructions to each data node for data synchronization according to the execution plan, the data synchronization method provided by the application is not limited by the cache of the management node, thereby consuming a lot of time. By pushing the execution plan to the data node, each data node can perform data synchronization operations in parallel, improving the speed and reliability of data synchronization.

[0064] The method of the above embodiment can be implemented by using a data synchronization device. The data synchronization device is described below in combination with Figure 5 , Figure 5 is a structural schematic diagram of a first embodiment of the data synchronization device provided by the present application.

[0065] As shown in Figure 5 , the data synchronization device 50 of the embodiment of the present application comprises a sending module 51, an analysis module 52, and an output module 53.

[0066] The sending module 51 is configured to send, in response to a data synchronization request, the data synchronization request from a management node to a data node.

[0067] The analysis module 52 is configured to analyze, by the data node, the data synchronization request to obtain an execution plan.

[0068] The output module 53 is configured to perform data filtering on data in a database according to the execution plan, and output the data filtered to be synchronized to a storage medium and / or other database.

[0069] The method of the above embodiment can be implemented by using a data synchronization device. The data synchronization device is described below in combination with Figure 6 , Figure 6 is a structural schematic diagram of a second embodiment of the data synchronization device provided by the present application. The data synchronization device 60 comprises a memory 61 and a processor 62. The memory 61 is configured to store program data. The processor 62 is configured to execute the program data to implement the following method:

[0070] In response to a data synchronization request, a management node sends the data synchronization request to a data node. The data node analyzes the data synchronization request to obtain an execution plan. The data node performs data filtering on data in a database according to the execution plan, and outputs the data filtered to be synchronized to a storage medium and / or other database.

[0071] Refer to Figure 7 , Figure 7 is a structural schematic diagram of an embodiment of a computer readable storage medium provided by the present application. The computer readable storage medium 70 stores program data 71. When the program data 71 is executed by a processor, the program data 71 is configured to implement the following method:

[0072] In response to a data synchronization request, a management node sends the data synchronization request to a data node. The data node analyzes the data synchronization request to obtain an execution plan. The data node performs data filtering on data in a database according to the execution plan, and outputs the data filtered to be synchronized to a storage medium and / or other database.

[0073] The embodiments of the present application are realized in the form of software function units and sold or used as independent products, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the whole or part of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0074] The above is only the embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation using the content of the specification and drawings, or direct or indirect application in other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. A method of data synchronization, the method comprising: The data synchronization method is applied to a database, and the database comprises a management node and a data node; the data synchronization method comprises: In response to a data synchronization request, the management node sends the data synchronization request to the data node; The data node analyzes the data synchronization request and obtains an execution plan; the execution plan comprises a synchronization statement corresponding to the data synchronization request; The data node performs data filtering on data in the database according to the execution plan, and outputs the data filtered and filtered to a storage medium and / or other database.

2. The data synchronization method of claim 1, wherein, after the data filtered and filtered is output to the storage medium and / or other database, the method further comprises: the data node sends a data synchronization result to the management node; if all the data synchronization results returned by the data nodes are successful, the management node sends a synchronization success signal to a client; if there is one data synchronization result returned by the data node that is a failure, the management node sends a synchronization failure signal to the client.

3. The data synchronization method of claim 2, wherein, the data synchronization method further comprises: the data node records the time and / or data number of outputting the data filtered and filtered; when there is one data synchronization result returned by the data node that is a failure, the data node continues to output the data filtered and filtered based on the time and / or data number.

4. A data synchronization method, characterized by, The data synchronization method is applied to a data synchronization system, and the data synchronization system comprises a first database, a business processing module, and a second database; the data synchronization method comprises: In response to a data synchronization request, the business processing module sends a data synchronization instruction to the first database; The first database outputs the data to be synchronized according to the data synchronization instruction, wherein the first database outputs the data to be synchronized in the manner of the data synchronization method of any one of claims 1 to 3; The business processing module transmits the data to be synchronized to the second database, so that the second database synchronizes the data to be synchronized.

5. The data synchronization method of claim 4, wherein, the data synchronization system further comprises a storage medium; the first database outputs the data to be synchronized according to the data synchronization instruction, comprising: the first database outputs the data to be synchronized to the storage medium according to the data synchronization instruction; the business processing module transmits the data to be synchronized to the second database, so that the second database synchronizes the data to be synchronized, comprising: the business processing module transmits the data to be synchronized from the storage medium to the second database, so that the second database synchronizes the data to be synchronized.

6. The data synchronization method of claim 5, wherein, after the business processing module transmits the data to be synchronized from the storage medium to the second database, the method further comprises: the storage medium deletes the data to be synchronized.

7. The data synchronization method of claim 4, further comprising: after the business processing module sends the data synchronization instruction to the first database, the method further comprises: the business processing module acquires a structure of a target table in the first database; the business processing module establishes a corresponding table in the second database according to the structure of the target table and a preset mapping table; the business processing module transmits the to-be-synchronized data to the second database, comprising: the business processing module transmits the to-be-synchronized data to the corresponding table in the second database.

8. The data synchronization method of claim 4, further comprising: the business processing module transmits the to-be-synchronized data to the second database, comprising: the business processing module transmits the to-be-synchronized data to the second database in batches.

9. A data synchronization apparatus, characterized by comprising: The data synchronization device comprises a memory and a processor coupled with the memory; wherein the memory is configured to store program data, and the processor is configured to execute the program data to implement the data synchronization method of any one of claims 1 to 8.

10. A computer storage medium, characterized in that, The computer storage medium is configured to store program data, and the program data, when executed by a computer, is configured to implement the data synchronization method of any one of claims 1 to 8.

Images