Data synchronization method, device, equipment and medium

Abstract

The present application discloses a data synchronization method, apparatus, device and medium, which relates to the field of data synchronization, including: after a data synchronization service integrated with several canal clients loads a filter file corresponding to any canal client in a persistent storage into a local memory, based on the filter file, the data obtained from the server queue corresponding to the canal server is filtered to obtain filtered data; the data in the server queue is the data obtained by the canal server from an upstream database; the filtered data is stored in a client queue corresponding to any canal client for data synchronization with a downstream database corresponding to any canal client; if a filter file change request is obtained through an interface in a file configuration page, the corresponding file in the persistent storage is changed, and the changed file is reloaded into the local memory to continue data filtering, thereby realizing data synchronization.

Description

Technical Field

[0001] The present invention relates to the field of data synchronization, and in particular to a data synchronization method, device, equipment and medium. Background Art

[0002] In the current data business, data is persistently stored in the MySQL database of each business system. Considering the limitations of the MySQL database, it is necessary to synchronize the data in the MySQL database to other databases to meet the use of different application scenarios, such as synchronizing to Hive, TiDB, Kudu, Hbase, ElasticSearch and other databases.

[0003] In the process of data synchronization, data filtering mechanisms are often involved. The current data filtering mechanism usually configures a data filtering list in the startup configuration file of the canal server to filter the data obtained from the MySQL database based on the data filtering list, and then synchronizes the filtered data to other databases. However, if the data filtering list needs to be changed, since the data filtering list is stored in the startup configuration file of the canal server, it is necessary to restart the canal server after the data filtering list is changed, or add canal-admin, which is used to provide overall configuration management and node operation and maintenance functions for the canal middleware, to reload the configuration of the data filtering list through canal-admin. However, the above two methods are heavyweight operations, which will seriously affect the synchronization of data during the restart or reload configuration process, and the canal server needs to run stably in actual work and rarely restarts. Therefore, how to achieve data synchronization without restarting the canal server is a problem that needs to be solved at present. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a data synchronization method, device, equipment and medium, which can realize dynamic changes to data synchronization filter files without restarting the canal server, and realize differentiated data synchronization for different downstream database systems. The specific scheme is as follows:

[0005] In a first aspect, the present application provides a data synchronization method, which is applied to a data synchronization service integrated with several canal clients, where different canal clients correspond to different downstream database systems and data synchronization filter files; wherein the method comprises:

[0006] After loading the target data synchronization filter file corresponding to any canal client in the preset persistent storage into the local preset memory, filtering the to-be-synchronized data obtained from the server queue corresponding to the canal server based on the target data synchronization filter file to obtain filtered data; wherein the data in the server queue is the data obtained by the canal server from the upstream database system;

[0007] Sending the filtered data to a target client queue corresponding to any canal client for storage, so that any canal client synchronizes the data stored in the target client queue to a target downstream database system corresponding to any canal client;

[0008] If a change request for the target data synchronization filter file is obtained through the first preset interface in the file configuration page, the target data synchronization filter file in the preset persistent storage is changed based on the change request, and the changed data synchronization filter file is reloaded into the preset memory so that the data to be synchronized can be filtered based on the changed data synchronization filter file.

[0009] Optionally, filtering the to-be-synchronized data obtained from the server queue corresponding to the canal server based on the target data synchronization filter file includes:

[0010] The target object created in the data synchronization service by any canal client obtains the data to be synchronized from the server queue corresponding to the canal server, and filters the data to be synchronized based on the target data synchronization filter file.

[0011] Optionally, before the target object created in the data synchronization service by any canal client obtains the data to be synchronized from the server queue corresponding to the canal server, the method further includes:

[0012] The second preset interface of any canal client is called to create the corresponding target object in the data synchronization service; wherein different canal clients correspond to different objects.

[0013] Optionally, the data in the server queue is data sent by the canal server to the server queue for storage; the data sent by the canal server to the server queue for storage is data obtained by the canal server after parsing the data change log sent by the upstream database system; the data change log is a log synchronized to the canal server by the upstream database system based on the master-slave synchronization mechanism when receiving the log request sent by the canal server.

[0014] Optionally, filtering the to-be-synchronized data obtained from the server queue corresponding to the canal server based on the target data synchronization filter file to obtain filtered data includes:

[0015] Filtering the data to be synchronized obtained from the server queue corresponding to the canal server based on the data whitelist in the target data synchronization filter file, so as to determine the filtered data corresponding to the data whitelist from the data to be synchronized;

[0016] The data whitelist includes any one or a combination of a database whitelist, a data table whitelist and a data field whitelist.

[0017] Optionally, filtering the to-be-synchronized data obtained from the server queue corresponding to the canal server based on the target data synchronization filter file to obtain filtered data includes:

[0018] Filtering the data to be synchronized obtained from the server queue corresponding to the canal server based on the data blacklist in the target data synchronization filter file, so as to delete the data corresponding to the data blacklist from the data to be synchronized, and determining the filtered data based on the remaining data in the data to be synchronized;

[0019] The data blacklist includes any one or a combination of a database blacklist, a data table blacklist and a data field blacklist.

[0020] Optionally, filtering the to-be-synchronized data obtained from the server queue corresponding to the canal server based on the target data synchronization filter file to obtain filtered data includes:

[0021] Acquire the data to be synchronized from the server queue corresponding to the canal server, and if there are multiple data to be synchronized, determine the first data in the data to be synchronized as the current data to be synchronized, and filter the current data to be synchronized based on the target data synchronization filter file to obtain the current filtered data, so as to send the current filtered data to the target client queue for storage;

[0022] Determine whether the current data to be synchronized is the last data in the data to be synchronized; if not, determine the next data in the data to be synchronized as the new current data to be synchronized, and jump again to the step of filtering the current data to be synchronized based on the target data synchronization filter file until the current data to be synchronized is the last data in the data to be synchronized, and then obtain new data to be synchronized from the server queue.

[0023] In a second aspect, the present application provides a data synchronization device, which is applied to a data synchronization service integrated with several canal clients, where different canal clients correspond to different downstream database systems and data synchronization filter files; wherein the device comprises:

[0024] A data filtering module is used to filter the to-be-synchronized data obtained from the server queue corresponding to the canal server based on the target data synchronization filtering file after loading the target data synchronization filtering file corresponding to any canal client in the preset persistent storage into the local preset memory, so as to obtain filtered data; wherein the data in the server queue is the data obtained by the canal server from the upstream database system;

[0025] A data synchronization module, used for sending the filtered data to a target client queue corresponding to any canal client for storage, so that any canal client synchronizes the data stored in the target client queue to a target downstream database system corresponding to any canal client;

[0026] A file change module is used to change the target data synchronization filter file in the preset persistent storage based on the change request if a change request for the target data synchronization filter file is obtained through the first preset interface in the file configuration page, and to reload the changed data synchronization filter file into the preset memory so as to filter the data to be synchronized based on the changed data synchronization filter file.

[0027] In a third aspect, the present application provides an electronic device, including:

[0028] Memory, used to store computer programs;

[0029] A processor is used to execute the computer program to implement the aforementioned data synchronization method.

[0030] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program, which implements the aforementioned data synchronization method when executed by a processor.

[0031] The data synchronization method of the present application is applied to a data synchronization service integrated with several canal clients, and different canal clients correspond to different downstream database systems and data synchronization filter files; wherein, the method comprises: after loading the target data synchronization filter file corresponding to any canal client in the preset persistent storage into the local preset memory, filtering the data to be synchronized obtained from the server queue corresponding to the canal server based on the target data synchronization filter file to obtain filtered data; wherein, the data in the server queue is the data obtained by the canal server from the upstream database system; sending the filtered data to the target client queue corresponding to any canal client for storage, so that any canal client synchronizes the data stored in the target client queue to the target downstream database system corresponding to any canal client; if a change request for the target data synchronization filter file is obtained through the first preset interface in the file configuration page, then changing the target data synchronization filter file in the preset persistent storage based on the change request, and reloading the changed data synchronization filter file into the preset memory, so as to filter the data to be synchronized based on the changed data synchronization filter file. It can be seen that the present application stores the data synchronization filter file in the persistent storage in advance, so that when data synchronization is performed based on the data synchronization service pre-integrated with several canal clients, the data synchronization filter file in the persistent storage is loaded into the local memory of the data synchronization service, so that the data synchronization service can perform data filtering based on the data synchronization filter file in the local memory. In this way, by migrating the data filtering operation from the original canal server to the data synchronization service integrated with several canal clients, the resource consumption of the canal server can be reduced; and when the data synchronization filter file needs to be changed, it is only necessary to obtain the file change request through the preset interface in the file configuration page provided by the data synchronization service, so that the change of the data synchronization filter file in the persistent storage can be realized, and then the changed data synchronization filter file is reloaded into the local memory of the data synchronization service, so that the dynamic change of the data synchronization filter file can be realized without restarting the canal server, and the data can continue to be filtered based on the changed data synchronization filter file. Further, since different canal clients in the present application correspond to different downstream database systems and different data synchronization filter files, different data synchronization filter files can be used to achieve differentiated data synchronization of different downstream database systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0033] Figure 1 A flow chart of a data synchronization method disclosed in this application;

[0034] Figure 2 A data synchronization architecture diagram disclosed in this application;

[0035] Figure 3 A data synchronization flow chart disclosed in this application;

[0036] Figure 4 A schematic diagram of the structure of a data synchronization device disclosed in this application;

[0037] Figure 5 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] Since both restarting the canal server and reloading the configuration of canal-admin are heavyweight operations, data synchronization will be seriously affected during the restart or reloading process, and the canal server needs to run stably in actual work and rarely restarts. To this end, the present application provides a data synchronization method that can dynamically change the data synchronization filter file without restarting the canal server, and realize differentiated data synchronization for different downstream database systems.

[0040] See also Figure 1 As shown, an embodiment of the present invention discloses a data synchronization method, which is applied to a data synchronization service integrated with several canal clients, and different canal clients correspond to different downstream database systems and data synchronization filter files; wherein the method includes:

[0041] Step S11: After loading the target data synchronization filter file corresponding to any canal client in the preset persistent storage into the local preset memory, the data to be synchronized obtained from the server queue corresponding to the canal server is filtered based on the target data synchronization filter file to obtain filtered data; wherein the data in the server queue is the data obtained by the canal server from the upstream database system.

[0042] In this embodiment, considering that different downstream database systems correspond to different data synchronization filter files, the present application configures a corresponding canal client for each downstream database system, integrates different canal clients corresponding to different downstream database systems into the data synchronization service, and pre-stores different data synchronization filter files corresponding to different downstream database systems into the preset persistent storage. When data synchronization is performed through a data synchronization service integrated with several canal clients, the target data synchronization filter file corresponding to any canal client in the preset persistent storage is loaded into the preset memory of the local data synchronization service, so as to filter the data to be synchronized obtained from the server queue corresponding to the canal server based on the target data synchronization filter file in the preset memory, and synchronize the filtered data to the target downstream database system corresponding to any canal client. Among them, the data in the server queue corresponding to the canal server is the data obtained by the canal server from the upstream database system.

[0043] Among them, canal middleware is an open source tool for data synchronization and data change capture, mainly used to solve the problem of data synchronization between databases. The upstream database system can be a MySQL database, and the downstream database system can include Hive, TiDB, Kudu, Hbase, ElasticSearch and other databases. The preset persistent storage can realize the persistent storage of data synchronization filter files, and the preset persistent storage can use Redis and so on.

[0044] For filtering the data to be synchronized, the data synchronization service specifically reads the target data synchronization filter file corresponding to any canal client from the preset persistent storage, loads the target data synchronization filter file into the local preset memory, and then calls the second preset interface of any canal client to create the corresponding target object in the data synchronization service. After the target object is created, the data synchronization service obtains the data to be synchronized from the server queue corresponding to the canal server through the target object created locally by any canal client, and filters the data to be synchronized based on the target data synchronization filter file in the local preset memory.

[0045] It should be noted that different canal clients correspond to different objects; among them, the object can select the CanalConnector object to connect to the canal server and obtain data from the server queue corresponding to the canal server.

[0046] It should also be noted that the data in the server queue corresponding to the canal server is the data sent by the canal server to the server queue for storage; and the data sent by the canal server to the server queue for storage is the data obtained by the canal server after parsing the data change log sent by the upstream database system; among which, the data change log is the log synchronized to the canal server based on the master-slave synchronization mechanism when the upstream database system receives the log request sent by the canal server.

[0047] Specifically, the upstream database system as the master database will write the changed data into the data change log, and the canal server will disguise itself as the slave database of the upstream database system and send a log request to the upstream database system; accordingly, when the upstream database system receives the log request sent by the canal server, it will push the data change log to the canal server based on the master-slave synchronization mechanism. After that, the canal server parses the data change log sent by the upstream database system and sends the parsed data to the server queue for storage.

[0048] The data change log may be a binlog (binary log), which is a log file in the MySQL database and is used to record all data change operations in the database. These data change operations may include database update, deletion, insertion and other operations.

[0049] Regarding the filtering of data to be synchronized, according to one embodiment, the data synchronization service can filter the data to be synchronized obtained from the server queue corresponding to the canal server based on the data whitelist in the target data synchronization filter file, so as to determine the filtered data corresponding to the data whitelist from the data to be synchronized, and then send the filtered data to the target client queue corresponding to any canal client for storage, so that any canal client can synchronize the data stored in the target client queue to the target downstream database system corresponding to any canal client. The data whitelist includes any one or a combination of database whitelist, data table whitelist and data field whitelist.

[0050] Regarding the filtering of data to be synchronized, according to another embodiment, the data synchronization service can filter the data to be synchronized obtained from the server queue corresponding to the canal server based on the data blacklist in the target data synchronization filter file, so as to delete the data corresponding to the data blacklist from the data to be synchronized, and determine the filtered data based on the remaining data in the data to be synchronized, and then send the filtered data to the target client queue corresponding to any canal client for storage, so that any canal client can synchronize the data stored in the target client queue to the target downstream database system corresponding to any canal client. The data blacklist includes any one or a combination of a database blacklist, a data table blacklist, and a data field blacklist.

[0051] Regarding the filtering of data to be synchronized, according to another embodiment, if the number of data to be synchronized obtained from the server queue corresponding to the canal server is multiple, it is necessary to filter the multiple data in the data to be synchronized one by one in sequence based on the target data synchronization filter file, and then send the filtered data to the target client queue corresponding to any canal client for storage, so that any canal client can synchronize the data stored in the target client queue to the target downstream database system corresponding to any canal client.

[0052] Specifically, if there are multiple data to be synchronized, the first data in the data to be synchronized is determined as the current data to be synchronized, and the current data to be synchronized is filtered based on the target data synchronization filter file to obtain the current filtered data, and then the current filtered data is sent to the target client queue for storage, so that any canal client can synchronize the data stored in the target client queue to the target downstream database system corresponding to any canal client. After completing the filtering of the current data to be synchronized, the data synchronization service will further determine whether the current data to be synchronized is the last data in the data to be synchronized. If not, the next data in the data to be synchronized is determined as the new current data to be synchronized, and jump back to the above step of filtering the current data to be synchronized based on the target data synchronization filter file until the current data to be synchronized is the last data in the data to be synchronized, and then obtain the new data to be synchronized from the server queue to continue data filtering and synchronization.

[0053] Step S12: sending the filtered data to a target client queue corresponding to any canal client for storage, so that any canal client synchronizes the data stored in the target client queue to a target downstream database system corresponding to any canal client.

[0054] In this embodiment, after filtering the data to be synchronized obtained from the server queue corresponding to the canal server based on the target data synchronization filter file corresponding to any canal client to obtain the filtered data, the data synchronization service sends the filtered data to the target client queue corresponding to any canal client for storage, so that any canal client synchronizes the data stored in the target client queue to the target downstream database system corresponding to any canal client. In this way, through the one-to-one correspondence between the canal client, the data synchronization filter file and the downstream database system, differential data synchronization of different downstream database systems can be achieved.

[0055] Step S13: If a change request for the target data synchronization filter file is obtained through the first preset interface in the file configuration page, the target data synchronization filter file in the preset persistent storage is changed based on the change request, and the changed data synchronization filter file is reloaded into the preset memory so as to filter the data to be synchronized based on the changed data synchronization filter file.

[0056] In this embodiment, when you want to change the data synchronization filter file in the preset persistent storage, you can obtain a change request for the data synchronization filter file through the first preset interface in the file configuration page provided by the data synchronization service, so as to change the corresponding file in the preset persistent storage based on the change request, and after the change is completed, reload the changed data synchronization filter file into the local preset memory of the data synchronization service, so that the data synchronization service can continue to filter the to-be-synchronized data obtained from the server queue based on the changed data synchronization filter file. Among them, the first preset interface in the file configuration page is an API (Application Programming Interface) in the form of REST exposed to the outside. In this way, this embodiment can easily and quickly implement changes to the data synchronization filter file in the persistent storage through the externally exposed interface in the file configuration page provided by the data synchronization service.

[0057] It can be seen that the present application stores the data synchronization filter file in the persistent storage in advance, so that when the data synchronization service is pre-integrated with several canal clients, the data synchronization filter file in the persistent storage is loaded into the local memory of the data synchronization service, so that the data synchronization service can filter the data based on the data synchronization filter file in the local memory. In this way, by migrating the data filtering operation from the original canal server to the data synchronization service integrated with several canal clients, the resource consumption of the canal server can be reduced; and when the data synchronization filter file needs to be changed, it is only necessary to obtain the file change request through the preset interface in the file configuration page provided by the data synchronization service, so that the change of the data synchronization filter file in the persistent storage can be realized, and then the changed data synchronization filter file is reloaded into the local memory of the data synchronization service, so that the dynamic change of the data synchronization filter file can be realized without restarting the canal server, and the data can continue to be filtered based on the changed data synchronization filter file. Further, since different canal clients in the present application correspond to different downstream database systems and different data synchronization filter files, different data synchronization filter files can be used to realize differential data synchronization of different downstream database systems.

[0058] See also Figure 2 and Figure 3 As shown, an embodiment of the present invention discloses a data synchronization method, which is applied to a data synchronization service integrated with several canal clients, and different canal clients correspond to different downstream database systems and data synchronization filter files; wherein the method includes:

[0059] The data synchronization service reads the target data synchronization filter file corresponding to any canal client from the preset persistent storage, loads the target data synchronization filter file into the local preset memory, and then calls the second preset interface of any canal client to create the corresponding target object in the data synchronization service. After the target object is created, the data synchronization service pulls the data to be synchronized from the server queue corresponding to the canal server through the target object created locally by any canal client, and filters the data to be synchronized based on the target data synchronization filter file in the local preset memory.

[0060] When the target data synchronization filter file contains a data whitelist and the number of data to be synchronized is multiple, the data synchronization service determines the first data in the data to be synchronized as the current data to be synchronized, and filters and matches the current data to be synchronized based on the data whitelist in the target data synchronization filter file. If the filtering and matching is successful, the current filtered data corresponding to the data whitelist is determined from the current data to be synchronized, and the current filtered data is sent to the target client queue corresponding to any canal client for storage, so that any canal client synchronizes the data stored in the target client queue to the target downstream database system corresponding to any canal client. Further, after completing the filtering and matching of the current data to be synchronized, the data synchronization service needs to determine whether the current data to be synchronized is the last data in the data to be synchronized. If the current data to be synchronized is not the last data in the data to be synchronized, it indicates that the matching of the data to be synchronized has not been completed. At this time, it is necessary to determine the next data in the data to be synchronized as the new current data to be synchronized, and jump again to the step of filtering and matching the current data to be synchronized based on the data whitelist in the target data synchronization filter file. If the current data to be synchronized is the last data in the data to be synchronized, it indicates that the matching of the data to be synchronized has been completed. At this time, it is necessary to obtain new data to be synchronized from the server queue corresponding to the canal server to continue data filtering and synchronization.

[0061] If the data synchronization service fails to filter and match the current data to be synchronized, the current data to be synchronized can be discarded, and it can be further determined whether the current data to be synchronized is the last data in the data to be synchronized. If the current data to be synchronized is not the last data in the data to be synchronized, it indicates that the matching of the data to be synchronized has not been completed. At this time, it is necessary to determine the next data in the data to be synchronized as the new current data to be synchronized, and jump again to the step of filtering and matching the current data to be synchronized based on the data whitelist in the target data synchronization filter file until the current data to be synchronized is the last data in the data to be synchronized, and then obtain new data to be synchronized from the server queue to continue data filtering and synchronization.

[0062] In addition, the data synchronization filter file in the preset persistent storage can be viewed and changed through the file configuration page provided by the data synchronization service. For changes to the data synchronization filter file, a change request for the data synchronization filter file is obtained through the first preset interface in the file configuration page provided by the data synchronization service, so that the corresponding file in the preset persistent storage is changed based on the change request, and after the change is completed, the changed data synchronization filter file is reloaded into the local preset memory of the data synchronization service, so that the data synchronization service continues to filter the data to be synchronized obtained from the server queue based on the changed data synchronization filter file.

[0063] It can be seen that the present application stores the data synchronization filter file in the persistent storage in advance, so that when the data synchronization service is pre-integrated with several canal clients, the data synchronization filter file in the persistent storage is loaded into the local memory of the data synchronization service, so that the data synchronization service can filter the data based on the data synchronization filter file in the local memory. In this way, by migrating the data filtering operation from the original canal server to the data synchronization service integrated with several canal clients, the resource consumption of the canal server can be reduced; and when the data synchronization filter file needs to be changed, it is only necessary to obtain the file change request through the preset interface in the file configuration page provided by the data synchronization service, so that the change of the data synchronization filter file in the persistent storage can be realized, and then the changed data synchronization filter file is reloaded into the local memory of the data synchronization service, so that the dynamic change of the data synchronization filter file can be realized without restarting the canal server, and the data can continue to be filtered based on the changed data synchronization filter file. Further, since different canal clients in the present application correspond to different downstream database systems and different data synchronization filter files, different data synchronization filter files can be used to realize differential data synchronization of different downstream database systems.

[0064] See also Figure 4 As shown, an embodiment of the present invention discloses a data synchronization device, which is applied to a data synchronization service integrated with several canal clients, and different canal clients correspond to different downstream database systems and data synchronization filter files; wherein, the device includes:

[0065] The data filtering module 11 is used to filter the to-be-synchronized data obtained from the server queue corresponding to the canal server based on the target data synchronization filtering file after loading the target data synchronization filtering file corresponding to any canal client in the preset persistent storage into the local preset memory, so as to obtain filtered data; wherein the data in the server queue is the data obtained by the canal server from the upstream database system;

[0066] A data synchronization module 12 is used to send the filtered data to a target client queue corresponding to any canal client for storage, so that any canal client synchronizes the data stored in the target client queue to a target downstream database system corresponding to any canal client;

[0067] The file change module 13 is used to change the target data synchronization filter file in the preset persistent storage based on the change request if a change request for the target data synchronization filter file is obtained through the first preset interface in the file configuration page, and reload the changed data synchronization filter file into the preset memory, so as to filter the data to be synchronized based on the changed data synchronization filter file.

[0068] It can be seen that the present application stores the data synchronization filter file in the persistent storage in advance, so that when data synchronization is performed based on the data synchronization service pre-integrated with several canal clients, the data synchronization filter file in the persistent storage is loaded into the local memory of the data synchronization service, so that the data synchronization service can perform data filtering based on the data synchronization filter file in the local memory. In this way, by migrating the data filtering operation from the original canal server to the data synchronization service integrated with several canal clients, the resource consumption of the canal server can be reduced; and when the data synchronization filter file needs to be changed, it is only necessary to obtain the file change request through the preset interface in the file configuration page provided by the data synchronization service, so that the change of the data synchronization filter file in the persistent storage can be realized, and then the changed data synchronization filter file is reloaded into the local memory of the data synchronization service, so that the dynamic change of the data synchronization filter file can be realized without restarting the canal server, and the data can continue to be filtered based on the changed data synchronization filter file. Further, since different canal clients in the present application correspond to different downstream database systems and different data synchronization filter files, different data synchronization filter files can be used to achieve differentiated data synchronization of different downstream database systems.

[0069] In some specific embodiments, the data filtering module 11 includes:

[0070] The first filtering unit is used to obtain the data to be synchronized from the server queue corresponding to the canal server through the target object created by any canal client in the data synchronization service, and filter the data to be synchronized based on the target data synchronization filter file.

[0071] In some specific embodiments, the data synchronization device further includes:

[0072] The object creation unit is used to call the second preset interface of any canal client to create the corresponding target object in the data synchronization service; wherein different canal clients correspond to different objects.

[0073] In some specific embodiments, the data in the server queue is data sent by the canal server to the server queue for storage; the data sent by the canal server to the server queue for storage is data obtained by the canal server after parsing the data change log sent by the upstream database system; the data change log is a log synchronized to the canal server by the upstream database system based on the master-slave synchronization mechanism when receiving the log request sent by the canal server.

[0074] In some specific embodiments, the data filtering module 11 includes:

[0075] A second filtering unit is used to filter the to-be-synchronized data obtained from the server queue corresponding to the canal server based on the data whitelist in the target data synchronization filtering file, so as to determine the filtered data corresponding to the data whitelist from the to-be-synchronized data;

[0076] The data whitelist includes any one or a combination of a database whitelist, a data table whitelist and a data field whitelist.

[0077] In some specific embodiments, the data filtering module 11 includes:

[0078] a third filtering unit, configured to filter the data to be synchronized obtained from the server queue corresponding to the canal server based on the data blacklist in the target data synchronization filtering file, so as to delete the data corresponding to the data blacklist from the data to be synchronized, and determine the filtered data based on the remaining data in the data to be synchronized;

[0079] The data blacklist includes any one or a combination of a database blacklist, a data table blacklist and a data field blacklist.

[0080] In some specific embodiments, the data filtering module 11 includes:

[0081] a fourth filtering unit, configured to obtain the data to be synchronized from the server queue corresponding to the canal server, and if there are multiple data to be synchronized, determine the first data in the data to be synchronized as the current data to be synchronized, and filter the current data to be synchronized based on the target data synchronization filter file to obtain the current filtered data, so as to send the current filtered data to the target client queue for storage;

[0082] A step jump unit is used to determine whether the current data to be synchronized is the last data in the data to be synchronized. If not, the next data in the data to be synchronized is determined as the new current data to be synchronized, and the step of filtering the current data to be synchronized based on the target data synchronization filter file is jumped again until the current data to be synchronized is the last data in the data to be synchronized, and then new data to be synchronized is obtained from the server queue.

[0083] Furthermore, the present application also discloses an electronic device. Figure 5 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.

[0084] Figure 5 A schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the data synchronization method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0085] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0086] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0087] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, and may be Windows Server, Netware, Unix, Linux, etc. In addition to computer programs that can be used to complete the data synchronization method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 may further include computer programs that can be used to complete other specific tasks.

[0088] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the aforementioned disclosed data synchronization method. The specific steps of the method can refer to the corresponding contents disclosed in the aforementioned embodiments, and will not be repeated here.

[0089] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0090] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0091] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0092] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0093] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data synchronization method, characterized in that: The method is applied to a data synchronization service integrated with several canal clients, where different canal clients correspond to different downstream database systems and data synchronization filter files; wherein the method comprises: After loading the target data synchronization filter file corresponding to any canal client in the preset persistent storage into the local preset memory, filtering the to-be-synchronized data obtained from the server queue corresponding to the canal server based on the target data synchronization filter file to obtain filtered data; wherein the data in the server queue is the data obtained by the canal server from the upstream database system; Sending the filtered data to a target client queue corresponding to any canal client for storage, so that any canal client synchronizes the data stored in the target client queue to a target downstream database system corresponding to any canal client; If a change request for the target data synchronization filter file is obtained through the first preset interface in the file configuration page, the target data synchronization filter file in the preset persistent storage is changed based on the change request, and the changed data synchronization filter file is reloaded into the preset memory so that the data to be synchronized can be filtered based on the changed data synchronization filter file.

2. The data synchronization method according to claim 1, characterized in that: The filtering of the to-be-synchronized data obtained from the server queue corresponding to the canal server based on the target data synchronization filter file includes: The target object created in the data synchronization service by any canal client obtains the data to be synchronized from the server queue corresponding to the canal server, and filters the data to be synchronized based on the target data synchronization filter file.

3. The data synchronization method according to claim 2, characterized in that: Before the target object created in the data synchronization service by any canal client obtains the data to be synchronized from the server queue corresponding to the canal server, the method further includes: The second preset interface of any canal client is called to create the corresponding target object in the data synchronization service; wherein different canal clients correspond to different objects.

4. The data synchronization method according to claim 1, characterized in that: The data in the server queue is the data sent by the canal server to the server queue for storage; the data sent by the canal server to the server queue for storage is the data obtained by the canal server after parsing the data change log sent by the upstream database system; the data change log is the log synchronized to the canal server by the upstream database system based on the master-slave synchronization mechanism when receiving the log request sent by the canal server.

5. The data synchronization method according to claim 1, characterized in that: The filtering of the to-be-synchronized data obtained from the server queue corresponding to the canal server based on the target data synchronization filter file to obtain filtered data includes: Filtering the data to be synchronized obtained from the server queue corresponding to the canal server based on the data whitelist in the target data synchronization filter file, so as to determine the filtered data corresponding to the data whitelist from the data to be synchronized; The data whitelist includes any one or a combination of a database whitelist, a data table whitelist and a data field whitelist.

6. The data synchronization method according to claim 1, characterized in that: The filtering of the to-be-synchronized data obtained from the server queue corresponding to the canal server based on the target data synchronization filter file to obtain filtered data includes: Filtering the data to be synchronized obtained from the server queue corresponding to the canal server based on the data blacklist in the target data synchronization filter file, so as to delete the data corresponding to the data blacklist from the data to be synchronized, and determining the filtered data based on the remaining data in the data to be synchronized; The data blacklist includes any one or a combination of a database blacklist, a data table blacklist and a data field blacklist.

7. The data synchronization method according to any one of claims 1 to 6, characterized in that: The filtering of the to-be-synchronized data obtained from the server queue corresponding to the canal server based on the target data synchronization filter file to obtain filtered data includes: Acquire the data to be synchronized from the server queue corresponding to the canal server, and if there are multiple data to be synchronized, determine the first data in the data to be synchronized as the current data to be synchronized, and filter the current data to be synchronized based on the target data synchronization filter file to obtain the current filtered data, so as to send the current filtered data to the target client queue for storage; Determine whether the current data to be synchronized is the last data in the data to be synchronized; if not, determine the next data in the data to be synchronized as the new current data to be synchronized, and jump again to the step of filtering the current data to be synchronized based on the target data synchronization filter file until the current data to be synchronized is the last data in the data to be synchronized, and then obtain new data to be synchronized from the server queue.

8. A data synchronization device, characterized in that: The device is applied to a data synchronization service integrating several canal clients, wherein different canal clients correspond to different downstream database systems and data synchronization filter files; wherein the device comprises: A data filtering module is used to filter the to-be-synchronized data obtained from the server queue corresponding to the canal server based on the target data synchronization filtering file after loading the target data synchronization filtering file corresponding to any canal client in the preset persistent storage into the local preset memory, so as to obtain filtered data; wherein the data in the server queue is the data obtained by the canal server from the upstream database system; A data synchronization module, used for sending the filtered data to a target client queue corresponding to any canal client for storage, so that any canal client synchronizes the data stored in the target client queue to a target downstream database system corresponding to any canal client; A file change module is used to change the target data synchronization filter file in the preset persistent storage based on the change request if a change request for the target data synchronization filter file is obtained through the first preset interface in the file configuration page, and to reload the changed data synchronization filter file into the preset memory so as to filter the data to be synchronized based on the changed data synchronization filter file.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the data synchronization method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program, which, when executed by a processor, implements the data synchronization method as described in any one of claims 1 to 7.

Images