Data synchronization method and device, electronic equipment and readable storage medium

By verifying and managing permissions during the data synchronization process, the problems of table structure, data consistency and sensitive data leakage during the data synchronization process in the prior art are solved, and safe and efficient data synchronization is achieved.

CN119046374APending Publication Date: 2024-11-29DUXIAOMAN TECH (BEIJING) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410997370.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-24
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

The existing technology cannot guarantee the consistency of online and offline table structures and data during data synchronization, and there is a risk of sensitive data leakage during synchronization.

Method used

By receiving data synchronization requests from offline databases, obtain cluster information of online databases, select target cluster and sub-database information, perform permission verification, ensure that offline databases have access permissions and perform data synchronization, use metadatabases to store permission information, and ensure data Security and consistency during synchronization.

Benefits of technology

The consistency of online and offline table structure and data during the data synchronization process is achieved, which avoids sensitive data leakage and ensures data security.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119046374A_ABST
    Figure CN119046374A_ABST
Patent Text Reader

Abstract

The invention provides a data synchronization method and device, electronic equipment and a readable storage medium, and the method comprises the steps: obtaining cluster information with a reading authority in an online database according to a data synchronization request, selecting a target cluster from the cluster information through an offline database, and selecting target sub-database information and target table information corresponding to the target cluster; by performing authority verification of data synchronization on the target sub-database and the target table, the security of the sensitive data in the online database is ensured, and the sensitive data in the online database is prevented from being leaked. After the verification is passed, the target sub-database and the target table data are synchronized to the offline database according to the data synchronization request, and data unification and table structure unification of the synchronous data do not need to be carried out for different online databases, so that the consistency of online and offline table structures and data in the data synchronization process is guaranteed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data synchronization method, device, electronic device and readable storage medium. Background Art

[0002] For the joint debugging environment database as an offline database, when different development ends need to jointly test different codes in the joint debugging environment, the joint debugging environment database needs to synchronize data with each online database to ensure that each module can work together correctly. The synchronization method is usually synchronized by a synchronization tool. The synchronization tool synchronization method uses open source structure synchronization data software to synchronize to the joint debugging environment database in real time. It cannot guarantee the consistency of the online and offline table structures and data during data synchronization. Summary of the invention

[0003] In view of this, embodiments of the present invention provide a data synchronization method, device, electronic device, and readable storage medium to solve the problem that the table structure and data consistency cannot be guaranteed during the data synchronization process.

[0004] According to one aspect of the present invention, there is provided a data synchronization method, comprising:

[0005] Receive data synchronization requests sent by offline databases;

[0006] In response to the data synchronization request, obtaining cluster information with read permission in the online database;

[0007] Based on the cluster information, receiving target cluster information selected by the offline database;

[0008] Obtaining sub-database information and table information corresponding to the target cluster information, wherein each cluster corresponds to a number of sub-databases, and each of the sub-databases corresponds to a number of tables;

[0009] Based on the sub-database information and the table information, receiving target sub-database information and target table information selected by the offline database;

[0010] Obtaining authority information corresponding to the target sub-database information and the target table information;

[0011] Determining the permission of the offline database to access the target sub-database and the target table according to the permission information;

[0012] When the offline database does not have permission to access the target sub-database and the target table, returning no-authority-access information to the offline database;

[0013] Alternatively, when the offline database has the authority to access the target sub-database and the target table, the data in the target sub-database and the target table are synchronized to the offline database according to the data synchronization request.

[0014] Optionally, after receiving the target cluster information selected by the offline database based on the cluster information, the method further includes:

[0015] Obtain configuration parameter information corresponding to the target cluster information;

[0016] The environmental parameters of the offline database are constructed according to the configuration parameter information.

[0017] Optionally, before obtaining the cluster information with read permission in the online database in response to the data synchronization request, the method further includes:

[0018] Obtaining a first permission to read data in the online database;

[0019] The information corresponding to the first authority is stored in a metadata database, where the metadata database is used to store attribute data of the data to be synchronized, and the attribute data includes authority data.

[0020] Optionally, after returning the unauthorized access information to the offline database, the method further includes:

[0021] Receiving a permission application request for the offline database;

[0022] In response to the permission application request, obtaining permission modification information of the online database;

[0023] When the permission modification information indicates that the offline database has permission to access the target sub-database and the target table, the first permission in the metadata database is modified.

[0024] According to a second aspect of the present invention, there is provided a data synchronization device, comprising:

[0025] A first receiving module, used to receive a data synchronization request sent by an offline database;

[0026] A first processing module, in response to the data synchronization request, obtains cluster information with read permission in the online database;

[0027] A second receiving module, configured to receive target cluster information selected by the offline database based on the cluster information;

[0028] A second processing module is used to obtain sub-database information and table information corresponding to the target cluster information, wherein each cluster corresponds to a number of sub-databases, and each of the sub-databases corresponds to a number of tables;

[0029] A third receiving module, configured to receive target sub-database information and target table information selected by the offline database based on the sub-database information and the table information;

[0030] A third processing module is used to obtain the authority information corresponding to the target sub-database information and the target table information;

[0031] A fourth processing module, configured to determine the permission of the offline database to access the target sub-database and the target table according to the permission information;

[0032] A fifth processing module, configured to return unauthorized access information to the offline database when the offline database does not have the authority to access the target sub-database and the target table;

[0033] The sixth processing module is used to synchronize the data in the target sub-database and the target table to the offline database according to the data synchronization request when the offline database has the authority to access the target sub-database and the target table.

[0034] Optionally, the data synchronization device further includes:

[0035] A fourth receiving module, used to obtain configuration parameter information corresponding to the target cluster information;

[0036] The seventh processing module is used to construct the environmental parameters of the offline database according to the configuration parameter information.

[0037] Optionally, the data synchronization device further includes:

[0038] A fifth receiving module, used to obtain a first permission to read data in the online database;

[0039] An eighth processing module is used to store the information corresponding to the first authority in a metadata database, where the metadata database is used to store attribute data of data to be synchronized, and the attribute data includes authority data.

[0040] Optionally, the data synchronization device further includes:

[0041] A sixth receiving module, configured to receive a permission application request for the offline database;

[0042] a seventh receiving module, in response to the permission application request, obtaining permission modification information of the online database;

[0043] The ninth processing module is used to modify the first permission in the metadata database when the permission modification information indicates that the offline database has permission to access the target sub-database and the target table.

[0044] According to a third aspect of the present invention, there is provided an electronic device, comprising:

[0045] Processor; and

[0046] Memory for storing programs,

[0047] The program includes instructions, and when the instructions are executed by a processor, the processor executes any one of the methods according to the first aspect of the present invention.

[0048] According to a fourth aspect of the present invention, there is provided a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute any method according to the first aspect of the present invention.

[0049] One or more technical solutions provided in the embodiments of the present invention obtain cluster information with read permissions in the online database according to the data synchronization request, and the offline database selects the target cluster from the cluster information, and selects the target sub-database information and target table information corresponding to the target cluster. By verifying the permissions for data synchronization of the target sub-database and the target table, the security of sensitive data in the online database is guaranteed, and the leakage of sensitive data in the online database is avoided. After the verification is passed, according to the data synchronization request, the target sub-database and target table data are synchronized to the offline database, and there is no need to unify the data and table structure of the synchronized data for different online databases, so as to ensure the consistency of the online and offline table structures and data during the data synchronization process. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Further details, features and advantages of the invention are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0051] Figure 1 A flow chart showing a data synchronization method according to an exemplary embodiment of the present invention is shown;

[0052] Figure 2 shows a filtering diagram of a data transmission system according to an exemplary embodiment of the present invention;

[0053] Figure 3 A schematic diagram of a joint debugging image library according to an exemplary embodiment of the present invention is shown;

[0054] Figure 4 A data synchronization principle diagram according to an exemplary embodiment of the present invention is shown;

[0055] Figure 5 A schematic diagram of a data transmission system according to an exemplary embodiment of the present invention is shown;

[0056] Figure 6 An implementation diagram of a data synchronization method according to an exemplary embodiment of the present invention is shown;

[0057] Figure 7 A schematic block diagram of a data synchronization device according to an exemplary embodiment of the present invention is shown;

[0058] Figure 8 A block diagram of an exemplary electronic device that can be used to implement an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0059] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0060] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0061] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc. mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0062] It should be noted that the modifications of "one" and "plurality" mentioned in the present invention are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0063] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes, and are not used to limit the scope of these messages or information.

[0064] The solution of the present invention is described below with reference to the accompanying drawings, and the technical solution provided by the embodiment of the present invention is explained in detail through specific embodiments and their application scenarios.

[0065] Data synchronization usually uses script synchronization and synchronization tool synchronization. In the coordinated development scenario, the script synchronization method manages the database scripts of each development end and regularly maintains the database table structure and data of the joint debugging environment. Since the script synchronization method is maintained by multiple development ends, it is more difficult to manage.

[0066] The synchronization tool uses open source structure synchronization data software to synchronize to the joint debugging environment database in real time. It cannot guarantee the consistency of online and offline table structures and data during data synchronization. In addition, when each business uses the open source synchronization tool for synchronization, it needs to apply for an account separately, and it cannot guarantee that private data will not be leaked during the synchronization process.

[0067] In view of this, the present invention provides a data synchronization method, device, electronic device and readable storage medium, wherein the data synchronization method is applied to any electronic device with data synchronization function, and the electronic device includes but is not limited to: personal mobile terminal device, computer or server device, etc.

[0068] like Figure 1 As shown, Figure 1 1 is a flow chart of a data synchronization method provided by an embodiment of the present invention. The data synchronization method may include the following steps S101 to S109:

[0069] S101, receiving a data synchronization request sent by an offline database.

[0070] As an implementation method, a data transmission system is used to synchronize data between the online database and the offline database. The data transmission system (DTS, Data Transmission Service) receives data synchronization requests sent by the offline database. When there is a need for data synchronization, the offline database generates a data synchronization request and sends it to the data transmission system.

[0071] As a specific implementation method, if the offline database is a joint debugging environment database, the online database is the development end of multiple APPs in the joint debugging environment, where the joint debugging environment is usually a development environment deployed on a local server, mainly used by APP developers to perform joint debugging of front-end and back-end interfaces to ensure normal parameter transmission. When there is a data synchronization requirement in the joint debugging environment database, a data synchronization request is sent to the data transmission system.

[0072] S102, in response to the data synchronization request, obtaining cluster information with read permission in the online database.

[0073] As an implementation method, after the data transmission system receives a data synchronization request sent by an offline database, it determines the corresponding online database based on the data synchronization request, and obtains the cluster information with read permission in the online database. The cluster information with read permission is pre-recorded in the metadata database of the data transmission system, and the cluster information includes the cluster name.

[0074] As a specific implementation method, after adding authorization in the data transmission system DTS to clusters D1, D2, etc. with read permissions in advance, the information that clusters D1, D2, etc. have read permissions is written into the metadata database of the data transmission system. After the data transmission system receives the data synchronization request sent by the offline database, it determines the corresponding online database according to the data synchronization request, and obtains a list of all cluster names with read permissions in the online database from the metadata.

[0075] As a specific implementation method, the online database includes a master database, a slave database, and a standby database. The slave database is a backup of the master database. When the master database changes, the slave database must be updated accordingly. The standby database is a database that is updated along with the slave database to prevent the normal data interaction tasks when the master and slave databases fail. Based on the operation status of the online database and the data synchronization request, the master database, slave database, or standby database is selected for data synchronization.

[0076] S103: Based on the cluster information, receive target cluster information selected by the offline database.

[0077] As an implementation method, after obtaining a list of all cluster names with read permissions in the online database, the list is sent to the offline database, and the offline database selects the target cluster information from the list of all cluster names. The selection principle can be to select the target cluster that is in an idle state.

[0078] S104, obtaining sub-database information and table information corresponding to the target cluster information, wherein each cluster corresponds to a number of sub-databases, and each sub-database corresponds to a number of tables.

[0079] As an implementation method, a cluster is a group of independent computers connected through a high-speed network. After determining the target cluster information, obtain the sub-database information and table information corresponding to the target cluster information, that is, obtain the information of several computers as sub-databases under the target cluster. Among them, the table (TABLE) is an object used to store data in the sub-database, which is a collection of structured data.

[0080] As a specific implementation method, the target cluster D1 is selected from the clusters D1, D2, etc. with read permissions, and all the sub-database names DB1, DB2, etc. under the target cluster D1 are obtained through the information of the target cluster D1, and the table names table1, table2, etc. in each sub-database are obtained. Among them, the sub-database information and table information under the clusters D1, D2, etc. with read permissions are all stored in the metadata database of the data transmission system.

[0081] S105: Receive target sub-database information and target table information selected by the offline database based on the sub-database information and the table information.

[0082] As an implementation method, after obtaining the target cluster information, the sub-database information and table information corresponding to the target cluster information are obtained, and the sub-database information and table information are sent to the offline database, and the offline database selects the target sub-database information and target table information from the sub-database information and table information.

[0083] S106, obtaining authority information corresponding to the target sub-database information and the target table information.

[0084] As an implementation method, the cluster information that the online database has read permissions for is pre-recorded in the metadata database of the data transmission system. Also recorded is the permission information of each sub-database and table corresponding to the cluster. This permission information is used to verify whether the current offline database has the permission to synchronize each sub-database and table, so as to prevent sensitive information stored in the sub-databases and tables from being leaked by data synchronization operations.

[0085] S107, determining the permission of the offline database to access the target sub-database and the target table according to the permission information.

[0086] As an implementation method, after obtaining the permission information of each sub-database and table corresponding to the target cluster, determine whether you have the permission to access the data in the target sub-database and target table. This permission information is set before data synchronization to limit sensitive information from being leaked to the offline database at will.

[0087] S108, when the offline database does not have the authority to access the target sub-database and the target table, returning the information of no authority to access to the offline database.

[0088] As an implementation method, if the offline database does not have the permission to access the target sub-database and the target table, or has the permission to access the target sub-database but does not have the permission to access the target table, the information that there is no permission to access the target sub-database and the target table is returned to the offline database, indicating that the offline database does not have the permission to synchronize the target sub-database and the target table data.

[0089] As a specific implementation method, when obtaining the configuration information of the target cluster, target sub-database, target table and target cluster, the joint debugging environment database needs to obtain the token authentication of the data transmission system DTS, where the token is a temporary token in computer identity authentication. When data synchronization is required, secondary authentication is required for the target sub-database and target table. If this permission is not available, the status reminder information status: 401 will be returned to the offline database to report errors and prompt missing permissions.

[0090] In this case, the offline database needs to call the permission application interface to apply for the required permissions for the structure synchronization and full synchronization of the database tables in the cluster for the joint debugging environment database. After approval by the online database, the joint debugging environment database is allowed to synchronize data with the offline database currently corresponding to it, and the permission information is updated in the metadata database of the data transmission system. Only then can the offline database configure the data synchronization task and perform a pre-check before data synchronization.

[0091] S109, or, in the case where the offline database has the authority to access the target sub-database and the target table, the data in the target sub-database and the target table are synchronized to the offline database according to the data synchronization request.

[0092] As an implementation method, after the joint debugging environment database, which is an offline database, obtains the required target cluster, target sub-database and target table, as well as the configuration parameters of the target cluster, it performs upstream and downstream configuration, callback configuration, full and incremental synchronization configuration and data synchronization task pre-check through the calling interface of the joint debugging environment database. After the pre-check passes, the data synchronization from the online database to the offline database is completed through the data transmission system.

[0093] As a specific implementation method, when obtaining the configuration information of the target cluster, target sub-database, target table and target cluster, the joint debugging environment database needs to obtain the token authentication of the data transmission system DTS, where the token is a temporary token in computer identity authentication. When it comes to data synchronization, secondary authentication is required for the target sub-database and target table. If this permission is available, the data synchronization task from the online database to the offline database is performed.

[0094] When the data synchronization task is completed or fails, the status of the data synchronization task is reported to the data transmission system. After receiving the status report message, the data transmission system generates a notification of the data synchronization task status success or running status through the joint debugging environment database.

[0095] As a specific implementation method, Figure 2As shown, when the data transmission system completes data synchronization from the online database to the offline database, the data in the online database is read to generate a core message CoreMsg, which is filtered by a filtering module in the data transmission system and written to the offline database after filtering.

[0096] The filtering is to distinguish the data in the online database when starting the synchronization task, and filter the distinguished data definition language (DDL) and data manipulation language (DML) separately. If it is data definition language DDL, it will be released directly and written into the downstream database through the writing module; if it is data manipulation language DML, it will be determined whether its column name is configured with sensitive fields. The column name refers to the name of a given column in a data table or data set. If a sensitive field is configured, this data manipulation language DML will be filtered out and data synchronization will not be performed, thereby achieving the purpose of protecting sensitive data.

[0097] In one embodiment of the present invention, after receiving target cluster information selected from an offline database based on the cluster information, S103 further includes:

[0098] S1031, obtaining configuration parameter information corresponding to the target cluster information;

[0099] S1032: Build environment parameters of the offline database according to the configuration parameter information.

[0100] As an implementation method, after determining the target cluster, obtain the configuration parameter information of the target cluster, including the cluster ID, cluster domain number, cluster priority, and description information of cluster member devices. According to the configuration parameter information, build a joint debugging MySQL environment consistent with the online relational database management system MySQL configuration.

[0101] In one embodiment of the present invention, S102, in response to the data synchronization request, before obtaining the cluster information with read permission in the online database, further includes:

[0102] S1021, obtaining the first permission to read data in the online database;

[0103] S1022: Store information corresponding to the first authority in a metadata repository, where the metadata repository is used to store attribute data of the data to be synchronized, where the attribute data includes authority data.

[0104] As an implementation method, before data synchronization, the first permission for reading data in each online database is pre-set, and the first permission is written into the metadata database of the data transmission system, so as to facilitate the query of the reading permission in the subsequent data synchronization. Among them, the first permission is used to characterize that several clusters in the online database have the reading permission. The metadata database is a warehouse for organizing, storing and managing data according to the data structure. In this embodiment, it is used to store the attribute data of the data to be synchronized, that is, the permission data of the data to be synchronized.

[0105] In one embodiment of the present invention, after returning the unauthorized access information to the offline database in S108, the method further includes:

[0106] S1081, receiving a permission application request for an offline database;

[0107] S1082, in response to the permission application request, obtaining permission modification information of the online database;

[0108] S1083: When the permission modification information indicates that the offline database has permission to access the target sub-database and the target table, modify the first permission in the metadata database.

[0109] As an implementation method, after the secondary authentication for the target sub-database and the target table, if the downstream database does not have the authority to access the target sub-database and the target table, or has the authority to access the target sub-database but does not have the authority to access the target table, the offline database is returned information that it has no authority to access the target sub-database and the target table, indicating that the offline database has no authority to synchronize the target sub-database and the target table data. At this time, the offline database is required to make a permission application request, and the corresponding online database modifies the permission information to grant the offline database access rights. And the first permission is modified in the metadata of the data transmission system according to the new permission information.

[0110] In one embodiment of the present invention, since the offline database Y is a joint debugging environment database, there is a problem of data update in the joint debugging environment database during data synchronization. For the data definition language DDL and the data manipulation language DML, the above-mentioned conflict events need to be resolved separately.

[0111] For the data manipulation language DML, if a new insert conflict event is encountered, the new insert of the column will be ignored. If an update conflict event is encountered, the updated data in the upstream database will overwrite the data in the downstream database.

[0112] For the data definition language DDL, the error code ignoring strategy of the relational database management system MySQL is adopted. When writing to MySQL as an offline database, the error code generated is captured, and then it is determined whether the error code can be ignored in the task. If it cannot be ignored, the error code is processed. If it can be ignored, the data synchronization process continues.

[0113] In one embodiment of the present invention, after the online database X synchronizes data to the offline database Y through the data transmission system, when the data of the online database X is incremented, the configuration information corresponding to the incremental data is obtained, and the configuration information is configured to synchronize only the data definition language DDL, and not the data manipulation language DML. Then, when the table structure of the online database X is subsequently changed, the changed table structure will be synchronized to the downstream database Y.

[0114] In one embodiment of the present invention, Figure 3 As shown in the figure, in some special scenarios, for downstream databases, there are situations where you need to repeatedly modify the column structure, modify a large amount of data, or only need to use a database snapshot at a certain time. In these cases, you need to use the joint debugging mirror library, including:

[0115] 1. Apply for database environment testing of the joint debugging environment database as an offline database.

[0116] 2. After the application process is approved, use the open source application k8s to schedule an empty database instance.

[0117] 3. Use MySQLdump, which comes with the relational database management system MySQL, to export logical data to an empty joint debugging mirror database. MySQLdump is a utility used by MySQL to transfer the storage database, and the joint debugging mirror database is a mirror database of the joint debugging environment database.

[0118] 4. Execute custom scripts in the mirror database (such as batch replacement of certain data, modification of column structure, etc.).

[0119] 5. The mirror database replaces the downstream database to provide a business testing environment for each module.

[0120] 6. After the test is completed, destroy the joint debugging mirror database.

[0121] In one embodiment of the present invention, in order to support the synchronization capability of the joint debugging environment database, it is necessary to pre-determine that the data transmission system has developed relevant functions, so as to support the joint debugging environment database to obtain data from the online database, create task processes, and task status callback notifications. The pre-developed content of the data transmission system is as follows:

[0122] 1. The data transmission system provides an open application programming interface (API) to enable operations such as creating and deleting tasks.

[0123] 2. The data transmission system provides a separate table structure subscription function, which can filter the specified type of data manipulation language DML.

[0124] 3. When synchronizing data from an online database, permission control steps are set, and the data from the online database can only be synchronized after the permission approval process.

[0125] 4. After data synchronization is completed or other task status changes, the data transmission system can perform task status callback.

[0126] 5. The data transmission system can automatically resolve conflicts. When data synchronization conflicts with other tasks, it will automatically resolve conflicts.

[0127] In one embodiment of the present invention, Figure 4 As shown, data synchronization includes three methods: structure migration, full data migration, and incremental data pulling, which can synchronize the data of the source database to be migrated as an online database to the target database to be migrated as an offline database.

[0128] The principle of structure migration is to execute SQL statements in the source database, obtain the current table structure, and execute it downstream.

[0129] The principle of full migration is to scan the data in segments in the source database, use the downstream table structure to splice it into SQL statements, and execute it downstream.

[0130] The principle of incremental data migration is to use the feature of MySQL that all addition, deletion and modification operations are recorded in the binary log, use the upstream table structure to parse the addition, deletion and modification operations recorded in the binary log, and restore them into SQL statements for execution downstream.

[0131] In one embodiment of the present invention, Figure 5 As shown, the distributed data synchronization system Databus is used as the data transmission system to synchronize data from the master database or slave database of the online database to the offline database. The offline database can be the relational database management system MySQL, the open source stream processing platform Kafka, or the search server ElasticSearch. Among them, the distributed data synchronization system Databus includes a front-end module, a server module, and a task synchronization module. Among them, the front-end module: provides users with interactive and visual configuration information. The server module: performs task configuration, task scheduling, high availability and other functions according to the configuration information. The task synchronization module: is used for specific data synchronization, and performs database and table data synchronization operations.

[0132] In one embodiment of the present invention, Figure 6 As shown, the joint debugging environment database is used as the offline database, and the joint debugging environment database sends a data synchronization request to the data transmission system DTS. The joint debugging environment database obtains the cluster information with read permissions in the online database, and obtains the configuration information of the corresponding cluster. At the same time, according to the cluster information with read permissions, the corresponding sub-database information and table information are obtained, and then the permission of the offline database to obtain the sub-database information and table information is secondary authenticated. After the authentication is passed, the data synchronization task is pre-checked. After the check is correct, the data synchronization between the offline database and the online database is performed. If the secondary authentication fails, the offline database submits a permission application in order to synchronize the data after authorization.

[0133] The data synchronization method provided in the embodiment of the present invention obtains cluster information with read permissions in the online database according to the data synchronization request, and the offline database selects the target cluster from the cluster information, and selects the target sub-database information and target table information corresponding to the target cluster. By verifying the permissions for data synchronization of the target sub-database and the target table, the security of sensitive data in the online database is guaranteed, and the leakage of sensitive data in the online database is avoided. After the verification is passed, according to the data synchronization request, the target sub-database and the target table data are synchronized to the offline database, and there is no need to unify the data and table structure of the synchronized data for different online databases, so as to ensure the consistency of the online and offline table structures and data during the data synchronization process.

[0134] Corresponding to the above embodiment, see Figure 7 The embodiment of the present invention further provides a data synchronization device 700, the data synchronization device 700 comprising:

[0135] The first receiving module 701 is used to receive a data synchronization request sent by an offline database;

[0136] The first processing module 702 obtains cluster information with read permission in the online database in response to the data synchronization request;

[0137] The second receiving module 703 is used to receive target cluster information selected by the offline database based on the cluster information;

[0138] The second processing module 704 is used to obtain sub-database information and table information corresponding to the target cluster information, wherein each cluster corresponds to a number of sub-databases, and each sub-database corresponds to a number of tables;

[0139] The third receiving module 705 is used to receive target sub-database information and target table information selected by the offline database based on the sub-database information and table information;

[0140] The third processing module 706 is used to obtain the authority information corresponding to the target sub-database information and the target table information;

[0141] The fourth processing module 707 is used to determine the permission of the offline database to access the target sub-database and the target table according to the permission information;

[0142] The fifth processing module 708 is used to return the information of no permission to the offline database when the offline database does not have the permission to access the target sub-database and the target table;

[0143] The sixth processing module 709 is used to synchronize the data in the target sub-database and the target table to the offline database according to the data synchronization request when the offline database has the permission to access the target sub-database and the target table.

[0144] Optionally, the data synchronization device 700 further includes:

[0145] The fourth receiving module 710 is used to obtain configuration parameter information corresponding to the target cluster information;

[0146] The seventh processing module 711 is used to construct the environmental parameters of the offline database according to the configuration parameter information.

[0147] Optionally, the data synchronization device 700 further includes:

[0148] A fifth receiving module 712, configured to obtain a first permission to read data in an online database;

[0149] The eighth processing module 713 is used to store the information corresponding to the first authority in a metadata database, where the metadata database is used to store attribute data of the data to be synchronized, where the attribute data includes authority data.

[0150] Optionally, the data synchronization device 700 further includes:

[0151] The sixth receiving module 714 is used to receive a permission application request from an offline database;

[0152] The seventh receiving module 715 obtains permission modification information of the online database in response to the permission application request;

[0153] The ninth processing module 716 is used to modify the first permission in the metadata database when the permission modification information indicates that the offline database has the permission to access the target sub-database and the target table.

[0154] The data synchronization device provided in the embodiment of the present invention obtains cluster information with read permission in the online database according to the data synchronization request, and the offline database selects the target cluster from the cluster information, and selects the target sub-database information and target table information corresponding to the target cluster. By verifying the permission of data synchronization for the target sub-database and the target table, the security of sensitive data in the online database is guaranteed, and the leakage of sensitive data in the online database is avoided. After the verification is passed, according to the data synchronization request, the target sub-database and the target table data are synchronized to the offline database, and there is no need to unify the data and table structure of the synchronized data for different online databases, so as to ensure the consistency of the table structure and data online and offline during the data synchronization process.

[0155] The exemplary embodiment of the present invention also provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to cause the electronic device to perform the method according to the embodiment of the present invention when executed by the at least one processor.

[0156] Exemplary embodiments of the present invention also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform a method according to an embodiment of the present invention.

[0157] An exemplary embodiment of the present invention further provides a computer program product, including a computer program, wherein the computer program is used to cause the computer to perform a method according to an embodiment of the present invention when executed by a processor of a computer.

[0158] refer to Figure 8 , a block diagram of an electronic device 800 that can be used as a server or client of the present invention will now be described, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0159] like Figure 8As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0160] A plurality of components in the electronic device 800 are connected to the I / O interface 805, including: an input unit 806, an output unit 807, a storage unit 808, and a communication unit 809. The input unit 806 may be any type of device capable of inputting information to the electronic device 800, and the input unit 806 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 807 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 808 may include, but is not limited to, a disk, an optical disk. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0161] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing module (CPU), a graphics processing module (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above. For example, in some embodiments, the data synchronization method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via the ROM 802 and / or the communication unit 809. In some embodiments, the computing unit 801 may be configured to perform the data synchronization method in any other appropriate manner (e.g., by means of firmware).

[0162] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data synchronization device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0163] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0164] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0165] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0166] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0167] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

Claims

1. A data synchronization method, It is characterized in that include: Receive data synchronization requests sent by offline databases; In response to the data synchronization request, obtaining cluster information with read permission in the online database; Based on the cluster information, receiving target cluster information selected by the offline database; Obtaining sub-database information and table information corresponding to the target cluster information, wherein each cluster corresponds to a number of sub-databases, and each of the sub-databases corresponds to a number of tables; Based on the sub-database information and the table information, receiving target sub-database information and target table information selected by the offline database; Obtaining authority information corresponding to the target sub-database information and the target table information; Determining the permission of the offline database to access the target sub-database and the target table according to the permission information; When the offline database does not have permission to access the target sub-database and the target table, returning no-authority-access information to the offline database; Alternatively, when the offline database has the authority to access the target sub-database and the target table, the data in the target sub-database and the target table are synchronized to the offline database according to the data synchronization request.

2. The data synchronization method according to claim 1, It is characterized in that After receiving the target cluster information selected by the offline database based on the cluster information, the method further includes: Obtain configuration parameter information corresponding to the target cluster information; The environmental parameters of the offline database are constructed according to the configuration parameter information.

3. The data synchronization method according to claim 1, It is characterized in that Before obtaining the cluster information with read permission in the online database in response to the data synchronization request, the method further includes: Obtaining a first permission to read data in the online database; The information corresponding to the first authority is stored in a metadata database, where the metadata database is used to store attribute data of the data to be synchronized, and the attribute data includes authority data.

4. The data synchronization method according to claim 3, It is characterized in that After returning the unauthorized access information to the offline database, the method further includes: Receiving a permission application request for the offline database; In response to the permission application request, obtaining permission modification information of the online database; When the permission modification information indicates that the offline database has permission to access the target sub-database and the target table, the first permission in the metadata database is modified.

5. A data synchronization device, It is characterized in that include: A first receiving module, used to receive a data synchronization request sent by an offline database; A first processing module, in response to the data synchronization request, obtains cluster information with read permission in the online database; A second receiving module, configured to receive target cluster information selected by the offline database based on the cluster information; A second processing module is used to obtain sub-database information and table information corresponding to the target cluster information, wherein each cluster corresponds to a number of sub-databases, and each of the sub-databases corresponds to a number of tables; A third receiving module, configured to receive target sub-database information and target table information selected by the offline database based on the sub-database information and the table information; A third processing module is used to obtain the authority information corresponding to the target sub-database information and the target table information; A fourth processing module, configured to determine the permission of the offline database to access the target sub-database and the target table according to the permission information; A fifth processing module, configured to return unauthorized access information to the offline database when the offline database does not have the authority to access the target sub-database and the target table; The sixth processing module is used to synchronize the data in the target sub-database and the target table to the offline database according to the data synchronization request when the offline database has the authority to access the target sub-database and the target table.

6. The data synchronization device according to claim 5, It is characterized in that The data synchronization device further includes: A fourth receiving module, used to obtain configuration parameter information corresponding to the target cluster information; The seventh processing module is used to construct the environmental parameters of the offline database according to the configuration parameter information.

7. The data synchronization device according to claim 5, It is characterized in that The data synchronization device further includes: A fifth receiving module, used to obtain a first permission to read data in the online database; An eighth processing module is used to store the information corresponding to the first authority in a metadata database, where the metadata database is used to store attribute data of data to be synchronized, and the attribute data includes authority data.

8. The data synchronization device according to claim 7, It is characterized in that The data synchronization device further includes: A sixth receiving module, configured to receive a permission application request for the offline database; a seventh receiving module, in response to the permission application request, obtaining permission modification information of the online database; The ninth processing module is used to modify the first permission in the metadata database when the permission modification information indicates that the offline database has permission to access the target sub-database and the target table.

9. An electronic device, include: processor; as well as Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 4.

10. A non-transitory computer-readable storage medium storing computer instructions, in, The computer instructions are used to make a computer execute the method according to any one of claims 1-4.