Data synchronization method and device, electronic equipment and computer storage medium

Abstract

The invention provides a data synchronization method and device, electronic equipment and a computer storage medium, after basic configuration information of a data synchronization task is received, anomaly detection is performed on a first data source and a second data source, and if an anomaly detection result of the first data source and an anomaly detection result of the second data source are normal, the data synchronization task is executed. If yes, for each to-be-synchronized data table in the first data source, converting the data type of data in the to-be-synchronized data table to obtain a target to-be-synchronized data table; wherein the data type of the data in the target to-be-synchronized data table is consistent with the data type of the data in the target data table in the second data source; for each target to-be-synchronized data table, generating a target secure transmission rule; and synchronizing data in the target to-be-synchronized data table to a target data table in the second data source by using the target secure transmission rule. Therefore, the purpose of adapting to constantly changing data sources and data formats is achieved, and the data quality after data synchronization is effectively improved.

Description

Technical Field

[0001] The present application relates to the technical field of data synchronization, and in particular to a data synchronization method, device, electronic device and computer storage medium. Background Art

[0002] At present, data connection tools (datax, sqoop, flume, etc.) are usually used in the process of data synchronization, but the data source in the data synchronization process only supports conventional structured data sources. However, the current big data integration system usually needs to deal with data from multiple heterogeneous data sources, which may be relational databases, NoSQL databases, file systems, Kafka streams, etc. Existing technologies are difficult to adapt to the ever-changing data sources and data formats, so data loss and data integrity damage may occur during the synchronization process, resulting in a decrease in data quality after data synchronization. Summary of the invention

[0003] In view of this, the present application provides a data synchronization method, device, electronic device and computer storage medium, which can adapt to the ever-changing data sources and data formats, and effectively improve the data quality after data synchronization.

[0004] The first aspect of the present application provides a data synchronization method, comprising:

[0005] Receive basic configuration information of a data synchronization task; wherein the basic configuration information of the data synchronization task includes information of a first data source and information of a second data source;

[0006] Performing anomaly detection on the first data source and the second data source respectively to obtain an anomaly detection result of the first data source and an anomaly detection result of the second data source; wherein the anomaly detection includes at least one of connection detection, performance detection and configuration detection;

[0007] If both the anomaly detection result of the first data source and the anomaly detection result of the second data source are normal, for each data table to be synchronized in the first data source, the data type of the data in the data table to be synchronized is converted to obtain a target data table to be synchronized; wherein the data type of the data in the target data table to be synchronized is consistent with the data type of the data in the target data table in the second data source;

[0008] For each of the target data tables to be synchronized, a target security transmission rule is generated; wherein the target security transmission rule is a security transmission rule corresponding to the target data table to be synchronized;

[0009] The target security transmission rule is used to synchronize the data in the target data table to be synchronized to the target data table in the second data source.

[0010] Optionally, generating a target security transmission rule for each target data table to be synchronized includes:

[0011] For each of the target data tables to be synchronized, generate the obfuscated fields and dynamic salt values ​​of the target data tables to be synchronized;

[0012] A hash operation is performed based on the identifier of the data table to be synchronized, the data type of the data in the data table to be synchronized, the data type of the data in the target data table in the second data source, the obfuscated field of the target data table to be synchronized and the dynamic salt value to obtain the target security transmission rule.

[0013] Optionally, the data synchronization method further includes:

[0014] In the process of synchronizing the data in the target data table to be synchronized to the target data table in the second data source by using the target security transmission rule, an adaptation analysis is performed to obtain an adaptation analysis result; wherein the adaptation analysis includes at least one of data existence analysis, data integrity analysis, data consistency analysis, data range analysis and data format analysis.

[0015] Optionally, after synchronizing the data in the target to-be-synchronized data table to the target data table in the second data source by using the target secure transmission rule, the method further includes:

[0016] For each data synchronization task, when the data synchronization task is completed, the data integrity of each target data table in the synchronized second data source is detected, and a data synchronization result is generated.

[0017] A second aspect of the present application provides a data synchronization device, including:

[0018] A receiving unit, configured to receive basic configuration information of a data synchronization task; wherein the basic configuration information of the data synchronization task includes information of a first data source and information of a second data source;

[0019] an anomaly detection unit, configured to perform anomaly detection on the first data source and the second data source respectively, to obtain an anomaly detection result of the first data source and an anomaly detection result of the second data source; wherein the anomaly detection includes at least one of connection detection, performance detection and configuration detection;

[0020] a conversion unit, configured to convert the data type of the data in each data table to be synchronized in the first data source to obtain a target data table to be synchronized if both the anomaly detection result of the first data source and the anomaly detection result of the second data source are normal; wherein the data type of the data in the target data table to be synchronized is consistent with the data type of the data in the target data table in the second data source;

[0021] A generating unit, configured to generate a target security transmission rule for each target data table to be synchronized; wherein the target security transmission rule is a security transmission rule corresponding to the target data table to be synchronized;

[0022] A synchronization unit is used to synchronize the data in the target data table to be synchronized to the target data table in the second data source by using the target security transmission rule.

[0023] Optionally, the generating unit includes:

[0024] A first generating subunit is used to generate, for each of the target data tables to be synchronized, an obfuscated field and a dynamic salt value of the target data table to be synchronized;

[0025] The second generating sub-unit is used to perform hash operation according to the identifier of the data table to be synchronized, the data type of the data in the data table to be synchronized, the data type of the data in the target data table in the second data source, the obfuscated field of the target data table to be synchronized and the dynamic salt value to obtain the target security transmission rule.

[0026] Optionally, the data synchronization device further includes:

[0027] An adaptation analysis unit is used to perform adaptation analysis in the process of synchronizing the data in the target data table to be synchronized to the target data table in the second data source by using the target security transmission rule, and obtain an adaptation analysis result; wherein the adaptation analysis includes at least one of data existence analysis, data integrity analysis, data consistency analysis, data range analysis and data format analysis.

[0028] Optionally, the data synchronization device further includes:

[0029] The integrity analysis unit is used to detect the data integrity of each target data table in the synchronized second data source for each data synchronization task when the data synchronization task is completed, and generate a data synchronization result.

[0030] A third aspect of the present application provides an electronic device, including:

[0031] one or more processors;

[0032] a storage device having one or more programs stored thereon;

[0033] When the one or more programs are executed by the one or more processors, the one or more processors implement the data synchronization method as described in any one of the first aspects.

[0034] A fourth aspect of the present application provides a computer storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the data synchronization method as described in any one of the first aspects is implemented.

[0035] As can be seen from the above scheme, the present application provides a data synchronization method, device, electronic device and computer storage medium. After receiving the basic configuration information of the data synchronization task, the first data source and the second data source are respectively detected for anomalies to obtain the anomaly detection result of the first data source and the anomaly detection result of the second data source; if the anomaly detection result of the first data source and the anomaly detection result of the second data source are both normal, then for each data table to be synchronized in the first data source, the data type of the data in the data table to be synchronized is converted to obtain a target data table to be synchronized; wherein the data type of the data in the target data table to be synchronized is consistent with the data type of the data in the target data table in the second data source; for each target data table to be synchronized, a target security transmission rule is generated; wherein the target security transmission rule is the security transmission rule corresponding to the target data table to be synchronized; and the data in the target data table to be synchronized is synchronized to the target data table in the second data source using the target security transmission rule. Thus, the purpose of adapting to the ever-changing data source and data format is achieved, and the data quality after data synchronization is effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0037] Figure 1 A specific flow chart of a data synchronization method provided in an embodiment of the present application;

[0038] Figure 2 A flowchart of a method for configuring a data source provided in another embodiment of the present application;

[0039] Figure 3 A schematic diagram of a data synchronization configuration method provided by another embodiment of the present application;

[0040] Figure 4 A flowchart of a method for configuring a data synchronization task provided by another embodiment of the present application;

[0041] Figure 5 An overall framework diagram of a method for implementing data synchronization provided by another embodiment of the present application;

[0042] Figure 6A schematic diagram of a data synchronization device provided by another embodiment of the present application;

[0043] Figure 7 A schematic diagram of an electronic device for implementing a data synchronization method provided in another embodiment of the present application. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0045] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0046] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0047] It should be noted that the concepts such as "first" and "second" mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0048] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0049] The present application embodiment provides a data synchronization method, such as Figure 1 As shown, the specific steps include:

[0050] S101. Receive basic configuration information of a data synchronization task.

[0051] The basic configuration information of the data synchronization task includes information of the first data source and information of the second data source. The data source includes but is not limited to relational databases, NoSQL databases, file systems, Kafka streams, etc., which are not limited here.

[0052] During the specific implementation of this application, a plug-in design is also provided to support various types of data source connectors. Users can easily add new data source support according to their needs. An open interface is also provided to provide rich application programming interfaces (Application Programming Interface, API) to facilitate integration with other systems and services to meet the needs of diverse application scenarios.

[0053] S102: Perform anomaly detection on the first data source and the second data source respectively to obtain an anomaly detection result of the first data source and an anomaly detection result of the second data source.

[0054] The anomaly detection includes at least one of connection detection, performance detection and configuration detection.

[0055] It is understandable that when the abnormality detection result indicates that an abnormality currently exists, a corresponding alarm message may be generated to prompt the staff, etc. to reconfigure the configuration with the abnormality.

[0056] In the specific implementation process of this application, the main goal of connection detection is to verify whether the data source is accessible, including but not limited to network connectivity, the validity of authentication information, etc., which are not limited here. You can use but are not limited to using ping or similar tools to check basic network connectivity and try to establish a connection using the provided authentication information, such as username / password, API key, etc., which are not limited here. For the database, a simple query can be executed to (select 1) to confirm read and write permissions. The final connection detection result indicators are connection success rate, response time, etc., which are not limited here. Specifically, when the connection success rate is greater than the success rate threshold, it means that the connection detection result is normal, and when it is less than the success rate threshold, it means that the connection detection result is abnormal; when the response time is greater than the response threshold, it means that the connection detection result is abnormal, and when it is less than the response threshold, it means that the connection detection result is normal.

[0057] The main goal of performance testing is to evaluate the performance of the data source under different load conditions to ensure that it can meet the requirements of the synchronization task. You can, but are not limited to, simulate data requests of different sizes (for example, the data size is set to the system default size, 5 times the default size, and 10 times the default size), measure the time and throughput of processing data, and observe whether there is a bottleneck. The final performance test result indicators are average response time, throughput, error rate, etc., which are not limited here.

[0058] The main goal of configuration detection is to ensure that all necessary configuration items are set correctly and in accordance with best practices. Configuration detection can be performed by, but is not limited to, checking key parameters in the configuration file, such as timeout settings, number of retries, etc., and verifying whether key parameters of different types of data sources are missing. This confirms whether there are enough resources allocated to data synchronization tasks, such as memory and CPU. The final configuration detection result indicators are configuration item compliance, resource configuration status, etc., which are not limited here. In this way, resource usage can be dynamically adjusted based on actual load conditions, optimizing the utilization efficiency of hardware and network resources.

[0059] In the specific implementation process of the present application, not only can the first data source and the second data source be respectively detected for anomalies after receiving the basic configuration information of the data synchronization task to obtain the anomaly detection result of the first data source and the anomaly detection result of the second data source, but also the first data source and the second data source can be detected for anomalies when the user enters the configuration management to configure the first data source and the second data source, such as Figure 2 As shown, after entering the configuration management, select the data source configuration, configure the source database (the database of the first data source), after the configuration is completed, check whether the first data source has an abnormality, if it is abnormal, reconfigure the source database, if normal, reselect the data source configuration, configure the target database (the database of the second data source), and similarly check whether the second data source has an abnormality, if abnormal, reconfigure the target database, if normal, complete the anomaly detection of the first data source and the second data source, and generate the anomaly detection results of the first data source and the anomaly detection results of the second data source.

[0060] like Figure 3 As shown, in the process of configuring data synchronization tasks in this application, you can choose single-table synchronization, multi-table synchronization and whole-database synchronization. Among them, single-table synchronization can configure field mapping relationships, type mapping relationships, filtering conditions, etc. Multi-table synchronization can be divided into single-database multi-table and sub-database sub-table.

[0061] like Figure 4 As shown, in the specific implementation process of the present application, the configuration process of the data synchronization task can be, but is not limited to, after selecting a real-time task or an offline task, select the source database (the first data source), then select the synchronized table (the data table to be synchronized in the first data source), then select the filtering condition to be operated, and then select the corresponding target library (the second data source), select a new target table or an existing target table (the target data table in the second data source), select the field mapping rule (all by default), and you can also select the monitoring alarm logic, as well as the person who needs to be notified when an abnormality occurs. After the configuration is completed, start the data synchronization task and monitor the data synchronization task.

[0062] S103: If both the anomaly detection result of the first data source and the anomaly detection result of the second data source are normal, for each data table to be synchronized in the first data source, convert the data type of the data in the data table to be synchronized to obtain a target data table to be synchronized.

[0063] The data type of the data in the target data table to be synchronized is consistent with the data type of the data in the target data table in the second data source.

[0064] In the specific implementation process of the present application, the basic file of the first data source can be parsed to obtain the data type of the data in each data table to be synchronized in the first data source. The same is true for the second data source. The basic file of the second data source is parsed to obtain the data type of the data in the target data table to be synchronized in the second data source. Therefore, the data type of the data in the data table to be synchronized in the first data source can be converted in advance to the data type of the data in the target data table to be synchronized in the second data source, and the corresponding data source can be adapted to complete field filtering or redefinition, data filtering, and parsing of user defined functions (UDF) in advance, which are not limited here.

[0065] S104: Generate target security transmission rules for each target data table to be synchronized.

[0066] The target security transmission rule is the security transmission rule corresponding to the target data table to be synchronized.

[0067] It is understandable that when data is synchronized between different systems, an encrypted transmission protocol is used to prevent data leakage or illegal access. This application will generate a security transmission rule corresponding to each target data table to be synchronized.

[0068] Optionally, in another embodiment of the present application, an implementation of step S104 includes:

[0069] For each target data table to be synchronized, the obfuscated fields and dynamic salt value of the target data table to be synchronized are generated. Then, a hash operation is performed based on the identifier of the data table to be synchronized, the data type of the data in the data table to be synchronized, the data type of the data in the target data table in the second data source, the obfuscated fields and dynamic salt value of the target data table to be synchronized, and the target security transmission rule is obtained.

[0070] It is understandable that, using the above method, even if the input parameters are the same, the generated secret key (target security transmission rule) will be different. Distributed management of secret keys is performed, and the generated secret key is divided into multiple fragments, which are stored in different locations and reassembled only when needed. Data verification is performed at the beginning and completion of data synchronization.

[0071] S105: Synchronize the data in the target data table to be synchronized to the target data table in the second data source by using the target security transmission rule.

[0072] In the specific implementation process of the present application, adaptation analysis is performed in the process of synchronizing the data in the target data table to be synchronized to the target data table in the second data source using the target security transmission rules, and the adaptation analysis results are obtained; wherein the adaptation analysis includes at least one of data existence analysis, data integrity analysis, data consistency analysis, data range analysis and data format analysis.

[0073] In the specific implementation process of this application, the main purpose of data existence analysis is to improve detection efficiency and accuracy. Specifically, it can be done by, but not limited to, checking whether a data item exists in the target data source, using BloomFilters to store the core fields of the source data, and then randomly extracting from the target library for quick verification, which is not limited here.

[0074] The main purpose of data integrity analysis is to enhance the depth and breadth of data verification. It can be done by, but not limited to, determining whether there are missing or empty fields, and using the Zero-Knowledge Proof (ZKP) principle to verify data integrity without revealing specific information. This is not limited here.

[0075] The main purpose of data consistency analysis is to strengthen data synchronization and consistency maintenance across systems. This can be done through, but not limited to, change data capture (CDC): capturing changes in the source system and propagating them to other systems in a timely manner to maintain data consistency. This is not limited here.

[0076] The main purpose of data range analysis is to more reasonably define a reasonable numerical range. This can be done by, but not limited to, using statistical methods to analyze historical data, automatically generate reasonable upper and lower limits, and check whether the data is within the expected range. This is not limited here.

[0077] The main purpose of data format analysis is to support more types of data formats and simplify the configuration process. It can automatically match the corresponding format, automatically check whether the data conforms to the expected format, such as date format, numeric format, etc., or provide a graphical interface to allow users to easily set different data format requirements, etc., which is not limited here. Non-technical personnel can also easily manage and adjust synchronization tasks.

[0078] It should be noted that in the process of synchronizing the data in the target data table to be synchronized to the target data table in the second data source using the target security transmission rule, if an abnormal situation is encountered, the operation will be automatically attempted to be restored or a prompt message will be generated to prompt manual intervention to ensure the continuity of data synchronization. The key components are designed with redundancy, so that even if a node fails, the system can still operate normally, improving the overall reliability and stability.

[0079] In the specific implementation process of this application, a single data synchronization task supports single-table synchronization, multi-table synchronization as archive table, multi-table synchronization as sub-library and sub-table, supports rapid creation of target table, and realizes field-level mapping configuration. In addition, data synchronization tasks also support flexible scheduling and real-time monitoring of operation status, operation tracking synchronization batch, synchronization status, total read volume, amount to be synchronized, amount synchronized, operation log, and SMS alarm for abnormal tasks.

[0080] In the specific implementation process of the present application, after synchronizing the data in the target data table to be synchronized to the target data table in the second data source using the target security transmission rules, it is possible to further detect the data integrity of each target data table in the synchronized second data source for each data synchronization task when the data synchronization task is completed, and generate a data synchronization result.

[0081] Specifically, each data synchronization task will trigger a set of verification rules after completion. This includes but is not limited to real-time or regular detection of data integrity in the data source, timely detection of data loss, data inconsistency, data corruption and other issues. An alarm mechanism is provided to notify relevant personnel to handle exceptions. In addition, error logs are analyzed and parsed, and an intuitive dashboard is created to display the key indicators and abnormal conditions of the task, so as to quickly locate the problem.

[0082] In the specific implementation process of this application, after the data synchronization task is completed, the change logs of the two synchronized data sources can continue to be monitored, and incremental data can be automatically captured to reduce the manpower and time consumed by manual synchronization, efficiently complete the automatic real-time synchronization of standby data, and improve the timeliness of business data. For frequently updated data sources, an incremental synchronization strategy is adopted to transmit only the changed parts, saving bandwidth and storage space.

[0083] During the specific implementation of the present application, metadata information can also be automatically updated to ensure that the data models of all data sources remain consistent and reduce errors caused by data structure mismatches.

[0084] like Figure 5 As shown, it is an overall framework diagram of a method for implementing data synchronization provided in an embodiment of the present application, which consists of four parts: data service products, big data cluster platform, self-developed big data API service, and data warehouse.

[0085] Data service products: products defined after business splitting based on business processes, business indicators, timing services, alarm services, scheduler services, etc.

[0086] Big data cluster platform: a technical platform that integrates data collection, storage, processing, analysis and visualization. It has the basic guarantee of elastic scaling, efficient operation and maintenance, scalability and high availability. In the specific implementation process of this application, it can use but is not limited to distributed computing frameworks such as Apache Spark or Flink, which supports efficient processing of large-scale data and significantly improves the speed of data synchronization.

[0087] Self-developed data API service: Customize, optimize and expand big data cluster platform tasks, and customize UDFs for each big data platform.

[0088] Data warehouse: A data warehouse is a system dedicated to integrating, storing and managing all types of enterprise data. Its main function is to improve data access and reporting, allowing users to easily access and report data from multiple sources, and quickly eliminate information islands. All historical data is stored in the warehouse, supporting more analysis scenarios in addition to event tables; at the same time, the computing pressure is transferred to the data warehouse, reducing the operating pressure on online businesses.

[0089] As can be seen from the above scheme, the present application provides a data synchronization method, which performs anomaly detection on the first data source and the second data source respectively after receiving the basic configuration information of the data synchronization task, and obtains the anomaly detection result of the first data source and the anomaly detection result of the second data source; if the anomaly detection result of the first data source and the anomaly detection result of the second data source are both normal, then for each data table to be synchronized in the first data source, the data type of the data in the data table to be synchronized is converted to obtain a target data table to be synchronized; wherein the data type of the data in the target data table to be synchronized is consistent with the data type of the data in the target data table in the second data source; for each target data table to be synchronized, a target security transmission rule is generated; wherein the target security transmission rule is the security transmission rule corresponding to the target data table to be synchronized; and the data in the target data table to be synchronized is synchronized to the target data table in the second data source using the target security transmission rule. Thus, the purpose of adapting to the ever-changing data source and data format is achieved, and the data quality after data synchronization is effectively improved.

[0090] The present application embodiment provides a data synchronization device, such as Figure 6 As shown, specifically including:

[0091] The receiving unit 601 is used to receive basic configuration information of the data synchronization task.

[0092] The basic configuration information of the data synchronization task includes information of the first data source and information of the second data source.

[0093] The anomaly detection unit 602 is used to perform anomaly detection on the first data source and the second data source respectively to obtain an anomaly detection result of the first data source and an anomaly detection result of the second data source.

[0094] The anomaly detection includes at least one of connection detection, performance detection and configuration detection.

[0095] The conversion unit 603 is used to convert the data type of the data in each data table to be synchronized in the first data source to obtain a target data table to be synchronized if the anomaly detection results of the first data source and the anomaly detection results of the second data source are both normal.

[0096] The data type of the data in the target data table to be synchronized is consistent with the data type of the data in the target data table in the second data source.

[0097] The generating unit 604 is used to generate a target security transmission rule for each target data table to be synchronized.

[0098] The target security transmission rule is the security transmission rule corresponding to the target data table to be synchronized.

[0099] Optionally, in another embodiment of the present application, an implementation of the generating unit 604 includes:

[0100] The first generating subunit is used to generate, for each target data table to be synchronized, an obfuscated field and a dynamic salt value of the target data table to be synchronized.

[0101] The second generating sub-unit is used to perform hash operation according to the identifier of the data table to be synchronized, the data type of the data in the data table to be synchronized, the data type of the data in the target data table in the second data source, the obfuscated field of the target data table to be synchronized and the dynamic salt value to obtain the target security transmission rule.

[0102] The specific working process of the units disclosed in the above embodiments of the present application can be found in the corresponding method embodiments, which will not be repeated here.

[0103] The synchronization unit 605 is used to synchronize the data in the target to-be-synchronized data table to the target data table in the second data source by using the target security transmission rule.

[0104] For the specific working process of the units disclosed in the above embodiments of the present application, please refer to the corresponding method embodiments, such as Figure 1 As shown, no further description is given here.

[0105] Optionally, in another embodiment of the present application, an implementation of the data synchronization device further includes:

[0106] The adaptation analysis unit is used to perform adaptation analysis in the process of synchronizing the data in the target to-be-synchronized data table to the target data table in the second data source by using the target security transmission rule, and obtain the adaptation analysis result.

[0107] Among them, the adaptation analysis includes at least one of data existence analysis, data integrity analysis, data consistency analysis, data range analysis and data format analysis.

[0108] The specific working process of the units disclosed in the above embodiments of the present application can be found in the corresponding method embodiments, which will not be repeated here.

[0109] Optionally, in another embodiment of the present application, an implementation of the data synchronization device further includes:

[0110] The integrity analysis unit is used to detect the data integrity of each target data table in the synchronized second data source for each data synchronization task when the data synchronization task is completed, and generate a data synchronization result.

[0111] The specific working process of the units disclosed in the above embodiments of the present application can be found in the corresponding method embodiments, which will not be repeated here.

[0112] As can be seen from the above scheme, the present application provides a data synchronization device, after the receiving unit 601 receives the basic configuration information of the data synchronization task, the anomaly detection unit 602 performs anomaly detection on the first data source and the second data source respectively, and obtains the anomaly detection result of the first data source and the anomaly detection result of the second data source; if the anomaly detection result of the first data source and the anomaly detection result of the second data source are both normal, then the conversion unit 603 converts the data type of the data in the data table to be synchronized for each data table to be synchronized in the first data source, and obtains the target data table to be synchronized; wherein the data type of the data in the target data table to be synchronized is consistent with the data type of the data in the target data table in the second data source; the generation unit 604 generates a target security transmission rule for each target data table to be synchronized; wherein the target security transmission rule is the security transmission rule corresponding to the target data table to be synchronized; the synchronization unit 605 uses the target security transmission rule to synchronize the data in the target data table to be synchronized to the target data table in the second data source. Thereby, the purpose of adapting to the constantly changing data source and data format is achieved, and the data quality after data synchronization is effectively improved.

[0113] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0114] Another embodiment of the present application provides an electronic device, such as Figure 7 As shown, including:

[0115] One or more processors 701.

[0116] The storage device 702 stores one or more programs.

[0117] When the one or more programs are executed by the one or more processors 701, the one or more processors 701 implement the data synchronization method as described in any one of the above embodiments.

[0118] Another embodiment of the present application provides a computer storage medium on which a computer program is stored, wherein when the computer program is executed by a processor, the data synchronization method as described in any one of the above embodiments is implemented.

[0119] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0120] It should be noted that the computer-readable medium mentioned above in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0121] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0122] Another embodiment of the present application provides a computer program product. When the computer program product is executed, it is used to execute any of the above-mentioned data synchronization methods.

[0123] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, the above-mentioned functions defined in the method of the embodiment of the present application are executed.

[0124] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in this application is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing this application.

[0125] Although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present application. Certain features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination.

[0126] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of application involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above application concept. For example, the above features are replaced with (but not limited to) technical features with similar functions applied in the present application.

Claims

1. A data synchronization method, characterized in that: include: Receive basic configuration information of a data synchronization task; wherein the basic configuration information of the data synchronization task includes information of a first data source and information of a second data source; Performing anomaly detection on the first data source and the second data source respectively to obtain an anomaly detection result of the first data source and an anomaly detection result of the second data source; wherein the anomaly detection includes at least one of connection detection, performance detection and configuration detection; If both the anomaly detection result of the first data source and the anomaly detection result of the second data source are normal, for each data table to be synchronized in the first data source, the data type of the data in the data table to be synchronized is converted to obtain a target data table to be synchronized; wherein the data type of the data in the target data table to be synchronized is consistent with the data type of the data in the target data table in the second data source; For each of the target data tables to be synchronized, a target security transmission rule is generated; wherein the target security transmission rule is a security transmission rule corresponding to the target data table to be synchronized; The target security transmission rule is used to synchronize the data in the target data table to be synchronized to the target data table in the second data source.

2. The data synchronization method according to claim 1, characterized in that: The step of generating a target security transmission rule for each target data table to be synchronized includes: For each of the target data tables to be synchronized, generate the obfuscated fields and dynamic salt values ​​of the target data tables to be synchronized; A hash operation is performed based on the identifier of the data table to be synchronized, the data type of the data in the data table to be synchronized, the data type of the data in the target data table in the second data source, the obfuscated field of the target data table to be synchronized and the dynamic salt value to obtain the target security transmission rule.

3. The data synchronization method according to claim 1, characterized in that: Also includes: In the process of synchronizing the data in the target data table to be synchronized to the target data table in the second data source by using the target security transmission rule, an adaptation analysis is performed to obtain an adaptation analysis result; wherein the adaptation analysis includes at least one of data existence analysis, data integrity analysis, data consistency analysis, data range analysis and data format analysis.

4. The data synchronization method according to claim 1, characterized in that: After synchronizing the data in the target to-be-synchronized data table to the target data table in the second data source by using the target security transmission rule, the method further includes: For each data synchronization task, when the data synchronization task is completed, the data integrity of each target data table in the synchronized second data source is detected, and a data synchronization result is generated.

5. A data synchronization device, characterized in that: include: A receiving unit, configured to receive basic configuration information of a data synchronization task; wherein the basic configuration information of the data synchronization task includes information of a first data source and information of a second data source; an anomaly detection unit, configured to perform anomaly detection on the first data source and the second data source respectively, to obtain an anomaly detection result of the first data source and an anomaly detection result of the second data source; wherein the anomaly detection includes at least one of connection detection, performance detection and configuration detection; a conversion unit, configured to convert the data type of the data in each data table to be synchronized in the first data source to obtain a target data table to be synchronized if both the anomaly detection result of the first data source and the anomaly detection result of the second data source are normal; wherein the data type of the data in the target data table to be synchronized is consistent with the data type of the data in the target data table in the second data source; A generating unit, configured to generate a target security transmission rule for each target data table to be synchronized; wherein the target security transmission rule is a security transmission rule corresponding to the target data table to be synchronized; A synchronization unit is used to synchronize the data in the target data table to be synchronized to the target data table in the second data source by using the target security transmission rule.

6. The data synchronization device according to claim 5, characterized in that: The generating unit comprises: A first generating subunit is used to generate, for each of the target data tables to be synchronized, an obfuscated field and a dynamic salt value of the target data table to be synchronized; The second generating sub-unit is used to perform hash operation according to the identifier of the data table to be synchronized, the data type of the data in the data table to be synchronized, the data type of the data in the target data table in the second data source, the obfuscated field of the target data table to be synchronized and the dynamic salt value to obtain the target security transmission rule.

7. The data synchronization device according to claim 5, characterized in that: Also includes: An adaptation analysis unit is used to perform adaptation analysis in the process of synchronizing the data in the target data table to be synchronized to the target data table in the second data source by using the target security transmission rule, and obtain an adaptation analysis result; wherein the adaptation analysis includes at least one of data existence analysis, data integrity analysis, data consistency analysis, data range analysis and data format analysis.

8. The data synchronization device according to claim 5, characterized in that: Also includes: The integrity analysis unit is used to detect the data integrity of each target data table in the synchronized second data source for each data synchronization task when the data synchronization task is completed, and generate a data synchronization result.

9. An electronic device, characterized in that: include: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the data synchronization method as claimed in any one of claims 1 to 4.

10. A computer storage medium, characterized in that: A computer program is stored thereon, wherein when the computer program is executed by a processor, the data synchronization method as claimed in any one of claims 1 to 4 is implemented.

Images