Data auditing method and device in data synchronization, electronic equipment and storage medium

Abstract

The invention provides a data auditing method and device in data synchronization, electronic equipment and a storage medium, and relates to the technical field of information, the method comprises the steps that target data is consumed from Topic, the target data comprises to-be-loaded data and a transmission control label, and the transmission control label is a label generated based on the target to-be-loaded data in a preset verification period; loading the to-be-loaded data in the target end database; when the transmission control tag is analyzed in the loading process, generating a verification tag based on the loaded first loading data; and auditing the first loading data and the target to-be-loaded data based on the transmission control label and the verification label. According to the data auditing method and device, the to-be-loaded data and the transmission control label are fused together to form the target data for transmission, so that in the process that the target end device loads the to-be-loaded data to the target end database, data auditing can be achieved based on the transmission control label, participation of a third party is not needed, and therefore the data auditing efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of information technology, and in particular to a data audit method, device, electronic device and storage medium in data synchronization. Background Art

[0002] In today's digital age, data has become one of the most valuable assets. As the core tool for storing and managing this data, the performance and reliability of the database directly affect the efficiency of enterprise operations and the quality of decision-making. However, as the scale of enterprises expands and business needs diversify, a single database is often difficult to meet all needs, and data synchronization technology is needed to ensure that the target database and the source database are consistent.

[0003] In the related art, data synchronization tools such as GoldenGate are usually used to complete data synchronization. During the data synchronization process, a third party is used to audit the data of the target database and the source database.

[0004] However, in the above-mentioned related technologies, when a third party is used to audit data, it involves the interaction between the target database and the source database and the third party, thereby reducing the efficiency of data auditing. Summary of the invention

[0005] The present application provides a data audit method, device, electronic device and storage medium in data synchronization, so as to solve the defect of reducing data audit efficiency in the prior art.

[0006] In a first aspect, an embodiment of the present application provides a data audit method in data synchronization, which is applied to a target end device, and the method includes: Consume target data from the Topic, where the target data includes data to be loaded and a transmission control tag. The data to be loaded is data published by the source device to the Topic and needs to be synchronized from the source database to the target database of the target device. The transmission control tag is a tag generated based on the target data to be loaded within a preset verification period. Loading the data to be loaded into the target database; When the transmission control tag is parsed during the loading process, a verification tag is generated based on the first loading data that has been loaded in the data to be loaded; Based on the transmission control tag and the verification tag, the first loading data and the target data to be loaded are audited to obtain a first audit result.

[0007] According to a data audit method in data synchronization provided by the present application, the first loading data and the target data to be loaded are audited based on the transmission control tag and the verification tag to obtain a first audit result, including: In the case where the transmission control tag is the same as the verification tag, determining that the first audit result is a successful audit; In the case where the transmission control tag is different from the verification tag, the first audit result is determined to be an audit failure, and an exception message is output, where the exception message is used to remind the user that the target data to be loaded is loaded abnormally.

[0008] According to a data audit method in data synchronization provided by the present application, the target data also includes a release scenario tag; When the transmission control tag is parsed during the loading process, generating a verification tag based on the first loading data that has been loaded in the data to be loaded includes: In the case where the publishing scenario tag represents a real-time publishing scenario, when the transmission control tag is parsed during the loading process, a verification tag is generated based on the first loaded data in the data to be loaded.

[0009] According to a data audit method in data synchronization provided by the present application, the method also includes: In the case where the release scenario label represents a timed release scenario, when the loading cycle arrives, at least one of the following audits is performed to obtain a second audit result; Auditing the second loading data loaded in the loading cycle and the data corresponding to the second loading data in the data to be loaded; Auditing the data structure of the second loaded data and the data structure of the data corresponding to the second loaded data in the data to be loaded; The data dictionary of the second loading data and the data dictionary of the data to be loaded corresponding to the second loading data are audited.

[0010] According to a data audit method in data synchronization provided by the present application, the data to be loaded includes incremental data, and the target data also includes incremental change statistics and incremental change types; The loading of the to-be-loaded data in the target database includes: Based on the incremental change statistical information and the incremental change type, the data to be loaded is loaded into the target database.

[0011] In a second aspect, an embodiment of the present application provides a data audit method in data synchronization, which is applied to a source device, and the method includes: Obtain the data to be loaded that needs to be synchronized from the source database of the source device to the target database of the target device; Generate a transmission control tag based on the target data to be loaded within a preset verification period; generating target data based on the data to be loaded and the transmission control tag; The target data is published to a Topic, and the target data is used to instruct a target-end device to consume the target data from the Topic, and to load the data to be loaded into the target-end database. When the transmission control tag is parsed during the loading process, a verification tag is generated based on the first loaded data, and based on the transmission control tag and the verification tag, the first loaded data and the target data to be loaded are audited to obtain a first audit result.

[0012] According to a data audit method in data synchronization provided by the present application, the data to be loaded includes incremental data; the method further includes: The incremental data is published to a target node representing the source device in a distributed network, the incremental data is used to instruct the target node to create a new block, the incremental data and a hash value generated based on the incremental data are stored in the new block, and the new block is sent to other nodes in the distributed network, the new block is used to instruct the other nodes to update the database of the other nodes based on the incremental data when determining that the new block is legal based on the hash value.

[0013] In a third aspect, an embodiment of the present application provides a data auditing device in data synchronization, including: A consumption unit, used to consume target data from a Topic, wherein the target data includes data to be loaded and a transmission control tag, wherein the data to be loaded is data published by a source device to the Topic and needs to be synchronized from a source database to a target database of a target device, and the transmission control tag is a tag generated based on the target data to be loaded within a preset verification period; A loading unit, used for loading the data to be loaded into the target database; A first generating unit, configured to generate a verification tag based on the first loaded data in the data to be loaded when the transmission control tag is parsed during the loading process; The first audit unit is used to audit the first loaded data and the target data to be loaded based on the transmission control tag and the verification tag to obtain a first audit result.

[0014] In a fourth aspect, an embodiment of the present application provides a data auditing device in data synchronization, including: An acquisition unit, used for acquiring data to be loaded that needs to be synchronized from a source database of a source device to a target database of a target device; A second generating unit, configured to generate a transmission control tag based on target data to be loaded within a preset verification period; A third generating unit, configured to generate target data based on the data to be loaded and the transmission control tag; A first publishing unit is used to publish the target data to a Topic, wherein the target data is used to instruct a target-end device to consume the target data from the Topic and load the data to be loaded into the target-end database. When the transmission control tag is parsed during the loading process, a verification tag is generated based on the first loaded data, and based on the transmission control tag and the verification tag, the first loaded data and the target data to be loaded are audited to obtain a first audit result.

[0015] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the data audit method in data synchronization as described in any of the above aspects is implemented.

[0016] In a sixth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data audit method in data synchronization as described in any of the above aspects.

[0017] In a seventh aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the data audit method in data synchronization as described in any of the above aspects.

[0018] The data audit method, device, electronic device and storage medium in data synchronization provided by the present application publish the target data including the data to be loaded and the transmission control tag to the Topic, the target end device consumes the target data from the Topic, loads the data to be loaded in the target end database, and generates a verification tag based on the first loaded data in the data to be loaded when the transmission control tag is parsed during the loading process, and audits the first loaded data and the target data to be loaded based on the transmission control tag and the verification tag to obtain a first audit result. It can be seen that the present application combines the data to be loaded and the transmission control tag to form the target data for transmission. In this way, when the target end device consumes the target data and loads the data to be loaded in the target data into the target end database, it can realize data audit based on the transmission control tag included in the target data without the participation of a third party, thereby improving the efficiency of data audit. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 This is one of the flow charts of the data audit method in data synchronization provided in the embodiment of the present application.

[0021] Figure 2 This is the second flow chart of the data audit method in data synchronization provided in the embodiment of the present application.

[0022] Figure 3 This is the third flow chart of the data audit method in data synchronization provided in the embodiment of the present application.

[0023] Figure 4 It is an overall flow chart of the data audit method in data synchronization provided in the embodiment of the present application.

[0024] Figure 5 It is an interaction diagram of the data audit method in data synchronization provided in an embodiment of the present application.

[0025] Figure 6 It is one of the structural diagrams of the data audit device in data synchronization provided in the embodiment of the present application.

[0026] Figure 7 This is the second structural diagram of the data audit device in data synchronization provided in the embodiment of the present application.

[0027] Figure 8 It is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0029] Combine the following Figure 1-Figure 5 The present invention describes a data auditing method in data synchronization.

[0030] Figure 1 This is one of the flow charts of the data audit method in data synchronization provided by the embodiment of the present application, which is applied to the target end device, such as Figure 1 As shown, the data audit method in data synchronization includes the following steps: Step 101: Consume target data from a Topic, wherein the target data includes data to be loaded and a transmission control tag. The data to be loaded is data published by a source device to the Topic and needs to be synchronized from the source database to the target database of the target device. The transmission control tag is a tag generated based on the target data to be loaded within a preset verification period.

[0031] The data in the source database can be static data or dynamic data. Topic is a Topic corresponding to Kafka. ZCBUS (Zero Copy Bus) can be integrated with Kafka to achieve data synchronization. ZCBUS is a high-performance, low-latency memory sharing technology that can achieve zero-copy transmission of memory data, avoid data replication between memory and external devices, and improve the efficiency and speed of data transmission. Kafka is a distributed stream processing platform that is widely used to build real-time data pipelines and stream applications. It can efficiently process large amounts of data streams and supports multiple languages ​​and platforms.

[0032] For example, after subscribing to consumption data, the target device can consume the target data from the Topic. The target data includes the data to be loaded and the transmission control tag. The data to be loaded can be the full data or incremental data of the source database. The full data and incremental data are stored in different Topics respectively. The full data and incremental data refer to the data transmission method during data synchronization or backup. In data synchronization, full data synchronization refers to synchronizing all data in the source database to the target database, while incremental data synchronization refers to synchronizing the newly added or modified data in the source database to the target database. Incremental data synchronization can reduce the time and cost of data transmission and improve synchronization efficiency.

[0033] When the data to be loaded is incremental data of the source database, the source device can use the CAPTURE plug-in to capture the incremental data in the source database based on a preset statistical period, and generate BSD format data from the captured incremental data. The BSD format data specifically includes the following fields: 1. Database type, 1 byte, which can include: 1) MYSQL; 2) ORACLE; 3) DB2; 4) SQLSERVER; 5) POSTGRESQL; 6) SYBASE.

[0034] 2. Operation type of incremental data, 1 byte, including: OPTYPE_INSERT 0 #Insert operation

[0035] OPTYPE_UPDATE 1 #Update operation

[0036] OPTYPE_DELETE 2 #delete operation

[0037] OPTYPE_DDL 3 #DDL operation

[0038] OPTYPE_TAG 4 #Full release start and end tag information, where 0 is the full release start tag, 1 is the full release end tag, and 2 is the full release exception tag.

[0039] OPTYPE_CONTROL 5 # When the full data starts, the full data initiated label operation added in the incremental data

[0040] # Real-time data comparison, providing transaction-level background automatic comparison

[0041] OPTYPE_INFO 6 #In the real-time publishing scenario, when capturing data in the source database, a transmission control tag is generated based on the data captured within a preset verification period (for example, 5 minutes), and the transmission control tag is sent regularly; OPTYPE_SQL 7 #In the scheduled publishing scenario, different pre-operations or post-operations may be required before or after the scheduled publishing task is executed. OPTYPE_SQL 7 indicates that the target device pre-operation or post-operation service is added to the scheduled publishing task setting, and SQL statement tags are stored separately. SQL statements are used to provide the verification service of the on-cloud and off-cloud data synchronization of the target device, that is, the audit service. The on-cloud here refers to the source database, and the off-cloud refers to the target database, including the audit of the data to be loaded, the audit of the data format, and the audit of the data dictionary, etc.; OPTYPE_DICT 8 #Send the data dictionary of the data to be loaded; OPTYPE_STATISTICS 9 #Data audit statistics.

[0042] 3. Flag bit flag, if it is an update operation, the last bit of the flag bit is 1, if it is a non-update operation, the last bit of the flag bit is 0; 4. Operation time: a string ending with 0

[0043] 5. Database name: a string ending with 0

[0044] 6. Table name: a string ending with 0

[0045] 7. Split column name: If it is 0, it means there is no split column. If it is Data Definition Language (DDL), the following represents the SQL statement. If it is DML, the column value is represented by a string ending with 0.

[0046] Specifically, a loop operation is performed according to the total number of columns to determine whether a string ending with 0 is a column name, and the length, data, numeric type or time type of the column data is converted into a string for storage to facilitate subsequent data auditing.

[0047] Among them, the insert operation can be represented by I (Insert), the update operation can be represented by U (Update), and the delete operation can be represented by D (Delete). When using the CAPTURE plug-in to capture I / U / D / DDL and other operations, the changes of each table in the source data stream are counted based on the preset statistical period. For example, the preset statistical period can be ten minutes. After the statistics are completed, the counted incremental data is published to the Topic, which stores the incremental data, the incremental change type corresponding to the incremental data (I / U / D / DDL, etc.) and the incremental change statistical information. The incremental change statistical information includes, for example, the location of the incremental change, the data source information of the incremental change, and the time of the incremental change, and initializes the statistical information of the table, recounts, and prepares for the next preset statistical period.

[0048] It should be noted that the transmission control tag can be included in the full data publishing service or the ETL processing service. In the full data publishing service, when the full data publishing starts, the tag is written to the incremental topic to mark that the full data publishing has started, and the full topic information (for example, the source of the full data, the target location, the publishing time, etc.) is written to the incremental topic. The purpose of this is to allow the target end device that subscribes to the incremental topic to obtain the detailed information of the full data publishing and perform appropriate processing accordingly. In the ETL processing service, ETL processing is divided into two parts: ETL custom publishing and ETL transparent transmission of the transmission control tags contained in the incremental data published by the superior.

[0049] Step 102: Load the data to be loaded into the target database.

[0050] For example, after acquiring the target data, the target-end device loads the data to be loaded into the target-end database.

[0051] Step 103: When the transmission control tag is parsed during the loading process, a verification tag is generated based on the first loaded data in the data to be loaded.

[0052] For example, in the process of loading the data to be loaded in the target database, if a transmission control tag is parsed from the target data, it means that reconciliation is required, and a tag generation algorithm is used to calculate the first loaded data in the data to be loaded to obtain a verification tag. The tag generation algorithm is the same as the tag generation algorithm used when generating the transmission control tag, and is a tag generation algorithm negotiated by the source device and the target device.

[0053] Step 104: Based on the transmission control tag and the verification tag, the first loading data and the target data to be loaded are audited to obtain a first audit result.

[0054] For example, when generating a check tag, the transmission control tag parsed from the target data is compared with the generated check tag, and a first audit result of the first loaded data and the target data to be loaded is determined based on the comparison result, thereby realizing data auditing during data synchronization.

[0055] It should be noted that when the target device performs a load operation, it can also count the number of successful I / U / D / DDL operations, and this application does not limit this.

[0056] The data audit method in data synchronization provided by the present application publishes the target data including the data to be loaded and the transmission control tag to the Topic, the target end device consumes the target data from the Topic, loads the data to be loaded in the target end database, and generates a verification tag based on the first loaded data in the data to be loaded when the transmission control tag is parsed during the loading process, and audits the first loaded data and the target data to be loaded based on the transmission control tag and the verification tag to obtain a first audit result. It can be seen that the present application combines the data to be loaded and the transmission control tag to form the target data for transmission. In this way, the target end device can realize data audit based on the transmission control tag included in the target data during the process of consuming the target data and loading the data to be loaded in the target data into the target end database, without the participation of a third party, thereby improving the efficiency of data audit, shortening the time of data synchronization, and improving the efficiency of data synchronization; in addition, since the present application realizes data audit during the data synchronization process, the correctness of data synchronization is improved.

[0057] In one embodiment, the above step 104 audits the first loading data and the target data to be loaded based on the transmission control tag and the verification tag to obtain a first audit result, which can be specifically implemented by the following steps: When the transmission control tag is identical to the verification tag, the first audit result is determined to be a successful audit; when the transmission control tag is different from the verification tag, the first audit result is determined to be a failed audit, and exception information is output, wherein the exception information is used to remind the user that the loading of the target data to be loaded is abnormal.

[0058] For example, the transmission control tag parsed from the target data is compared with the generated verification tag. When the transmission control tag and the verification tag are the same, it means that the target database has successfully loaded the target data to be loaded, and the first audit result is determined to be audit success, and the data loading service continues; when the transmission control tag and the verification tag are different, it means that the target database has failed to load the target data to be loaded, and the first audit result is determined to be audit failure, and at the same time, abnormal information is output to remind the user that the loading of the target data to be loaded is abnormal.

[0059] When the target data to be loaded is abnormal, you can handle it by the following methods: Check the synchronization task: First, check whether the synchronization task is configured correctly, including whether the source database, target database, synchronization mode, synchronization cycle, etc. are configured correctly. If there is a problem with the synchronization task, it needs to be repaired in time to ensure the correctness of data synchronization.

[0060] Check the target database: If the synchronization data source is normal, you need to further check whether the target database is correct, including whether the connection, permissions, table structure, etc. of the target database are correct. If there is a problem with the target database, it needs to be repaired in time to ensure correct data synchronization.

[0061] Data rollback: If the data has been synchronized to the target database, but there are exceptions or errors, you can consider rolling back the data. Rollback can be achieved by restoring backup data or deleting abnormal data.

[0062] Data repair: If the data has been synchronized to the target database but there are errors, you can consider repairing the data. Repair can be achieved by modifying the data or adding missing data.

[0063] Record exception information: During the data synchronization process, if there is an exception or error, you need to record the exception information in a timely manner, including the exception type, exception time, exception location, etc. Recording exception information can help engineers quickly locate the problem and handle it.

[0064] Strengthen data quality management: In order to avoid data synchronization anomalies, it is necessary to strengthen data quality management, including the management of synchronization data sources, data cleaning and data verification. Only by ensuring data quality can the correct synchronization of data be ensured.

[0065] It should be noted that after each reconciliation is completed, all reconciliation information will be cleared and the counting will start again in order to prepare for the next reconciliation.

[0066] In this embodiment, the audit result can be automatically determined based on the comparison between the transmission control tag and the verification tag, thereby improving the efficiency of determining the audit result.

[0067] In one embodiment, the target data further includes a release scenario tag; when the transmission control tag is parsed during the loading process in the above step 103, a verification tag is generated based on the first loaded data in the data to be loaded, which can be specifically implemented by the following steps: In the case where the publishing scenario tag represents a real-time publishing scenario, when the transmission control tag is parsed during the loading process, a verification tag is generated based on the first loaded data in the data to be loaded.

[0068] Among them, the real-time publishing scenario is a scenario that has high requirements for the real-time performance of data synchronization.

[0069] For example, when the publishing scenario tag is OPTYPE_INFO 6, it is determined that the publishing scenario tag represents a real-time publishing scenario. In the real-time publishing scenario, when a transmission control tag is parsed during the loading process, a verification tag is generated based on the first loaded data in the data to be loaded.

[0070] In this embodiment, when the publishing scene tag represents a real-time publishing scene, when the transmission control tag is parsed during the loading process, a verification tag is generated based on the first loaded data in the data to be loaded, and based on the transmission control tag and the verification tag, a rapid audit of the first loaded data and the target data to be loaded is achieved.

[0071] In one embodiment, the data audit method in data synchronization further includes the following steps: In the case where the release scenario label represents a timed release scenario, when the loading cycle arrives, at least one of the following audits is performed to obtain a second audit result; Audit the second loading data loaded during the loading cycle and the data corresponding to the second loading data in the data to be loaded; audit the data structure of the second loading data and the data structure of the data corresponding to the second loading data in the data to be loaded; audit the data dictionary of the second loading data and the data dictionary of the data corresponding to the second loading data in the data to be loaded.

[0072] Among them, the scheduled publishing scenario is a scenario with low requirements for the real-time performance of data synchronization, such as the synchronization scenario of telephone bills.

[0073] For example, when the publishing scenario label is OPTYPE_SQL 7, it is determined that the publishing scenario label represents a timed publishing scenario. In the timed publishing scenario, when the loading cycle arrives, the target end device will audit at least one of the second loaded data loaded in the loading cycle and the data corresponding to the second loaded data in the data to be loaded, the data structure of the second loaded data and the data structure of the data corresponding to the second loaded data in the data to be loaded, and the data dictionary of the second loaded data and the data dictionary of the data corresponding to the second loaded data in the data to be loaded to obtain a second audit result.

[0074] In this embodiment, when the release scenario label represents a timed release scenario, at least one of the second loaded data loaded during the loading cycle and the data corresponding to the second loaded data in the data to be loaded, the data structure of the second loaded data and the data structure of the data corresponding to the second loaded data in the data to be loaded, and the data dictionary of the second loaded data and the data dictionary of the data corresponding to the second loaded data in the data to be loaded are audited to improve the accuracy of data auditing.

[0075] In one embodiment, the data to be loaded includes incremental data, and the target data also includes incremental change statistics and incremental change types; the above step 102 loads the data to be loaded in the target database, which can be specifically implemented by the following steps: Based on the incremental change statistical information and the incremental change type, the data to be loaded is loaded into the target database.

[0076] For example, taking the incremental change type as an update operation, if the incremental change statistical information includes that the data to be loaded is updated data, specifically, the data in the second row of Table 1 is updated, then when the target device loads the data to be loaded, it updates the second row of data in the table corresponding to Table 1 in the target database to the data to be loaded.

[0077] In this embodiment, the data to be loaded can be loaded based on the incremental change statistical information and the incremental change type, thereby realizing automatic loading of the data to be loaded and improving data synchronization efficiency.

[0078] Figure 2 This is a second flow chart of a data audit method in data synchronization provided by an embodiment of the present application, which is applied to a source device, such as Figure 2 As shown, the data audit method in data synchronization includes the following steps: Step 201: Obtain the data to be loaded that needs to be synchronized from the source database of the source device to the target database of the target device.

[0079] For example, the data to be loaded can be the full data or incremental data of the source database. A trigger can be added to each table in the source database. The trigger detects the data in the table based on the preset statistical period. When the data in the table changes, the trigger will encapsulate the change information in the table into a message and write it to the Kafka Topic. The change information in the table is the incremental data. Consumers can subscribe to this Topic through Kafka's consumer application programming interface (API) to obtain the table change information in real time, that is, consume incremental data from the Topic. In this way, the data changes in the table can be monitored in real time.

[0080] Step 202: Generate a transmission control tag based on the target data to be loaded within a preset verification period.

[0081] For example, the preset verification period can be represented by the field OPTYPE_INFO 6. For example, the preset verification period is 5 minutes. When capturing data in the source database, a transmission control tag is generated based on the target data to be loaded captured within the preset verification period, that is, a transmission control tag is generated for each preset verification period. For the target device, an audit is performed each time a transmission control tag is parsed.

[0082] Step 203: Generate target data based on the data to be loaded and the transmission control tag.

[0083] For example, when the data to be loaded and the transmission control tag are acquired, the data to be loaded and the transmission control tag are packaged to obtain the target data.

[0084] Step 204: publish the target data to the Topic, wherein the target data is used to instruct the target-end device to consume the target data from the Topic and load the data to be loaded into the target-end database. When the transmission control tag is parsed during the loading process, a verification tag is generated based on the first loaded data, and based on the transmission control tag and the verification tag, the first loaded data and the target data to be loaded are audited to obtain a first audit result.

[0085] For example, when the target data is obtained, the target data is published to the Topic to facilitate the target end device to consume the target data in the Topic. The specific execution method of the target end device after obtaining the target data can refer to the description of steps 102 to 104 above, and this application will not repeat it here.

[0086] The data audit method in data synchronization provided by the present application publishes the target data including the data to be loaded and the transmission control tag to the Topic, the target end device consumes the target data from the Topic, loads the data to be loaded in the target end database, and generates a verification tag based on the first loaded data in the data to be loaded when the transmission control tag is parsed during the loading process, and audits the first loaded data and the target data to be loaded based on the transmission control tag and the verification tag to obtain a first audit result. It can be seen that the present application combines the data to be loaded and the transmission control tag to form the target data for transmission. In this way, when the target end device consumes the target data and loads the data to be loaded in the target data into the target end database, it can realize data audit based on the transmission control tag included in the target data without the participation of a third party, thereby improving the efficiency of data audit.

[0087] This application also stores incremental data and incremental change statistical information in the blockchain, applies blockchain technology to data synchronization scenarios, and achieves more efficient and secure data synchronization by storing incremental data and incremental change statistical information in a distributed network.

[0088] In one embodiment, the data to be loaded includes incremental data; Figure 3 This is a flowchart of the data audit method in data synchronization provided by the embodiment of the present application. Figure 3 As shown, the data audit method in data synchronization also includes the following steps: Step 205: publish the incremental data to a target node representing the source device in a distributed network, wherein the incremental data is used to instruct the target node to create a new block, store the incremental data and a hash value generated based on the incremental data in the new block, and send the new block to other nodes in the distributed network, wherein the new block is used to instruct the other nodes to update the database of the other nodes based on the incremental data when determining that the new block is legal based on the hash value.

[0089] For example, all nodes that need to be synchronized are connected to a distributed network, and each node has a complete copy of the database. When the target node representing the source device receives the incremental data, the preset hash algorithm is used to calculate the hash value of the incremental data, and the incremental data and the corresponding hash value are stored in a new block. The new block is linked to the previous block in the target node to obtain the blockchain corresponding to the target node. Since the blockchain has the characteristics of decentralization and immutability, the new block is sent to other nodes in the distributed network, so that other nodes can confirm the legitimacy of the new block by verifying the hash value of the new block. If other nodes recognize this new block, the incremental data stored in the new block will be updated to the database corresponding to the other nodes to achieve data synchronization, and the new block will be linked to the previous block of other nodes to obtain the blockchain corresponding to other nodes to achieve blockchain synchronization. The other nodes here refer to all nodes in the distributed network except the target node, so that the new block is updated to the database of each node in the entire distributed network. This ensures that the database of each node is the same and any changes can be synchronized correctly. In the blockchain, each node has a public and private key pair for digital signature and verification operations. This will ensure that only authorized nodes can record changes to the blockchain and other nodes can verify the legitimacy of these changes. By using this encryption technology, unauthorized changes can be prevented, thereby achieving higher security.

[0090] It should be noted that step 205 may be performed before or after any step after step 201, and the present invention does not limit this. Figure 3 In the description, the execution of step 205 after step 204 is taken as an example.

[0091] It should be noted that the incremental change statistical information corresponding to the incremental data can also be stored in the blockchain. The specific method is similar to the above-mentioned method of storing the incremental data in the blockchain, and this application will not go into details here.

[0092] It should be noted that when the target node representing the source device receives incremental data, a new blockchain can also be created to use the new blockchain to record the new state of the incremental data. This application does not limit this.

[0093] In this embodiment, blockchain is used as the underlying technology for data synchronization, which can provide a more efficient, secure, fast and decentralized data synchronization method. Each node can check and confirm the legitimacy of each other's data, thereby ensuring that each node in the distributed network has the same database, ultimately achieving data synchronization and further improving the security of data synchronization.

[0094] Figure 4is an overall flow chart of the data audit method in data synchronization provided by the embodiment of the present application, such as Figure 4 As shown, the CAPTURE plug-in captures the incremental data in the source database, the incremental change type (I / U / D / DDL, etc.) corresponding to the incremental data, and the incremental change statistics based on the preset statistical period, and publishes the target data generated based on the incremental data, the incremental change type corresponding to the incremental data, and the incremental change statistics to the corresponding Topic of Kafka. The description in the Topic includes the incremental data, the incremental change type corresponding to the incremental data, and the incremental change statistics, and a separate offset is used to store the incremental change statistics, which serves as a unique identifier or pointer. The target device consumes the target data. In the process of loading incremental data in the target database, the number of successful I / U / D / DDL operations is counted. When encountering statistical information, that is, when a loading cycle is completed, the loaded data is compared with the data consumed from the Topic, and whether the loading is normal is determined based on the comparison results. If the loading is normal, the data loading service continues.

[0095] Figure 5 is an interactive diagram of a data audit method in data synchronization provided by an embodiment of the present application, such as Figure 5 As shown, the data audit method in data synchronization is applied to the source device and the target device, and includes the following steps: Step 501: The source-end device obtains data to be loaded that needs to be synchronized from a source-end database of the source-end device to a target-end database of the target-end device.

[0096] Step 502: The source device generates a transmission control tag based on the target data to be loaded within a preset verification period, and generates target data based on the data to be loaded and the transmission control tag.

[0097] Step 503: The source device publishes the target data to the Topic.

[0098] Step 504: The target device consumes target data from the Topic.

[0099] Step 505: The target-end device loads the data to be loaded into the target-end database, and when the transmission control tag is parsed during the loading process, generates a verification tag based on the first loaded data in the data to be loaded.

[0100] Step 506: The target-end device audits the first loaded data and the target data to be loaded based on the transmission control tag and the verification tag to obtain a first audit result.

[0101] Step 507: The source device publishes the incremental data to a target node representing the source device in a distributed network, where the incremental data is used to instruct the target node to create a new block, store the incremental data and a hash value generated based on the incremental data in the new block, and send the new block to other nodes in the distributed network, where the new block is used to instruct the other nodes to update the database of the other nodes based on the incremental data when determining that the new block is legal based on the hash value.

[0102] It should be noted that step 507 may be executed before or after any step after step 501, and this application does not limit this.

[0103] The data auditing device in data synchronization provided by the present application is described below. The data auditing device in data synchronization described below and the data auditing method in data synchronization described above can refer to each other.

[0104] Figure 6 It is one of the structural diagrams of the data audit device in data synchronization provided by the embodiment of the present application, which is applied to the target end device, such as Figure 6 As shown, the data audit device 600 in data synchronization includes a consumption unit 601, a loading unit 602, a first generation unit 603 and a first audit unit 604; wherein: The consumption unit 601 is used to consume target data from the Topic, wherein the target data includes data to be loaded and a transmission control tag. The data to be loaded is data published by the source device to the Topic and needs to be synchronized from the source database to the target database of the target device. The transmission control tag is a tag generated based on the target data to be loaded within a preset verification period. A loading unit 602, configured to load the data to be loaded into the target database; A first generating unit 603 is configured to generate a verification tag based on the first loaded data in the data to be loaded when the transmission control tag is parsed during the loading process; The first audit unit 604 is used to audit the first loading data and the target data to be loaded based on the transmission control tag and the verification tag to obtain a first audit result.

[0105] The data audit device in data synchronization provided by the present application publishes the target data including the data to be loaded and the transmission control tag to the Topic, the target end device consumes the target data from the Topic, loads the data to be loaded in the target end database, and generates a verification tag based on the first loaded data in the data to be loaded when the transmission control tag is parsed during the loading process, and audits the first loaded data and the target data to be loaded based on the transmission control tag and the verification tag to obtain a first audit result. It can be seen that the present application combines the data to be loaded and the transmission control tag to form the target data for transmission. In this way, when the target end device consumes the target data and loads the data to be loaded in the target data into the target end database, it can realize data audit based on the transmission control tag included in the target data without the participation of a third party, thereby improving the efficiency of data audit.

[0106] Based on any of the above embodiments, the first audit unit 604 is specifically used for: In the case where the transmission control tag is the same as the verification tag, determining that the first audit result is a successful audit; In the case where the transmission control tag is different from the verification tag, the first audit result is determined to be an audit failure, and an exception message is output, where the exception message is used to remind the user that the target data to be loaded is loaded abnormally.

[0107] Based on any of the above embodiments, the target data further includes a release scenario tag; and the first generating unit 603 is specifically configured to: In the case where the publishing scenario tag represents a real-time publishing scenario, when the transmission control tag is parsed during the loading process, a verification tag is generated based on the first loaded data in the data to be loaded.

[0108] Based on any of the above embodiments, the data auditing device 600 in data synchronization further includes: A second audit unit is used to perform at least one of the following audits when the loading cycle arrives, to obtain a second audit result when the release scenario label represents a timed release scenario; Auditing the second loading data loaded in the loading cycle and the data corresponding to the second loading data in the data to be loaded; Auditing the data structure of the second loaded data and the data structure of the data corresponding to the second loaded data in the data to be loaded; The data dictionary of the second loading data and the data dictionary of the data to be loaded corresponding to the second loading data are audited.

[0109] Based on any of the above embodiments, the data to be loaded includes incremental data, and the target data also includes incremental change statistics and incremental change types; the loading unit 602 is specifically used to: Based on the incremental change statistical information and the incremental change type, the data to be loaded is loaded into the target database.

[0110] Figure 7 This is a second structural diagram of a data audit device in data synchronization provided by an embodiment of the present application, which is applied to a source device, such as Figure 7 As shown, the data audit device 700 in data synchronization includes an acquisition unit 701, a second generation unit 702, a third generation unit 703 and a first publishing unit 704; wherein: An acquisition unit 701 is used to acquire data to be loaded that needs to be synchronized from a source database of a source device to a target database of a target device; A second generating unit 702, configured to generate a transmission control tag based on target data to be loaded within a preset verification period; A third generating unit 703, configured to generate target data based on the data to be loaded and the transmission control tag; The first publishing unit 704 is used to publish the target data to the Topic, and the target data is used to instruct the target-end device to consume the target data from the Topic, and load the data to be loaded into the target-end database. When the transmission control tag is parsed during the loading process, a verification tag is generated based on the first loaded data, and based on the transmission control tag and the verification tag, the first loaded data and the target data to be loaded are audited to obtain a first audit result.

[0111] The data audit device in data synchronization provided by the present application publishes the target data including the data to be loaded and the transmission control tag to the Topic, the target end device consumes the target data from the Topic, loads the data to be loaded in the target end database, and generates a verification tag based on the first loaded data in the data to be loaded when the transmission control tag is parsed during the loading process, and audits the first loaded data and the target data to be loaded based on the transmission control tag and the verification tag to obtain a first audit result. It can be seen that the present application combines the data to be loaded and the transmission control tag to form the target data for transmission. In this way, when the target end device consumes the target data and loads the data to be loaded in the target data into the target end database, it can realize data audit based on the transmission control tag included in the target data without the participation of a third party, thereby improving the efficiency of data audit.

[0112] Based on any of the above embodiments, the data to be loaded includes incremental data; the data auditing device 700 in data synchronization further includes: A second publishing unit is used to publish the incremental data to a target node representing the source device in a distributed network, the incremental data is used to instruct the target node to create a new block, store the incremental data and a hash value generated based on the incremental data in the new block, and send the new block to other nodes in the distributed network, the new block is used to instruct the other nodes to update the database of the other nodes based on the incremental data when determining that the new block is legal based on the hash value.

[0113] Figure 8 is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present application, such as Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the data audit method in data synchronization, the method comprising: consuming target data from a Topic, the target data comprising data to be loaded and a transmission control tag, the data to be loaded is data in the target database of the target device that is published by the source device to the Topic and needs to be synchronized from the source database to the target device, the transmission control tag is a tag generated based on the target data to be loaded within a preset verification period; loading the data to be loaded in the target database; when the transmission control tag is parsed during the loading process, generating a verification tag based on the first loaded data in the data to be loaded; based on the transmission control tag and the verification tag, auditing the first loaded data and the target data to be loaded to obtain a first audit result.

[0114] Alternatively, the processor 810 may call the logic instructions in the memory 830 to execute the following method: Acquire the data to be loaded that needs to be synchronized from the source database of the source device to the target database of the target device; generate a transmission control tag based on the target data to be loaded within a preset verification period; generate target data based on the data to be loaded and the transmission control tag; publish the target data to a Topic, wherein the target data is used to instruct the target device to consume the target data from the Topic, and load the data to be loaded in the target database; when the transmission control tag is parsed during the loading process, generate a verification tag based on the first loaded data, and audit the first loaded data and the target data to be loaded based on the transmission control tag and the verification tag to obtain a first audit result.

[0115] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0116] On the other hand, the present application also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the data audit method in data synchronization provided by the above methods, the method including: consuming target data from a Topic, the target data including data to be loaded and a transmission control tag, the data to be loaded is data published by a source device to the Topic and needs to be synchronized from the source database to the target database of the target device, the transmission control tag is a tag generated based on the target data to be loaded within a preset verification period; loading the data to be loaded in the target database; when the transmission control tag is parsed during the loading process, generating a verification tag based on the first loaded data already loaded in the data to be loaded; based on the transmission control tag and the verification tag, auditing the first loaded data and the target data to be loaded to obtain a first audit result.

[0117] Alternatively, when the computer program is executed by a processor, the following method is implemented: Acquire the data to be loaded that needs to be synchronized from the source database of the source device to the target database of the target device; generate a transmission control tag based on the target data to be loaded within a preset verification period; generate target data based on the data to be loaded and the transmission control tag; publish the target data to a Topic, wherein the target data is used to instruct the target device to consume the target data from the Topic, and load the data to be loaded in the target database; when the transmission control tag is parsed during the loading process, generate a verification tag based on the first loaded data, and audit the first loaded data and the target data to be loaded based on the transmission control tag and the verification tag to obtain a first audit result.

[0118] On the other hand, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the data audit method in data synchronization provided by the above-mentioned methods, the method comprising: consuming target data from a Topic, the target data comprising data to be loaded and a transmission control tag, the data to be loaded being data in a target database published by a source device to the Topic that needs to be synchronized from a source database to a target device, the transmission control tag being a tag generated based on the target data to be loaded within a preset verification period; loading the data to be loaded in the target database; when the transmission control tag is parsed during the loading process, generating a verification tag based on the first loaded data already loaded in the data to be loaded; based on the transmission control tag and the verification tag, auditing the first loaded data and the target data to be loaded to obtain a first audit result.

[0119] Alternatively, when the program instructions are executed by a computer, the computer can implement the following method: Acquire the data to be loaded that needs to be synchronized from the source database of the source device to the target database of the target device; generate a transmission control tag based on the target data to be loaded within a preset verification period; generate target data based on the data to be loaded and the transmission control tag; publish the target data to a Topic, wherein the target data is used to instruct the target device to consume the target data from the Topic, and load the data to be loaded in the target database; when the transmission control tag is parsed during the loading process, generate a verification tag based on the first loaded data, and audit the first loaded data and the target data to be loaded based on the transmission control tag and the verification tag to obtain a first audit result.

[0120] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0121] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data audit method in data synchronization, characterized in that: Applied to a target end device, the method comprises: Consume target data from the Topic, where the target data includes data to be loaded and a transmission control tag. The data to be loaded is data published by the source device to the Topic and needs to be synchronized from the source database to the target database of the target device. The transmission control tag is a tag generated based on the target data to be loaded within a preset verification period. Loading the data to be loaded into the target database; When the transmission control tag is parsed during the loading process, a verification tag is generated based on the first loading data that has been loaded in the data to be loaded; Based on the transmission control tag and the verification tag, the first loading data and the target data to be loaded are audited to obtain a first audit result.

2. The data audit method in data synchronization according to claim 1, characterized in that: The auditing of the first loading data and the target data to be loaded based on the transmission control tag and the verification tag to obtain a first audit result includes: In the case where the transmission control tag is the same as the verification tag, determining that the first audit result is a successful audit; In the case where the transmission control tag is different from the verification tag, the first audit result is determined to be an audit failure, and an exception message is output, where the exception message is used to remind the user that the target data to be loaded is loaded abnormally.

3. The data audit method in data synchronization according to claim 1, characterized in that: The target data also includes a release scenario tag; When the transmission control tag is parsed during the loading process, generating a verification tag based on the first loading data that has been loaded in the data to be loaded includes: In the case where the publishing scenario tag represents a real-time publishing scenario, when the transmission control tag is parsed during the loading process, a verification tag is generated based on the first loaded data in the data to be loaded.

4. The data audit method in data synchronization according to claim 3, characterized in that: The method further comprises: In the case where the release scenario label represents a timed release scenario, when the loading cycle arrives, at least one of the following audits is performed to obtain a second audit result; Auditing the second loading data loaded in the loading cycle and the data corresponding to the second loading data in the data to be loaded; Auditing the data structure of the second loaded data and the data structure of the data corresponding to the second loaded data in the data to be loaded; The data dictionary of the second loading data and the data dictionary of the data to be loaded corresponding to the second loading data are audited.

5. The data audit method in data synchronization according to any one of claims 1 to 4, characterized in that: The data to be loaded includes incremental data, and the target data also includes incremental change statistics and incremental change types; The loading of the to-be-loaded data in the target database includes: Based on the incremental change statistical information and the incremental change type, the data to be loaded is loaded into the target database.

6. A data audit method in data synchronization, characterized in that: Applied to a source device, the method comprises: Obtain the data to be loaded that needs to be synchronized from the source database of the source device to the target database of the target device; Generate a transmission control tag based on the target data to be loaded within a preset verification period; generating target data based on the data to be loaded and the transmission control tag; The target data is published to a Topic, and the target data is used to instruct a target-end device to consume the target data from the Topic, and to load the data to be loaded into the target-end database. When the transmission control tag is parsed during the loading process, a verification tag is generated based on the first loaded data, and based on the transmission control tag and the verification tag, the first loaded data and the target data to be loaded are audited to obtain a first audit result.

7. The data audit method in data synchronization according to claim 6, characterized in that: The data to be loaded includes incremental data; the method further includes: The incremental data is published to a target node representing the source device in a distributed network, the incremental data is used to instruct the target node to create a new block, the incremental data and a hash value generated based on the incremental data are stored in the new block, and the new block is sent to other nodes in the distributed network, the new block is used to instruct the other nodes to update the database of the other nodes based on the incremental data when determining that the new block is legal based on the hash value.

8. A data auditing device in data synchronization, characterized in that: include: A consumption unit, used to consume target data from a Topic, wherein the target data includes data to be loaded and a transmission control tag, wherein the data to be loaded is data published by a source device to the Topic and needs to be synchronized from a source database to a target database of a target device, and the transmission control tag is a tag generated based on the target data to be loaded within a preset verification period; A loading unit, used for loading the data to be loaded into the target database; A first generating unit, configured to generate a verification tag based on the first loaded data in the data to be loaded when the transmission control tag is parsed during the loading process; The first audit unit is used to audit the first loaded data and the target data to be loaded based on the transmission control tag and the verification tag to obtain a first audit result.

9. A data auditing device in data synchronization, characterized in that: include: An acquisition unit, used for acquiring data to be loaded that needs to be synchronized from a source database of a source device to a target database of a target device; A second generating unit, configured to generate a transmission control tag based on target data to be loaded within a preset verification period; A third generating unit, configured to generate target data based on the data to be loaded and the transmission control tag; A first publishing unit is used to publish the target data to a Topic, wherein the target data is used to instruct a target-end device to consume the target data from the Topic and load the data to be loaded into the target-end database. When the transmission control tag is parsed during the loading process, a verification tag is generated based on the first loaded data, and based on the transmission control tag and the verification tag, the first loaded data and the target data to be loaded are audited to obtain a first audit result.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, it implements the data audit method in data synchronization as described in any one of claims 1 to 5, or implements the data audit method in data synchronization as described in claim 6 or 7.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the data audit method in data synchronization as described in any one of claims 1 to 5, or implements the data audit method in data synchronization as described in claim 6 or 7.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the data audit method in data synchronization as described in any one of claims 1 to 5, or implements the data audit method in data synchronization as described in claim 6 or 7.

Images