A data reconciliation method, device, equipment and storage medium

Abstract

This application discloses a data reconciliation method, apparatus, device, and storage medium, relating to the field of data synchronization. The method includes: acquiring a source reconciliation statement, a source log table, and a source business table provided by a data provider, and a target reconciliation statement, a target log table, and a target business table provided by a data receiver; comparing the source data in the source log table and the source business table based on the data provider's source reconciliation statement to obtain a pre-reconciliation result; synchronizing the source data to the data receiver's target business table based on the pre-reconciliation result; and comparing the obtained data synchronization result with the target data in the target business table based on the source reconciliation statement to obtain a post-reconciliation result. Pre-reconciliation during data synchronization ensures the accuracy of the data provider's business data. Furthermore, the data logs provided by both the data provider and the data receiver allow for rapid identification and location of discrepancies when discrepancies occur, ensuring normal business operation and improving platform stability.

Description

Technical Field

[0001] This invention relates to the field of data synchronization, and in particular to a data reconciliation method, apparatus, device, and storage medium. Background Technology

[0002] With the rapid development of big data in recent years, more and more industries are focusing on data applications. In today's world where data empowers all sectors, data openness and sharing are in line with the times, and ensuring the accuracy of data flow is extremely important. Data from different sources, after being synchronized by ETL (Extract-Transform-Load) tools, falls into different target data sources. The CDC (Change Data Capture) data synchronization technology used in this process is also widely adopted.

[0003] However, during the aforementioned data synchronization process, it is difficult to guarantee the real-time and accurate flow of data. Problems frequently arise, such as unsynchronized source data, discrepancies between the target and source data volumes, and difficulties in quickly locating differences within large datasets. When these issues occur, both the data provider and the data recipient must perform data backtracking. In datasets of hundreds of millions, locating discrepancies during synchronization can be challenging due to these problems. If the discrepancies cannot be located, a resynchronization process is often necessary, resulting in lengthy synchronization times and wasted time and manpower. Therefore, providing a data synchronization solution that ensures both the accuracy of data provided by the data provider and the accurate receipt of data by the data recipient is a problem that needs to be solved in this field. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a data reconciliation method, apparatus, device, and storage medium that can perform pre-reconciliation during data synchronization, ensuring the accuracy of business data from the data provider. Furthermore, the data provider and data receiver provide data logs to quickly identify and locate discrepancies when they occur, ensuring normal business operation and improving platform stability. The specific solution is as follows:

[0005] Firstly, this application provides a data reconciliation method, including:

[0006] The system obtains source reconciliation statement table, source log table, and source business table provided by the data provider, and target reconciliation statement table, target log table, and target business table provided by the data receiver. The source business table and the target business table are used to store business data, the source reconciliation statement table and the target reconciliation statement table are used to store the data information and reconciliation information of the business data, and the source log table and the target log table are used to store the business primary key of the business data.

[0007] Based on the source reconciliation statement table of the data provider, the source log table and the source business table are compared with the source data to obtain the pre-reconciliation result;

[0008] Based on the prior reconciliation results, the source data is synchronized to the target business table of the data recipient, and the target data in the target business table is compared with the obtained data synchronization results and the source reconciliation statement to obtain the post-reconciliation results.

[0009] Optionally, synchronizing the source data to the target business table of the data recipient based on the prior reconciliation result includes:

[0010] Using a pre-defined streaming data processing system as a data processing tool, the source data is synchronized to the target business table based on the pre-reconciliation results.

[0011] Optionally, before comparing the source data in the source log table and the source business table based on the source reconciliation statement table from the data provider, the method further includes:

[0012] The batch number of the source statement table is obtained incrementally using the preset batch acquisition component;

[0013] Accordingly, synchronizing the source data to the target business table of the data recipient based on the prior reconciliation results includes:

[0014] Based on the prior reconciliation results, the source data batch is synchronized to the target business table of the data recipient according to the batch number.

[0015] Optionally, the step of comparing the source data in the source log table and the source business table based on the source reconciliation statement table from the data provider to obtain the pre-reconciliation result includes:

[0016] Determine the first batch data volume corresponding to each batch number in the source reconciliation form, and determine the second batch data volume of the source data for each batch based on the source log table and the source business table, and determine whether the first batch data volume and the second batch data volume are the same;

[0017] Based on the judgment result, filter out the first source data corresponding to the first batch number with the same data volume and the second source data corresponding to the second batch number with different data volume from the source business table;

[0018] The prior reconciliation status in the target reconciliation statement table corresponding to the first source data is updated to be consistent, and the prior reconciliation status in the target reconciliation statement table corresponding to the second source data is updated to be inconsistent, so as to synchronize the first source data.

[0019] Optionally, synchronizing the source data to the target business table of the data recipient includes:

[0020] The associated data between the source log table and the source business table is extracted using a preset extraction component;

[0021] The associated data is written to the target business table using a preset table writing component, and the target log table is updated based on the associated data.

[0022] Optionally, the step of comparing the obtained data synchronization results and the source reconciliation statement with the target data in the target business table to obtain the post-reconciliation results includes:

[0023] Start the reconciliation task according to the preset reconciliation task start time, and obtain the target batch number corresponding to the reconciliation task;

[0024] Determine the target data volume corresponding to the target batch number in the target business table and the source data volume corresponding to the target batch number in the source reconciliation form, and determine whether the target data volume and the source data volume are the same to obtain the post-reconciliation result.

[0025] Optionally, determining whether the target data volume and the source data volume are the same to obtain the post-reconciliation result includes:

[0026] If they are the same, then the target batch number is determined to be consistent with the accounting records;

[0027] If they are different, the data difference between the target data volume and the source data volume is determined based on the business primary key, and a corresponding data difference record is generated according to the data difference in order to supplement the data of the target batch number.

[0028] Secondly, this application provides a data reconciliation device, comprising:

[0029] The data acquisition module is used to acquire source reconciliation statement table, source log table, and source business table provided by the data provider, and target reconciliation statement table, target log table, and target business table provided by the data receiver; the source business table and the target business table are used to store business data, the source reconciliation statement table and the target reconciliation statement table are used to store the data information and reconciliation information of the business data, and the source log table and the target log table are used to store the business primary key of the business data;

[0030] The pre-reconciliation module is used to compare the source data in the source log table and the source business table with the source reconciliation statement table of the data provider to obtain the pre-reconciliation result;

[0031] The data synchronization module is used to synchronize the source data to the target business table of the data recipient based on the prior reconciliation result, and to obtain the post-reconciliation result by comparing the obtained data synchronization result with the target data in the target business table based on the source reconciliation statement.

[0032] Thirdly, this application provides an electronic device, which includes a processor and a memory; wherein the memory is used to store a computer program, which is loaded and executed by the processor to implement the aforementioned data reconciliation method.

[0033] Fourthly, this application provides a computer-readable storage medium for storing a computer program that, when executed by a processor, implements the aforementioned data reconciliation method.

[0034] This application obtains source reconciliation statement tables, source log tables, and source business tables provided by the data provider, and target reconciliation statement tables, target log tables, and target business tables provided by the data receiver. The source and target business tables store business data, while the source and target reconciliation statement tables store data information and reconciliation information. The source and target log tables store the business primary keys of the business data. Based on the source data in the source log tables and source business tables, a pre-reconciliation result is obtained by comparing the source data in the source reconciliation statement tables with the source data in the source log tables. Then, based on the pre-reconciliation result, the source data is synchronized to the target business tables of the data receiver. Finally, based on the obtained data synchronization result and the source reconciliation statement tables, the target data in the target business tables is compared with the target data in the target business tables to obtain a post-reconciliation result. This pre-reconciliation during data synchronization ensures the accuracy of the data provider's business data and avoids inconsistencies between the target and source data volumes. Furthermore, the data provider and data receiver provide data logs to quickly identify and locate discrepancies when discrepancies occur, allowing for timely data correction and ensuring the normal operation of the data synchronization process, thus improving platform stability. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0036] Figure 1 A flowchart of a data reconciliation method provided in this application;

[0037] Figure 2This application provides a data reconciliation flowchart based on preset components;

[0038] Figure 3 This application provides a data reconciliation flowchart based on streaming data processing;

[0039] Figure 4 A flowchart illustrating a specific data reconciliation method provided in this application;

[0040] Figure 5 A pre-reconciliation logic diagram provided for this application;

[0041] Figure 6 This application provides a pre-reconciliation result display diagram;

[0042] Figure 7 This application provides a diagram showing the results of a post-event reconciliation statement.

[0043] Figure 8 A schematic diagram of a data reconciliation device provided in this application;

[0044] Figure 9 This application provides a structural diagram of an electronic device. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] During data synchronization, discrepancies can be quickly located, but due to the difficulty in ensuring real-time and accurate data flow, problems frequently arise such as unsynchronized source data, inconsistencies between target and source data volumes, and the challenge of quickly locating discrepancies within large datasets. This application addresses these issues by batching source data and providing reconciliation statements to reduce the volume of data synchronized in a single session. Furthermore, pre-synchronization reconciliation ensures the accuracy of business data from the data provider, and both the data provider and receiver provide data logs to quickly identify and locate discrepancies when they occur.

[0047] See Figure 1 As shown in the figure, an embodiment of the present invention discloses a data reconciliation method, including:

[0048] Step S11: Obtain the source reconciliation statement table, source log table, and source business table provided by the data provider, and the target reconciliation statement table, target log table, and target business table provided by the data receiver; the source business table and the target business table are used to store business data, the source reconciliation statement table and the target reconciliation statement table are used to store the data information and reconciliation information of the business data, and the source log table and the target log table are used to store the business primary key of the business data.

[0049] This embodiment aims to address the challenges of large data volumes, long synchronization times, and difficulty in locating discrepancies during CDC data synchronization. It proposes a reconciliation scheme for the data synchronization process, where the data provider offers the source reconciliation statement, source log table, and source business table. The data receiver offers the target reconciliation statement, target log table, and target business table. It should also be noted that, if... Figure 2 As shown, the batch number of the source reconciliation statement table can be incrementally obtained using the preset batch acquisition component. By batching the data, the reconciliation statement provided by the data provider after batching and summarizing the source data can be obtained, reducing the data volume and time of each synchronization. Pre-synchronization during the data synchronization process ensures the accuracy of the data provider's business data. Furthermore, both the data provider and the data receiver provide data logs. When discrepancies occur, the logs from both parties can be used to quickly identify the discrepancies in batches, locate the discrepancies, and promptly supplement the discrepancies, ensuring normal business operation, improving platform stability, and ensuring efficient and accurate data synchronization.

[0050] Specifically, the source business table provided by the data provider is used to store business data. In addition to business fields, it also includes fields such as primary key ID (Identity document) and entry time. The aforementioned business primary key ID is a unique record identifier for the business data and can use a UUID (Universally Unique Identifier) ​​primary key strategy. The aforementioned source business table is shown in Table 1:

[0051] Table 1

[0052]

[0053] Table 2

[0054] Field Name Field identifier Field type Field length Primary key identifier non-empty Unique identifier for reconciliation records SD_ID VARCHAR 64 primary key yes Table name SD_TABLE_NAME VARCHAR 100 yes Batch number SD_BATCH VARCHAR 20 yes Batch Count SD_COUNT INT yes Batch Date SD_DATE DATETIME yes

[0055] Understandably, the source reconciliation statement table provided by the data provider, as shown in Table 2 above, is used to record the data for each batch in the business data table, including fields such as the unique identifier of the reconciliation record, the business table name, the batch number, the number of records in the batch, and the batch date. Specifically, the unique identifier of the reconciliation record is the primary key, using a UUID primary key strategy; the table name is the name of the business data table; the batch number should be unique within the same business table; the number of records in the batch is the number of data records in that batch, and should be consistent with the actual number of updated records in the business data table for that batch; the batch date is the time when the reconciliation data was written to the reconciliation table, used as the incremental column of the reconciliation statement, and uses the value of a database time function, in the format YYYY-MM-DD hh:mm:ss.

[0056] In this embodiment, the source log table provided by the data provider is used to record the batch number and corresponding primary key of each batch of business data. The primary key of the business data is uniquely identified by the primary key of the source business data table, and the batch number is the batch number of the source reconciliation statement table, as shown in Table 3.

[0057] Table 3

[0058]

[0059] The target business table provided by the data receiver is used to store the synchronized source business data, and must contain at least the business primary key column of the source business table. The aforementioned business primary key ID is a unique record identifier for the business data, and the data originates from the source business table. See Table 4 for details:

[0060] Table 4

[0061]

[0062] The target reconciliation statement table provided by the data recipient is used to record information and reconciliation details for each batch of data synchronized by the task. In addition to the source reconciliation table fields, it also includes fields such as pre-reconciliation status, data volume in the database, post-reconciliation status, and entry time. Specifically, the pre-reconciliation status is the reconciliation result of the source data during data synchronization; the pre-reconciliation data volume is the batch data volume retrieved from the source business table and source log table; the data volume in the database is the batch data volume existing in the target table after data synchronization; and the post-reconciliation status is the reconciliation result between the source reconciliation statement data volume and the target data volume for the same batch. See Table 5 for details.

[0063] Table 5

[0064] Field Name Field identifier Field type Field length Primary key identifier non-empty Unique identifier for reconciliation records SD_ID VARCHAR 64 composite primary key yes Table name SD_TABLE_NAME VARCHAR 64 yes Batch number SD_BATCH VARCHAR 20 yes Batch Count SD_COUNT INT 20 yes Batch Date SD_DATE DATETIME yes Pre-reconciliation status PRE_STATUS VARCHAR 2 no Pre-reconciliation data volume PRE_COUNT BIGINT 20 no Post-event reconciliation results SD_STATUS VARCHAR 2 no Inbound data volume WRITE_COUNT BIGINT 20 no

[0065] The target log table provided by the data recipient is used to record the batch number and corresponding primary key of each batch of business data received. The primary key uniquely identifies the source business data table, and the batch number is the batch number from the source reconciliation statement table, as shown in Table 6.

[0066] Table 6

[0067]

[0068] Step S12: Based on the source reconciliation statement table of the data provider, compare the source data in the source log table and the source business table to obtain the pre-reconciliation result.

[0069] In this embodiment, the source data in the source log table and source business table can be compared with the source reconciliation statement table of the data provider to obtain the pre-reconciliation result. By calculating and comparing the source reconciliation statement maintained by the data provider with the source business table, the accuracy of the source reconciliation statement data volume statistics can be guaranteed. During the data synchronization process, the data integration platform can use the pre-reconciliation function to provide early warnings of inconsistencies between the data provider's reconciliation statement and business data, thus ensuring data accuracy from the source.

[0070] Step S13: Based on the prior reconciliation result, synchronize the source data to the target business table of the data recipient, and compare the target data in the target business table with the obtained data synchronization result and the source reconciliation statement to obtain the post-reconciliation result.

[0071] In this embodiment, after prior verification in the above steps, the source data can be synchronized to the target business table of the data recipient based on the prior reconciliation results. The target data in the target business table is then compared with the obtained data synchronization results and the source reconciliation statement to obtain the subsequent reconciliation results. Furthermore, a pre-set streaming data processing system can be used as a data processing tool to synchronize the source data to the target business table based on the prior reconciliation results. Specifically, as follows... Figure 3 As shown, NIFI is used as an ETL tool to implement the above data synchronization process. Based on the above steps, the source data can be synchronized to the target business table of the data receiver in batches according to the obtained batch number. After pre-verification, NIFI extracts the batch of business data into the target business table, and simultaneously obtains the data maintenance target log table after data entry. Through the above scheme, after data synchronization via ETL tool, post-event reconciliation is performed, and the target reconciliation statement can provide the synchronization reconciliation results for each batch of data. The data receiver calculates and statistically analyzes the data falling into the target business table and compares it with the source reconciliation statement to obtain the reconciliation results. This reflects the stability of data synchronization, ensures the accuracy of target data, and uses post-event reconciliation to calculate the statistical data volume of the target table and the source table, intuitively expressing the data synchronization results.

[0072] This embodiment acquires the source reconciliation statement, source log table, and source business table provided by the data provider, and the target reconciliation statement, target log table, and target business table provided by the data receiver. Based on the source data in the source log table and source business table, a pre-reconciliation result is obtained by comparing the source data in the source reconciliation statement with the source data in the source log table. Then, based on the pre-reconciliation result, the source data is synchronized to the target business table of the data receiver. Finally, based on the obtained data synchronization result and the source reconciliation statement, the target data in the target business table is compared with the target data in the target business table to obtain a post-reconciliation result. In this way, the data provider can batch-compile the source data, providing reconciliation statements to batch the data, reducing the data volume and time of each synchronization. Pre-reconciliation during the data synchronization process ensures the accuracy of the data provider's business data and avoids inconsistencies between the target and source data volumes. Furthermore, the data provider and data receiver provide data logs, allowing for rapid identification and location of discrepancies when discrepancies occur, enabling timely data correction and ensuring the normal operation of the data synchronization business, thus improving platform stability.

[0073] As can be seen from the previous embodiment, this application can perform pre-reconciliation during data synchronization to ensure the accuracy of the data provider's business data. Furthermore, the data provider and data receiver provide data logs for post-reconciliation, quickly confirming discrepancies in batches. Next, this embodiment will describe in detail the process of implementing reconciliation and post-reconciliation. See [link to previous embodiment]. Figure 4 As shown, this embodiment of the invention discloses a specific data reconciliation method, including:

[0074] Step S21: Based on the source reconciliation statement from the data provider, compare the source data in the source log table and the source business table to obtain the pre-reconciliation result.

[0075] like Figure 5As shown, in this embodiment, the first batch data volume corresponding to each batch number in the source reconciliation form can be determined first. Then, based on the source log table and source business table, the second batch data volume of each batch's source data can be determined. It is then determined whether the first batch data volume and the second batch data volume are the same. Based on the determination result, the first source data corresponding to the first batch number with the same data volume and the second source data corresponding to the second batch number with different data volume are filtered from the source business table. The pre-reconciliation status in the target reconciliation table corresponding to the first source data is updated to consistent, and the pre-reconciliation status in the target reconciliation table corresponding to the second source data is updated to inconsistent, so as to synchronize the first source data. It is understood that pre-reconciliation is required during data synchronization. When synchronizing the current batch data, the batch volume of each batch in the source reconciliation table is calculated to be consistent with the corresponding batch data volume in the source business table. If the current batch's pre-reconciliation is consistent, the batch will proceed with subsequent data extraction normally, and the [pre-reconciliation status] field of the target reconciliation table will be updated to "consistent"; if inconsistent, the batch will not be extracted, and the [pre-reconciliation status] field of the platform reconciliation table will be updated to "inconsistent," ultimately resulting in the following... Figure 6 The results of the prior reconciliation are shown.

[0076] Step S22: Based on the prior reconciliation results, extract the associated data between the source log table and the first source data in the source business table using a preset extraction component; write the associated data into the target business table using a preset write component, and update the target log table based on the associated data.

[0077] In this embodiment, as Figure 2 As shown, based on the pre-reconciliation results, a preset extraction component can be used to extract the associated data between the source log table and the first source data in the source business table. A preset write component is then used to write the associated data into the target business table. Finally, the target log table is updated based on the associated data to complete data synchronization. The aforementioned pre-reconciled data is extracted using the extraction component to associate the source log table with the source business table. The write component then first writes the extracted data into the target business table to ensure that the data in the log table is already present in the target business table. Finally, the write component writes the data written to the target business table into the target log table.

[0078] Step S23: Start the reconciliation task according to the preset reconciliation task start time, obtain the target batch number corresponding to the reconciliation task; determine the target data volume corresponding to the target batch number in the target business table and the source data volume corresponding to the target batch number in the source reconciliation form, and determine whether the target data volume and the source data volume are the same, to obtain the post-reconciliation result.

[0079] In this embodiment, after data synchronization, a reconciliation task is initiated according to a preset reconciliation task start time. The target batch number corresponding to the reconciliation task is obtained, and the target data volume corresponding to the target batch number in the target business table and the source data volume corresponding to the target batch number in the source reconciliation form are determined. It is then determined whether the target data volume and the source data volume are the same to obtain the post-reconciliation result. If the data volumes are the same, the reconciliation of the target batch number is considered consistent. If the data volumes are different, the data difference between the target data volume and the source data volume is determined based on the business primary key, and a corresponding data difference record is generated to supplement the data for the target batch number. For example, a scheduled task is initiated at 0:00 every day to obtain tasks with the reconciliation function enabled. For the current task, instances with reconciliation statuses of "not reconciled," "successful," or "terminated" are obtained. The amount of data added to the task instance is then obtained and compared with the total source data of all batches under that instance in the source reconciliation table. If the total is consistent, the reconciliation of the relevant batches of that instance is considered consistent, and the reconciliation status of the target reconciliation form is maintained. If the sums are inconsistent, count the data volume of each batch of target tables separately, compare it with the source quantity in the source reconciliation statement for that batch, obtain the reconciliation result for each batch, and maintain the target reconciliation statement. The final result is as follows: Figure 7 The post-reconciliation results are shown. Furthermore, when inconsistencies exist between reconciliation batches, the source and target log tables are compared to identify discrepancies. By calculating the differences based on the primary key, records showing discrepancies between the source and target are obtained for subsequent data supplementation. In this way, data with reconciliation anomalies can be quickly identified by performing batch-wide data collision analysis on the target log tables of both the data provider and data receiver. This significantly reduces the workload for operations personnel in locating discrepancies, ensuring efficient and accurate data synchronization, reflecting the true value of the data, and guaranteeing the normal operation of the business.

[0080] In this embodiment, the source data in the source log table and source business table are compared with the source reconciliation statement table provided by the data provider to obtain a pre-reconciliation result. Based on the pre-reconciliation result, the associated data of the first source data in the source log table and source business table is extracted using a preset extraction component; the associated data is written to the target business table using a preset table writing component, and the target log table is updated based on the associated data. The reconciliation task is started according to the preset reconciliation task start time, and the target batch number corresponding to the reconciliation task is obtained; the target data volume corresponding to the target batch number in the target business table and the source data volume corresponding to the target batch number in the source reconciliation form are determined, and it is judged whether the target data volume and the source data volume are the same to obtain the post-reconciliation result. In this way, the accuracy of the data provider's reconciliation statement is ensured by pre-reconciliation, avoiding disputes between the two parties caused by data differences from the source, and the statistical analysis of the data volume of the target table and the source table is used by post-reconciliation to intuitively express the data synchronization result. When there are differences after data synchronization, the difference records are quickly located using bilateral log tables and the data is supplemented in a timely manner, which greatly improves the operation and maintenance efficiency, reduces the operation and maintenance cost, provides intuitive results of data synchronization, and ensures the normal operation of the business.

[0081] See Figure 8 As shown in the illustration, this application also discloses a data reconciliation device, comprising:

[0082] Data acquisition module 11 is used to acquire source reconciliation statement table, source log table, and source business table provided by the data provider, and target reconciliation statement table, target log table, and target business table provided by the data receiver; the source business table and the target business table are used to store business data, the source reconciliation statement table and the target reconciliation statement table are used to store data information and reconciliation information of the business data, and the source log table and the target log table are used to store the business primary key of the business data;

[0083] The pre-reconciliation module 12 is used to compare the source data in the source log table and the source business table with the source reconciliation statement table of the data provider to obtain the pre-reconciliation result;

[0084] The data synchronization module 13 is used to synchronize the source data to the target business table of the data recipient based on the prior reconciliation result, and to compare the target data in the target business table with the obtained data synchronization result and the source reconciliation statement to obtain the post-reconciliation result.

[0085] In this embodiment, source reconciliation statements, source log tables, and source business tables provided by the data provider, and target reconciliation statements, target log tables, and target business tables provided by the data receiver are obtained. Based on the source data in the source log tables and source business tables, a pre-reconciliation result is obtained by comparing the source data in the source reconciliation statements with the source data in the source log tables. Then, based on the pre-reconciliation result, the source data is synchronized to the target business table of the data receiver. Finally, based on the obtained data synchronization result and the source reconciliation statements, the target data in the target business table is compared with the target data in the target business table to obtain a post-reconciliation result. In this way, the data provider can batch-compile the source data, providing reconciliation statements to batch the data, reducing the data volume and time of each synchronization. Pre-reconciliation during the data synchronization process ensures the accuracy of the data provider's business data and avoids inconsistencies between the target and source data volumes. Furthermore, the data provider and data receiver provide data logs, allowing for rapid identification and location of discrepancies when discrepancies occur, enabling timely data correction and ensuring the normal operation of the data synchronization process.

[0086] In some specific embodiments, the data synchronization module 13 specifically includes:

[0087] The first data synchronization unit is used to use a preset streaming data processing system as a data processing tool to synchronize the source data to the target business table based on the pre-reconciliation results.

[0088] In some specific embodiments, the pre-reconciliation module 12 further includes:

[0089] The first batch number acquisition unit is used to incrementally acquire the batch number of the source statement table using a preset batch acquisition component;

[0090] Accordingly, the data synchronization module 13 specifically includes:

[0091] The second data synchronization unit is used to synchronize the source data batch to the target business table of the data recipient based on the batch number according to the pre-reconciliation result.

[0092] In some specific embodiments, the pre-reconciliation module 12 specifically includes:

[0093] The data judgment unit is used to determine the first batch data volume corresponding to each batch number in the source reconciliation form, and to determine the second batch data volume of the source data for each batch according to the source log table and the source business table, and to determine whether the first batch data volume and the second batch data volume are the same;

[0094] The data filtering unit is used to filter out the first source data corresponding to the first batch number with the same data volume and the second source data corresponding to the second batch number with different data volume from the source business table according to the judgment result.

[0095] The status update unit is used to update the pre-reconciliation status in the target reconciliation statement table corresponding to the first source data to be consistent, and to update the pre-reconciliation status in the target reconciliation statement table corresponding to the second source data to be inconsistent, so as to synchronize the first source data.

[0096] In some specific embodiments, the data synchronization module 13 specifically includes:

[0097] The data extraction unit is used to extract the associated data of the first source data in the source log table and the source business table through a preset extraction component;

[0098] The data update unit is used to write the associated data into the target business table using a preset table writing component, and update the target log table based on the associated data.

[0099] In some specific embodiments, the data synchronization module 13 specifically includes:

[0100] The second batch number acquisition unit is used to start a reconciliation task according to a preset reconciliation task start time and obtain the target batch number corresponding to the reconciliation task.

[0101] The post-reconciliation submodule is used to determine the target data volume corresponding to the target batch number in the target business table and the source data volume corresponding to the target batch number in the source reconciliation form, and to determine whether the target data volume and the source data volume are the same, so as to obtain the post-reconciliation result.

[0102] In some specific embodiments, the post-event reconciliation submodule specifically includes:

[0103] The result judgment unit is used to determine that the target batch number is consistent when the post-reconciliation results are the same.

[0104] The difference determination unit is used to determine the data difference between the target data volume and the source data volume based on the business primary key when the post-reconciliation results are different, and to generate a corresponding data difference record according to the data difference so as to supplement the data of the target batch number.

[0105] Furthermore, embodiments of this application also disclose an electronic device, Figure 9 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0106] Figure 9 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the data reconciliation method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0107] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0108] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0109] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the data reconciliation method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0110] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed data reconciliation method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0111] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0112] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0113] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0114] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0115] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data reconciliation method, characterized in that, include: The system obtains source reconciliation statement table, source log table, and source business table provided by the data provider, and target reconciliation statement table, target log table, and target business table provided by the data receiver. The source business table and the target business table are used to store business data, the source reconciliation statement table and the target reconciliation statement table are used to store the data information and reconciliation information of the business data, and the source log table and the target log table are used to store the business primary key of the business data. Based on the source reconciliation statement table of the data provider, the source log table and the source business table are compared with the source data to obtain the pre-reconciliation result; Based on the prior reconciliation results, the source data is synchronized to the target business table of the data recipient, and the target data in the target business table is compared with the obtained data synchronization results and the source reconciliation statement to obtain the post-reconciliation results. Before comparing the source data in the source log table and the source business table based on the source reconciliation statement table from the data provider, the method further includes: The batch number of the source statement table is obtained incrementally using the preset batch acquisition component; Accordingly, synchronizing the source data to the target business table of the data recipient based on the prior reconciliation results includes: Based on the pre-reconciliation results, the source data batch is synchronized to the target business table of the data recipient according to the batch number; Furthermore, the preliminary reconciliation result is obtained by comparing the source data in the source log table and the source business table with the source reconciliation statement table from the data provider, including: Determine the first batch data volume corresponding to each batch number in the source reconciliation form, and determine the second batch data volume of the source data for each batch based on the source log table and the source business table, and determine whether the first batch data volume and the second batch data volume are the same; Based on the judgment result, filter out the first source data corresponding to the first batch number with the same data volume and the second source data corresponding to the second batch number with different data volume from the source business table; The prior reconciliation status in the target reconciliation statement table corresponding to the first source data is updated to be consistent, and the prior reconciliation status in the target reconciliation statement table corresponding to the second source data is updated to be inconsistent, so as to synchronize the first source data. The step of synchronizing the source data to the target business table of the data recipient includes: The associated data between the source log table and the source business table is extracted using a preset extraction component; The associated data is written to the target business table using a preset table writing component, and the target log table is updated based on the associated data; Accordingly, the process of obtaining post-reconciliation results by comparing the obtained data synchronization results and the source reconciliation statement with the target data in the target business table includes: Start the reconciliation task according to the preset reconciliation task start time, and obtain the target batch number corresponding to the reconciliation task; Determine the target data volume corresponding to the target batch number in the target business table and the source data volume corresponding to the target batch number in the source reconciliation form, and determine whether the target data volume and the source data volume are the same to obtain the post-reconciliation result.

2. The data reconciliation method according to claim 1, characterized in that, The step of synchronizing the source data to the target business table of the data recipient based on the prior reconciliation results includes: Using a pre-defined streaming data processing system as a data processing tool, the source data is synchronized to the target business table based on the pre-reconciliation results.

3. The data reconciliation method according to claim 1, characterized in that, The step of determining whether the target data volume and the source data volume are the same, and obtaining the post-reconciliation result, includes: If they are the same, then the target batch number is determined to be consistent with the accounting records; If they are different, the data difference between the target data volume and the source data volume is determined based on the business primary key, and a corresponding data difference record is generated according to the data difference in order to supplement the data of the target batch number.

4. A data reconciliation device, characterized in that, include: The data acquisition module is used to acquire source reconciliation statement table, source log table, and source business table provided by the data provider, and target reconciliation statement table, target log table, and target business table provided by the data receiver; the source business table and the target business table are used to store business data, the source reconciliation statement table and the target reconciliation statement table are used to store the data information and reconciliation information of the business data, and the source log table and the target log table are used to store the business primary key of the business data; The pre-reconciliation module is used to compare the source data in the source log table and the source business table with the source reconciliation statement table of the data provider to obtain the pre-reconciliation result; The data synchronization module is used to synchronize the source data to the target business table of the data recipient based on the prior reconciliation result, and to compare the target data in the target business table with the obtained data synchronization result and the source reconciliation statement to obtain the post-reconciliation result; The pre-reconciliation module further includes: The first batch number acquisition unit is used to incrementally acquire the batch number of the source statement table using a preset batch acquisition component; Accordingly, the data synchronization module includes: The second data synchronization unit is used to synchronize the source data batch to the target business table of the data recipient based on the batch number according to the pre-reconciliation result. Furthermore, the pre-reconciliation module includes: The data judgment unit is used to determine the first batch data volume corresponding to each batch number in the source reconciliation form, and to determine the second batch data volume of the source data for each batch according to the source log table and the source business table, and to determine whether the first batch data volume and the second batch data volume are the same; The data filtering unit is used to filter out the first source data corresponding to the first batch number with the same data volume and the second source data corresponding to the second batch number with different data volume from the source business table according to the judgment result. The status update unit is used to update the prior reconciliation status in the target reconciliation statement table corresponding to the first source data to be consistent, and to update the prior reconciliation status in the target reconciliation statement table corresponding to the second source data to be inconsistent, so as to synchronize the first source data. The data synchronization module includes: The data extraction unit is used to extract the associated data of the first source data in the source log table and the source business table through a preset extraction component; The data update unit is used to write the associated data into the target business table using a preset table writing component, and update the target log table based on the associated data; Accordingly, the data synchronization module includes: The second batch number acquisition unit is used to start a reconciliation task according to a preset reconciliation task start time and obtain the target batch number corresponding to the reconciliation task. The post-reconciliation submodule is used to determine the target data volume corresponding to the target batch number in the target business table and the source data volume corresponding to the target batch number in the source reconciliation form, and to determine whether the target data volume and the source data volume are the same, so as to obtain the post-reconciliation result.

5. An electronic device, characterized in that, The electronic device includes a processor and a memory; wherein the memory is used to store a computer program, which is loaded and executed by the processor to implement the data reconciliation method as described in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the data reconciliation method as described in any one of claims 1 to 3.

Images