Full-link reconciliation system based on four-side recorded data

Abstract

The invention discloses a full-link reconciliation system based on four-side recorded data, and the system comprises a data reconciliation module which is used for setting data collection and data reconciliation into an independent operation mode in a library table data record; in the structured data record, separating a structured file reconciliation detail result table from a library table data reconciliation detail result table; the consistency rule verification module is used for outputting a data reconciliation structure and carrying out consistency comparison by querying a committed data volume, a collected data volume, an in-lake data volume and a front library table data volume; the result display module is used for performing page display on a result of consistent account checking, and adding a collection frequency field on a display page to display the collection condition of the collection task; and the reconciliation table rule filling module is used for setting the statistical period of the reconciliation data volume of the front-end processor and enabling the statistical caliber of the reconciliation data volume of the front-end processor to be consistent with the caliber of the collected data extraction. According to the method, the consistency of the reconciliation quantities of the four-party data is analyzed more accurately, and the uniformity and integrity of the four-party data are ensured.

Description

Technical Field

[0001] The present invention relates to an account reconciliation system, and in particular to a full-link account reconciliation system based on four-party record data. Background Art

[0002] In order to standardize data collection and ensure the timeliness and integrity of data, relevant departments regularly carry out data reconciliation. Data reconciliation refers to the consistency check of the quantity and structure of data during the data collection process, aiming to ensure the accuracy and integrity of the data during the collection process. However, there are some shortcomings in the current data reconciliation work, mainly focusing on the data volume, data structure and the execution of the reconciliation procedure. The main problems include: 1. Data volume statistics lead to inconsistent reconciliation a). When the statistical value of [Data Volume] is 0, it may be because the reconciliation records were not written into the reconciliation table in time before the collection task was executed, resulting in the statistical value of 0 during reconciliation, causing inconsistency in reconciliation.

[0003] b). If the time interval in the reconciliation table is not set accurately (for example, the time interval is smaller than the actual reconciliation period), the statistical value of [Data Volume] will be smaller than the actual value, which will also cause inconsistency in the reconciliation among the four parties.

[0004] 2. Reconciliation data is not updated in a timely manner After modifying the reconciliation data, if the collection task or reconciliation task is not re-executed, the data reconciliation results will not be updated, resulting in subsequent reconciliation results that are still incorrect and cannot reflect the latest collection situation.

[0005] 3. The fields in the reconciliation table do not match the actual situation If the [total_num] field in the reconciliation table is inconsistent with the configuration of the collection method, the incremental value and the full value may be confused. For example, if [total_num] counts the incremental value, and the collection task is configured as full collection, the counted [data volume] will be inconsistent with the [collected data volume] and [input data volume].

[0006] 4. The problem of repeated execution of collection tasks When the collection task is executed repeatedly, the [collected data volume] and [input data volume] may exceed the actual value of the [pre-database table data volume], or even become an integer multiple of it, resulting in inconsistent reconciliation results among the four parties.

[0007] 5. The reconciliation process is blocked The current reconciliation program has a design bottleneck. Especially when the amount of data in the front-end database table is too large (for example, more than 200 million records), data statistics take a long time, which may block the entire reconciliation process, resulting in the inability to continue subsequent reconciliation work and ultimately the inability to output the reconciliation results.

[0008] 6. Difficulty maintaining reconciliation table fields Some fields in the reconciliation table (such as collection frequency, directory information, etc.) are required, but it is difficult for developers to obtain this information. They need to be filled in manually based on the collection task information, which increases the complexity of maintenance work and the risk of errors.

[0009] These problems indicate that the current data collection and reconciliation work has a lot of room for improvement in terms of process design, system execution, and field configuration. If not addressed, these problems may affect the accuracy of data reconciliation, which will have a negative impact on the overall data governance work. Optimizing the reconciliation process, improving field configuration and program design can effectively improve the quality and efficiency of data collection work. Summary of the invention

[0010] The technical problem to be solved by the present invention is to provide a full-link reconciliation system based on four-party record data, which can more accurately analyze the consistency of the four-party data reconciliation volume and ensure the unity and integrity of the four-party data.

[0011] The technical solution adopted by the present invention to solve the above-mentioned technical problems is to provide a full-link reconciliation system based on four-party record data, including a data reconciliation module: in the library table data record, data collection and data reconciliation are set to independent operation modes; in the structured data record, the structured file reconciliation detail result table and the library table data reconciliation detail result table are separated independently; a consistency rule verification module: when performing consistency verification on the four-party record data, the data reconciliation structure is output, and consistency comparison is performed by querying the commission data volume, the collection data volume, the lake data volume, and the front-end library table data volume; a result display module: in the front-end management subsystem, the consistent reconciliation results are displayed on the page, and a collection frequency field is added to the display page to display the collection status of the collection task; a reconciliation table rule filling module: the statistical period of the front-end reconciliation data volume is set to: day, week, and month, and the statistical caliber of the front-end reconciliation data volume is made consistent with the caliber of the collection data extraction.

[0012] Furthermore, the data reconciliation module separates the existing reconciliation logic from the collection task, and adds a query logic associated with the data directory information corresponding to the collection task, so that the reconciliation logic and the collection task run independently.

[0013] Furthermore, the consistency rule verification module includes: the comparison rule for minute and hour incremental tasks is: the amount of commissioned data = the amount of data entering the lake; other comparison rules are: the amount of commissioned data = the amount of collected data = the amount of data entering the lake; if the comparison results are consistent, it indicates that the reconciliation results are consistent; otherwise, a reconciliation inconsistency alarm is generated.

[0014] Furthermore, the result display module displays the reconciliation-consistent result on the business data monitoring-data reconciliation page under the monitoring and alarm module.

[0015] Furthermore, the reconciliation table rule filling module also provides filling rules and additional precautions for total_num, jhpt_start_time and jhpt_end_time.

[0016] Compared with the prior art, the present invention has the following beneficial effects: the full-link reconciliation system based on four-party record data provided by the present invention is optimized from four aspects: data reconciliation logic, consistency verification rules, result display page, and reconciliation table filling rules. It can more accurately analyze the consistency of the four-party data reconciliation volume and ensure the unity and integrity of the four-party data. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is an architecture diagram of the full-link reconciliation system based on four-party record data of the present invention; Figure 2 This is a flow chart of file data reconciliation of the full-link reconciliation system of the present invention; Figure 3 This is a flowchart of the database table data volume reconciliation of the full-link reconciliation system of the present invention. DETAILED DESCRIPTION

[0018] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0019] Figure 1 This is an architecture diagram of the full-link reconciliation system based on four-party record data of the present invention.

[0020] See also Figure 1 The full-link reconciliation system based on four-party record data provided by the present invention includes: Data reconciliation module: In the database table data record, data collection and data reconciliation are set to independent operation mode; in the structured data record, the structured file reconciliation detail result table and the database table data reconciliation detail result table are separated; Consistency rule verification module: When performing consistency verification on the four-party record data, the data reconciliation structure is output, and consistency comparison is performed by querying the amount of commissioned data, the amount of collected data, the amount of data entering the lake, and the amount of data in the front-end database table; Result display module: In the front-end management subsystem, the reconciliation results are displayed on the page, and a collection frequency field is added to the display page to display the collection status of the collection task; Reconciliation table rule filling module: Set the statistical period of the front-end reconciliation data volume to: daily, weekly, and monthly, and make the statistical caliber of the front-end reconciliation data volume consistent with the caliber of the collected data extraction.

[0021] The present invention optimizes data reconciliation logic, consistency verification rules, result display page, and reconciliation table filling rules in four aspects: 1. Optimize the reconciliation logic: In the database table data record, the original synchronous operation mode of "data collection + data reconciliation" is transformed into an independent operation mode; in the structured data record, the structured file reconciliation detail result table and the database table data reconciliation detail result table are separated. This method is to separate the existing reconciliation logic from the collection task, add the collection task corresponding data directory information association query logic, so that the two can run independently, solve problem 1, and also avoid the problem of task blocking affecting the subsequent program operation in problem 6.

[0022] 2. Optimization of data reconciliation consistency verification rules: When performing consistency verification on the four-party data, output the data reconciliation structure, and perform consistency comparison by querying "commissioned office data volume", "collected data volume", "lake data volume", and "front-end library table data volume". This method is to solve the problem of optimizing the four-party reconciliation consistency verification rules in the front-end machine management system and solve problem 5.

[0023] The rules are as follows: a. The comparison rule for minute-hour incremental tasks is: the amount of commissioned data = the amount of data entering the lake; b. Otherwise, the comparison rule is: the amount of commissioned data = the amount of collected data = the amount of data entering the lake; c. If the comparison results are consistent, it means that the reconciliation results are consistent; otherwise, an inconsistent reconciliation alarm is generated.

[0024] 3. Optimization of reconciliation structure display: In the front-end management subsystem, the consistent reconciliation results are displayed on the "Business Data Monitoring-Data Reconciliation" page under the "Monitoring and Alarm" module. According to the optimization of the reconciliation logic, it is necessary to add the "Collection Frequency" field on this page to indicate the collection status of the collection task.

[0025] 4. Reconciliation data filling specification: According to the optimized reconciliation plan, the statistical cycles of the front-end reconciliation data volume (i.e. the responsible party data volume) are: daily, weekly, and monthly. If the reconciliation results need to be consistent, the statistical caliber of the responsible party data volume and the caliber of the collected data extraction should be consistent. This method can further explain the filling rules of [total_num], jhpt_start_time / jhpt_end_time and supplementary precautions to solve problems 2, 3, and 4.

[0026] The optimized file data reconciliation logic of the present invention is as follows Figure 2 As shown in the figure, the optimized database table data volume reconciliation logic is as follows Figure 3 The specific advantages are as follows: 1. Improve productivity: Through data reconciliation and consistency verification, ensure the consistency of data among various business systems, reduce duplication of work or correction process caused by data errors, and improve overall production efficiency.

[0027] 2. Improve quality: Data consistency ensures the accuracy of information, thereby improving the quality of decision-making. Accurate data helps reduce errors and improve the stability and reliability of business processes.

[0028] 3. Enhanced accuracy: Through code processing and reconciliation mechanisms, every data point can be accurate, thereby improving the accuracy of data processing and reducing data deviation or error.

[0029] 4. Improve efficiency: Automated data reconciliation reduces the time and labor costs of manual verification, improves data processing efficiency, and makes the business system run more smoothly.

[0030] Although the present invention has been disclosed as above with preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art may make some modifications and improvements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be based on the definition of the claims.

Claims

1. A full-link reconciliation system based on four-party record data, characterized in that: include: Data reconciliation module: In the database table data records, data collection and data reconciliation are set to independent operation mode; In the structured data record, separate the structured file reconciliation detail result table and the library table data reconciliation detail result table; Consistency rule verification module: When performing consistency verification on the four-party record data, the data reconciliation structure is output, and consistency comparison is performed by querying the amount of commissioned data, the amount of collected data, the amount of data entering the lake, and the amount of data in the front-end database table; Result display module: In the front-end management subsystem, the reconciliation results are displayed on the page, and a collection frequency field is added to the display page to display the collection status of the collection task; Reconciliation table rule filling module: Set the statistical period of the front-end reconciliation data volume to: daily, weekly, and monthly, and make the statistical caliber of the front-end reconciliation data volume consistent with the caliber of the collected data extraction.

2. The full-link reconciliation system based on four-party record data as claimed in claim 1, characterized in that: The data reconciliation module separates the existing reconciliation logic from the collection task, adds the collection task corresponding data directory information association query logic, and enables the reconciliation logic and the collection task to run independently.

3. The full-link reconciliation system based on four-party record data as claimed in claim 1, characterized in that: The consistency rule verification module includes: The comparison rule for minute and hour incremental tasks is: the amount of commissioned data = the amount of data entering the lake; other comparison rules are: the amount of commissioned data = the amount of collected data = the amount of data entering the lake; if the comparison results are consistent, it means that the reconciliation results are consistent; otherwise, a reconciliation inconsistency alarm is generated.

4. The full-link reconciliation system based on four-party record data as claimed in claim 1, characterized in that: The result display module displays the reconciliation-consistent results on the business data monitoring-data reconciliation page under the monitoring and alarm module.

5. The full-link reconciliation system based on four-party record data as claimed in claim 1, characterized in that: The reconciliation table rule filling module also provides filling rules and additional precautions for total_num, jhpt_start_time and jhpt_end_time.

Images