Data checking method and device, equipment, storage medium and program product

Abstract

The invention discloses a data checking method and device, equipment, a storage medium and a program product, and relates to the technical field of big data. The method comprises the following steps: a first server can check at least one piece of business data of multiple batches in a target batch group according to a preset first check rule to obtain each first check result, and sends the business data to a second server under the condition that the first check result indicates that the business data meets the first check rule; the second server checks the service data according to a preset second check rule to obtain a second check result; and under the condition that the second check result indicates that the service data meets the second check rule, determining that the service data is submitted successfully, and storing the service data in the database of the second server. According to the embodiment of the invention, the quality of data sent by the submission end to the supervision end is improved, and the influence of problem data in the supervision end on subsequent data supervision analysis is reduced.

Description

Technical Field

[0001] The present application belongs to the field of big data technology, and in particular relates to a data verification method, device, equipment, storage medium and program product. Background Art

[0002] Regulators collect data reported by financial institutions to achieve the goal of financial supervision, ensure the stability and security of the financial market, and promote the healthy and sustainable development of financial institutions. However, when regulators collect data reported by financial institutions, not only is the quality of the collected data poor, but the amount of data collected is also large, resulting in low efficiency in data processing by regulators. In order to improve the efficiency of data processing by regulators, data needs to be verified. The existing technology is that regulators conduct post-data quality verification on the submitted data, but such poor quality data has already been submitted to the regulator, which has a greater impact on subsequent data supervision and analysis. Summary of the invention

[0003] The embodiments of the present application provide a data verification method, apparatus, device, storage medium and program product, which can improve the quality of data sent from the reporting end to the supervision end and reduce the impact of problematic data in the supervision end on subsequent data supervision analysis.

[0004] In a first aspect, an embodiment of the present application provides a data verification method, which is applied to a first server, and the method includes:

[0005] Obtain at least one business data of multiple batches in the target batch group;

[0006] According to a preset first checking rule, each of the business data is checked to obtain each first checking result, wherein the first checking result is used to indicate whether the business data satisfies the first checking rule;

[0007] When the first verification result indicates that the business data satisfies the first verification rule, the business data is sent to the second server so that the second server verifies the business data according to the preset second verification rule to obtain a second verification result, and the second verification result is used to indicate whether the business data satisfies the second verification rule; when the second verification result indicates that the business data satisfies the second verification rule, it is determined that the business data is successfully submitted, and the business data is stored in the database of the second server.

[0008] In a second aspect, an embodiment of the present application provides another data verification method, which is applied to a second server, and the method includes:

[0009] receiving the business data sent by the first server when a first verification result indicates that the business data satisfies a preset first verification rule, the first verification result being a result obtained by the first server verifying each of the business data in at least one business data of a plurality of batches in the target batch group according to the first verification rule, the first verification result being used to indicate whether the business data satisfies the first verification rule;

[0010] According to a preset second checking rule, the business data is checked to obtain a second checking result, where the second checking result is used to indicate whether the business data satisfies the second checking rule;

[0011] When the second verification result indicates that the business data satisfies the second verification rule, it is determined that the business data is successfully submitted, and the business data is stored in the database of the second server.

[0012] In a third aspect, an embodiment of the present application provides a data checking device, which is applied to a first server, and the device includes:

[0013] An acquisition module, used for acquiring at least one business data of multiple batches in a target batch group;

[0014] A first checking module, used to check each of the business data according to a preset first checking rule to obtain each first checking result, wherein the first checking result is used to indicate whether the business data satisfies the first checking rule;

[0015] The first sending module is used to send the business data to the second server when the first verification result indicates that the business data satisfies the first verification rule, so that the second server verifies the business data according to the preset second verification rule to obtain a second verification result, and the second verification result is used to indicate whether the business data satisfies the second verification rule; when the second verification result indicates that the business data satisfies the second verification rule, it is determined that the business data is successfully submitted, and the business data is stored in the database of the second server.

[0016] In a fourth aspect, an embodiment of the present application provides another data checking device, which is applied to a second server, and the device includes:

[0017] A first receiving module, used to receive the business data sent by the first server when a first verification result indicates that the business data satisfies a preset first verification rule, the first verification result being a result obtained by the first server verifying each of the business data in at least one business data of multiple batches in the target batch group according to the first verification rule, and the first verification result is used to indicate whether the business data satisfies the first verification rule;

[0018] A second checking module, used to check the business data according to a preset second checking rule to obtain a second checking result, wherein the second checking result is used to indicate whether the business data satisfies the second checking rule;

[0019] A determination module is used to determine that the business data is successfully submitted and store the business data in a database of the second server when the second verification result indicates that the business data satisfies the second verification rule.

[0020] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the data verification method as described in any one of the above is implemented.

[0021] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the data verification method as described in any one of the above is implemented.

[0022] In a seventh aspect, an embodiment of the present application provides a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes any one of the data verification methods described above.

[0023] The data verification method, device, equipment, storage medium and program product of the embodiment of the present application, the first server can verify at least one business data of multiple batches in the target batch group according to the preset first verification rule, obtain each first verification result, and send the business data to the second server when the first verification result indicates that the business data meets the first verification rule; then the second server verifies the business data according to the preset second verification rule to obtain the second verification result; when the second verification result indicates that the business data meets the second verification rule, it is determined that the business data is successfully submitted, and the business data is stored in the database of the second server. In this way, in the embodiment of the present application, by performing a pre-verification before submitting the business data in the first server, not only some problematic data can be screened out, but also the verification pressure on the supervisory end can be reduced, and the submitted business data can be verified again in the second server to further discover problematic data, thereby improving the quality of data sent from the submitting end to the supervisory end, and reducing the impact of problematic data in the supervisory end on subsequent data supervision and analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0025] Figure 1 is an architecture diagram of a data verification system provided in an embodiment of the present application;

[0026] Figure 2 It is a flowchart of a data verification method provided in an embodiment of the present application;

[0027] Figure 3 It is a design diagram of the overall verification solution provided in the embodiment of the present application;

[0028] Figure 4 It is a schematic diagram of the overall verification process provided by the embodiment of the present application;

[0029] Figure 5 is a design diagram of a pre-check tool provided in an embodiment of the present application;

[0030] Figure 6 is a flow chart of a pre-check tool provided in an embodiment of the present application;

[0031] Figure 7 It is a design diagram of the in-process verification scenario provided in the embodiment of the present application;

[0032] Figure 8 It is a design diagram of the post-verification scenario provided in the embodiment of the present application;

[0033] Fig. 9 is a design diagram of a verification engine tool provided in an embodiment of the present application;

[0034] Fig.10 is a processing flow chart of the verification engine tool provided in an embodiment of the present application;

[0035] Fig.11 It is a structural schematic diagram of a data checking device provided in an embodiment of the present application;

[0036] Fig.12 is a schematic diagram of the structure of another data checking device provided in an embodiment of the present application;

[0037] Fig.13 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0039] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0040] Regulators collect data reported by financial institutions to achieve the goal of financial supervision, ensure the stability and security of the financial market, and promote the healthy and sustainable development of financial institutions. However, when regulators collect data reported by financial institutions, not only is the quality of the collected data poor, but the amount of data collected is also large, resulting in low efficiency in data processing by regulators. In order to improve the efficiency of data processing by regulators, data needs to be verified. The existing technology is that regulators conduct post-data quality verification on the submitted data, but such poor quality data has already been submitted to the regulator, which has a greater impact on subsequent data supervision and analysis.

[0041] The acquisition, storage, use, and processing of data in the technical solution of this application are in compliance with the relevant provisions of national laws and regulations. It should be noted that in the embodiments of this application, some existing solutions in the industry such as certain software, components, and models may be mentioned, and they should be considered as exemplary. Their purpose is only to illustrate the feasibility of the implementation of the technical solution of this application, but it does not mean that the applicant has or will necessarily use the solution.

[0042] In order to solve the problems in the prior art, the embodiments of the present application provide a data checking method, device, equipment, storage medium and program product. The data checking system provided by the embodiments of the present application is first introduced below.

[0043] Figure 1 FIG. 1 shows an architecture diagram of a data checking system applicable to an embodiment of the present application. Figure 1 As shown, the data checking system 100 may include: a first server 101 and a second server 102 .

[0044] The first server 101 may refer to a financial institution server that reports business data. Financial institution servers are computing devices that provide services specifically for financial enterprises such as banks, securities companies, and insurance companies. They are used to process transactions, store data, and run financial software and applications. They usually have the ability to collect data in real time or at regular intervals, and store the data locally or send it to other servers for further processing and analysis. There may be multiple first servers, and different first servers may correspond to different financial institutions.

[0045] The second server 102 may refer to a regulatory agency server that collects business data submitted by financial institutions. The regulatory agency server stores and supervises a large amount of business data, including financial institution reports, market transaction data, customer information, etc., and processes and analyzes the business data submitted by financial institutions to monitor market dynamics, detect abnormal transactions, assess risks, etc. These analyses help regulatory agencies formulate policies, conduct risk warnings, and take regulatory measures.

[0046] The second server 102 can be connected to at least one first server 101 in communication. Specifically, the communication can be performed using an Ethernet or wireless network connection. Of course, the communication method is not limited thereto, and can also be other network connection communication methods, which are not specifically limited here.

[0047] The data verification method provided in the embodiment of the present application is introduced below.

[0048] Figure 2 The flowchart of the data verification method provided by an embodiment of the present application is shown. Optionally, the method of the embodiment of the present application can be applied to the above Figure 1 The first server and the second server shown. Figure 2 As shown, a data checking method 200 may include the following steps S201 to S205:

[0049] S201, a first server obtains at least one business data of multiple batches in a target batch group;

[0050] S202: The first server verifies each business data according to a preset first verification rule to obtain each first verification result, where the first verification result is used to indicate whether the business data satisfies the first verification rule;

[0051] S203: When the first verification result indicates that the business data satisfies the first verification rule, the first server sends the business data to the second server;

[0052] S204, the second server receives the service data sent by the first server;

[0053] S205. The second server verifies the business data according to a preset second verification rule to obtain a second verification result, where the second verification result is used to indicate whether the business data satisfies the second verification rule.

[0054] S206: When the second verification result indicates that the business data satisfies the second verification rule, the second server determines that the business data has been successfully submitted, and stores the business data in a database of the second server.

[0055] In the data verification method of the embodiment of the present application, the first server can verify at least one business data of multiple batches in the target batch group according to the preset first verification rule, obtain each first verification result, and send the business data to the second server when the first verification result indicates that the business data meets the first verification rule; then the second server verifies the business data according to the preset second verification rule to obtain the second verification result; when the second verification result indicates that the business data meets the second verification rule, it is determined that the business data is successfully submitted, and the business data is stored in the database of the second server. In this way, in the embodiment of the present application, by performing a pre-verification before submitting the business data in the first server, not only some problematic data can be screened out, but also the verification pressure on the regulatory end can be reduced, and the submitted business data can be verified again in the second server to further discover problematic data, thereby improving the quality of data sent from the submitting end to the regulatory end, and reducing the impact of problematic data in the regulatory end on subsequent data supervision and analysis.

[0056] In S201, the target batch group is a batch group to be submitted in the first server. The target batch group may include multiple batches, and each batch includes at least one business data.

[0057] The above-mentioned business data may refer to various types of data generated by financial institutions in the course of daily operations. These data reflect information on the financial institutions' business activities, customer transactions, market conditions, risk management, etc.

[0058] In S202, the first check result may be used to indicate whether the business data satisfies a first check rule. The first check rule may include a null check sub-rule, a length check sub-rule and a type check sub-rule.

[0059] In the above, each business data is checked according to the preset first check rule to obtain each first check result. Exemplarily, each business data can be checked according to the null value check sub-rule in the preset first check rule to obtain each first check sub-result, and the first check sub-result is used to indicate whether there is a null value in the business data; each business data is checked according to the length check sub-rule in the first check rule to obtain each second check sub-result, and the second check sub-result is used to indicate whether the data length of the business data is equal to the preset length; each business data is checked according to the type check sub-rule in the first check rule to obtain each third check sub-result, and the third check sub-result is used to indicate whether the data type in the business data meets the preset type. The first check result includes the first check sub-result, the second check sub-result and the third check sub-result.

[0060] In S203, the business data is sent to the second server. Exemplarily, the business data may be sent to the second server via a wireless network connection.

[0061] In S204, the second server receives the service data sent by the first server. For example, the second server may receive the service data sent by the first server via a wireless network connection.

[0062] In S205, the second check result can be used to indicate whether the business data meets the second check rule. The second check rule includes data consistency check sub-rule, data consistency check sub-rule, data uniqueness check sub-rule, data timeliness check sub-rule and data accuracy check sub-rule.

[0063] In the above, the business data is checked according to the preset second check rule to obtain the second check result. Exemplarily, the business data can be checked according to the data consistency check sub-rule in the preset second check rule to obtain the first check sub-result, and the first check sub-result is used to indicate whether the first server from which the business data is source corresponds to the financial institution from which the business data is sourced; the business data is checked according to the data consistency check sub-rule in the second check rule to obtain the second check sub-result, and the second check sub-result is used to indicate whether the correspondence between the business primary key values ​​in the business data satisfies the preset correspondence; the business data is checked according to the data uniqueness check sub-rule in the second check rule Verify and obtain a third verification sub-result, which is used to indicate whether each business primary key value in the business data is unique; verify the business data according to the data timeliness verification sub-rule in the second verification rule to obtain a fourth verification sub-result, which is used to indicate whether the time information in the business data is within the preset target time period; verify the business data according to the data accuracy verification sub-rule in the second verification rule to obtain a fifth verification sub-result, which is used to indicate whether each business primary key value in the business data meets the preset value range. The second verification result includes the first verification sub-result, the second verification sub-result, the third verification sub-result, the fourth verification sub-result and the fifth verification sub-result.

[0064] In S206, since the business data meets the first check rule and the second check rule, it can be determined that the business data is successfully reported, and the business data is stored in the database of the second server, thereby improving the data quality sent by the reporting end to the supervision end.

[0065] In some embodiments, the above S202 may specifically include:

[0066] According to the null value check sub-rule in the preset first check rule, each business data is checked to obtain each first check sub-result, where the first check sub-result is used to indicate whether there is a null value in the business data;

[0067] According to the length check sub-rule in the first check rule, each service data is checked to obtain each second check sub-result, where the second check sub-result is used to indicate whether the data length of the service data is equal to the preset length;

[0068] According to the type check sub-rule in the first check rule, each business data is checked to obtain each third check sub-result, and the third check sub-result is used to indicate whether the data type in the business data meets the preset type.

[0069] The first check result includes a first syndrome result, a second syndrome result, and a third syndrome result.

[0070] The first check result may include a first syndrome result, a second syndrome result, and a third syndrome result. The first syndrome result may be used to indicate whether there is a null value in the business data. The second syndrome result may be used to indicate whether the data length of the business data is equal to a preset length. The third syndrome result may be used to indicate whether the data type in the business data satisfies a preset type, and the preset type may include a numeric value, a character, and the like.

[0071] In the embodiment of the present application, a basic check is performed on the business data according to the null check sub-rule, length check sub-rule and type check sub-rule in the preset first check rule to obtain the first check sub-result, the second check sub-result and the third check sub-result. Such a basic check in advance can accurately screen out business data with basic problems and reduce the check pressure on the regulatory end.

[0072] In some embodiments, the above S205 may specifically include:

[0073] According to the data consistency check sub-rule in the preset second check rule, the business data is checked to obtain a first check sub-result, and the first check sub-result is used to indicate whether the first server from which the business data is derived corresponds to the financial institution from which the business data is derived;

[0074] According to the data consistency check sub-rule in the second check rule, the business data is checked to obtain a second check sub-result, where the second check sub-result is used to indicate whether the correspondence between the business primary key values ​​in the business data satisfies a preset correspondence;

[0075] According to the data uniqueness check sub-rule in the second check rule, the business data is checked to obtain a third check sub-result, where the third check sub-result is used to indicate whether each business primary key value in the business data is unique;

[0076] According to the data timeliness check sub-rule in the second check rule, the business data is checked to obtain a fourth check sub-result, where the fourth check sub-result is used to indicate whether the time information in the business data is within a preset target time period;

[0077] According to the data accuracy check sub-rule in the second check rule, the business data is checked to obtain a fifth check sub-result, which is used to indicate whether each business primary key value in the business data meets a preset value range.

[0078] The second check result includes a first check sub-result, a second check sub-result, a third check sub-result, a fourth check sub-result and a fifth check sub-result.

[0079] The above-mentioned second check result may include a first check sub-result, a second check sub-result, a third check sub-result, a fourth check sub-result and a fifth check sub-result. Among them, the first check sub-result may be used to indicate whether the first server from which the business data is derived corresponds to the financial institution from which the business data is derived. The second check sub-result may be used to indicate whether the correspondence between the business primary key values ​​in the business data satisfies a preset correspondence, such as whether the brand primary key value corresponds to the territorial primary key value. The third check sub-result may be used to indicate whether each business primary key value in the business data is unique. The fourth check sub-result may be used to indicate whether the time information in the business data is within a preset target time period. The fifth check sub-result may be used to indicate whether each business primary key value in the business data satisfies a preset numerical range.

[0080] In an embodiment of the present application, the business data is checked according to the data consistency check sub-rule, data consistency check sub-rule, data uniqueness check sub-rule, data timeliness check sub-rule and data accuracy check sub-rule in the preset second check rule to obtain the first check sub-result, the second check sub-result, the third check sub-result, the fourth check sub-result and the fifth check sub-result. Such in-process logical check can accurately screen out business data with logical problems and further improve the data quality sent from the reporting end to the regulatory end.

[0081] As an implementation of the present application, in order to further improve the quality of data sent from the reporting end to the supervision end, before the above S205, the above method may further include:

[0082] The second server verifies the business data according to the preset first verification rule to obtain a third verification result, where the third verification result is used to indicate whether the business data satisfies the first verification rule;

[0083] The above S205 may specifically include:

[0084] When the third verification result indicates that the business data satisfies the first verification rule, the second server verifies the business data according to the preset second verification rule to obtain a second verification result.

[0085] The first check rule in the second server may be the same as the first check rule in the first server.

[0086] The third check result may be used to indicate whether the business data in the second server satisfies the first check rule.

[0087] In the embodiment of the present application, since it may be possible that the first server does not enforce prior basic verification of the business data, it is necessary to use the second server to perform a basic verification on the business data again according to the preset first verification rules to obtain a third verification result. When the third verification result indicates that the business data satisfies the first verification rule, the business data is verified according to the preset second verification rules to obtain a second verification result. This can accurately screen out business data with basic and logical problems in the second server, and further improve the data quality sent from the reporting end to the regulatory end.

[0088] As another implementation of the present application, in order to reduce the impact of post-verification on business data, after S206, the method further includes:

[0089] The second server performs mirroring processing on at least one business data in the database to obtain at least one business mirror data;

[0090] The second server evaluates each business mirror data according to a preset evaluation standard rule to obtain an evaluation result of each business mirror data, wherein the evaluation result includes abnormal information in the business mirror data that does not meet the evaluation standard rule;

[0091] The second server generates a data quality assessment report for the target batch group according to the evaluation result of at least one business mirror data;

[0092] The second server sends the data quality assessment report to the first server.

[0093] The first server receives the data quality assessment report sent by the second server.

[0094] The at least one business mirror data may be data obtained by mirroring at least one business data in the database, wherein the at least one business data in the database is at least one business data in the target batch group that satisfies the first check rule and the second check rule.

[0095] The above-mentioned evaluation standard rules may include the null value check sub-rule, length check sub-rule and type check sub-rule in the first check rule, and the data consistency check sub-rule, data consistency check sub-rule, data uniqueness check sub-rule, data timeliness check sub-rule and data accuracy check sub-rule in the second check rule.

[0096] The above evaluation results may include abnormal information in the business mirror data that does not meet the evaluation standard rules, wherein the abnormal information may be that the time information in the business data is not in the target time period, or that the business primary key value in the business data is in an alarm value interval within a preset value range.

[0097] The data quality assessment report of the target batch group is generated based on the evaluation result of at least one business image data. For example, the business image data can be evaluated using data quality assessment tools such as Apache Griffin, DataCleaner, etc. These tools can help identify problems in the data and provide a detailed assessment report.

[0098] In an embodiment of the present application, the second server mirrors at least one business data in the database to obtain at least one business mirror data. The use of this read-write separation mechanism can reduce the impact of post-verification on business data, and then evaluate each business mirror data according to preset evaluation standard rules to obtain an evaluation result of each business mirror data. Finally, a data quality assessment report for the target batch group is generated and sent to the first server, supporting the financial institution corresponding to the first server to re-submit the revised business data to ensure the integrity of the submitted data.

[0099] In order to facilitate the understanding of the data verification method in the embodiment of the present application, the actual application process of this data verification method is described as follows:

[0100] (1) Overall verification solution: This verification solution is suitable for semi-structured data, structured data formats, large-scale data synchronous and asynchronous reporting mechanisms, and verification scenarios with high data quality requirements. It provides a lightweight offline verification tool and a general verification tool based on a database engine to meet the data verification solution for the entire process of institutional data reporting, and the efficiency of each process can be greatly improved.

[0101] Please see the attachment for the overall verification plan Figure 3 , providing lightweight pre-verification tools to facilitate institutions to verify offline rules in advance; providing database verification engine tools to support data verification capabilities during and after data submission.

[0102] Please see the attachment for the overall verification process Figure 4The data submitted by the institution shall first undergo pre-verification to reduce the error rate of reporting, and then undergo in-process verification. If passed, it can be put into the database. If failed, it shall be submitted again. The data put into the database shall be post-verified, and the results of the post-verification shall be notified to the institution to support the institution to submit again.

[0103] (2) In the pre-verification scenario, before the regulated institution (equivalent to the first server mentioned above) officially submits the data, the regulated institution will initiate a data verification on the data to be submitted to ensure that the data submitted this time meets the local requirements. The pre-verification tool can assist the regulated institution to check the basic problems of the data in advance, improve the quality of data submission, and reduce the frequency of data submission.

[0104] a) Tool positioning: A lightweight, easy-to-deploy, and easy-to-execute client-side verification tool, and the defined verification rules must be consistent with the platform's in-process verification rules.

[0105] b) Standard input: Based on the attribute list, the verification specification document and verification rule document (equivalent to the first verification rule mentioned above) are derived to support the parsing and verification of data files submitted by the institution.

[0106] c) Execution engine: Based on the requirements of the tool positioning, sqlLite was finally selected as the execution engine, which can be implemented quickly and has high batch execution efficiency.

[0107] For the design of the pre-verification tool, please refer to Appendix 5. The core technology is to design the verification template, and the verification rules can quickly execute the verification task through the verification template.

[0108] Please see the attachment for the process of the pre-check tool Figure 6 ,The pre-verification process includes three parts of data acquisition, data verification, and data calculation, to achieve the function of rapid response and feedback.

[0109] (3) In the on-site verification scenario, after the regulated institution formally submits the data, the regulatory institution (equivalent to the second server mentioned above) automatically initiates a data verification for the data submitted this time. Data that passes the verification can be stored in the database, and data that fails the verification needs to be modified by the regulatory institution before being submitted again to ensure that the data submitted this time meets the global requirements. For details, please see the attachment Figure 7 .

[0110] a) Basic verification: instance granularity. Verification rules are maintained in standard parsing metadata, maintained in the database, saved to memory when the service is started, and rule verification is performed in memory during data parsing (equivalent to the above).

[0111] b) Logical verification: Batch group granularity, the regulatory agency determines the relevance of the internal data of the batch group submitted this time. The verification rules are maintained in the metadata of the verification engine, database maintenance, and the verification engine interface is called to implement the rules (equivalent to the second verification rule mentioned above) verification.

[0112] (4) In the post-verification scenario, the regulatory agency will initiate a data verification on the existing data that has been successfully submitted by the regulated institution. Based on the results of this verification, a data quality assessment report will be generated, which can intuitively reflect the overall quality of the data submitted by each regulated financial institution. For details, please see the attachment Figure 8 .

[0113] Inventory verification is initiated in batches at T+1 every morning. In order to reduce the impact of inventory verification on the original business data, a read-write separation mechanism is adopted to initiate data replication tasks that extract common fields such as the data to be verified and the business primary key, and call the verification engine for the T+1 business mirror data.

[0114] (5) Verification engine tool. The verification engine is a rule engine developed based on a database. It is mainly used to perform quality verification on structured data saved in the database. It implements the configuration and execution of verification rules and supports both real-time verification and batch verification. Its main advantages are as follows:

[0115] a) Flexible import, support for batch import of rules using rule templates, and support for adding, deleting, modifying and checking individual rules in the configuration interface.

[0116] b) Flexible configuration, support for custom templates, simplified configuration of general and simple rules, and support for custom SQL to configure complex logic rules.

[0117] c) Flexible adaptation to databases. The implementation principle of the rule engine is to configure and splice intercepted SQL through rules, and then execute the intercepted SQL to implement data verification. The verification engine adapts to standard SQL statements, supports verification libraries of multiple different databases, and implements data verification in various scenarios.

[0118] Please see the attached for the design of the verification engine tool Fig. 9 .

[0119] Please see the attached for the processing flow of the verification engine tool Fig.10 .

[0120] In the embodiment of the present application, an offline, lightweight verification tool is designed. All pre-verification rules are encapsulated in the tool. All reporting agencies have unified the verification caliber. The collection end updates the verification rules and can quickly transmit them to the reporting agency by changing the pre-verification tool. In addition, an in-process verification is also designed. For performance issues, memory verification and database verification tools are designed based on single-item or non-single-item inspections, which solves performance problems and batch rule configuration problems. The verification engine tool provided by this application is designed with a verification template. By enumerating all verification scenarios, a verification template is designed. The scenarios and templates correspond one to one. Through simple configuration, the configuration time of the rules is greatly reduced.

[0121] Based on the data checking method provided in the above embodiment, the present application also provides a specific implementation of a data checking device. Please refer to the following embodiment.

[0122] like Fig.11 As shown, a data checking device 1100 provided in an embodiment of the present application is applied to a first server. The device 1100 may include the following modules: an acquisition module 1101 , a first checking module 1102 and a first sending module 1103 .

[0123] An acquisition module 1101 is used to acquire at least one business data of multiple batches in a target batch group;

[0124] A first checking module 1102, used to check each business data according to a preset first checking rule to obtain each first checking result, where the first checking result is used to indicate whether the business data satisfies the first checking rule;

[0125] The first sending module 1103 is used to send the business data to the second server when the first verification result indicates that the business data meets the first verification rule, so that the second server verifies the business data according to the preset second verification rule to obtain a second verification result, and the second verification result is used to indicate whether the business data meets the second verification rule; when the second verification result indicates that the business data meets the second verification rule, it is determined that the business data is successfully submitted, and the business data is stored in the database of the second server.

[0126] In the data verification device of the embodiment of the present application, the first server can verify at least one business data of multiple batches in the target batch group according to the preset first verification rule, obtain each first verification result, and send the business data to the second server when the first verification result indicates that the business data meets the first verification rule; then the second server verifies the business data according to the preset second verification rule to obtain the second verification result; when the second verification result indicates that the business data meets the second verification rule, it is determined that the business data is successfully submitted, and the business data is stored in the database of the second server. In this way, in the embodiment of the present application, by performing a pre-verification before submitting the business data in the first server, not only some problematic data can be screened out, but also the verification pressure on the supervisory end can be reduced, and the submitted business data can be verified again in the second server to further discover problematic data, thereby improving the quality of data sent from the submitting end to the supervisory end, and reducing the impact of problematic data in the supervisory end on subsequent data supervision and analysis.

[0127] In some embodiments, the first checking module 1102 may specifically include:

[0128] A first checking unit, used to check each business data according to a null value check sub-rule in a preset first checking rule to obtain each first check sub-result, where the first check sub-result is used to indicate whether there is a null value in the business data;

[0129] A second checking unit, used to check each service data according to the length check sub-rule in the first checking rule to obtain each second check sub-result, where the second check sub-result is used to indicate whether the data length of the service data is equal to a preset length;

[0130] The third checking unit is used to check each business data according to the type check sub-rule in the first checking rule to obtain each third check sub-result, and the third check sub-result is used to indicate whether the data type in the business data meets the preset type.

[0131] The first check result includes a first syndrome result, a second syndrome result, and a third syndrome result.

[0132] As an implementation of the present application, in order to ensure the integrity of the reported data, the above-mentioned device 1100 may also include:

[0133] The second receiving module is used to receive a data quality assessment report sent by the second server, where the data quality assessment report is a report for a target batch group generated based on an evaluation result of at least one business mirror data, where the at least one business mirror data is data obtained by mirroring at least one business data in a database, and the evaluation result is a result obtained by evaluating the business mirror data based on preset evaluation standard rules, where the evaluation result includes abnormal information in the business mirror data that does not meet the evaluation standard rules.

[0134] like Fig.12 As shown, another data checking device 1200 provided in an embodiment of the present application is applied to a second server. The device 1200 may include the following modules: a first receiving module 1201 , a second checking module 1202 and a determining module 1203 .

[0135] The first receiving module 1201 is used to receive business data sent by the first server when a first verification result indicates that the business data meets a preset first verification rule, the first verification result is a result obtained by the first server verifying each business data in at least one business data of multiple batches in the target batch group according to the first verification rule, and the first verification result is used to indicate whether the business data meets the first verification rule;

[0136] A second checking module 1202, used to check the business data according to a preset second checking rule to obtain a second checking result, where the second checking result is used to indicate whether the business data satisfies the second checking rule;

[0137] The determination module 1203 is used to determine that the business data is successfully submitted and store the business data in the database of the second server if the second verification result indicates that the business data meets the second verification rule.

[0138] In the data verification device of the embodiment of the present application, the first server can verify at least one business data of multiple batches in the target batch group according to the preset first verification rule, obtain each first verification result, and send the business data to the second server when the first verification result indicates that the business data meets the first verification rule; then the second server verifies the business data according to the preset second verification rule to obtain the second verification result; when the second verification result indicates that the business data meets the second verification rule, it is determined that the business data is successfully submitted, and the business data is stored in the database of the second server. In this way, in the embodiment of the present application, by performing a pre-verification before submitting the business data in the first server, not only some problematic data can be screened out, but also the verification pressure on the supervisory end can be reduced, and the submitted business data can be verified again in the second server to further discover problematic data, thereby improving the quality of data sent from the submitting end to the supervisory end, and reducing the impact of problematic data in the supervisory end on subsequent data supervision and analysis.

[0139] In some embodiments, the second checking module 1202 may specifically include:

[0140] a fourth verification unit, configured to verify the business data according to the data consistency verification sub-rule in the preset second verification rule, and obtain a first verification sub-result, wherein the first verification sub-result is used to indicate whether the first server from which the business data is derived corresponds to the financial institution from which the business data is derived;

[0141] A fifth checking unit, used to check the business data according to the data consistency check sub-rule in the second checking rule to obtain a second checking sub-result, where the second checking sub-result is used to indicate whether the correspondence between the business primary key values ​​in the business data satisfies a preset correspondence;

[0142] a sixth checking unit, configured to check the business data according to the data uniqueness check sub-rule in the second checking rule to obtain a third checking sub-result, wherein the third checking sub-result is used to indicate whether each business primary key value in the business data is unique;

[0143] a seventh checking unit, configured to check the business data according to the data timeliness check sub-rule in the second checking rule to obtain a fourth checking sub-result, wherein the fourth checking sub-result is used to indicate whether the time information in the business data is within a preset target time period;

[0144] an eighth checking unit, configured to check the business data according to the data accuracy checking sub-rule in the second checking rule to obtain a fifth checking sub-result, wherein the fifth checking sub-result is used to indicate whether each business primary key value in the business data satisfies a preset value range.

[0145] The second check result includes a first check sub-result, a second check sub-result, a third check sub-result, a fourth check sub-result and a fifth check sub-result.

[0146] As an implementation of the present application, in order to further improve the quality of data sent from the reporting end to the supervision end, the above-mentioned device 1200 may also include:

[0147] A third checking module, used to check the business data according to the preset first checking rule to obtain a third checking result, and the third checking result is used to indicate whether the business data satisfies the first checking rule;

[0148] The second checking module 1202 is specifically configured to check the business data according to a preset second checking rule to obtain a second checking result when the third checking result indicates that the business data satisfies the first checking rule.

[0149] As another implementation of the present application, in order to reduce the impact of post-verification on business data, the above-mentioned device 1200 may also include:

[0150] A processing module, used for performing mirror processing on at least one business data in the database to obtain at least one business mirror data;

[0151] An evaluation module is used to evaluate each business mirror data according to a preset evaluation standard rule to obtain an evaluation result of each business mirror data, wherein the evaluation result includes abnormal information in the business mirror data that does not meet the evaluation standard rule;

[0152] A generating module, used for generating a data quality assessment report of a target batch group according to an evaluation result of at least one business mirror data;

[0153] The second sending module is used to send the data quality assessment report to the first server.

[0154] Fig.13 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.

[0155] The electronic device may include a processor 1301 and a memory 1302 storing computer program instructions.

[0156] Specifically, the processor 1301 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0157] The memory 1302 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 1302 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 1302 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 1302 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 1302 is a non-volatile solid-state memory.

[0158] In certain embodiments, the memory 1302 may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage media device, an optical storage media device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Thus, in general, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to an aspect of the present disclosure.

[0159] The processor 1301 implements any one of the data checking methods in the above embodiments by reading and executing the computer program instructions stored in the memory 1302 .

[0160] In one example, the electronic device may further include a communication interface 1303 and a bus 1310. Fig.13 As shown, the processor 1301, the memory 1302, and the communication interface 1303 are connected via a bus 1310 and communicate with each other.

[0161] The communication interface 1303 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0162] Bus 1310 includes hardware, software or both, and the parts of electronic equipment are coupled to each other.For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 1310 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.

[0163] The electronic device can execute the data verification method in the embodiment of the present application, thereby realizing the combination Figure 2 , Fig.11 and Fig.12 The data verification method and device described.

[0164] In addition, in combination with the data checking method in the above embodiments, the present application embodiment can provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the data checking methods in the above embodiments is implemented.

[0165] In combination with the data verification method in the above embodiment, an embodiment of the present application may provide a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes any one of the data verification methods above.

[0166] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0167] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0168] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0169] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0170] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A data verification method, characterized in that: Applied to a first server, the method comprises: Obtain at least one business data of multiple batches in the target batch group; According to a preset first checking rule, each of the business data is checked to obtain each first checking result, wherein the first checking result is used to indicate whether the business data satisfies the first checking rule; When the first verification result indicates that the business data satisfies the first verification rule, the business data is sent to the second server so that the second server verifies the business data according to the preset second verification rule to obtain a second verification result, and the second verification result is used to indicate whether the business data satisfies the second verification rule; when the second verification result indicates that the business data satisfies the second verification rule, it is determined that the business data is successfully submitted, and the business data is stored in the database of the second server.

2. The method according to claim 1, characterized in that The business data is checked according to the preset first checking rule to obtain the first checking results, including: According to the null value check sub-rule in the preset first check rule, each of the business data is checked to obtain each first check sub-result, where the first check sub-result is used to indicate whether there is a null value in the business data; According to the length check sub-rule in the first check rule, each of the service data is checked to obtain each second check sub-result, where the second check sub-result is used to indicate whether the data length of the service data is equal to a preset length; According to the type check sub-rule in the first check rule, each of the business data is checked to obtain each third check sub-result, wherein the third check sub-result is used to indicate whether the data type in the business data satisfies a preset type, The first check result includes the first syndrome result, the second syndrome result, and the third syndrome result.

3. The method according to claim 1, characterized in that The method further comprises: Receive a data quality assessment report sent by the second server, wherein the data quality assessment report is a report of the target batch group generated according to an evaluation result of at least one business mirror data, wherein the at least one business mirror data is data obtained by mirroring at least one of the business data in the database, and the evaluation result is a result obtained by evaluating the business mirror data according to preset evaluation standard rules, and the evaluation result includes abnormal information in the business mirror data that does not meet the evaluation standard rules.

4. A data checking method, characterized in that: Applied to the second server, the method comprises: receiving the business data sent by the first server when a first verification result indicates that the business data satisfies a preset first verification rule, the first verification result being a result obtained by the first server verifying each of the business data in at least one business data of a plurality of batches in the target batch group according to the first verification rule, the first verification result being used to indicate whether the business data satisfies the first verification rule; According to a preset second checking rule, the business data is checked to obtain a second checking result, where the second checking result is used to indicate whether the business data satisfies the second checking rule; When the second verification result indicates that the business data satisfies the second verification rule, it is determined that the business data is successfully submitted, and the business data is stored in the database of the second server.

5. The method according to claim 4, characterized in that The checking of the business data according to the preset second checking rule to obtain a second checking result includes: According to the data consistency check sub-rule in the preset second check rule, the business data is checked to obtain a first check sub-result, wherein the first check sub-result is used to indicate whether the first server from which the business data is derived corresponds to the financial institution from which the business data is derived; According to the data consistency check sub-rule in the second check rule, the business data is checked to obtain a second check sub-result, where the second check sub-result is used to indicate whether the correspondence between the business primary key values ​​in the business data satisfies a preset correspondence; According to the data uniqueness check sub-rule in the second check rule, the business data is checked to obtain a third check sub-result, wherein the third check sub-result is used to indicate whether each business primary key value in the business data is unique; According to the data timeliness check sub-rule in the second check rule, the business data is checked to obtain a fourth check sub-result, where the fourth check sub-result is used to indicate whether the time information in the business data is within a preset target time period; According to the data accuracy check sub-rule in the second check rule, the business data is checked to obtain a fifth check sub-result, wherein the fifth check sub-result is used to indicate whether each of the business primary key values ​​in the business data meets a preset value range. The second check result includes the first check sub-result, the second check sub-result, the third check sub-result, the fourth check sub-result and the fifth check sub-result.

6. The method according to claim 4, characterized in that Before verifying the business data according to the preset second verification rule to obtain the second verification result, the method further includes: According to the preset first checking rule, the business data is checked to obtain a third checking result, where the third checking result is used to indicate whether the business data satisfies the first checking rule; The checking of the business data according to the preset second checking rule to obtain a second checking result includes: When the third verification result indicates that the business data satisfies the first verification rule, the business data is verified according to a preset second verification rule to obtain a second verification result.

7. The method according to claim 4, characterized in that After determining that the service data is successfully submitted and storing the service data in a database of the second server, the method further includes: Performing mirroring processing on at least one of the business data in the database to obtain at least one business mirror data; According to a preset evaluation standard rule, each of the business mirror data is evaluated to obtain an evaluation result of each of the business mirror data, wherein the evaluation result includes abnormal information in the business mirror data that does not meet the evaluation standard rule; Generating a data quality assessment report for the target batch group according to the evaluation result of the at least one business mirror data; The data quality assessment report is sent to the first server.

8. A data checking device, characterized in that: Applied to a first server, the device comprises: An acquisition module, used for acquiring at least one business data of multiple batches in a target batch group; A first checking module, used to check each of the business data according to a preset first checking rule to obtain each first checking result, wherein the first checking result is used to indicate whether the business data satisfies the first checking rule; The first sending module is used to send the business data to the second server when the first verification result indicates that the business data satisfies the first verification rule, so that the second server verifies the business data according to the preset second verification rule to obtain a second verification result, and the second verification result is used to indicate whether the business data satisfies the second verification rule; when the second verification result indicates that the business data satisfies the second verification rule, it is determined that the business data is successfully submitted, and the business data is stored in the database of the second server.

9. A data checking device, characterized in that: Applied to the second server, the device comprises: A first receiving module, used to receive the business data sent by the first server when a first verification result indicates that the business data satisfies a preset first verification rule, the first verification result being a result obtained by the first server verifying each of the business data in at least one business data of multiple batches in the target batch group according to the first verification rule, and the first verification result is used to indicate whether the business data satisfies the first verification rule; A second checking module, used to check the business data according to a preset second checking rule to obtain a second checking result, wherein the second checking result is used to indicate whether the business data satisfies the second checking rule; A determination module is used to determine that the business data is successfully submitted and store the business data in a database of the second server when the second verification result indicates that the business data satisfies the second verification rule.

10. An electronic device, characterized in that: The device comprises: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the data checking method according to any one of claims 1 to 7 is implemented.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the data checking method according to any one of claims 1 to 7 is implemented.

12. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the data checking method according to any one of claims 1 to 7.

Images