Data quality improvement treatment method and system

Abstract

The invention provides a data quality improvement treatment method and system. The data quality improvement treatment method comprises the following steps of performing incremental data extraction; checking the incremental data to obtain an incremental problem list; transferring the incremental problem list; processing increment problem data in the increment problem list to obtain an increment problem processing result; and initiating a data supplementation task for the incremental problem processing result to complete the improvement and treatment of the incremental data quality. According to the technical scheme, the data access script is configured to be seamlessly connected with the Hive database, and the task of incremental data extraction is automatically executed; a high-efficiency query language is written by using SparkSQL, and a large-scale data set is processed by using the powerful memory computing capability of the SparkSQL, so that the high efficiency and accuracy of a data processing flow are ensured.

Description

Background Art

[0002] With the continuous development of business, higher requirements are put forward for the quality verification and data control of incremental data. The traditional incremental data governance method can no longer meet the relevant needs. It often requires offline communication, has high maintenance costs, low efficiency, and cannot handle problematic data in a timely and effective manner. In order to avoid data inaccuracy caused by human intervention and data security leakage, it is urgent to invent a data quality improvement governance method. Summary of the invention

[0003] The present application provides a data quality improvement management method and system for achieving accurate data processing and improving data quality through fully automated verification and management of the system.

[0004] In a first aspect, a data quality improvement governance method is provided, comprising the following steps:

[0005] Perform incremental data extraction;

[0006] Checking the incremental data to obtain an incremental problem list;

[0007] Circulating the incremental problem list;

[0008] Processing the incremental problem data in the incremental problem list to obtain incremental problem processing results;

[0009] A data supplementary reporting task is initiated for the incremental problem processing results to improve and govern the incremental data quality.

[0010] In the above technical scheme, incremental data is extracted; the incremental data is checked to obtain an incremental problem list; the incremental problem list is circulated; the incremental problem data in the incremental problem list is processed to obtain incremental problem processing results; a data supplementary reporting task is initiated for the incremental problem processing results to improve and manage the incremental data quality; the data acquisition script is configured to seamlessly connect with the Hive database to automatically execute the incremental data extraction task; SparkSQL is used to write an efficient query language, and its powerful in-memory computing capabilities are used to process large-scale data sets to ensure the high efficiency and accuracy of the data processing process; accurate data processing is achieved and data quality is improved through fully automated verification and management of the system.

[0011] In a specific implementation plan, the steps for extracting incremental data are as follows: configuring data acquisition scripts, business types, and data acquisition tasks according to different calibers, creating different data extraction tasks, and extracting incremental data.

[0012] In a specific implementation scheme, the step of checking the incremental data to obtain the incremental problem list is specifically as follows:

[0013] Define verification rules according to different data acquisition tasks and specific business needs and requirements, and configure corresponding verification rule sets;

[0014] After the data is collected, a verification task is automatically created according to the pre-configured rule set, and the corresponding verification results are generated to obtain an incremental problem list.

[0015] In a specific implementation scheme, the step of circulating the incremental problem list is specifically as follows:

[0016] The incremental problem lists of different types and sources shall be circulated to the relevant persons in charge respectively.

[0017] In a specific implementation scheme, the step of processing the incremental question data in the incremental question list to obtain the incremental question processing result is specifically:

[0018] The incremental problem data in the incremental problem list of different types and sources are modified and processed, and after the processing is completed, the incremental problem status is modified to processed to obtain the incremental problem processing result.

[0019] In a specific implementation plan, a data supplement task is initiated for the incremental problem processing results to complete the steps of improving and managing the incremental data quality, specifically:

[0020] For incremental problem data whose status is processed, a data supplementary reporting task is initiated, and the data is pulled again for verification and reporting.

[0021] In a specific implementation scheme, the incremental data is stored using a big data storage method.

[0022] In a specific implementation scheme, the incremental problem list is circulated at the system level, and circulation reminders are issued.

[0023] In a specific implementation plan, the circulation reminder methods include enterprise WeChat reminders and email reminders.

[0024] Secondly, a data quality improvement governance system is provided, including:

[0025] An incremental data extraction module, used for performing incremental data extraction;

[0026] A verification module, used to verify the incremental data and obtain an incremental problem list;

[0027] A circulation module, used for circulating the incremental problem list;

[0028] A processing module, used for processing the incremental problem data in the incremental problem list to obtain an incremental problem processing result;

[0029] The task supplementary reporting module is used to initiate a data supplementary reporting task for the incremental problem processing results, so as to improve and manage the incremental data quality.

[0030] In the above technical scheme, incremental data is extracted; the incremental data is checked to obtain an incremental problem list; the incremental problem list is circulated; the incremental problem data in the incremental problem list is processed to obtain incremental problem processing results; a data supplementary reporting task is initiated for the incremental problem processing results to improve and manage the incremental data quality; the data acquisition script is configured to seamlessly connect with the Hive database to automatically execute the incremental data extraction task; SparkSQL is used to write an efficient query language, and its powerful in-memory computing capabilities are used to process large-scale data sets to ensure the high efficiency and accuracy of the data processing process; accurate data processing is achieved and data quality is improved through fully automated verification and management of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A flowchart of a data quality improvement management method provided in an embodiment of the present application;

[0032] Figure 2 A specific flow chart of the data quality improvement management method provided in the embodiment of the present application;

[0033] Figure 3 A schematic diagram of a task for providing a number of data in an embodiment of the present application;

[0034] Figure 4 A schematic diagram of a verification rule set provided in an embodiment of the present application;

[0035] Figure 5 A schematic diagram of the verification tasks and verification results provided in the embodiment of the present application;

[0036] Figure 6 A schematic diagram of an incremental question list provided for an embodiment of the present application;

[0037] Figure 7 A schematic diagram of the flow of the incremental question list provided in the embodiment of the present application;

[0038] Figure 8 A structural block diagram of the data quality improvement management system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] The present application is further described in detail below through the accompanying drawings and embodiments. Through these descriptions, the characteristics and advantages of the present application will become clearer and more specific.

[0040] The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise noted.

[0041] In addition, the technical features involved in different embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0042] To facilitate understanding of the data quality improvement governance method and system provided in the embodiment of the present application, its application scenario is first explained. The data quality improvement governance method and system provided in the embodiment of the present application are used to achieve accurate data processing and improve data quality through full-automatic verification and governance of the system. With the continuous development of business, higher requirements are put forward for the quality verification and data control of incremental data. The traditional incremental data governance method can no longer meet the relevant needs, and offline communication is often required, with high maintenance costs, low efficiency and inability to process problem data in a timely and effective manner. In order to avoid data inaccuracy caused by human intervention and the leakage of data security, it is urgent to invent a data quality improvement governance method. For this reason, the embodiment of the present application provides a data quality improvement governance method and system, so as to achieve accurate data processing and improve data quality through full-automatic verification and governance of the system. It is described in detail with embodiments in conjunction with specific drawings below.

[0043] refer to Figures 1 to 8 , Figure 1 A flowchart of a data quality improvement management method provided in an embodiment of the present application; Figure 2 A specific flow chart of the data quality improvement management method provided in the embodiment of the present application;

[0044] Figure 3 A schematic diagram of a task for providing a number of data in an embodiment of the present application; Figure 4 A schematic diagram of a verification rule set provided in an embodiment of the present application; Figure 5 A schematic diagram of the verification tasks and verification results provided in the embodiment of the present application; Figure 6 A schematic diagram of an incremental question list provided for an embodiment of the present application; Figure 7 A schematic diagram of the flow of the incremental question list provided in the embodiment of the present application; Figure 8 A structural block diagram of the data quality improvement management system provided in an embodiment of the present application.

[0045] exist Figure 1In the embodiment of the present application, a data quality improvement management method is provided, comprising the following steps:

[0046] Perform incremental data extraction;

[0047] Checking the incremental data to obtain an incremental problem list;

[0048] Circulating the incremental problem list;

[0049] Processing the incremental problem data in the incremental problem list to obtain incremental problem processing results;

[0050] A data supplementary reporting task is initiated for the incremental problem processing results to improve and govern the incremental data quality.

[0051] In the above technical scheme, incremental data is extracted; the incremental data is checked to obtain an incremental problem list; the incremental problem list is circulated; the incremental problem data in the incremental problem list is processed to obtain incremental problem processing results; a data supplementary reporting task is initiated for the incremental problem processing results to improve and manage the incremental data quality; the data acquisition script is configured to seamlessly connect with the Hive database to automatically execute the incremental data extraction task; SparkSQL is used to write an efficient query language, and its powerful in-memory computing capabilities are used to process large-scale data sets to ensure the high efficiency and accuracy of the data processing process; accurate data processing is achieved and data quality is improved through fully automated verification and management of the system.

[0052] In this embodiment, the data quality improvement and governance method is a data quality improvement and governance method based on T+1 incremental data and verification, which specifically includes:

[0053] During the incremental data extraction process, the script can automatically identify new or changed data to reduce the burden of data processing and improve system performance;

[0054] We have established sound data verification rules, created corresponding rule sets according to different businesses, checked the data after acquisition, and used Spark's data processing function to batch verify the accuracy and completeness of the data;

[0055] Based on the verification results, the system automatically generates a list of related to-do tasks and notifies the corresponding business operators so that data problems can be handled in a timely manner to ensure data quality;

[0056] After the data is processed, a corresponding data extraction task is created on the second day to recheck and supplement the data;

[0057] By configuring data retrieval scripts, it can seamlessly connect with the Hive database and automatically perform incremental data extraction tasks. Use SparkSQL to write efficient query languages ​​and use its powerful in-memory computing capabilities to process large-scale data sets, ensuring the high efficiency and accuracy of the data processing process.

[0058] In a specific implementation plan, the steps for extracting incremental data are as follows: configure data extraction scripts, business types, and data extraction tasks according to different calibers, create different data extraction tasks, and extract incremental data. Figure 3 The data collection tasks include finance, consulting, group insurance core, core, and core-universal interest settlement. The data collection scripts and data collection tasks can be flexibly configured according to different businesses and calibers.

[0059] In a specific implementation scheme, the step of checking the incremental data to obtain the incremental problem list is specifically as follows:

[0060] refer to Figure 4 , define verification rules according to different data acquisition tasks and specific business needs and requirements, and configure the corresponding verification rule set;

[0061] refer to Figure 5 After the data is obtained, the verification task is automatically created according to the pre-configured rule set, and the corresponding verification results are generated to obtain an incremental problem list.

[0062] In this embodiment, configurable verification rules and rule sets facilitate maintenance and rule changes; a set of rules can verify existing and incremental data by replacing keywords according to different options.

[0063] In a specific implementation scheme, the step of circulating the incremental problem list is specifically as follows:

[0064] refer to Figure 6 and Figure 7 , the incremental problem lists of different types and sources will be circulated to the relevant persons in charge respectively.

[0065] In a specific implementation scheme, the step of processing the incremental question data in the incremental question list to obtain the incremental question processing result is specifically:

[0066] refer to Figure 7 , modify and process the incremental problem data in the incremental problem list of different types and sources, and after the processing is completed, modify the incremental problem status to processed to obtain the incremental problem processing result.

[0067] In a specific implementation plan, a data supplement task is initiated for the incremental problem processing results to complete the steps of improving and managing the incremental data quality, specifically:

[0068] For incremental problem data whose status is processed, a data supplementary reporting task is initiated, and the data is pulled again for verification and reporting.

[0069] In a specific implementation scheme, the incremental data is stored in a big data storage method, which is convenient for data query and storage and more efficient.

[0070] In a specific implementation scheme, the incremental problem list is circulated at the system level, and circulation reminders are issued.

[0071] In a specific implementation plan, the circulation reminder methods include enterprise WeChat reminder method and email reminder method. The circulation of problem data is realized at the system level, and multiple reminder methods such as enterprise WeChat and email are used to greatly reduce the time cost and communication cost of offline processing.

[0072] In the above technical solution, 1. Data acquisition scripts and tasks can be flexibly configured according to different businesses and calibers; 2. Configurable verification rules and rule sets are convenient for maintenance and rule changes; a set of rules can replace keywords according to different options to verify existing and incremental data; 3. The use of big data storage methods facilitates data query and storage, and is more efficient; 4. The flow of problem data is realized at the system level, and various reminder methods such as corporate WeChat and emails are used to greatly reduce the time cost and communication cost of offline processing.

[0073] exist Figure 8 In the embodiment of the present application, a data quality improvement management system is also provided, including:

[0074] An incremental data extraction module, used for performing incremental data extraction;

[0075] A verification module, used to verify the incremental data and obtain an incremental problem list;

[0076] A circulation module, used for circulating the incremental problem list;

[0077] A processing module, used for processing the incremental problem data in the incremental problem list to obtain an incremental problem processing result;

[0078] The task supplementary reporting module is used to initiate a data supplementary reporting task for the incremental problem processing results, so as to improve and manage the incremental data quality.

[0079] In the above technical scheme, incremental data is extracted; the incremental data is checked to obtain an incremental problem list; the incremental problem list is circulated; the incremental problem data in the incremental problem list is processed to obtain incremental problem processing results; a data supplementary reporting task is initiated for the incremental problem processing results to improve and manage the incremental data quality; the data acquisition script is configured to seamlessly connect with the Hive database to automatically execute the incremental data extraction task; SparkSQL is used to write an efficient query language, and its powerful in-memory computing capabilities are used to process large-scale data sets to ensure the high efficiency and accuracy of the data processing process; accurate data processing is achieved and data quality is improved through fully automated verification and management of the system.

[0080] refer to Figure 2 In this embodiment, the data of each system is synchronized through the data center, and the corresponding data extraction script is configured according to different caliber requirements to extract data. The system automatically checks the data quality of the extracted T+1 incremental data through different verification rules, analyzes and processes the verification results, records the problem data list and processes the problem data. The processing personnel only need to process the problem data on the page.

[0081] Through the system's fully automated verification and management, accurate data processing can be achieved, and it has the following advantages: 1. Multi-link data processing; 2. Data scalability; 3. Timeliness; 4. Reduced costs; 5. Reduced offline communication and increased efficiency.

[0082] Those skilled in the art will appreciate that the present application may be implemented as a system, method or computer program product.

[0083] Therefore, the present disclosure may be specifically implemented in the following forms, namely: it may be completely hardware, it may be completely software (including firmware, resident software, microcode, etc.), or it may be a combination of hardware and software, generally referred to herein as a "circuit", "module" or "system". In addition, in some embodiments, the present application may also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium may contain computer-readable program code.

[0084] Any combination of one or more computer-readable media can be used. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this document, computer-readable storage media can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, device or device.

[0085] Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art can change, modify, replace and modify the above embodiments within the scope of the present application. On this basis, the present application can be replaced and improved in many ways, all of which fall within the scope of protection of the present application.

Claims

1. A data quality improvement management method, characterized in that: The following steps are involved: Perform incremental data extraction; Checking the incremental data to obtain an incremental problem list; Circulating the incremental problem list; Processing the incremental problem data in the incremental problem list to obtain incremental problem processing results; A data supplementary reporting task is initiated for the incremental problem processing results to improve and govern the incremental data quality.

2. The data quality improvement management method according to claim 1 is characterized in that: The steps for incremental data extraction are as follows: configure data acquisition scripts, business types, and data acquisition tasks according to different calibers, create different data extraction tasks, and perform incremental data extraction.

3. The data quality improvement management method according to claim 2 is characterized in that: The steps of checking the incremental data to obtain the incremental problem list are specifically as follows: Define verification rules according to different data acquisition tasks and specific business needs and requirements, and configure corresponding verification rule sets; After the data is collected, a verification task is automatically created according to the pre-configured rule set, and the corresponding verification results are generated to obtain an incremental problem list.

4. The data quality improvement management method according to claim 3 is characterized in that: The steps of circulating the incremental problem list are specifically as follows: The incremental problem lists of different types and sources shall be circulated to the relevant persons in charge respectively.

5. The data quality improvement management method according to claim 4 is characterized in that: The step of processing the incremental question data in the incremental question list to obtain the incremental question processing result is specifically: The incremental problem data in the incremental problem list of different types and sources are modified and processed, and after the processing is completed, the incremental problem status is modified to processed to obtain the incremental problem processing result.

6. The data quality improvement management method according to claim 5 is characterized in that: Initiate a data supplement task for the incremental problem processing results to complete the steps of improving and managing the incremental data quality, specifically: For incremental problem data whose status is processed, a data supplementary reporting task is initiated, and the data is pulled again for verification and reporting.

7. The data quality improvement management method according to claim 6 is characterized in that: The incremental data is stored in a big data manner.

8. The data quality improvement management method according to claim 7 is characterized in that: The incremental problem list is circulated at the system level, and circulation reminders are issued.

9. The data quality improvement management method according to claim 8 is characterized in that: The methods of circulation reminder include enterprise WeChat reminder and email reminder.

10. A data quality improvement management system, characterized in that: include: An incremental data extraction module, used for performing incremental data extraction; A verification module, used to verify the incremental data and obtain an incremental problem list; A circulation module, used for circulating the incremental problem list; A processing module, used for processing the incremental problem data in the incremental problem list to obtain an incremental problem processing result; The task supplementary reporting module is used to initiate a data supplementary reporting task for the incremental problem processing results, so as to improve and manage the incremental data quality.

Images