Data quality rule generation method and device, storage medium and program product

Abstract

The invention discloses a data quality rule generation method and device, a storage medium and a program product. The method comprises the steps of obtaining metadata registered by a target object; all rule templates matched with the metadata are screened from a quality rule template library, a rule template set is obtained, template types in the quality rule template library comprise standard rule templates and constraint rule templates, the standard rule templates comprise templates used for setting data standards, and the constraint rule templates comprise templates used for setting data standards. The constraint rule template comprises a template used for setting constraint conditions; and generating a data quality rule based on the rule template set and the metadata, the data quality rule including a detection rule for performing quality detection on data generated by a target system of the financial institution. Through the data quality rule generation method and device, the problem that the generation efficiency of the data quality rule is low due to the fact that the data quality rule is generated in a manual compiling mode in related technologies is solved.

Description

Technical Field

[0001] The present application relates to the field of big data, and more specifically, to a method, device, storage medium and program product for generating data quality rules. Background Art

[0002] At present, in the process of digital transformation, enterprises generally form a set of tools for enterprise-level data management, such as metadata management tools and data quality management tools. These tools have independent positioning, and enterprises continue to use these tools for corresponding data management in the process of operation and production.

[0003] In the related technology, the data quality rules in the data quality management tool are maintained by the data governance personnel themselves, lacking unified standards and high cost of rule management. Specifically, first, when the data governance personnel write data quality rules, the rule query statements written for specific data requirements are not the same, which may lead to incomplete coverage of statement judgment conditions and errors in rule writing; second, data quality rules need to be changed synchronously according to changes in business scenarios or data standards, and the data quality rules managed and operated manually need to be actively adjusted by the data governance personnel for updates, which may lead to omissions; third, for the same type of data, the pattern of data quality rules is generally consistent, and different users need to deploy the same rules for different forms. There is a lot of repetitive work, which wastes a lot of manpower costs.

[0004] With regard to the problem of low efficiency in generating data quality rules by manually writing them in related technologies, no effective solution has been proposed so far. Summary of the invention

[0005] The main purpose of the present application is to provide a method, device, storage medium and program product for generating data quality rules, so as to solve the problem of low efficiency in generating data quality rules by manually writing data quality rules in related technologies.

[0006] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a method for generating data quality rules is provided. The method comprises: obtaining metadata registered by a target object; screening all rule templates matching the metadata from a quality rule template library to obtain a rule template set, wherein the template types in the quality rule template library include: standard rule templates and constraint rule templates, wherein the standard rule templates include: templates for setting data standards, and the constraint rule templates include: templates for setting constraint conditions; based on the rule template set and the metadata, generating data quality rules, wherein the data quality rules include: detection rules for performing quality detection on data generated by the target system of a financial institution.

[0007] Furthermore, all rule templates matching the metadata are screened from the quality rule template library to obtain a rule template set, including: matching the data fields in the metadata with the data fields of all rule templates in the quality rule template library to obtain a matching result; based on the matching result, determining all rule templates in the quality rule template library matching the metadata to obtain a rule template set.

[0008] Furthermore, based on the rule template set and the metadata, data quality rules are generated, including: using a target interface to visually display all rule templates in the rule template set, and using the target interface to receive a selection result of the target object selecting a rule template in the rule template set; based on the metadata, replacing the data field in the rule template selected by the target object indicated by the selection result to obtain the data quality rules.

[0009] Furthermore, after screening all rule templates matching the metadata from the quality rule template library to obtain a rule template set, it includes: acquiring the incremental data generated by the target system, and performing data quality detection on the incremental data based on the data quality rules to obtain detection results; generating a data quality report based on the detection results, and sending the data quality report to a target terminal, wherein the target terminal is used to display the data quality report.

[0010] Furthermore, based on the data quality rules, a data quality check is performed on the incremental data to obtain a test result, including: obtaining a partition logic for the incremental data, wherein the partition logic includes: a partition logic for logically partitioning the incremental data in the target system; based on the data quality rules and the partition logic, a parallel detection strategy is used to perform a data quality check on the incremental data to obtain the test result.

[0011] Furthermore, before performing data quality detection on the incremental data based on the data quality rules and obtaining the detection results, it also includes: obtaining the execution status and dependencies of all jobs in the target system; determining whether the jobs on which the incremental data depends have been completed based on the execution status and dependencies of all jobs in the target system; and when the jobs on which the incremental data depends have been completed, performing data quality detection on the incremental data based on the data quality rules and obtaining the detection results.

[0012] Furthermore, after generating data quality rules based on the rule template set and the metadata, it also includes: obtaining detection data obtained by quality detection of data generated by the target system in a historical time period; based on the detection data, adjusting the detection strategy of quality detection, wherein the detection strategy includes at least one of the following: parallelism and detection frequency of parallel detection.

[0013] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a device for generating data quality rules is provided. The device includes: an acquisition unit, which is used to acquire the metadata registered by the target object; a screening unit, which is used to screen all rule templates matching the metadata from the quality rule template library to obtain a rule template set, wherein the template types in the quality rule template library include: standard rule templates and constraint rule templates, wherein the standard rule template includes: a template for setting data standards, and the constraint rule template includes: a template for setting constraint conditions; a generation unit, which is used to generate data quality rules based on the rule template set and the metadata, wherein the data quality rules include: a detection rule for performing quality detection on the data generated by the target system of the financial institution.

[0014] Furthermore, the screening unit includes: a matching subunit, used to match the data fields in the metadata with the data fields of all rule templates in the quality rule template library to obtain a matching result; and a determination subunit, used to determine all rule templates in the quality rule template library that match the metadata based on the matching result to obtain a rule template set.

[0015] Furthermore, the generation unit includes: a first processing sub-unit, used to use the target interface to visually display all rule templates in the rule template set, and use the target interface to receive the selection result of the target object selecting the rule template in the rule template set; a second processing sub-unit, used to replace the data field in the rule template selected by the target object indicated by the selection result based on the metadata, so as to obtain the data quality rule.

[0016] Furthermore, the data quality rule generation device also includes: a first processing unit, used to obtain the incremental data generated by the target system after screening all rule templates matching the metadata from the quality rule template library to obtain a rule template set, and perform data quality detection on the incremental data based on the data quality rules to obtain a detection result; a second processing unit, used to generate a data quality report based on the detection result, and send the data quality report to a target terminal, wherein the target terminal is used to display the data quality report.

[0017] Furthermore, the first processing unit includes: an acquisition subunit, used to acquire the partitioning logic of the incremental data, wherein the partitioning logic includes: partitioning logic for logically partitioning the incremental data in the target system; a detection subunit, used to perform data quality detection on the incremental data based on the data quality rules and the partitioning logic, using a parallel detection strategy to obtain the detection result.

[0018] Furthermore, the data quality rule generation device also includes: a third processing unit, used to obtain the execution status and dependencies of all jobs in the target system before performing data quality detection on the incremental data based on the data quality rules and obtaining the detection results; a determination unit, used to determine whether the jobs on which the incremental data depends have been completed based on the execution status and dependencies of all jobs in the target system; and an execution unit, used to perform the step of performing data quality detection on the incremental data based on the data quality rules and obtaining the detection results when the jobs on which the incremental data depends have been completed.

[0019] Furthermore, the data quality rule generation device also includes: a fourth processing unit, which is used to obtain detection data obtained by quality detection of data generated by the target system in a historical time period after generating data quality rules based on the rule template set and the metadata; an adjustment unit, which is used to adjust the detection strategy of quality detection based on the detection data, wherein the detection strategy includes at least one of the following: parallelism and detection frequency of parallel detection.

[0020] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored executable program, wherein when the executable program is run, the device where the computer-readable storage medium is located is controlled to execute the method for generating data quality rules.

[0021] According to another aspect of the present application, an electronic device is provided, comprising: a memory storing an executable program; and a processor for running the program, wherein the method for generating data quality rules is executed when the program is running.

[0022] According to another aspect of the present application, a computer program product is provided, comprising computer instructions, which implement the steps of the method for generating data quality rules when executed by a processor.

[0023] In an embodiment of the present application, metadata registered by the target object is obtained; all rule templates matching the metadata are screened from the quality rule template library to obtain a rule template set, wherein the template types in the quality rule template library include: standard rule templates, constraint rule templates, standard rule templates include: templates for setting data standards, and constraint rule templates include: templates for setting constraint conditions; based on the rule template set and metadata, data quality rules are generated, wherein the data quality rules include: detection rules for quality detection of data generated by the target system of the financial institution, thereby solving the technical problem of low efficiency in generating data quality rules by manually writing in the related art. In the present invention, data quality rules are automatically generated based on metadata and the quality rule template library, thereby avoiding the low efficiency of manually writing data quality rules in the related art, thereby achieving the technical effect of improving the efficiency of generating data quality rules. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0025] Figure 1 A hardware structure block diagram of a computer terminal for implementing a method for generating data quality rules is shown;

[0026] Figure 2 is a flow chart of a method for generating data quality rules according to an embodiment of the present application;

[0027] Figure 3 is a schematic diagram of a device for generating data quality rules according to an embodiment of the present application;

[0028] Figure 4 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] It should be noted that the data quality rule generation method, device, storage medium and program product in the present application can be used in the financial technology field to generate data quality detection rules, and can also be used in any field other than the financial technology field to generate data quality detection rules. The application of the data quality rule generation method, device, storage medium and program product is not limited.

[0032] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0033] Metadata: refers to data that describes data. It is information about the characteristics of the data itself and is used to provide context, meaning, and additional details required to manage the data. Metadata can be used to make data easier to discover, understand, and manage, and is essential for data governance, data quality, and data security.

[0034] Data quality rules: A series of clear standards and requirements established by an enterprise or organization to ensure that its data meets specific quality standards. Data quality rules help identify, measure, monitor and improve data quality to ensure the accuracy, completeness, consistency, credibility and availability of data.

[0035] Job dependency: refers to the task identifier for data processing during the processing of big data forms, which is the information to determine whether the data processing has been completed.

[0036] It should be noted that the collected information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this application are information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data are in compliance with relevant laws, regulations and standards, necessary confidentiality measures are taken, and public order and good customs are not violated, and corresponding operation entrances are provided for users to choose to authorize or refuse. For example, an interface is set up between this system and relevant users or institutions to provide users with corresponding operation entrances for users to choose to agree or refuse the results of automated decision-making; if the user chooses to refuse, the expert decision-making process will be entered.

[0037] Embodiment 1

[0038] According to an embodiment of the present application, a method embodiment of a method for generating data quality rules is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0039] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a method for generating data quality rules. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0040] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0041] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the method for generating data quality rules in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the method for generating data quality rules described above is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0042] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0043] The display may be, for example, a touch screen liquid crystal display (LCD), which may enable a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0044] Under the above operating environment, this application provides Figure 2 The method for generating data quality rules is shown. Figure 2 It is a flowchart of a method for generating data quality rules according to the first embodiment of the present application.

[0045] Step S201, obtaining metadata registered by the target object.

[0046] The above-mentioned metadata may include: metadata information of the target object registering the data on the metadata management platform, such as metadata such as data source, table, field, etc. The target object may also specify the business meaning of the data, data classification, sensitivity level, etc. The above-mentioned target object may be a data quality manager of a financial institution.

[0047] Step S202, filter all rule templates matching the metadata from the quality rule template library to obtain a rule template set, wherein the template types in the quality rule template library include: standard rule templates, constraint rule templates, the standard rule templates include: templates for setting data standards, and the constraint rule templates include: templates for setting constraint conditions.

[0048] The above-mentioned standard rule template can be a template for setting data standards. The design of the rule template for the data standard (i.e., the standard rule template) can specifically include: using a structured method to organize the template. The standard rule template in the quality rule template library can include multiple data standard items. Each data standard item (such as "customer ID") can have one or more corresponding rule templates. The rule template can directly map the attributes and constraints in the data standard. For example, the standard of the "customer ID" data element may include that it should be an integer, unique and non-empty, and the corresponding rule template is to check whether the field conforms to the integer type, uniqueness and non-empty.

[0049] Standard rule template example: SELECT [customer ID] FROM library name. table name WHERE customer ID NOT LIKE '[0-9]*' OR COUNT (DISTINCT customer ID) < (SELECT COUNT ([customer ID]) FROM library name. table name) OR [customer ID] IS NULL.

[0050] In the standard rule template, users can be allowed to focus only on the definition of data standards, and then specific quality inspection rules can be automatically generated based on these definitions, for example, replacing fields such as table name and customer ID in the rule template.

[0051] Constraint rule templates may include: templates for setting constraint conditions. Specifically, the data constraint rule template library may cover a series of common constraint conditions in database design, such as primary key (ensuring uniqueness and non-null), foreign key (ensuring reference integrity), unique constraint, etc. The template structure remains highly consistent and is easy to understand and expand.

[0052] For example, the template of the primary key constraint can be: SELECT [primary key field] FROM library name.table name GROUP BY [primary key field] HAVING COUNT (*)>1.

[0053] In this embodiment, by dynamically replacing the field names and table names in the constraint rule template, different data tables and fields can be adapted, which greatly simplifies the rule customization and application process.

[0054] In this embodiment, after the target object registers the metadata information of the data through the metadata management platform, the matching relationship between the metadata and the rule template library can be automatically identified. For example, if the target object specifies a field in the metadata that is associated with the "customer ID" data standard, it can immediately match all rule templates under the application of this standard to obtain a rule template set.

[0055] Step S203: Generate data quality rules based on the rule template set and metadata, wherein the data quality rules include: detection rules for performing quality detection on data generated by the target system of the financial institution.

[0056] For example, all rule templates in the rule template collection can be displayed through the target interface. The target object can easily establish associations between metadata in the rule templates displayed on the target interface by dragging, clicking, etc., including but not limited to data lineage, dependencies, etc., and then obtain data quality rules by dynamically replacing field names and table names in the rule templates.

[0057] In this embodiment, through the above steps, data quality rules are automatically generated based on metadata and quality rule template library, avoiding the situation of manually writing data quality rules in related technologies and low efficiency, thereby achieving the technical effect of improving the efficiency of generating data quality rules. Further, the technical problem of manually writing data quality rules in related technologies and low efficiency of generating data quality rules is solved.

[0058] In this embodiment, data quality rules are automatically generated using the metadata of structured data, and rule bases can be formed in the fields of data standards, data constraints, data dictionaries, etc. The user only needs to register the metadata information of the data, and specific data quality rules can be automatically generated based on the metadata and rule base template entered by the user. The user only needs to simply set the frequency of running the data quality rules to achieve automatic scanning of the rules, thereby automatically identifying common data quality problems, such as null values, duplicate records, logical errors (such as negative age), data inconsistencies, non-dictionary values, etc., and providing the specific location and description of the problem.

[0059] Optionally, in the method for generating data quality rules provided in an embodiment of the present application, all rule templates that match the metadata are screened from the quality rule template library to obtain a rule template set, including: matching the data fields in the metadata with the data fields of all rule templates in the quality rule template library to obtain a matching result; based on the matching result, determining all rule templates in the quality rule template library that match the metadata to obtain a rule template set.

[0060] In this embodiment, after the target object registers the metadata of the data field in the metadata management platform, the metadata can be read and parsed first. The metadata includes but is not limited to the type of the field (integer, string, date, etc.), the constraints of the field (such as whether it is the primary key, whether null values ​​are allowed, etc.), the data standards associated with the field (such as the specific format and rules that the customer ID should follow), and the dictionary value range of the field. By parsing these metadata, the characteristics and requirements of each data field can be understood. Then the data field in the metadata is matched with all the rule template fields in the quality rule template library. Specifically, it can be done by identifying the attributes of the metadata field and the conditions of the template field in the rule template library. For example, if the metadata shows that a certain field is a date type and requires to follow the format of "YYYY-MM-DD", then the rule template related to the standardization of the date format can be found. For the field designated as the primary key, the rule template related to the uniqueness and non-emptiness check can be found. Through this comparison of attributes and conditions, the purpose of being able to intelligently identify whether the rule template is applicable to the current metadata field is achieved.

[0061] For each data field of metadata that needs to be quality checked, one or more rule templates can be matched to obtain a rule template set.

[0062] For example, when users register metadata information of data through the metadata management platform, the matching relationship between this information and the rule template library can be automatically identified. Users do not need to manually write complex SQL queries or rule logic. They only need to associate data elements with data standards and select applicable data constraints through simple interface operations to automatically generate data quality inspection rules for specific forms and fields.

[0063] For example, if a user specifies a field to be associated with the "Customer ID" data standard, the system immediately applies all rule templates under the standard, generates inspection rules for the field, and adjusts them according to the scan scope (full or incremental data) selected by the user.

[0064] 1. Data standard rule template:

[0065] Sample template 1, date format standardization:

[0066] Template ID: DQS_STD_DATE_FORMAT;

[0067] Description: Ensure that all date fields follow a consistent format (e.g., YYYY-MM-DD).

[0068] Rule content: SELECT COUNT(*)FROM${Table_Name}WHERE NOT REGEXP_LIKE(${Date_Column},'^\d{4}-\d{2}-\d{2}$').

[0069] Sample template 2, numerical field precision control:

[0070] Template ID: DQS_STD_NUMERIC_PRECISION;

[0071] Description: Checks whether numeric fields comply with predefined precision and decimal places limits;

[0072] Rule content: SELECT COUNT(*)FROM${Table_Name}WHERE(${Numeric_Column}NOTBETWEEN${Min_Value}AND${Max_Value})OR(LENGTH(TRIM(TRAILING'0'FROM TO_CHAR(${Numeric_Column},'FM99999999999999999D99')))-INSTR(TO_CHAR(${Numeric_Column},

[0073] 'FM999999999999999999D99'),'.')-1)>${Decimal_Places}.

[0074] 2. Data constraint rule template:

[0075] Sample template 3, uniqueness constraint check:

[0076] Template ID: DQC_CONSTRAINT_UNIQUE_CHECK;

[0077] Description: Verify whether the value in the specified column is unique to prevent data duplication;

[0078] Rule content: SELECT COUNT(DISTINCT${Unique_Column})! =COUNT(${Unique_Column})FROM${Table_Name}.

[0079] Sample template 4, foreign key reference integrity check:

[0080] Template ID: DQC_CONSTRAINT_FOREIGN_KEY;

[0081] Description: Ensure that the foreign key values ​​in the table have corresponding primary key values ​​in the referenced table;

[0082] Rule content: SELECT COUNT(*)FROM${Table_Name}T1 3LEFT JOIN${Reference_Table}T2 ON T1.${FK_Column}=T2.${PK_Column}WHERE T2.${PK_Column}IS NULL.

[0083] By matching rule templates that match metadata from the data quality rule template library, data quality can be effectively monitored and managed to ensure that the data meets established standards and constraints, thereby supporting more reliable data quality analysis.

[0084] Optionally, in the method for generating data quality rules provided in an embodiment of the present application, data quality rules are generated based on a rule template set and metadata, including: using a target interface to visually display all rule templates in the rule template set, and using the target interface to receive the selection result of the target object selecting a rule template in the rule template set; based on the metadata, replacing the data fields in the rule template selected by the target object indicated by the selection result to obtain the data quality rules.

[0085] The above-mentioned target interface can be used to visually display the rule template set selected from the rule template library. By intuitively displaying the rule templates matching the metadata registered by the target object through the target interface, the target object can understand the purpose and inspection content of each template.

[0086] The target object can view the rule template collection through the target interface and select the applicable rule template according to the specific needs of the data. The target object may choose the rule template based on business understanding, data characteristics and quality goals to ensure that the rule template covers all necessary data quality dimensions.

[0087] In this embodiment, the target object selection result, that is, the list of rule templates that the target object wants to apply to a specific data object (such as a database table or field) can also be received through the target interface. Then, the data field placeholder in the rule template selected by the target object can be replaced with the specific field name specified by the metadata. At the same time, the conditions of the rule can be adjusted according to other information in the metadata (such as the specific value of the data standard or constraint) to ensure that the generated data quality rules can accurately detect the data quality. Finally, the generated data quality rules can be deployed to the data quality check engine and automatically executed according to the scanning frequency and scope (such as full or incremental data) set by the user.

[0088] Through the automated generation and deployment of data quality rules, the work of data governance personnel is simplified and the accuracy and efficiency of data quality checks are improved.

[0089] For example, when users register metadata on the metadata management platform, they do not need to consider specific data quality rules, but only need to consider how the metadata of the data is associated with standards and constraints. Data governance work has been customized by users and actively triggered, that is, automatically triggered according to the metadata registered by users. After users enter metadata, they can customize the forms and fields to be scanned for data quality through the target interface, and then automatically generate specific data quality rules based on the user's input metadata information and the set scanning scope through rule template merging, and deploy them, including:

[0090] 1. Metadata registration and association: First, a flexible metadata model is established to support users to register metadata information such as data sources, tables, fields, etc., and to allow users to specify the business meaning of data, data classification, sensitivity level, etc., and associate with enterprise data standards or business glossaries. In this embodiment, a target interface can also be provided, through which users can easily establish associations between metadata, including but not limited to data lineage, dependency, etc., by dragging, clicking, etc.

[0091] 2. Data quality rule template library: Use the pre-built quality rule template library to cover common data quality issues, such as null value checking, uniqueness verification, data format verification, range checking, etc. Each rule template should be configurable to adapt to different scenarios. After the user completes the metadata registration and selects the form and field to be scanned, the applicable rule template can be automatically selected according to the user's options, and the user can further refine the rule parameters (such as the proportion of fields that allow null values, specific requirements for data formats, etc.).

[0092] 3. Rule template merging and deployment: Automatically merge the rule templates and customized parameters selected by the user to generate a specific data quality inspection plan. For complex business needs, users are supported to customize the rule combination logic through a simple logic editor. The generated data quality rules can be automatically deployed to the data quality inspection engine without manual intervention, reducing the deployment cost and time of data quality rules.

[0093] Optionally, in the method for generating data quality rules provided in an embodiment of the present application, after screening all rule templates that match the metadata from the quality rule template library to obtain a rule template set, it includes: obtaining the incremental data generated by the target system, and performing data quality detection on the incremental data based on the data quality rules to obtain the detection results; generating a data quality report based on the detection results, and sending the data quality report to the target terminal, wherein the target terminal is used to display the data quality report.

[0094] In this embodiment, new or changed data in a big data environment can be identified through advanced watermark mechanisms or version control features (such as event time stamping or version tracking). These mechanisms can accurately locate new data generated since the last data quality scan, ensuring that the scan is only performed on the latest data portion, thereby improving efficiency and reducing resource consumption. In this embodiment, the execution status of related tasks in the data processing pipeline can also be checked. Through job dependency tracking technology, it is confirmed that all upstream tasks (such as ETL jobs) have been successfully completed and the generated data has been fully processed. After the data has been processed, the data quality detection of the incremental data can be started, ensuring that the data quality detection is based on complete and latest data, avoiding the impact of unprocessed or partially processed data on the detection results.

[0095] In this embodiment, data quality detection can be performed on incremental data based on data quality rules to detect whether the incremental data meets the preset quality standards and constraints. For example, it can be detected whether the incremental data date field follows the specified format, whether the numeric field is within the allowed precision range, and whether the primary key field is unique and non-empty. After the incremental data positioning data quality detection is completed, the detection results can be analyzed and summarized to identify problems in data quality, such as null values, duplicate records, logical errors, data inconsistencies, non-dictionary values, etc., and generate a data quality report. In this data quality report, an overview of the problem, a detailed problem list, an analysis of the impact of the problem, and recommended corrective measures can be included. The data quality report can provide data governance personnel with comprehensive data quality status feedback to help them quickly locate problems and take action.

[0096] In this embodiment, the generated data quality report can be sent to a pre-set target terminal, which can be a desktop or mobile device used by a data engineer, data analyst, or member of the data governance team. The sending method can be email, an internal messaging system, or directly pushed to a specific application interface. After the target terminal receives the data quality report, the report content can be displayed through a customized interface or reporting tool. The display method should be intuitive and easy to understand, and can highlight the severity and priority of the problem so that users can quickly understand the data quality status and take appropriate measures.

[0097] For example, use timestamps or version control features (event time, version control) in the data lake / warehouse to identify incremental data. Ensure that each scan targets only the newly arrived data. Track the execution status and dependencies of each job in the data processing pipeline through the metadata management system to ensure that the incremental data has been fully processed.

[0098] In this embodiment, the tool can also be integrated with the API of the big data processing platform to monitor the status of the upstream ETL job in real time. The execution of the data quality rule is started only when all dependent jobs are successfully completed and the incremental data is ready. The resource manager based on the big data platform implements flexible scheduling to ensure that the data quality inspection task runs during resource idle or off-peak hours to avoid competing for resources with key businesses.

[0099] After the data quality test is completed, the data quality problems can be automatically summarized, the severity and scope of the problem can be determined through data analysis, and the abnormal data can be marked to obtain a data quality report. The data quality report can then be sent to the relevant responsible persons via email, SMS or internal corporate communication tools, including problem details, impact assessment and recommended corrective measures, thereby improving the management efficiency of data quality management.

[0100] Optionally, in the method for generating data quality rules provided in an embodiment of the present application, data quality detection is performed on incremental data based on the data quality rules to obtain detection results, including: obtaining partition logic for the incremental data, wherein the partition logic includes: partition logic for logically partitioning the incremental data in the target system; based on the data quality rules and the partition logic, a parallel detection strategy is used to perform data quality detection on the incremental data to obtain detection results.

[0101] In this embodiment, the specific data quality rules required for this quality scan are dynamically generated according to the characteristics of the incremental data and the pre-defined rule templates, and then the data quality detection is performed according to the data quality rules. For example, if the incremental data contains new fields, the corresponding data format verification rules are automatically added. In order to speed up the scanning process, the data quality detection can be performed in parallel according to the partition logic of the incremental data (such as date partition) to obtain the detection results. It should be noted that each partition runs as an independent task, making full use of the distributed computing power to improve the detection efficiency of data quality detection.

[0102] Optionally, in the method for generating data quality rules provided in the embodiment of the present application, before performing data quality testing on the incremental data based on the data quality rules to obtain the test results, it also includes: obtaining the execution status and dependencies of all jobs in the target system; based on the execution status and dependencies of all jobs in the target system, determining whether the jobs on which the incremental data depends have been completed; when the jobs on which the incremental data depends have been completed, performing data quality testing on the incremental data based on the data quality rules to obtain the test results.

[0103] In this embodiment, the job execution status and dependency relationship can be obtained. For example, the execution status of all jobs in the target system can be monitored in real time. This can be achieved by integrating the API or monitoring tools provided by the target system, and it can ensure that the latest status of each job is obtained, including whether it has started, whether it is being executed, whether it has been completed, and whether there are any errors or exceptions. Since a data field or form may depend on the execution results of other jobs. Therefore, the dependencies between these jobs can be recorded and tracked through a metadata management system or other dependency tracking mechanisms. For example, it can be identified which job output data is required for the current data quality detection, as well as the execution order and status of these jobs.

[0104] After obtaining the execution status and dependencies of all jobs, you can check whether the jobs that the incremental data depends on have all been executed. This can be achieved by comparing the list of dependent jobs recorded in the metadata with the actual job completion status to ensure that all upstream data processing tasks have been successfully completed, thereby ensuring that the downstream incremental data quality detection is based on a complete and accurate data set.

[0105] After confirming that all jobs that the incremental data depends on have been executed, the execution of data quality detection tasks can be intelligently scheduled to perform data quality detection on the incremental data, ensuring that quality detection is performed during sufficient resources or non-business peak hours to avoid affecting business operations.

[0106] Before detecting incremental data, the integrity and consistency of the data are ensured by monitoring the job status and tracking dependencies, thereby improving the accuracy and reliability of data quality detection.

[0107] Optionally, in the method for generating data quality rules provided in an embodiment of the present application, after generating data quality rules based on a set of rule templates and metadata, it also includes: obtaining detection data obtained by performing quality detection on data generated by a target system in a historical time period; based on the detection data, adjusting a detection strategy for quality detection, wherein the detection strategy includes at least one of the following: parallelism and detection frequency of parallel detection.

[0108] In this embodiment, the scanning strategy is automatically adjusted according to the historical execution of data quality detection, such as adjusting the parallelism and optimizing resource allocation to ensure scanning efficiency. According to the data quality inspection results and actual business feedback, the data quality rules are continuously optimized to reduce false positives and false negatives and improve the effectiveness and accuracy of the rules.

[0109] For example, the test data collected from the historical data quality test can be automatically collected, including the test results of the historical time period, the types of problems encountered in the test process, the frequency of the problems, the severity of the problems, and the resource conditions consumed by the test. If the historical test data shows that the execution of some rule sets can be parallelized and the resources allow, the parallelism of the parallel detection can be improved, that is, more rules are executed in parallel at the same time point, thereby speeding up the detection speed. On the contrary, if the parallelism is too high, resulting in resource competition or reduced efficiency, the parallelism can be reduced. In the present embodiment, it can also be judged whether some quality rules need to increase the detection frequency to respond to the rapid changes in data quality, or reduce the detection frequency to save resources. For example, if the historical data shows that the error rate of a certain data field is low and relatively stable, the detection frequency of the field can be adjusted from daily detection to weekly detection.

[0110] When adjusting the detection strategy, you can adjust the detection strategy according to the resource usage to ensure the effective use of resources. For example, during non-business peak hours, you can increase the resource allocation for data quality detection to perform more frequent or more complex data quality detection tasks, and reduce the parallelism of detection tasks when resources are tight. By adjusting and optimizing the detection strategy, you can adjust the pertinence and efficiency of data quality detection according to the changing trend of data quality, thereby maximizing detection efficiency and making reasonable use of system resources while ensuring data quality.

[0111] In this embodiment, data quality detection rules can be automatically generated based on metadata, and data quality detection can be automatically performed, thereby improving the efficiency of data quality detection and the accuracy of detection results. Specifically, through the preset data standards and data constraint rule template library, users only need to register metadata to automatically generate data quality rules, which greatly reduces the labor cost and error rate of rule formulation and realizes standardized management. It allows users to easily associate metadata with data standards and data constraints, automatically trigger data quality checks, and make data governance more efficient and proactive. According to the characteristics of the big data environment, an incremental data scanning strategy is adopted, relying on job dependency status monitoring, intelligently scheduling rule execution timing, effectively utilizing computing resources, and ensuring stable performance. It can cover enterprise-defined quality standards, adapt to different business scenarios, provide comprehensive data quality assurance, and reduce data problems.

[0112] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0113] Embodiment 2

[0114] The embodiment of the present application also provides a device for generating data quality rules. It should be noted that the device for generating data quality rules in the embodiment of the present application can be used to execute the method for generating data quality rules provided in the embodiment of the present application. The device for generating data quality rules provided in the embodiment of the present application is introduced below.

[0115] According to an embodiment of the present application, a device for implementing the above-mentioned method for generating data quality rules is also provided, such as Figure 3 As shown, the device includes: an acquisition unit 31, a screening unit 32 and a generation unit 33.

[0116] Wherein, the acquisition unit 31 is used to acquire the metadata registered by the target object;

[0117] A screening unit 32 is used to screen all rule templates matching the metadata from the quality rule template library to obtain a rule template set, wherein the template types in the quality rule template library include: standard rule templates and constraint rule templates, the standard rule template includes: a template for setting data standards, and the constraint rule template includes: a template for setting constraint conditions;

[0118] The generating unit 33 is used to generate data quality rules based on the rule template set and metadata, wherein the data quality rules include: detection rules for performing quality detection on data generated by the target system of the financial institution.

[0119] The data quality rule generation device provided in the embodiment of the present application can obtain the metadata registered by the target object through the acquisition unit 31, and filter all the rule templates matching the metadata from the quality rule template library through the screening unit 32 to obtain a rule template set, wherein the template types in the quality rule template library include: standard rule templates, constraint rule templates, the standard rule template includes: templates for setting data standards, and the constraint rule template includes: templates for setting constraint conditions, and the data quality rules are generated based on the rule template set and the metadata through the generation unit 33, wherein the data quality rules include: detection rules for quality detection of data generated by the target system of the financial institution. Thus, the technical problem of low efficiency in generating data quality rules by manually writing in the related art is solved. In this embodiment, data quality rules are automatically generated based on metadata and the quality rule template library, avoiding the situation of low efficiency in manually writing data quality rules in the related art, thereby achieving the technical effect of improving the generation efficiency of data quality rules.

[0120] Optionally, in the data quality rule generation device provided in the embodiment of the present application, the screening unit includes: a matching subunit, used to match the data fields in the metadata with the data fields of all rule templates in the quality rule template library to obtain a matching result; a determination subunit, used to determine all rule templates in the quality rule template library that match the metadata based on the matching result to obtain a rule template set.

[0121] Optionally, in the data quality rule generation device provided in the embodiment of the present application, the generation unit includes: a first processing sub-unit, used to use the target interface to visually display all rule templates in the rule template set, and use the target interface to receive the selection result of the target object selecting the rule template in the rule template set; a second processing sub-unit, used to replace the data field in the rule template selected by the target object indicated by the selection result based on metadata to obtain the data quality rule.

[0122] Optionally, in the data quality rule generation device provided in the embodiment of the present application, the data quality rule generation device also includes: a first processing unit, used to obtain the incremental data generated by the target system after screening all rule templates matching the metadata from the quality rule template library to obtain a rule template set, and perform data quality detection on the incremental data based on the data quality rules to obtain a detection result; a second processing unit, used to generate a data quality report based on the detection result, and send the data quality report to the target terminal, wherein the target terminal is used to display the data quality report.

[0123] Optionally, in the data quality rule generation device provided in the embodiment of the present application, the first processing unit includes: an acquisition sub-unit, used to acquire the partition logic of the incremental data, wherein the partition logic includes: partition logic for logically partitioning the incremental data in the target system; a detection sub-unit, used to perform data quality detection on the incremental data based on the data quality rules and the partition logic, using a parallel detection strategy to obtain a detection result.

[0124] Optionally, in the data quality rule generation device provided in the embodiment of the present application, the data quality rule generation device also includes: a third processing unit, used to obtain the execution status and dependencies of all jobs in the target system before performing data quality detection on the incremental data based on the data quality rules and obtaining the detection results; a determination unit, used to determine whether the jobs on which the incremental data depends have been executed based on the execution status and dependencies of all jobs in the target system; and an execution unit, used to perform a step of performing data quality detection on the incremental data based on the data quality rules and obtaining the detection results when the jobs on which the incremental data depends have been executed.

[0125] Optionally, in the data quality rule generation device provided in the embodiment of the present application, the data quality rule generation device also includes: a fourth processing unit, which is used to obtain detection data obtained by quality detection of data generated by the target system in a historical time period after generating data quality rules based on a rule template set and metadata; an adjustment unit, which is used to adjust a detection strategy for quality detection based on the detection data, wherein the detection strategy includes at least one of the following: parallelism and detection frequency of parallel detection.

[0126] It should be noted that the acquisition unit 31, the screening unit 32 and the generation unit 33 correspond to steps S201 to S203 in the first embodiment, and the examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in the first embodiment. It should be noted that the modules or units may be hardware components or software components stored in a memory (e.g., the memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n), and the modules may also be part of the device and run in the computer terminal 10 provided in the first embodiment.

[0127] Embodiment 3

[0128] An embodiment of the present application may provide an electronic device, Figure 4 is a structural block diagram of an electronic device according to an embodiment of the present application. Figure 4 As shown, the electronic device may include: one or more ( Figure 4(only one is shown) processor 1002, memory 1004, storage controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0129] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0130] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain the metadata registered by the target object; filter all rule templates that match the metadata from the quality rule template library to obtain a rule template set, wherein the template types in the quality rule template library include: standard rule templates, constraint rule templates, standard rule templates include: templates for setting data standards, and constraint rule templates include: templates for setting constraint conditions; based on the rule template set and the metadata, generate data quality rules, wherein the data quality rules include: detection rules for performing quality detection on data generated by the target system of the financial institution.

[0131] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: filter all rule templates that match the metadata from the quality rule template library to obtain a rule template set, including: matching the data fields in the metadata with the data fields of all rule templates in the quality rule template library to obtain a matching result; based on the matching result, determine all rule templates in the quality rule template library that match the metadata to obtain a rule template set.

[0132] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: generate data quality rules based on the rule template set and metadata, including: using the target interface to visually display all rule templates in the rule template set, and using the target interface to receive the selection results of the target object selecting the rule template in the rule template set; based on the metadata, replace the data field in the rule template selected by the target object indicated by the selection result to obtain the data quality rules.

[0133] The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: after screening all rule templates that match the metadata from the quality rule template library to obtain a rule template set, including: obtaining the incremental data generated by the target system, and performing data quality detection on the incremental data based on the data quality rules to obtain the detection results; generating a data quality report based on the detection results, and sending the data quality report to the target terminal, wherein the target terminal is used to display the data quality report.

[0134] The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: perform data quality detection on the incremental data based on data quality rules to obtain detection results, including: obtaining the partition logic of the incremental data, wherein the partition logic includes: partition logic for logically partitioning the incremental data in the target system; based on the data quality rules and the partition logic, use a parallel detection strategy to perform data quality detection on the incremental data to obtain detection results.

[0135] The processor can call the information and application programs stored in the memory through the transmission device to execute the following steps: before performing data quality detection on the incremental data based on data quality rules and obtaining the detection results, it also includes: obtaining the execution status and dependencies of all jobs in the target system; based on the execution status and dependencies of all jobs in the target system, determining whether the jobs on which the incremental data depends have been executed; when the jobs on which the incremental data depends have been executed, performing data quality detection on the incremental data based on data quality rules and obtaining the detection results.

[0136] The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: after generating data quality rules based on the rule template set and metadata, it also includes: obtaining the detection data obtained by quality detection of the data generated by the target system in the historical time period; based on the detection data, adjusting the detection strategy of the quality detection, wherein the detection strategy includes at least one of the following: the parallelism of parallel detection and the detection frequency.

[0137] By adopting the embodiment of the present application, data quality rules are automatically generated based on metadata and a quality rule template library, thereby avoiding the situation in which data quality rules are manually written in the related art and the efficiency is low, thereby achieving the technical effect of improving the efficiency of generating data quality rules. This further solves the technical problem in the related art that data quality rules are generated manually and the efficiency of generating data quality rules is low.

[0138] It can be understood by those skilled in the art that Figure 4The structure shown is for illustration only, and the electronic device may also be a smart phone, a tablet computer, a PDA, a mobile Internet device (MID), a PAD or other terminal device. Figure 4 The structure of the electronic device is not limited. Figure 4 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 4 Different configurations are shown.

[0139] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0140] Embodiment 4

[0141] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the method for generating data quality rules provided in the first embodiment.

[0142] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0143] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing the steps of the method for generating data quality rules.

[0144] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0145] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0146] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0147] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0148] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0149] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0150] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for generating data quality rules, characterized in that: include: Get the metadata of the target object registration; Screening all rule templates matching the metadata from a quality rule template library to obtain a rule template set, wherein the template types in the quality rule template library include: standard rule templates and constraint rule templates, the standard rule template includes: a template for setting data standards, and the constraint rule template includes: a template for setting constraint conditions; Based on the rule template set and the metadata, data quality rules are generated, wherein the data quality rules include: detection rules for performing quality detection on data generated by a target system of a financial institution.

2. The generation method according to claim 1, characterized in that: All rule templates matching the metadata are screened from the quality rule template library to obtain a rule template set, including: Matching the data fields in the metadata with the data fields of all rule templates in the quality rule template library to obtain a matching result; Based on the matching result, all rule templates in the quality rule template library that match the metadata are determined to obtain a rule template set.

3. The generation method according to claim 1, characterized in that: Generating data quality rules based on the rule template set and the metadata includes: Using the target interface to visually display all the rule templates in the rule template set, and using the target interface to receive the selection result of the target object selecting the rule template in the rule template set; Based on the metadata, a data field in the rule template selected by the target object indicated by the selection result is replaced to obtain the data quality rule.

4. The generation method according to claim 1, characterized in that: After all rule templates matching the metadata are screened from the quality rule template library to obtain a rule template set, the method includes: Acquire the incremental data generated by the target system, and perform data quality detection on the incremental data based on the data quality rule to obtain a detection result; A data quality report is generated based on the detection result, and the data quality report is sent to a target terminal, wherein the target terminal is used to display the data quality report.

5. The generation method according to claim 4, characterized in that: Performing data quality detection on the incremental data based on the data quality rule to obtain a detection result includes: Acquire the partition logic of the incremental data, wherein the partition logic includes: partition logic for logically partitioning the incremental data in the target system; Based on the data quality rules and the partition logic, a parallel detection strategy is adopted to perform data quality detection on the incremental data to obtain the detection result.

6. The generation method according to claim 4, characterized in that: Before performing data quality detection on the incremental data based on the data quality rule to obtain the detection result, the method further includes: Obtaining the execution status and dependencies of all jobs in the target system; Based on the execution status and dependency relationship of all jobs in the target system, determining whether the job on which the incremental data depends has been completed; When the job on which the incremental data depends is completed, a step of performing a data quality check on the incremental data based on the data quality rule to obtain a check result is performed.

7. The generation method according to claim 1, characterized in that: After generating the data quality rules based on the rule template set and the metadata, the method further includes: Acquire test data obtained by performing quality testing on data generated by the target system during a historical period; Based on the detection data, a detection strategy for quality detection is adjusted, wherein the detection strategy includes at least one of the following: parallelism of parallel detection and detection frequency.

8. A device for generating data quality rules, characterized in that: include: An acquisition unit, used for acquiring metadata registered by a target object; A screening unit, used for screening all rule templates matching the metadata from a quality rule template library to obtain a rule template set, wherein the template types in the quality rule template library include: standard rule templates and constraint rule templates, the standard rule templates include: templates for setting data standards, and the constraint rule templates include: templates for setting constraint conditions; A generating unit is used to generate data quality rules based on the rule template set and the metadata, wherein the data quality rules include: detection rules for performing quality detection on data generated by a target system of a financial institution.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is run, the device where the computer-readable storage medium is located is controlled to execute the method for generating data quality rules according to any one of claims 1 to 7.

10. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method for generating data quality rules according to any one of claims 1 to 7 are implemented.

Images