Data quality management method and apparatus

Abstract

The application discloses a data quality management method and device, and belongs to the technical field of data processing. The data quality management method comprises the following steps: receiving a first input of a user, wherein the first input is used for inputting a target configuration; in response to the first input, a data model is constructed based on the target configuration; a data source corresponding to a target layer in a data warehouse is associated to a corresponding position of the data model and field mapping is performed; at least part of data in the data source corresponding to the target layer is checked based on a target data checking rule, and a quality checking report is obtained. The data quality management method can unify the data quality standard, ensure the consistency of data processing in the subsequent application process, unify the data storage structure, directly associate the data when the data of multiple systems is called, and realize the data sharing between different systems.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and in particular relates to a data quality management method and apparatus. Background Technology

[0002] With the development of big data applications, enterprises possess increasingly large and complex data assets, requiring effective data quality management to realize the business value of the data. Commonly used data quality management methods fail to unify data quality standards, resulting in fragmented data standards, long implementation cycles and high governance costs. Furthermore, inconsistent data storage structures lead to inability to directly link data when accessing data from multiple systems, hindering data sharing between different systems. Summary of the Invention

[0003] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes a data quality management method and apparatus that can unify data quality standards, ensure consistency in data processing during subsequent applications, unify data storage structures, and directly associate data when accessing data from multiple systems, thereby achieving data sharing between different systems.

[0004] In a first aspect, this application provides a data quality management method, the method comprising:

[0005] Receive the user's first input, which is used to input the target configuration;

[0006] In response to the first input, a data model is constructed based on the target configuration;

[0007] Associate the data source corresponding to the target layer in the data warehouse with the corresponding position in the data model and perform field mapping;

[0008] Based on the target data auditing rules, at least a portion of the data in the data source corresponding to the target layer is audited to obtain a quality inspection report.

[0009] According to the data quality management method provided in this application embodiment, by setting the target configuration of data through user input, the data quality standard can be unified, ensuring the consistency of data processing in subsequent applications. This guarantees the standardized production of data from the source and saves the cost of subsequent data application and processing. In addition, by constructing a data model based on the target configuration and data structure and associating the data model with the data source, the data storage structure is unified. When calling data from multiple systems, the data can be directly associated, realizing data sharing between different systems. Furthermore, when using data, the corresponding type of data source can be quickly retrieved, saving the cost of subsequent data application and processing and facilitating effective data management by users.

[0010] One embodiment of the data quality management method of this application includes a first input comprising a first sub-input and a second sub-input; the first sub-input is used to input a standard code, and the second sub-input is used to input a data meta-standard.

[0011] One embodiment of the data quality management method of this application includes associating the data source corresponding to the target layer in the data warehouse to the corresponding position in the data model and performing field mapping, comprising:

[0012] The data source corresponding to the target layer is determined from the data sources corresponding to multiple layers in the data warehouse, and the association relationship between the data model and the data source corresponding to the target layer is established.

[0013] Based on the aforementioned relationship, the data source associated with the data model is obtained; the data source corresponds to a physical table field;

[0014] Based on the logical table fields and physical table fields corresponding to the data model, a field mapping relationship is established between the data model and the data source.

[0015] One embodiment of the data quality management method of this application, before auditing at least a portion of the data in the data source corresponding to the target layer based on the target data auditing rules and obtaining a quality inspection report, the method further includes:

[0016] Receive a second input from the user; the second input is used to input at least one of the following: the audit rule name, audit type, audit component, and audit field of the data source corresponding to the target layer;

[0017] In response to the second input, target data audit rules corresponding to the data source are generated.

[0018] One embodiment of the data quality management method of this application includes a data warehouse comprising multiple data sources, each data source corresponding to a different auditing rule, wherein the multiple data sources and the multiple different auditing rules are in one-to-one correspondence, and the step of auditing at least a portion of the data in the data source corresponding to the target layer based on the target data auditing rule to obtain a quality inspection report includes:

[0019] Based on the multiple different audit rules, at least a portion of the data from the multiple data sources are audited to obtain multiple different quality inspection reports, and the multiple different quality inspection reports correspond one-to-one with the multiple different audit rules.

[0020] One embodiment of the data quality management method of this application, before constructing a data model based on the target configuration in response to the first input, the method further includes:

[0021] The target configuration is reviewed, and if the target configuration is approved, the release status of the target configuration is changed to "released"; the released target configuration is used to build the data model.

[0022] If the target configuration fails the review, the release status of the target configuration will be changed to unreleased, and the target configuration will be updated.

[0023] Secondly, this application provides a data quality management device, which includes:

[0024] The first processing module is used to receive the user's first input, which is used to input the target configuration.

[0025] The second processing module is used to construct a data model based on the target configuration in response to the first input;

[0026] The third processing module is used to associate the data source corresponding to the target layer in the data warehouse with the corresponding position in the data model and perform field mapping;

[0027] The fourth processing module is used to audit at least a portion of the data in the data source corresponding to the target layer based on the target data audit rules, and obtain a quality inspection report.

[0028] According to the data quality management device provided in this application embodiment, by setting the target configuration of data through user input, the data quality standard can be unified, ensuring the consistency of data processing in subsequent applications. This guarantees the standardized production of data from the source and saves the cost of subsequent data application and processing. In addition, by constructing a data model based on the target configuration and data structure and associating the data model with the data source, the data storage structure is unified. When calling data from multiple systems, the data can be directly associated, realizing data sharing between different systems. Furthermore, when using data, the corresponding type of data source can be quickly retrieved, saving the cost of subsequent data application and processing and facilitating effective data management by users.

[0029] Thirdly, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the data quality management method as described in the first aspect above.

[0030] Fourthly, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data quality management method as described in the first aspect above.

[0031] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the data quality management method as described in the first aspect above.

[0032] The above-described one or more technical solutions in the embodiments of this application have at least one of the following technical effects:

[0033] By setting target configurations for data through user input, data quality standards can be unified, ensuring consistency in data processing during subsequent applications. This guarantees standardized data production from the source, saving costs associated with subsequent data application and processing. Furthermore, by constructing data models based on target configurations and data structures, and associating these models with data sources, a unified data storage structure is achieved. When accessing data from multiple systems, data can be directly linked, enabling data sharing between different systems. Moreover, when using data, the corresponding data source type can be quickly retrieved, saving costs associated with subsequent data application and processing, and facilitating effective data management for users.

[0034] Furthermore, by establishing a connection between the data model and the data source in the target layer of the data warehouse, and by establishing a field mapping relationship between the data model and the data source, the data storage structure is unified. When calling data from multiple systems, data can be directly associated, realizing data sharing between different systems. Moreover, when using data, the corresponding data source can be quickly retrieved, saving the cost of subsequent data application and processing, and facilitating users to effectively manage data.

[0035] Furthermore, by having users input standard codes and data element standards to generate target configurations, data quality inspection rules are modularized. Users can complete the target configuration simply by dragging and dropping. In subsequent applications, data can be generated based on the target configuration, unifying data standards, data definitions, and different people's understanding of data. This ensures the consistency of data processing during the application process, guarantees standardized data production from the source, saves costs for subsequent data application and processing, and facilitates effective data management for users.

[0036] Furthermore, by inputting at least one of the audit rule name, audit type, audit component, and audit field of the data source corresponding to the target layer, target data audit rules can be generated. Different data audit rules can be generated based on different types of data sources, so that different types of data sources can be audited according to different data audit rules. Data can be audited according to data characteristics, ensuring the accuracy of audit results.

[0037] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0038] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0039] Figure 1 This is one of the flowcharts illustrating the data quality management method provided in the embodiments of this application;

[0040] Figure 2 This is one of the schematic diagrams illustrating the principle of the data quality management method provided in the embodiments of this application;

[0041] Figure 3 This is a second schematic diagram illustrating the principle of the data quality management method provided in the embodiments of this application;

[0042] Figure 4 This is the third schematic diagram illustrating the principle of the data quality management method provided in this application embodiment;

[0043] Figure 5 This is the fourth schematic diagram illustrating the principle of the data quality management method provided in the embodiments of this application;

[0044] Figure 6 This is the fifth schematic diagram illustrating the principle of the data quality management method provided in the embodiments of this application;

[0045] Figure 7 This is the sixth schematic diagram illustrating the principle of the data quality management method provided in this application embodiment;

[0046] Figure 8 This is the seventh schematic diagram illustrating the principle of the data quality management method provided in the embodiments of this application;

[0047] Figure 9 This is the eighth schematic diagram illustrating the principle of the data quality management method provided in this application embodiment;

[0048] Figure 10 This is the ninth schematic diagram illustrating the principle of the data quality management method provided in the embodiments of this application;

[0049] Figure 11 This is the tenth schematic diagram illustrating the principle of the data quality management method provided in the embodiments of this application;

[0050] Figure 12 This is a second schematic flowchart of the data quality management method provided in the embodiments of this application;

[0051] Figure 13 This is eleventh of the schematic diagrams illustrating the principle of the data quality management method provided in the embodiments of this application;

[0052] Figure 14 This is the twelfth schematic diagram illustrating the principle of the data quality management method provided in the embodiments of this application;

[0053] Figure 15 This is the thirteenth schematic diagram illustrating the principle of the data quality management method provided in the embodiments of this application;

[0054] Figure 16 This is the third flowchart illustrating the data quality management method provided in this application embodiment;

[0055] Figure 17 This is the fourteenth schematic diagram illustrating the principle of the data quality management method provided in the embodiments of this application;

[0056] Figure 18 This is the fifteenth schematic diagram illustrating the principle of the data quality management method provided in the embodiments of this application;

[0057] Figure 19 This is the sixteenth schematic diagram illustrating the principle of the data quality management method provided in the embodiments of this application;

[0058] Figure 20 This is the seventeenth schematic diagram illustrating the principle of the data quality management method provided in this application embodiment;

[0059] Figure 21 This is the eighteenth schematic diagram illustrating the principle of the data quality management method provided in this application embodiment;

[0060] Figure 22 This is the nineteenth schematic diagram illustrating the principle of the data quality management method provided in the embodiments of this application;

[0061] Figure 23 This is a schematic diagram of the data quality management device provided in the embodiments of this application;

[0062] Figure 24 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0063] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0064] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0065] The following is combined with Figures 1 to 22 This application describes a data quality management method according to an embodiment.

[0066] It should be noted that the entity executing the data quality management method can be a server, a data quality management device, or a user's terminal, including but not limited to mobile terminals and non-mobile terminals.

[0067] For example, mobile terminals include, but are not limited to, mobile phones, PDA smart terminals, tablets, and in-vehicle smart terminals; non-mobile terminals include, but are not limited to, PCs.

[0068] like Figure 1 As shown, the data quality management method includes steps 110, 120, 130 and 140.

[0069] This data quality management method can be applied to scenarios involving the management of government data or enterprise operational data, or it can be applied to other data management scenarios, which are not limited in this application.

[0070] Step 110: Receive the user's first input, which is used to input the target configuration.

[0071] In this step, the first input is used to input the target configuration.

[0072] The first input can be in at least one of the following ways:

[0073] Firstly, the first input can be a touch operation, including but not limited to click, swipe, and press operations.

[0074] In this embodiment, receiving the user's first input can be receiving the user's touch operation on the display area of ​​the terminal screen.

[0075] To reduce the rate of user error, the effective area of ​​the first input can be limited to a specific area, such as the upper middle area of ​​the data model building interface; or, while the data model building interface is displayed, a target control can be displayed on the current interface, and touching the target control will enable the first input; or the first input can be set to a series of taps on the display area within a target time interval.

[0076] Secondly, the first input can be a physical button input.

[0077] In this embodiment, the terminal is equipped with physical buttons on its body that correspond to the data model being constructed. The first input from the user can be the user pressing the corresponding physical button; the first input can also be a combination of pressing multiple physical buttons simultaneously.

[0078] Thirdly, the first input can be voice input.

[0079] In this implementation, the terminal can trigger the display of the input target configuration interface when it receives a voice message such as "build a data model".

[0080] Of course, in other embodiments, the first input may also be in other forms, including but not limited to character input, etc., which can be determined according to actual needs, and this application embodiment does not limit it.

[0081] The target configuration is for the relevant configurations used to standardize data standards.

[0082] Based on the target configuration, data with unified data standards and consistent data definitions can be obtained.

[0083] The data can be government big data or enterprise operation data, or other types of data; this application does not limit the scope of the data.

[0084] Users can input target configurations based on standard data requirements.

[0085] The target configuration may include one or both of the standard code and data element standards, which can be used to define the target configuration.

[0086] In some embodiments, the first input may include: a first sub-input and a second sub-input.

[0087] In this embodiment, the first sub-input and the second sub-input can be the same touch operation as the first input, physical button input, voice input, or any other feasible input method, which will not be described in detail here.

[0088] The first sub-input is used to input standard codes.

[0089] The standard code is the range of values ​​for a data standard. You can set the content and range of data that can be selected in the standard code. For example, you can set the encoding value in the standard code. Based on the encoding value, you can obtain data that is within the range of that encoding value.

[0090] The second sub-input is used to input the data element standard.

[0091] Data element standards can be created based on information such as standard classification, standard name, contextual description, business definition, standard terminology, standard documents, and custom attributes.

[0092] The standard classification can include national standards, government standards, and industry standards, and can be customized based on user needs.

[0093] The standard name can include a standard Chinese name and an English name. The Chinese name is used to represent a unique name for the business definition, and the English name is used as a reference for the standard mapping.

[0094] Contextual descriptions are used to specify or describe the context or application procedures in which the name is used and generated.

[0095] Business definitions are used to describe the attributes of data elements, and the attributes of various data elements can be distinguished based on business definitions.

[0096] Standard terminology refers to terminology specific to data standards.

[0097] Standard documents are documents that serve as the source of data standards.

[0098] Users can customize business attribute names and corresponding attribute values ​​according to their needs.

[0099] In practice, users can input relevant information about data standard classification to create standard classifications. For example, they can input the parent standard classification and the standard classification name to create one.

[0100] Standard classification is used to classify and manage standards of different categories or levels.

[0101] like Figure 2 As shown, standard classifications can include industry standards, national standards, and government standards, etc.

[0102] Industry standards can include computer industry standards, real estate industry standards, and tourism industry standards, among others.

[0103] When creating a standard category, if the standard category has a "parent standard category", it can be used as a subcategory; if the standard category does not have a "parent standard category", it can be used as a root node.

[0104] For example, when a computer industry standard has a corresponding superior standard classification, the computer industry standard is treated as a subclass of the industry standard; when a national standard does not have a corresponding superior standard classification, the national standard is treated as the root node.

[0105] like Figure 3 As shown, after creating a standard category, you can enter information such as the standard category, document number, document name, and document description, and upload the file to create a standard document.

[0106] It can generate multiple standard document data sets and categorize these sets of standard document data sets based on standard classifications. Figure 4 Examples are provided for some of the standard documents included in the national standard category.

[0107] like Figure 5 As shown, you can select the standard category and the standard document to be associated, and enter the Chinese name, English name and term description to create standard terms.

[0108] It can generate multiple sets of standard terminology data and categorize them based on standard classification. Figure 6 Examples are provided for some of the standard terms included in the national standard categories.

[0109] like Figure 7 As shown, you can select the standard category, the standard terminology to be referenced, and the associated standard document, and enter the standard code name, standard code description, standard code representation, encoding method, encoding value, encoding name, and encoding description to create the standard code.

[0110] After creating the standard code, you can configure the business attributes and technical attributes of the data element standard to create the data element standard.

[0111] Specifically, such as Figure 9 As shown, you can select the standard category, the referenced standard terminology, and the associated standard document, and enter the Chinese name, English name, contextual description, business definition, and custom attributes of the data element standard to complete the business attribute configuration of the data element standard;

[0112] Then select the data type, whether null values ​​are allowed, whether duplicates are allowed, and the standard code to be referenced, and enter the data length to complete the technical attribute configuration of the data element standard, such as... Figure 10 As shown;

[0113] Data element standards are created based on business attribute configuration and technical attribute configuration.

[0114] According to the data quality management method provided in the embodiments of this application, a target configuration is generated by the user inputting standard codes and data element standards. The data quality inspection rules are componentized, and the user can complete the target configuration simply by dragging and dropping. In subsequent applications, data can be generated based on the target configuration, which unifies data standards, data definitions, and different people's understanding of data, ensuring the consistency of data processing in the application process. It guarantees the standardized production of data from the source, saves the cost of subsequent data application and processing, and facilitates users to effectively manage data.

[0115] like Figure 12 As shown, in some embodiments, prior to step 120, the data quality management method may further include:

[0116] The target configuration is reviewed and approved. If the target configuration is approved, the release status of the target configuration is changed to "released". The released target configuration is used to build the data model.

[0117] If the target configuration fails the review, change the release status of the target configuration to unreleased and update the target configuration.

[0118] In this embodiment, auditing the target configuration can include determining whether the target configuration meets the user's application and management needs.

[0119] The release status of a target configuration can include both released and unreleased. The release status of a target configuration can be determined based on whether the target configuration has passed review.

[0120] If the target configuration is approved, the release status of the target configuration will be changed to "released".

[0121] The published target configuration is used to build the data model.

[0122] If the target configuration fails the review, change the release status of the target configuration to unreleased, update the target configuration, and resubmit the updated target configuration to the review system.

[0123] Reviewing target configurations can include at least one of manual review and automated system review.

[0124] The following is a detailed explanation of how to review target configurations manually.

[0125] In actual implementation, such as Figure 13 As shown, when submitting the target configuration to the review system, you can select the corresponding reviewer and enter the review reason;

[0126] Display the target configurations pending review in the "My Review" list, such as... Figure 14 As shown, the status of the target configuration to be reviewed is "Under Review".

[0127] When the approval opinion is "approved", the approval status of the target configuration is changed to "approved" and the release status of the target configuration is changed to "released".

[0128] When the approval opinion is "rejected", the approval status of the target configuration is changed to "rejected", and the release status of the target configuration is changed to "not released".

[0129] Specifically, such as Figure 8 As shown, after creating the standard code in the target configuration, submit the standard code to the review system. After submission, a standard code data of version V1 with a status of "unreleased" will be generated.

[0130] The standard code will take effect once it has passed review.

[0131] like Figure 11 As shown, after creating the data meta standard in the target configuration, submit the data meta standard to the review system. After submission, a V1 version of data meta standard data with the status of "unpublished" will be generated.

[0132] The data element standard will take effect once it has been approved.

[0133] If the release status of the target configuration changes to "unreleased", update the target configuration and resubmit the updated target configuration to the review system.

[0134] The following is a detailed explanation of how the system automatically reviews target configurations.

[0135] In some embodiments, programs or code related to review conditions can be set in the system, and the release status of the target configuration can be automatically changed to "released" when the target configuration meets the review conditions.

[0136] According to the data quality management method provided in the embodiments of this application, by reviewing the target configuration and updating the target configuration if the review fails, the subsequent data model can be built based on the approved target configuration. This can obtain data standards that meet the user's application and management needs, ensure the consistency of subsequent data processing, and facilitate the user's effective management of data.

[0137] Step 120: In response to the first input, construct a data model based on the target configuration.

[0138] In this step, a data model can be built based on the target configuration, or a data model can be created based on user-defined settings; this application does not impose any limitations on this.

[0139] In applications, data models can be built based on the structure of the data.

[0140] The data model can interface with the original table to collect data from the original table.

[0141] The data model is used to describe the static characteristics and constraints of business attributes. The static characteristics of business attributes can be table fields in the data table, which are used to distinguish data from a business perspective. Constraints are used to standardize data. For example, if the constraint is that the attribute length is less than 100, data that exceeds this range is considered non-compliant data.

[0142] In actual implementation, such as Figure 15 As shown, you can enter the Chinese name, English name, description information, and model fields to create a data model.

[0143] The model fields can be generated based on the target configuration or based on user input.

[0144] Model fields can include elements such as the English name, Chinese name, data type, length, whether it is nullable, reference standard, and reference index.

[0145] When model fields are generated based on target configuration, a list of data element standards with a publication status of "published" can be obtained. Then, the required data element standard can be selected from multiple "published" data element standards for import. Based on the imported data element standard, the name, data type, length, and whether it is non-null attributes in the data element standard are loaded into the model fields of the data model.

[0146] Step 130: Associate the data source corresponding to the target layer in the data warehouse with the corresponding location in the data model and perform field mapping.

[0147] In this step, the data sources in the data warehouse can be categorized, that is, the data warehouse can be divided into multiple layers.

[0148] A data warehouse includes multiple data sources.

[0149] The data warehouse can include the Operation Data Store (ODS), Data Warehouse Details (DWD), Data Warehouse Service (DWS), Data Warehouse Topic (DWT), Data Mart (DM), and Application Data Service (ADS).

[0150] The data models associated with data sources at different layers may be located in different places. You can associate the data source corresponding to the target layer in the data warehouse with the corresponding location of the data model.

[0151] Based on the fields in the data source corresponding to the target layer and the fields in the data model, the field mapping between the data source and the data model corresponding to the target layer can be completed.

[0152] In some embodiments, step 130 may include:

[0153] Identify the data source corresponding to the target layer from the data sources corresponding to multiple layers in the data warehouse, and establish the relationship between the data model and the data source corresponding to the target layer;

[0154] The data source associated with the data model is obtained based on the relationship; the data source corresponds to physical table fields;

[0155] Based on the logical table fields and physical table fields corresponding to the data model, establish the field mapping relationship between the data model and the data source.

[0156] In this embodiment, the data model corresponds to a logical table, and the fields of the logical table corresponding to the data model may include the logical English name, the logical Chinese name, and the data type, etc.

[0157] The data source includes physical tables, which are the actual tables. The fields of the physical table corresponding to the data source can include the physical English name, the physical Chinese name, and the data type, etc.

[0158] Based on logical table fields and physical table fields, the field mapping relationship between the data model and the data source can be established manually or automatically. The choice can be made based on user needs, and this application does not impose any restrictions.

[0159] In actual implementation, such as Figure 16 As shown, when there is no connection between the data model and the data source, a physical table can be generated to create the connection;

[0160] If a relationship has already been established between the data model and the data source, a data source can be selected from the data warehouse to create the relationship.

[0161] After association, such as Figure 17 As shown, you can manually map data based on the logical table fields corresponding to the data model by dragging the arrow to the physical table fields.

[0162] Alternatively, automatic field mapping can be triggered, where the system iterates through the logical table fields and matches them with the physical table fields based on the Chinese and English names of the attributes to complete the field mapping relationship between the data model and the data source corresponding to the target layer.

[0163] According to the data quality management method provided in the embodiments of this application, by establishing an association between the data model and the data source of the target layer in the data warehouse, and establishing a field mapping relationship between the data model and the data source, the data storage structure is unified. When calling data from multiple systems, the data can be directly associated, realizing data sharing between different systems. Moreover, when using data, the corresponding data source can be quickly retrieved, saving the cost of subsequent data application and processing, and facilitating users to effectively manage data.

[0164] Step 140: Based on the target data audit rules, audit at least a portion of the data in the data source corresponding to the target layer and obtain a quality inspection report.

[0165] In this step, the target data auditing rules are used to audit the data.

[0166] Different data models correspond to different data auditing rules.

[0167] Based on the data model, the target data audit rule can be obtained from multiple data audit rules.

[0168] Quality inspection reports may include at least some non-compliant data.

[0169] Different data auditing rules correspond to different quality inspection reports.

[0170] In actual implementation, based on the target data auditing rules, at least a portion of the data can be audited immediately; or a task scheduler can be configured to audit at least a portion of the data, such as... Figure 21 As shown, the task scheduler is configured by setting the scheduling cycle status and scheduling cycle.

[0171] It should be noted that if the dataset has an empty number of metadata records, meaning there is no data in the data source, then the data source does not need to be audited.

[0172] After the audit operation is triggered, the audit status corresponding to this data source changes to "in progress";

[0173] Once at least a portion of the data has been successfully audited, the audit status for that data source will change to "execution successful".

[0174] If an error occurs during the audit process, the audit status corresponding to that data source will change to "execution failed".

[0175] In some embodiments, prior to step 140, the data quality management method may further include:

[0176] Receive the user's second input; the second input is used to input at least one of the following: the audit rule name, audit type, audit component, and audit field of the data source corresponding to the target layer;

[0177] In response to the second input, the target data audit rules corresponding to the data source are generated.

[0178] In this embodiment, the second input is used to input at least one of the following: the audit rule name, audit type, audit component, and audit field of the data source corresponding to the target layer.

[0179] The second input can be the same as the first input, such as touch operation, physical button input, voice input, or any other feasible input method, which will not be elaborated here.

[0180] In response to the second input, target data audit rules corresponding to the data source can be generated.

[0181] In actual implementation, such as Figure 18 As shown, you can enter descriptive information and select a data model to create a data quality audit record;

[0182] like Figure 19 As shown, input at least one of the following: audit rule name, audit type, audit component, and audit field of the data source associated with the data model, and generate the target data audit rule.

[0183] The types of audits can include: analyzers, filters, writers, and extenders.

[0184] The analyzer is used to select one or more fields and analyze different types of fields. For example, the analyzer can analyze the number of uppercase and lowercase letters in a string, the maximum value of a number, the minimum value of a number, the average value of a number, the maximum date and the minimum date, etc.

[0185] Filters are used to filter data. They can be used to compare data by selecting fields, operators (=, !=, <, <=, >, >=, like, in), and filter values ​​to achieve filtering.

[0186] The writer is used to view and download data audit rules. The writer can write data to a CSV (Comma-Separated Values) file.

[0187] The extender is used to analyze table normalization, field normalization, database table storage, field count, record count, mobile phone verification, ID card verification, and custom SQL (Structured Query Language) processing.

[0188] Different types of audits correspond to different audit components.

[0189] like Figure 20 As shown, the analyzer can include a custom character analyzer, a string analyzer, a date analyzer, a number analyzer, an integrity analyzer, and a Boolean analyzer.

[0190] Filters can include comparison filters and maximum row count (MaxRows) filters.

[0191] The writer can include a CSV writer.

[0192] Extenders can include character normalizers, storage capacity generators, numeric calculators, table normalizers, mobile phone number validators, field counters, record counters, custom SQL processors, and identity number validators.

[0193] According to the data quality management method provided in the embodiments of this application, by inputting at least one of the audit rule name, audit type, audit component and audit field of the data source corresponding to the target layer, a target data audit rule is generated. Different data audit rules can be generated based on different types of data sources, so as to audit different types of data sources according to different data audit rules, and to audit data according to data characteristics, thereby ensuring the accuracy of audit results.

[0194] During the research and development process, the inventors discovered that commonly used data quality management methods cannot unify data quality standards, resulting in chaotic data standards, long implementation and governance cycles, high governance costs, and inconsistent data storage structures. This leads to data that cannot be directly linked when accessing data from multiple systems, making it difficult to share data between different systems.

[0195] In this application, by setting the target configuration of data through user input, the data quality standard can be unified, ensuring the consistency of data processing in subsequent applications, guaranteeing the standardized production of data from the source, and saving the cost of subsequent data application and processing.

[0196] In addition, a data model is built based on the target configuration and data structure, and the data model is associated with the data source, which unifies the data storage structure. When calling data from multiple systems, the data can be directly associated, realizing data sharing between different systems. Moreover, when using data, the corresponding data source can be quickly retrieved, saving the cost of subsequent data application and processing, and making it easier for users to effectively manage data.

[0197] According to the data quality management method provided in this application embodiment, by setting the target configuration of data through user input, the data quality standard can be unified, ensuring the consistency of data processing in subsequent applications. This guarantees the standardized production of data from the source and saves the cost of subsequent data application and processing. In addition, by constructing a data model based on the target configuration and data structure and associating the data model with the data source, the data storage structure is unified. When calling data from multiple systems, the data can be directly associated, realizing data sharing between different systems. Furthermore, when using data, the corresponding type of data source can be quickly retrieved, saving the cost of subsequent data application and processing and facilitating effective data management by users.

[0198] In some embodiments, the data warehouse includes multiple data sources, each corresponding to a different audit rule, with a one-to-one correspondence between the multiple data sources and the multiple different audit rules. Step 140 may include:

[0199] Auditing is conducted on at least a portion of data from multiple data sources based on multiple different auditing rules, resulting in multiple different quality inspection reports. Each of these quality inspection reports corresponds one-to-one with a different auditing rule.

[0200] In this embodiment, auditing at least some data from multiple data sources based on different auditing rules can yield different quality inspection reports, and each of the multiple different quality inspection reports corresponds one-to-one with a different auditing rule.

[0201] A data warehouse includes multiple data sources, each with its own set of audit rules. The data sources and the audit rules are in a one-to-one correspondence.

[0202] In application, target data auditing rules can be selected based on the characteristics of the data source corresponding to the target layer.

[0203] In some embodiments, after obtaining multiple different quality inspection reports, the data quality management method may further include:

[0204] Receives third input from the user for a target quality inspection report among multiple different quality inspection reports;

[0205] In response to the third input, the target quality inspection report is displayed.

[0206] In this embodiment, the third input can be the same as the first input, such as touch operation, physical button input, voice input, or any other feasible input method, which will not be described in detail here.

[0207] The target quality inspection report can be any of several different quality inspection reports.

[0208] In response to a third input, a target quality inspection report can be displayed.

[0209] The application can receive third-party input from users for any of the multiple different quality inspection reports to view the corresponding target quality inspection report. In other words, users can switch between different audit rules to view the corresponding quality inspection report.

[0210] In actual implementation, such as Figure 22 As shown, the system can generate corresponding quality inspection reports based on different inspection rules, and can also export the quality inspection reports as HTML format documents.

[0211] The quality inspection report includes non-compliant data. Based on the quality inspection report, non-compliant data can be updated and corrected in a timely manner, ensuring the standardization of the data.

[0212] According to the data quality management method provided in the embodiments of this application, at least a portion of the data from multiple data sources are audited based on multiple different audit rules to obtain multiple different quality inspection reports. The method also receives a third input from the user on a target quality inspection report among the multiple different quality inspection reports to display the target quality inspection report. Different audit rules can be switched to view the corresponding quality inspection report, which facilitates the user to update and correct non-compliant data in a timely manner based on the quality inspection report, ensuring the standardization of the data and thereby improving the business value of the data.

[0213] The data quality management device provided in this application is described below. The data quality management device described below can be referred to in correspondence with the data quality management method described above.

[0214] The data quality management method provided in this application can be executed by a data quality management device. This application uses an example of a data quality management device executing the data quality management method to illustrate the data quality management device provided in this application.

[0215] This application also provides a data quality management device.

[0216] like Figure 23 As shown, the data quality management device includes: a first processing module 2310, a second processing module 2320, a third processing module 2330, and a fourth processing module 2340.

[0217] The first processing module 2310 is used to receive the user's first input, which is used to input the target configuration.

[0218] The second processing module 2320 is used to construct a data model based on the target configuration in response to the first input;

[0219] The third processing module 2330 is used to associate the data source corresponding to the target layer in the data warehouse with the corresponding position in the data model and perform field mapping;

[0220] The fourth processing module 2340 is used to audit at least a portion of the data in the data source corresponding to the target layer based on the target data audit rules and obtain a quality inspection report.

[0221] According to the data quality management device provided in this application embodiment, by setting the target configuration of data through user input, the data quality standard can be unified, ensuring the consistency of data processing in subsequent applications. This guarantees the standardized production of data from the source and saves the cost of subsequent data application and processing. In addition, by constructing a data model based on the target configuration and data structure and associating the data model with the data source, the data storage structure is unified. When calling data from multiple systems, the data can be directly associated, realizing data sharing between different systems. Furthermore, when using data, the corresponding type of data source can be quickly retrieved, saving the cost of subsequent data application and processing and facilitating effective data management by users.

[0222] In some embodiments, the first input includes a first sub-input and a second sub-input; the first processing module 2310 may also be used to enable the first sub-input to be used for inputting standard code and the second sub-input to be used for inputting data element standard.

[0223] In some embodiments, the third processing module 2330 can also be used for:

[0224] Identify the data source corresponding to the target layer from the data sources corresponding to multiple layers in the data warehouse, and establish the relationship between the data model and the data source corresponding to the target layer;

[0225] The data source associated with the data model is obtained based on the relationship; the data source corresponds to physical table fields;

[0226] Based on the logical table fields and physical table fields corresponding to the data model, establish the field mapping relationship between the data model and the data source.

[0227] In some embodiments, before auditing at least a portion of the data in the data source corresponding to the target layer based on the target data auditing rules and obtaining a quality inspection report, the data quality management device may further include:

[0228] The fifth processing module is used to receive the user's second input; the second input is used to input at least one of the following: the audit rule name, audit type, audit component, and audit field of the data source corresponding to the target layer;

[0229] The sixth processing module is used to generate target data audit rules corresponding to the data source in response to the second input.

[0230] In some embodiments, the data warehouse includes multiple data sources, each corresponding to a different audit rule, with a one-to-one correspondence between the multiple data sources and the multiple different audit rules. The fourth processing module 2140 can also be used for:

[0231] Auditing is conducted on at least a portion of data from multiple data sources based on multiple different auditing rules, resulting in multiple different quality inspection reports. Each of these quality inspection reports corresponds one-to-one with a different auditing rule.

[0232] In some embodiments, before constructing the data model based on the target configuration in response to the first input, the data quality management apparatus may further include:

[0233] The seventh processing module is used to review the target configuration. If the target configuration is approved, the release status of the target configuration will be changed to "released". The released target configuration is used to build the data model.

[0234] The eighth processing module is used to change the release status of the target configuration to unreleased and update the target configuration if the target configuration fails to pass the review.

[0235] The data quality management device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0236] The data quality management device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system used.

[0237] The data quality management device provided in this application embodiment can achieve... Figures 1 to 22 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0238] In some embodiments, such as Figure 24 As shown, this application embodiment also provides an electronic device 2400, including a processor 2401, a memory 2402, and a computer program stored in the memory 2402 and executable on the processor 2401. When the program is executed by the processor 2401, it implements the various processes of the above-described data quality management method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0239] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0240] On the other hand, this application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the various processes of the above-described data quality management method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0241] In another aspect, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the various processes of the above-described data quality management method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0242] In another aspect, this application embodiment provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-described data quality management method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0243] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0244] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0245] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0246] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data quality management method, characterized in that, include: Receive the user's first input, which is used to input the target configuration; In response to the first input, a data model is constructed based on the target configuration; Associate the data source corresponding to the target layer in the data warehouse with the corresponding position in the data model and perform field mapping; Based on the target data auditing rules, at least a portion of the data in the data source corresponding to the target layer is audited to obtain a quality inspection report; The step of associating the data source corresponding to the target layer in the data warehouse with the corresponding position in the data model and performing field mapping includes: The data source corresponding to the target layer is determined from the data sources corresponding to multiple layers in the data warehouse, and the association relationship between the data model and the data source corresponding to the target layer is established. Based on the aforementioned relationship, the data source associated with the data model is obtained; the data source corresponds to a physical table field; Based on the logical table fields and physical table fields corresponding to the data model, a field mapping relationship is established between the data model and the data source.

2. The data quality management method according to claim 1, characterized in that, The first input includes: a first sub-input and a second sub-input; the first sub-input is used to input standard codes, and the second sub-input is used to input data element standards.

3. The data quality management method according to any one of claims 1-2, characterized in that, Before auditing at least a portion of the data in the data source corresponding to the target layer based on the target data auditing rules and obtaining a quality inspection report, the method further includes: Receive a second input from the user; the second input is used to input at least one of the following: the audit rule name, audit type, audit component, and audit field of the data source corresponding to the target layer; In response to the second input, target data audit rules corresponding to the data source are generated.

4. The data quality management method according to claim 3, characterized in that, The data warehouse includes multiple data sources, each corresponding to a different auditing rule. Each data source and each auditing rule has a one-to-one correspondence. The step of auditing at least a portion of the data in the data source corresponding to the target layer based on the target data auditing rule to obtain a quality inspection report includes: Based on the multiple different audit rules, at least a portion of the data from the multiple data sources are audited to obtain multiple different quality inspection reports, and the multiple different quality inspection reports correspond one-to-one with the multiple different audit rules.

5. The data quality management method according to any one of claims 1-2, characterized in that, Before constructing the data model based on the target configuration in response to the first input, the method further includes: The target configuration is reviewed, and if the target configuration is approved, the release status of the target configuration is changed to "released"; the released target configuration is used to build the data model. If the target configuration fails the review, the release status of the target configuration will be changed to unreleased, and the target configuration will be updated.

6. A data quality management device, characterized in that, include: The first processing module is used to receive the user's first input, which is used to input the target configuration. The second processing module is used to construct a data model based on the target configuration in response to the first input; The third processing module is used to associate the data source corresponding to the target layer in the data warehouse with the corresponding position in the data model and perform field mapping; The fourth processing module is used to audit at least a portion of the data in the data source corresponding to the target layer based on the target data audit rules, and obtain a quality inspection report; The step of associating the data source corresponding to the target layer in the data warehouse with the corresponding position in the data model and performing field mapping includes: The data source corresponding to the target layer is determined from the data sources corresponding to multiple layers in the data warehouse, and the association relationship between the data model and the data source corresponding to the target layer is established. Based on the aforementioned relationship, the data source associated with the data model is obtained; the data source corresponds to a physical table field; Based on the logical table fields and physical table fields corresponding to the data model, a field mapping relationship is established between the data model and the data source.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the data quality management method as described in any one of claims 1-5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the data quality management method as described in any one of claims 1-5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the data quality management method as described in any one of claims 1-5.

Images