A data source-based data quality checking method, device and medium

Abstract

The embodiment of the specification discloses a data quality inspection method and device based on a data source, and a medium, relates to the technical field of quality inspection, and the method comprises the following steps: acquiring a plurality of target inspection data and user-pre-set quality inspection requirement information, and determining the attribution data source information corresponding to each target inspection data; integrating the plurality of target inspection data to generate a target inspection data set, determining a data quality rule set corresponding to the target inspection data set based on the quality inspection requirement information and the attribution data source information corresponding to each target inspection data; performing data quality detection on the target inspection data set through the data quality rule set to determine a quality detection result set, and the quality detection result set comprises a plurality of problem data and the quality problem type corresponding to each problem data; and analyzing the quality detection result set according to the data source type corresponding to each target inspection data to generate data quality inspection display information.

Description

TECHNICAL FIELD

[0001] The present specification relates to the technical field of quality inspection, and particularly relates to a data source-based data quality inspection method, device and medium. BACKGROUND

[0002] In the era of big data, the accuracy and reliability of data are crucial for decision-making, business operation and market analysis. However, due to the diversity of data sources, the complexity of data formats and various factors in the data processing process, data quality problems are common. Traditional data quality inspection methods often rely on manual review, which is not only inefficient but also prone to errors, making it difficult to meet the needs of large-scale data processing.

[0003] In addition, different data source types (such as relational databases, NoSQL databases, CSV files, Excel tables, etc.) have different data structures and characteristics, and need to be designed specifically for data quality rules. However, in existing data quality inspection tools, there is often a lack of flexible support for different data source types, resulting in the inability to effectively apply pre-set data quality rules, thereby affecting the accuracy and comprehensiveness of data quality inspection.

[0004] Therefore, the existing data quality inspection method is difficult to efficiently integrate data of multiple data source types, and does not consider the characteristics of different data source types, resulting in a tedious and time-consuming data quality inspection process. SUMMARY

[0005] One or more embodiments of the present specification provide a data source-based data quality inspection method, device and medium, which solve the technical problem that the existing data quality inspection method is difficult to efficiently integrate data of multiple data source types and does not consider the characteristics of different data source types, resulting in a tedious and time-consuming data quality inspection process.

[0006] One or more embodiments of the present specification adopt the following technical solutions:

[0007] One or more embodiments of the present specification provide a data source-based data quality inspection method, the method comprising: acquiring a plurality of target inspection data and user-previously-set quality inspection requirement information, and determining attribution data source information corresponding to each of the target inspection data, wherein the attribution data source information comprises a data source type corresponding to an attribution data source, and the quality inspection requirement information comprises a preset data quality rule corresponding to at least one data source type; integrating the plurality of target inspection data to generate a target inspection data set, determining a data quality rule set corresponding to the target inspection data set based on the quality inspection requirement information and the attribution data source information corresponding to each of the target inspection data; performing data quality detection on the target inspection data set through the data quality rule set to determine a quality detection result set, wherein the quality detection result set comprises a plurality of problem data and a quality problem type corresponding to each of the problem data; and analyzing the quality detection result set according to the data source type corresponding to each of the target inspection data to generate data quality inspection display information.

[0008] One or more embodiments of the present specification provide a data source-based data quality inspection device, comprising:

[0009] at least one processor; and

[0010] a memory in communication connection with the at least one processor; wherein

[0011] the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above method.

[0012] One or more embodiments of the present specification provide a non-volatile computer storage medium, which stores computer executable instructions, and the computer executable instructions are configured to perform the above method.

[0013] The above at least one technical solution adopted by the embodiments of the present specification can achieve the following beneficial effects: Through the above technical solution, by acquiring the quality inspection requirement information set by the user in advance, including the preset data quality rules for specific data source types, the inspection process is more accurately focused on the data quality problems that the user is concerned about, avoiding the blindness problem that may exist in traditional quality inspection, and improving the pertinence and effectiveness of the inspection; the corresponding attribution data source information of each target inspection data, including the data source type, can be automatically identified, and this automatic identification function reduces manual intervention and speeds up the data integration speed; based on the quality inspection requirement information set by the user in advance and the attribution data source information corresponding to each target inspection data, the corresponding data quality rule set is accurately matched and applied, and the accurate matching ensures the pertinence and effectiveness of the inspection process, fully considers the characteristics of different data source types, determines the corresponding data quality rules for each data source type, so that the inspection process is more in line with the actual demand, and the accuracy and reliability of the inspection are improved. In addition, not only the quality of a single data point is concerned, but also a target inspection data set is formed by integrating multiple target inspection data, and detection is performed based on a comprehensive data quality rule set, and the comprehensive and systematic inspection method helps to find potential data quality problems and ensure the overall quality of the data; the finally generated data quality inspection display information is presented in the form of charts, reports and the like, so that the inspection result is more intuitive and easy to understand, the data quality status can be quickly understood through these display information, the problem type and distribution can be identified, and support is provided for subsequent data processing and decision-making. BRIEF DESCRIPTION OF DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present specification or the prior art, a brief introduction will be given below to the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments described in the present specification, and those skilled in the art can also obtain other drawings according to these drawings without creative labor. In the drawings:

[0015] Figure 1 A flowchart of a data source-based data quality inspection method provided by the embodiments of the present specification;

[0016] Figure 2 A structural schematic diagram of a data source-based data quality inspection device provided by the embodiments of the present specification. DETAILED DESCRIPTION

[0017] In order for those skilled in the art to better understand the technical solutions in the specification, the technical solutions in the specification will be clearly and completely described below in conjunction with the drawings in the specification. Obviously, the described embodiments are only a part of the embodiments of the specification, not all the embodiments. Based on the embodiments of the specification, all other embodiments obtained by those skilled in the art without creative labor should belong to the protection scope of the specification.

[0018] The embodiment of the specification provides a data quality checking method based on a data source. It should be noted that the execution subject in the embodiment of the specification can be a server or any device with data processing capability. Figure 1 A flowchart of a data quality checking method based on a data source provided by the embodiment of the specification is shown in FIG. 1, which mainly includes the following steps: Figure 1

[0019] In step S101, a plurality of target checking data and user-previously-set quality checking requirement information are acquired, and the attribution data source information corresponding to each target checking data is determined.

[0020] The attribution data source information includes the data source type corresponding to the attribution data source, and the quality checking requirement information includes the preset data quality rule corresponding to at least one data source type.

[0021] In an embodiment of the specification, first, data that needs to be checked for quality is collected from each possible data source. These data come from databases, files, API interfaces and other data sources. The data source type here can be a MySQL database, an Oracle database, a CSV file, an Excel file or data returned by a specific API interface. The user-previously-set quality checking requirement is acquired from a user interface or a configuration file, including the preset data quality rule that should be followed by each data source type. It should be noted that in the actual data quality checking process, the target checking data involved are numerous, and the user cannot set the quality rule one by one. The user can choose to set the quality rule for the data of certain data sources. The quality checking requirement information here includes the preset data quality rule corresponding to at least one data source type. After the plurality of target checking data is collected, the data source type label is added to each target checking data according to the data source of each target checking data, and the attribution data source information is determined. These labels will be used for subsequent matching with the preset rule.

[0022] In step S102, the plurality of target checking data is integrated to generate a target checking data set, and the data quality rule set corresponding to the target checking data set is determined based on the quality checking requirement information and the attribution data source information corresponding to each target checking data. ​

[0023] The plurality of target inspection data is integrated to generate a target inspection data set, specifically including: performing a data cleaning operation on the plurality of target inspection data to determine a plurality of processed target inspection data; determining data information of each of the processed target inspection data, wherein the data information includes data type information and data unit information; performing standardization processing on the target inspection data according to the data information of each of the processed target inspection data to generate a plurality of standardized target inspection data, and constructing a target inspection data set, wherein the standardization processing includes uniform data format, uniform data unit, classification data coding standardization, and numerical standard or normalization.

[0024] In one embodiment of the present specification, the integration of multiple target inspection data to generate a unified, standardized target inspection data set involves multiple key steps such as data cleaning, data information determination, and standardization processing, aiming to improve the consistency and comparability of the data, and to provide a data basis for subsequent data quality inspection. The purpose of data cleaning operation is to remove noise, errors and redundant information in the data, and to ensure the accuracy and integrity of the data. First, identify and delete invalid data such as null values, duplicate values, and obviously erroneous data records. For identifiable errors (such as spelling errors, format errors, etc.), manual or automatic correction is performed. According to the characteristics of the data, choose the appropriate missing value processing method, such as filling in the default value, using interpolation method or deleting the record with more missing values. Determine the data information of the processed target inspection data, extract the key information for each processed target inspection data, in order to carry out subsequent standardization processing. Identify the type of each data (such as numerical, text, date, etc.), and record the corresponding data type information. For numerical data, identify and record its unit information (such as meters, kilograms, seconds, etc.), in order to unify the units. Convert the processed target inspection data into a unified format and unit, to ensure the consistency and comparability of the data. Unify the format of all data to a format suitable for subsequent processing, such as unifying the date format to YYYY-MM-DD, the time format to HH:MM:SS, etc. Unify the units of numerical data to the same standard, for example, convert all length units to meters, all weight units to kilograms, etc. For categorical data (such as gender, region, etc.), use a unified coding standard for coding to facilitate subsequent data processing and analysis. For numerical data, standardize or normalize as needed to eliminate dimensional differences and bring data to the same order of magnitude for comparison and analysis. Standardization usually involves subtracting the mean and dividing by the standard deviation, while normalization is to scale the data between 0 and 1. Integrate multiple standard target inspection data processed by the above steps into a unified data set. Integrate all processed standard target inspection data into a data set according to a certain logical order (such as time order, data source order, etc.). Verify the integrated data set to ensure the integrity and accuracy of the data, and the logical consistency between the data. Store the verified data set in the preset data storage medium for subsequent data quality inspection and analysis.

[0025] By the technical solution, through the data cleaning step, the noise, error and redundant information in the data are removed, the error data is corrected, and the missing value is processed, thereby significantly improving the accuracy and integrity of the data, which is the basis for data analysis and quality check, and ensures the reliability of subsequent work; the unified data format, unit, coding and standardization or normalization of numerical value in the standardization processing step eliminate the inconsistency and dimension difference between the data, so that the data is more standardized and unified, facilitating cross-system and cross-platform sharing and analysis; the integrated target inspection data set reduces the redundancy and repetition of the data, improves the efficiency of data processing, and at the same time, the standardization processing makes the subsequent data quality check, analysis and mining work more convenient and efficient.

[0026] Based on the quality inspection requirement information and the attribution data source information corresponding to each target inspection data, a data quality rule set corresponding to the target inspection data set is determined, specifically including: obtaining attribution data source identification information corresponding to each target inspection data in the target inspection data set, and classifying according to the attribution data source identification information to generate an attribution data source list corresponding to the target inspection data set; through the preset data quality rule corresponding to each specified data source type in the quality inspection requirement information and the attribution data source list, setting the preset data quality rule for a plurality of specified target inspection data corresponding to the specified data source type in the attribution data source list, generating a quality rule identifier; based on the quality rule identifier, updating the rule state in the attribution data source list to determine at least one other data source type except the specified data source type; according to the preset data quality rule corresponding to each specified data source type, predicting the quality rule of the other data source type to generate a predicted data quality rule corresponding to each other data source type; through the predicted data quality rule and the preset data quality rule, a data quality rule set corresponding to the target inspection data set is determined.

[0027] In an embodiment of the present specification, based on the ownership data source information of the target inspection data set and the user preset quality inspection requirement information, a data quality rule set suitable for the data set is dynamically generated. First, the ownership data source identification information (such as data source ID, name, etc.) corresponding to each target inspection data in the target inspection data set is extracted. The target inspection data is classified according to the identification information, and an ownership data source list is generated, which contains all different data sources in the data set and their corresponding multiple target inspection data. According to the user preset quality inspection requirement information, the preset data quality rules corresponding to each specified data source type are found. For multiple specified target inspection data corresponding to the specified data source type in the ownership data source list, the corresponding preset data quality rules are set, and a quality rule identifier (such as rule ID, name, etc.) is generated for subsequent tracking and management. Based on the set quality rule identifier, the rule state is updated in the ownership data source list, and the data source types for which the preset data quality rules have been explicitly set are marked. Next, at least one other data source type is determined in addition to the specified data source types. These types of data sources may not have direct preset rules, but can be obtained through prediction. For each other data source type, according to its data characteristics, business logic or relationship with other known data sources, the data quality rules that may be applicable to it are predicted, and the prediction result will generate the predicted data quality rules corresponding to each other data source type. Finally, the preset data quality rules of the specified data source types and the predicted data quality rules of the other data source types obtained through prediction are combined to generate the data quality rule set corresponding to the target inspection data set, which contains the data quality rules of all data source types in the data set.

[0028] By setting the corresponding data quality rules for each data source type in the data set (including direct specification and prediction), the comprehensiveness of data quality inspection is ensured, and potential quality problems caused by missing data source types are reduced; the preset data quality rules are usually formulated based on business requirements and best practices, and have high accuracy and reliability, at the same time, the predicted data quality rules obtained through the prediction algorithm are also accurate, thereby improving the accuracy of the entire data quality rule set; the generation of the data quality rule set realizes the automatic configuration and prediction of the rules, reduces the possibility of manual intervention and errors, and improves the efficiency and accuracy of data processing.

[0029] According to the preset data quality rules corresponding to each specified data source type, the quality rules of the other data source types are predicted to generate the predicted data quality rules corresponding to each of the other data source types, specifically including: converting the plurality of preset data quality rules to determine the rule feature information corresponding to each of the preset data quality rules; constructing a rule training data set according to the rule feature information corresponding to each of the preset data quality rules and the plurality of specified data source types; training a predetermined classification model through the rule training data set to adjust the model parameters and determine a rule prediction model; and predicting the quality rules of the other data source types according to the rule prediction model to generate the predicted data quality rules corresponding to each of the other data source types.

[0030] In one embodiment of the present specification, first, a plurality of preset data quality rules are analyzed to extract the rule features and the data source type features corresponding to each rule. Feature extraction is one of the key steps in model training, and it is necessary to ensure that the extracted features can accurately reflect the essential characteristics of the rules and avoid introducing noise or redundant information. Abstract rules are converted into quantifiable features for subsequent model training. In addition, in order to ensure the comprehensiveness of the training data set, data quality rules of other data source types (such as SQL databases, NoSQL databases, CSV files, etc.) can also be derived from existing systems. The rules include integrity rules (such as null check), accuracy rules (such as range check), consistency rules (such as cross-field check), etc. to ensure that the collected rules are structured and contain sufficient metadata such as rule type, check condition, expected result, etc. Clean the rule data, remove duplicates, correct errors or inconsistent rule descriptions, and convert the rules into a format suitable for machine learning algorithms, such as feature vectors or matrices. Analyze the rule data to extract features useful for rule prediction. These features may include rule type, checked field, used function (such as regular expression, mathematical operation), threshold, etc. For text type rule descriptions, natural language processing (NLP) techniques can be used to extract semantic features.

[0031] According to the extracted rule feature information and the known specified data source type, a rule training data set is constructed, containing multiple samples, each sample consisting of rule feature information and corresponding data source type label. Provide sufficient data support for model training, so that the model can learn the association between rules and data source types. Select a predetermined classification model (such as decision tree, random forest, neural network, etc.), and use the constructed rule training data set to train the model. During training, by adjusting the parameters of the model (such as learning rate, iteration times, etc.), the model can accurately map the rule feature information to the corresponding data source type. Through training, a rule prediction model that can accurately predict the data quality rules corresponding to the data source type is obtained. For other unknown data source types, first extract the feature information related to the data source type. Then, input these feature information into the trained rule prediction model, and the model will output the predicted data quality rules. Automatically generate applicable data quality rules for unknown data source types for data quality checking.

[0032] By the predicted data quality rule and the preset data quality rule, a data quality rule set corresponding to the target inspection data set is determined, specifically including: applying the preset data quality rule of the specified data source type to the corresponding target inspection data, and establishing a specified rule mapping relationship between the preset data quality rule and the specified data source type; applying the predicted data quality rule to the target inspection data of the other data source type, and establishing other rule mapping relationships between the predicted data quality rule and the other data source type; according to multiple preset data quality rules, the specified rule mapping relationship, the predicted data quality rule and the other rule mapping relationship, a data quality rule set corresponding to the target inspection data set is determined.

[0033] In an embodiment of the present specification, the preset data quality rules of the specified data source type and the predicted data quality rules of the other data source types are combined to construct a data quality rule set corresponding to the target inspection data set. First, apply the preset data quality rules of the specified data source type to the corresponding target inspection data. For each specified data source type, check all target inspection data under this type through the preset data quality rules to ensure that the data quality of the specified data source type meets the established standards and requirements. While applying the preset data quality rules, establish the mapping relationship between these rules and the specified data source type. The mapping relationship can be in the form of a simple key-value pair, where the key is the identification of the data source type and the value is the preset data quality rule set corresponding to the type, which is convenient for subsequent management and query, and ensures that each data source type can quickly find its corresponding data quality rules.

[0034] Next, the predicted data quality rules of other data source types are applied to the corresponding target inspection data, and for each unknown or directly unspecified rule data source type, all target inspection data under this type are checked by the predicted data quality rules. The coverage of data quality inspection is extended, and certain data quality inspection can be performed even for unknown data source types. Similar to the application of preset data quality rules, a mapping relationship between the predicted data quality rules and other data source types is established, and the mapping relationship can also be in the form of a key-value pair, but the key is the unknown or predicted data source type identifier, and the value is the predicted data quality rule set corresponding to the type, which provides the basis for data quality inspection for unknown data source types and ensures that these rules can be correctly applied and managed.

[0035] Finally, according to the plurality of preset data quality rules, the specified rule mapping relationship, the predicted data quality rules and the other rule mapping relationship, the data quality rule set corresponding to the target inspection data set is integrated and determined, the set contains data quality rules of all data source types (including specified and predicted) and their mapping relationship with the data source types, which provides a comprehensive and unified data quality inspection standard for the entire target inspection data set, ensuring the comprehensiveness and accuracy of data quality inspection.

[0036] Step S103, performing data quality detection on the target inspection data set through the data quality rule set to determine a quality detection result set.

[0037] The quality detection result set includes a plurality of problem data and a quality problem type corresponding to each problem data.

[0038] The data quality rule set is used to perform data quality detection on the target inspection data set to determine a quality detection result set, specifically including: determining a plurality of preset data quality rules, a specified rule mapping relationship, a plurality of predicted data quality rules and other rule mapping relationships in the data quality rule set; traversing each target inspection data in the target inspection data set, and performing item-by-item inspection on the target inspection data corresponding to the specified data source type according to the plurality of preset data quality rules and the specified rule mapping relationship in the data quality rule set; performing item-by-item inspection on the target inspection data corresponding to other data source types according to the plurality of predicted data quality rules and the other rule mapping relationship; judging whether each target inspection data meets the corresponding data quality rule, recording the problem data and the corresponding quality problem type that do not meet the corresponding data quality rule, to generate the quality detection result set.

[0039] In one embodiment of the present specification, first, all elements contained in the data quality rule set need to be clearly defined, including multiple preset data quality rules, specified rule mapping relationships, multiple predictive data quality rules, and other rule mapping relationships. These elements collectively constitute the basis for quality detection of the data set. It is ensured that these rules can be accurately referenced and applied in the subsequent data quality detection process, thereby guaranteeing the accuracy and comprehensiveness of the detection results.

[0040] Next, each target inspection data in the target inspection data set needs to be traversed. This means that each item of data in the data set will be checked one by one to ensure that nothing is missed. Through comprehensive traversal of the data set, all potential data quality problems can be found, providing a basis for subsequent rule application and problem recording. For target inspection data corresponding to specified data source types, each item is checked according to the preset data quality rules and specified rule mapping relationships in the data quality rule set, including verifying the completeness, accuracy, consistency, and other aspects of the data to ensure that the data quality meets the established standards. For target inspection data corresponding to other data source types, each item is checked according to the predictive data quality rules and other rule mapping relationships. Predictive rules are based on historical data of similar data sources and business logic to infer, improving the coverage of data quality detection and avoiding data omissions in the data quality checking process. In the process of applying data quality rules, it needs to be determined whether each target inspection data meets the corresponding data quality rules. For data that does not meet the rules, it is considered as problem data, and its relevant information needs to be recorded, including the data itself and the corresponding quality problem type. By recording the problem data and its quality problem type, the quality problems existing in the data set and their distribution can be clearly understood, providing guidance for subsequent data cleaning and correction. Finally, all recorded problem data and its quality problem type are summarized to generate a quality detection result set. The result set is the direct output of the data quality detection process, and by generating the quality detection result set, the results of data quality detection can be intuitively displayed to help relevant personnel quickly understand the data quality status and take appropriate measures for improvement.

[0041] Step S104, according to the data source type corresponding to each target inspection data, the quality detection result set is analyzed to generate data quality inspection display information.

[0042] According to the data source type corresponding to each target inspection data, the quality detection result set is analyzed to generate data quality inspection display information, specifically including: according to the data source type corresponding to each target inspection data source, the quality detection result set is classified and counted to determine a plurality of quality problem statistical indicators corresponding to each data source type, wherein the quality problem statistical indicators include the total number of quality problems and the proportion data corresponding to each quality problem type; through the plurality of quality problem statistical indicators corresponding to each data source type, a data quality inspection report is generated to generate the data quality inspection display information based on the data quality inspection report.

[0043] In an embodiment of the present specification, the type of data source is determined, including databases (such as MySQL, Oracle), files (such as CSV, Excel), API interface returned data, etc. According to the difference of data storage and acquisition method, the data source is classified. According to the data source type corresponding to each target inspection data, the problem data in the quality detection result set is classified, and the problem data belonging to the same data source type is grouped into a group for subsequent statistics and analysis. Through classification statistics, the problems and their distribution in data quality of each data source type can be clearly seen, and dictionaries or database tables in programming languages (such as Python) can be used to record the quality problem quantity and type distribution of each data source type. For each data source type, a plurality of quality problem statistical indicators are determined, including the total number of quality problems, i.e. the number of all problem data under the data source type; and the proportion data corresponding to each quality problem type, i.e. the proportion of a specific problem type in the total problem data. Through these problem statistical indicators, the data quality status of each data source type can be quantitatively evaluated, and the main problem type can be identified. Based on the results of the above classification statistics and index calculation, a data quality inspection report is generated. This report should contain detailed quality problem statistical information of each data source type, including the total number of problems, the proportion of each problem type, possible cause analysis and suggested improvement measures, etc. The data quality inspection report is an important document for showing the data quality inspection results to relevant personnel, which helps users quickly understand the data quality status. Finally, based on the data quality inspection report, data quality inspection display information is generated. These information can be presented in the form of charts, dashboards, etc. to help users more intuitively understand the data quality status. Through the data quality inspection display information, users can quickly obtain key data quality indicators and compare the differences between different data source types, so as to make more wise decisions.

[0044] Based on the data quality inspection report, the data quality inspection display information is generated, specifically including: determining a plurality of quality problem statistical indicators corresponding to each data source type in the data quality inspection report; generating a plurality of quality problem data distribution heat display information corresponding to each data source type according to the problem data corresponding to each data source type and the quality problem type corresponding to each problem data; generating a problem type proportion pie chart through the proportion data corresponding to each quality problem type in each data source type; generating a plurality of problem quantity bar charts corresponding to each data source type based on the total number of quality problems in each data source type; and determining the data quality inspection display information through the quality problem data distribution heat display information, the problem type proportion pie chart, and the problem quantity bar chart.

[0045] In an embodiment of the present specification, a plurality of quality problem statistical indicators corresponding to each data source type are determined, including the total number of quality problems, the number of each quality problem type, and their respective proportions, etc. A heat display chart is generated according to the problem data corresponding to each data source type and the quality problem type corresponding to each problem data. This chart usually represents the difference in data density with color depth, where the deeper the color, the more data points in that area, i.e. a certain quality problem type is more common in that data source type. Through the heat display chart, the user can intuitively see the distribution of various quality problem types in different data source types, and quickly identify the main problem areas. For each data source type, a pie chart is generated according to the proportion data corresponding to each quality problem type, where each sector in the pie chart represents a quality problem type, and its size is proportional to the proportion of that type in the total problem data. Through the pie chart, the user can clearly see the proportion of various quality problem types in each data source type, and understand which problems are primary and which are secondary. Based on the total number of quality problems in each data source type, a bar chart is generated. Each bar in the bar chart represents a data source type, and its height is proportional to the total number of problems in that type. Through the bar chart, the user can intuitively compare the number of problems between different data source types and identify which data source types perform poorly in data quality. The quality problem data distribution heat display information, the problem type proportion pie chart, and the problem quantity bar chart generated above are integrated to form complete data quality inspection display information, which can be presented to the user in the form of a dashboard, a report, or an online platform. By integrating multiple visual charts, a comprehensive and intuitive data quality inspection display interface is provided for the user to quickly understand the data quality status.

[0046] By converting complex data quality check results into intuitive visual charts such as heat maps, pie charts, and bar charts, users can more quickly understand and grasp the data quality situation. Intuitiveness not only improves the efficiency of information transmission, but also reduces the difficulty of understanding, so that non-technical personnel can easily obtain key information. The data quality check display information clearly shows the problem distribution of various data source types, the proportion of problem types, and the number of problems, etc. This helps users quickly identify the main sources and types of data quality problems, and helps to focus limited resources on solving the most critical problems, improving the efficiency and relevance of problem solving.

[0047] Through the above technical solutions, by obtaining the quality check requirement information set by the user in advance, including the preset data quality rules for specific data source types, the check process is more accurately focused on the data quality problems that the user is concerned about, avoiding the blindness that may exist in traditional quality checks, and improving the relevance and effectiveness of the check. The automatic identification of the attribution data source information corresponding to each target check data, including the data source type, reduces manual intervention and speeds up data integration. Based on the quality check requirement information set by the user in advance and the attribution data source information corresponding to each target check data, the corresponding data quality rule set is accurately matched and applied, and the accuracy of the check process is ensured. The relevance and effectiveness of the check process, fully considering the characteristics of different data source types, determining the corresponding data quality rules for each data source type, making the check process more in line with actual needs, and improving the accuracy and reliability of the check. In addition, not only the quality of a single data point is concerned, but also multiple target check data are integrated to form a target check data set, and a comprehensive data quality rule set is used for detection. The comprehensive and systematic check method helps to find potential data quality problems and ensure the overall quality of the data. The finally generated data quality check display information is presented in the form of charts, reports, etc. The check results are more intuitive and easy to understand, and the data quality situation can be quickly understood through these display information to identify problem types and distribution, providing support for subsequent data processing and decision-making.

[0048] The embodiments of the present specification also provide a data source-based data quality check device, as shown in Figure 2 The device includes at least one processor and a memory communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the above method.

[0049] The embodiments of the present specification also provide a non-volatile computer storage medium storing computer executable instructions configured to perform the above method.

[0050] Each of the embodiments described in the specification can be described in progressive form, and the same or similar parts among the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the device, apparatus, and non-transitory computer storage medium embodiments, the description is relatively simple because they are substantially similar to the method embodiments, and the relevant parts can be referred to the part of the method embodiments.

[0051] The above describes specific embodiments of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different than the order in which they are recited in the embodiments and still achieve the desired result. In addition, the processes depicted in the accompanying drawings do not necessarily require the particular order shown or sequential order to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous or possible.

[0052] The devices and media provided by the embodiments of the specification are one-to-one corresponding to the methods, and therefore, the devices and media also have similar beneficial technical effects to the methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be described here.

[0053] Those skilled in the art will appreciate that embodiments of the specification can be provided as methods, systems, or computer program products. Therefore, the specification can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) having computer usable program code embodied therein.

[0054] The specification is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the specification. It should be understood that each flow and / or block in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in a flow or multiple flows and / or blocks Figure 1 The apparatus for performing the functions specified in a flow or multiple flows and / or blocks.

[0055] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the Figure 1 function specified in the flow or flows and / or blocks Figure 1 of the block or blocks.

[0056] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the Figure 1 function specified in the flow or flows and / or blocks Figure 1 of the block or blocks.

[0057] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0058] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) and / or cache memory, non-volatile memory, such as read-only memory (ROM), EPROM, and / or flash memory, etc. The memory is an example of computer readable media.

[0059] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD), or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to computing devices. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carrier waves.

[0060] It is also to be noted that the terms "comprising", "including", and any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises a... " does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the recited element.

[0061] The foregoing is merely illustrative of the various embodiments of this disclosure and the description is intended to provide further support to the patent claims. The embodiments of the disclosure, including the best mode thereof, do not limit the patent claims. Any modification, equivalent replacement, or improvement not departing from the spirit and principles of the one or more embodiments of the disclosure should be included in the patent claims.

Claims

1. A method for data quality checking based on data sources, characterized in that, The method comprises: acquiring a plurality of target inspection data and user pre-set quality inspection requirement information, and determining the attribution data source information corresponding to each of the target inspection data, wherein the attribution data source information comprises a data source type corresponding to an attribution data source, and the quality inspection requirement information comprises a pre-set data quality rule corresponding to at least one data source type; integrating the plurality of target inspection data to generate a target inspection data set, determining a data quality rule set corresponding to the target inspection data set based on the quality inspection requirement information and the attribution data source information corresponding to each of the target inspection data; performing data quality detection on the target inspection data set through the data quality rule set to determine a quality detection result set, wherein the quality detection result set comprises a plurality of problem data and a quality problem type corresponding to each of the problem data; analyzing the quality detection result set according to the data source type corresponding to each of the target inspection data to generate data quality inspection display information; determining the data quality rule set corresponding to the target inspection data set based on the quality inspection requirement information and the attribution data source information corresponding to each of the target inspection data, specifically comprising: acquiring attribution data source identification information corresponding to each of the target inspection data in the target inspection data set, and classifying according to the attribution data source identification information to generate an attribution data source list corresponding to the target inspection data set; setting the pre-set data quality rule for a plurality of specified target inspection data corresponding to a specified data source type in the attribution data source list through the pre-set data quality rule corresponding to each specified data source type in the quality inspection requirement information and the attribution data source list, and generating a quality rule identifier; based on the quality rule identifier, updating the rule state in the attribution data source list to determine at least one other data source type other than the specified data source type; according to the pre-set data quality rule corresponding to each specified data source type, predicting the quality rule of the other data source type to generate a predicted data quality rule corresponding to each of the other data source types; determining the data quality rule set corresponding to the target inspection data set through the predicted data quality rule and the pre-set data quality rule; according to the pre-set data quality rule corresponding to each specified data source type, predicting the quality rule of the other data source type to generate a predicted data quality rule corresponding to each of the other data source types, specifically comprising: converting a plurality of the pre-set data quality rules to determine rule feature information corresponding to each of the pre-set data quality rules; constructing a rule training data set according to the rule feature information corresponding to each of the pre-set data quality rules and a plurality of the specified data source types; training a pre-determined classification model through the rule training data set to adjust model parameters and determine a rule prediction model; According to the rule prediction model, the quality rules of the other data source types are predicted to generate predicted data quality rules corresponding to each of the other data source types; Through the predicted data quality rules and the preset data quality rules, a data quality rule set corresponding to the target inspection data set is determined, specifically including: Applying the preset data quality rules of the specified data source type to the corresponding target inspection data, and establishing a specified rule mapping relationship between the preset data quality rules and the specified data source type; Applying the predicted data quality rules to the target inspection data of the corresponding other data source types, and establishing other rule mapping relationships between the predicted data quality rules and the other data source types; According to the plurality of preset data quality rules, the specified rule mapping relationship, the predicted data quality rules and the other rule mapping relationships, the data quality rule set corresponding to the target inspection data set is determined; According to the data source type corresponding to each of the target inspection data, the quality detection result set is analyzed to generate data quality inspection display information, specifically including: According to the data source type corresponding to each of the target inspection data, the quality detection result set is classified and counted to determine a plurality of quality problem statistical indicators corresponding to each data source type, wherein the quality problem statistical indicators include the total number of quality problems and the proportion data corresponding to each quality problem type; Through the plurality of quality problem statistical indicators corresponding to each data source type, a data quality inspection report is generated, and based on the data quality inspection report, the data quality inspection display information is generated.

2. The data source based data quality checking method of claim 1, wherein, The plurality of target inspection data is integrated to generate a target inspection data set, specifically including: Performing data cleaning operations on the plurality of target inspection data to determine a plurality of processed target inspection data; Determining data information of each of the processed target inspection data, wherein the data information includes data type information and data unit information; According to the data information of each of the processed target inspection data, the target inspection data is standardized to generate a plurality of standard target inspection data, and a target inspection data set is constructed, wherein the standardization includes unified data format, unified data unit, classification data coding standardization and numerical standard or normalization.

3. The data source based data quality checking method of claim 1, wherein, Through the data quality rule set, data quality detection is performed on the target inspection data set to determine a quality detection result set, specifically including: Determining a plurality of preset data quality rules, a specified rule mapping relationship, a plurality of predicted data quality rules and other rule mapping relationships in the data quality rule set; Iterating through each of the target inspection data in the target inspection data set, and performing item-by-item inspection on the target inspection data corresponding to the specified data source type according to the plurality of preset data quality rules and the specified rule mapping relationship in the data quality rule set; According to the plurality of predicted data quality rules and the other rule mapping relationships, item-by-item inspection is performed on the target inspection data corresponding to the other data source types; According to the plurality of preset data quality rules, the specified rule mapping relationship, the predicted data quality rules and the other rule mapping relationships, the data quality rule set corresponding to the target inspection data set is determined; According to the data source type corresponding to each of the target inspection data, the quality detection result set is analyzed to generate data quality inspection display information, specifically including: According to the data source type corresponding to each of the target inspection data, the quality detection result set is classified and counted to determine a plurality of quality problem statistical indicators corresponding to each data source type, wherein the quality problem statistical indicators include the total number of quality problems and the proportion data corresponding to each quality problem type; Through the plurality of quality problem statistical indicators corresponding to each data source type, a data quality inspection report is generated, and based on the data quality inspection report, the data quality inspection display information is generated. The plurality of target inspection data is integrated to generate a target inspection data set, specifically including: Performing data cleaning operations on the plurality of target inspection data to determine a plurality of processed target inspection data; Determining data information of each of the processed target inspection data, wherein the data information includes data type information and data unit information; According to the data information of each of the processed target inspection data, the target inspection data is standardized to generate a plurality of standard target inspection data, and a target inspection data set is constructed, wherein the standardization includes unified data format, unified data unit, classification data coding standardization and numerical standard or normalization. determine whether each of the target inspection data meets the corresponding data quality rule, record problem data and corresponding quality problem type which do not meet the corresponding data quality rule, to generate the quality detection result set.

4. The data source based data quality checking method of claim 1, wherein, generate the data quality inspection display information based on the data quality inspection report, specifically including: determine a plurality of quality problem statistical indicators corresponding to each data source type in the data quality inspection report; generate a plurality of quality problem data distribution heat display information corresponding to each data source type according to the problem data corresponding to each data source type and the quality problem type corresponding to each problem data; generate a problem type proportion pie chart through the proportion data corresponding to each quality problem type in each data source type; generate a problem number bar chart of a plurality of data source types based on the total number of quality problems in each data source type; determine the data quality inspection display information through the quality problem data distribution heat display information, the problem type proportion pie chart and the problem number bar chart.

5. A data source based data quality checking apparatus, characterized by, The device comprises: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-4.

6. A non-transitory computer storage medium storing computer-executable instructions that, when executed, cause a computer to perform: The computer executable instructions are configured to perform the method of any one of claims 1-4.

Images