Data quality inspection method and device based on data source and medium

Abstract

The embodiment of the invention discloses a data quality inspection method and device based on a data source and a medium, and relates to the technical field of quality inspection.The method comprises the steps that multiple pieces of target inspection data and quality inspection demand information preset by a user are obtained, and attribution data source information corresponding to each piece of target inspection data is determined; integrating the multiple pieces of target inspection data to generate a target inspection data set, and determining a data quality rule set corresponding to the target inspection data set based on the quality inspection demand information and the attribution data source information corresponding to each piece of target inspection data; data quality detection is conducted on the target inspection data set through the data quality rule set, a quality detection result set is determined, and the quality detection result set comprises multiple pieces of problem data and a quality problem type corresponding to each piece of problem data; and according to the data source type corresponding to each piece of target inspection data, analyzing the quality inspection result set to generate data quality inspection display information.

Description

Technical Field

[0001] The present invention relates to the technical field of quality inspection, and in particular to a data quality inspection method, device and medium based on a data source. Background Art

[0002] In the era of big data, the accuracy and reliability of data are crucial for decision-making, business operations, and market analysis. However, due to the diversity of data sources, the complexity of data formats, and various factors in the data processing process, data quality issues are common. Traditional data quality inspection methods often rely on manual review, which is not only inefficient but also prone to errors and cannot meet the needs of large-scale data processing.

[0003] In addition, different data source types (such as relational databases, NoSQL databases, CSV files, Excel tables, etc.) have different data structures and characteristics, and data quality rules need to be designed specifically. However, existing data quality inspection tools often lack flexible support for different data source types, resulting in the inability to effectively apply preset data quality rules, thus affecting the accuracy and comprehensiveness of data quality inspections.

[0004] Therefore, existing data quality checking methods are difficult to efficiently integrate data from multiple data source types and do not consider the characteristics of different data source types, resulting in a cumbersome and time-consuming data quality checking process. Summary of the invention

[0005] One or more embodiments of the present specification provide a data quality checking method, device and medium based on a data source, which are used to solve the following technical problems: Therefore, existing data quality checking methods are difficult to efficiently integrate data of multiple data source types and do not consider the characteristics of different data source types, resulting in a cumbersome and time-consuming data quality checking process.

[0006] One or more embodiments of this specification adopt the following technical solutions:

[0007] One or more embodiments of the present specification provide a data quality inspection method based on a data source, the method comprising: acquiring a plurality of target inspection data and quality inspection requirement information preset by a user, and determining the belonging data source information corresponding to each of the target inspection data, wherein the belonging data source information includes the data source type corresponding to the belonging data source, and the quality inspection requirement information includes at least one preset data quality rule corresponding to the data source type; integrating the plurality of target inspection data to generate a target inspection data set, and determining a data quality rule set corresponding to the target inspection data set based on the quality inspection requirement information and the belonging data source information corresponding to each of the target inspection data; performing data quality inspection on the target inspection data set through the data quality rule set to determine a quality inspection result set, wherein the quality inspection result set includes a plurality of problem data and a quality problem type corresponding to each of the problem data; analyzing the quality inspection result set according to the data source type corresponding to each of the target inspection data to generate data quality inspection display information.

[0008] One or more embodiments of the present specification provide a data quality inspection device based on a data source, including:

[0009] at least one processor; and,

[0010] a memory communicatively connected to the at least one processor; wherein,

[0011] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above method.

[0012] One or more embodiments of the present specification provide a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured to execute the above method.

[0013] At least one of the above-mentioned technical solutions adopted in the embodiments of this specification can achieve the following beneficial effects: through the above-mentioned technical solution, by obtaining the quality inspection requirement information pre-set by the user, including the preset data quality rules for specific data source types, the inspection process can be more accurately focused on the data quality issues that the user cares about, avoiding the blindness problems that may exist in traditional quality inspections, and improving the pertinence and effectiveness of the inspection; it can automatically identify the attribution data source information corresponding to each target inspection data, including the data source type. This automatic identification function reduces manual intervention and speeds up data integration; based on the quality inspection requirement information pre-set by the user and the attribution data source information corresponding to each target inspection data, the corresponding data quality rule set is accurately matched and applied. The accurate matching ensures the pertinence and effectiveness of the inspection process, fully considers the characteristics of different data source types, and determines the corresponding data quality rules for each data source type, so that the inspection process is more in line with actual needs and improves the accuracy and reliability of the inspection. In addition, it not only focuses on the quality of a single data point, but also forms a target inspection data set by integrating multiple target inspection data, and performs inspections based on a comprehensive set of data quality rules. The comprehensive and systematic inspection method helps to discover potential data quality issues and ensure the overall quality of the data. The final generated data quality inspection display information is presented in the form of charts, reports, etc., making the inspection results more intuitive and easy to understand. Through these display information, you can quickly understand the data quality status, identify the problem type and distribution, and provide support for subsequent data processing and decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art description. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. In the drawings:

[0015] Figure 1 A flowchart of a data quality inspection method based on a data source provided in an embodiment of this specification;

[0016] Figure 2 A schematic diagram of the structure of a data quality inspection device based on a data source provided in an embodiment of this specification. DETAILED DESCRIPTION

[0017] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0018] The embodiment of this specification provides a data quality inspection method based on a data source. It should be noted that the execution subject in the embodiment of this specification can be a server or any device with data processing capabilities. Figure 1 A flow chart of a data quality inspection method based on a data source provided in an embodiment of this specification is shown as follows: Figure 1 As shown, it mainly includes the following steps:

[0019] Step S101, obtaining a plurality of target inspection data and quality inspection requirement information preset by a user, and determining the attribution data source information corresponding to each target inspection data.

[0020] The attribution data source information includes a data source type corresponding to the attribution data source, and the quality inspection requirement information includes at least one preset data quality rule corresponding to the data source type;

[0021] In one embodiment of the present specification, first, data that needs to be quality checked is collected from various possible data sources. These data come from a variety of data sources such as databases, files, API interfaces, etc. The data source type here can be a MySQL database, an Oracle database, a CSV file, an Excel file, or data returned by a specific API interface. The quality inspection requirements pre-set by the user are obtained from the user interface or configuration file, including the preset data quality rules that each data source type should follow. It should be noted that in the actual data quality inspection process, there are many target detection data involved, and the user cannot set quality rules one by one. The quality rules can be set for data from certain data sources. The quality inspection requirement information here includes at least one preset data quality rule corresponding to a data source type. After collecting multiple target inspection data, according to the data source corresponding to each target inspection data, a data source type label is added to each target inspection data to determine the belonging data source information. These labels will be used for subsequent matching with preset rules.

[0022] Step S102 , integrating multiple target inspection data to generate a target inspection data set, and determining a data quality rule set corresponding to the target inspection data set based on quality inspection requirement information and attribution data source information corresponding to each target inspection data set.

[0023] The multiple target inspection data are integrated to generate a target inspection data set, specifically including: performing data cleaning operations on the multiple target inspection data to determine multiple target inspection data after processing; determining data information of each of the target inspection data after processing, wherein the data information includes data type information and data unit information; performing standardization processing on the target inspection data according to the data information of each of the target inspection data after processing to generate multiple standard target inspection data, and construct a target inspection data set, wherein the standardization processing includes unified data format, unified data unit, classification data coding standardization and numerical standardization or normalization.

[0024] In one embodiment of the present specification, multiple target inspection data are integrated to generate a unified and standardized target inspection data set, involving multiple key steps such as data cleaning, data information determination, and standardization processing, aiming to improve the consistency and comparability of data and provide a data basis for subsequent data quality inspection. The purpose of the data cleaning operation is to remove noise, errors and redundant information in the data to ensure the accuracy and integrity of the data. First, invalid data such as null values, duplicate values, and obviously wrong data records are identified and deleted. For identifiable errors (such as spelling errors, format errors, etc.), manual or automatic corrections are performed. According to the characteristics of the data, a suitable missing value processing method is selected, such as filling default values, using interpolation, or deleting records with more missing values. The data information of the processed target inspection data is determined, and key information is extracted for each processed target inspection data for subsequent standardization processing. The type of each data (such as numeric type, text type, date type, etc.) is identified, and the corresponding data type information is recorded. For numeric data, its unit information (such as meter, kilogram, second, etc.) is identified and recorded to facilitate the unification of units. Convert the processed target inspection data into a unified format and unit to ensure data consistency and comparability. Unify the format of all data into a format suitable for subsequent processing, such as the date format is unified into YYYY-MM-DD, the time format is unified into HH:MM:SS, etc. Unify the units of numerical data into the same standard, such as converting all length units into meters, all weight units into kilograms, etc. For categorical data (such as gender, region, etc.), use a unified coding standard for encoding to facilitate subsequent data processing and analysis. For numerical data, standardize or normalize as needed to eliminate dimensional differences and make the data at the same level for easy comparison and analysis. Standardization usually involves subtracting the mean from the value and dividing it by the standard deviation, while normalization is to scale the value to between 0 and 1. Integrate multiple standard target inspection data processed by the above steps into a unified data set. Integrate all processed standard target inspection data into a data set in a certain logical order (such as time order, data source order, etc.). Verify the integrated data set to ensure the integrity and accuracy of the data, as well as the logical consistency between the data. The verified data set is stored in the preset data storage medium for subsequent data quality inspection and analysis.

[0025] Through the above technical solution, through the data cleaning step, the noise, errors and redundant information in the data are removed, the erroneous data are corrected, and the missing values ​​are processed, thereby significantly improving the accuracy and completeness of the data, which is the basis for data analysis and quality inspection, and ensures the reliability of subsequent work; the unified data format, unit, coding and standardization or normalization of numerical values ​​in the standardization processing step eliminate the inconsistency and dimensional differences between the data, making the data more standardized and unified, and facilitating cross-system and cross-platform sharing and analysis; the integrated target inspection data set reduces data redundancy and duplication, improves the efficiency of data processing, and at the same time, standardized processing also makes subsequent data quality inspection, analysis and mining more convenient and efficient.

[0026] Based on the quality inspection requirement information and the belonging data source information corresponding to each target inspection data, determine the data quality rule set corresponding to the target inspection data set, specifically including: obtaining the belonging data source identification information corresponding to each target inspection data in the target inspection data set, and classifying according to the belonging data source identification information to generate a belonging data source list corresponding to the target inspection data set; setting the preset data quality rule for multiple specified target inspection data corresponding to the specified data source type in the belonging data source list through the preset data quality rule corresponding to each specified data source type in the quality inspection requirement information and the belonging data source list, and generating a quality rule identifier; based on the quality rule identifier, updating the rule status in the belonging data source list to determine at least one other data source type except the specified data source type; predicting the quality rule of the other data source type according to the preset data quality rule corresponding to each specified data source type, and generating a predicted data quality rule corresponding to each of the other data source types; determining the data quality rule set corresponding to the target inspection data set through the predicted data quality rule and the preset data quality rule.

[0027] In one embodiment of the present specification, a data quality rule set applicable to the data set is dynamically generated based on the attribution data source information of the target inspection data set and the quality inspection requirement information preset by the user. First, the attribution data source identification information (such as data source ID, name, etc.) corresponding to each target inspection data is extracted from the target inspection data set. The target inspection data is classified according to these identification information to generate an attribution data source list, which contains all different data sources in the data set and their corresponding multiple target inspection data. According to the quality inspection requirement information preset by the user, the preset data quality rules corresponding to each specified data source type are found. For multiple specified target inspection data corresponding to the specified data source type in the attribution data source list, the corresponding preset data quality rules are set, and a quality rule identifier (such as rule ID, name, etc.) is generated for subsequent tracking and management. Based on the set quality rule identifier, the rule status is updated in the attribution data source list, and the data source type for which the preset data quality rules have been clearly set is marked. Next, at least one other data source type other than these specified data source types is determined. These types of data sources may not have direct preset rules, but can be obtained through prediction. For each other data source type, predict the data quality rules that may be applicable to it based on its data characteristics, business logic, or relationship with other known data sources. The prediction results will generate the predicted data quality rules corresponding to each other data source type. Finally, merge the preset data quality rules of the specified data source type and the predicted data quality rules of other data source types to generate a data quality rule set corresponding to the target inspection data set. The set contains the data quality rules of all data source types in the data set.

[0028] By setting corresponding data quality rules for each data source type in the data set (including directly specified and obtained through prediction), the comprehensiveness of data quality checks is ensured and potential quality problems caused by missing data source types are reduced; preset data quality rules are usually formulated based on business needs and best practices, with high accuracy and reliability. At the same time, the predictive data quality rules obtained through the prediction algorithm also strive to be accurate, thereby improving the accuracy of the entire data quality rule set; the generation of data quality rule sets realizes the automated configuration and prediction of rules, reduces the possibility of human intervention and errors, and improves the efficiency and accuracy of data processing.

[0029] According to the preset data quality rules corresponding to each specified data source type, the quality rules of the other data source types are predicted, and the predicted data quality rules corresponding to each of the other data source types are generated, which specifically includes: converting multiple preset data quality rules to determine the rule feature information corresponding to each of the preset data quality rules; constructing a rule training data set according to the rule feature information corresponding to each of the preset data quality rules and multiple specified data source types; training a predetermined classification model through the rule training data set to adjust the model parameters and determine the rule prediction model; according to the rule prediction model, the quality rules of the other data source types are predicted, and the predicted data quality rules corresponding to each of the other data source types are generated.

[0030] In one embodiment of the present specification, first, multiple preset data quality rules are analyzed to extract the rule features and corresponding data source type features corresponding to each rule. Feature extraction is one of the key steps in model training. It is necessary to ensure that the extracted features can accurately reflect the essential features of the rules and avoid introducing noise or redundant information. Convert the abstract rules into quantifiable features for subsequent model training. In addition, in order to ensure the comprehensiveness of the training data set, data quality rules for other data source types (such as SQL databases, NoSQL databases, CSV files, etc.) can also be exported from the existing system. The rules include integrity rules (such as non-empty checks), accuracy rules (such as range checks), consistency rules (such as cross-field checks), etc., ensuring that the collected rules are structured and contain sufficient metadata, such as rule types, check conditions, expected results, etc. Clean the rule data, remove duplicates, correct errors or inconsistent rule descriptions, and convert the rules into a format suitable for machine learning algorithm processing, such as feature vectors or matrices. Analyze the rule data and extract features that are useful for rule prediction. These features may include the type of rule, the field checked, the function used (such as regular expressions, mathematical operations), thresholds, etc. For text-based rule descriptions, natural language processing (NLP) technology can be used to extract semantic features.

[0031] According to the extracted rule feature information and the known specified data source type, a rule training data set is constructed, which contains multiple samples, each of which consists of rule feature information and the corresponding data source type label. Provide sufficient data support for model training so that the model can learn the association between rules and data source types. Select a predetermined classification model (such as decision tree, random forest, neural network, etc.) and train the model using the constructed rule training data set. During the training process, by adjusting the parameters of the model (such as learning rate, number of iterations, etc.), the model can accurately map the rule feature information to the corresponding data source type. Through training, a rule prediction model that can accurately predict the data quality rules corresponding to the data source type is obtained. For other unknown data source types, first extract the feature information related to the data source type. Then, input these feature information into the trained rule prediction model, and the model will output the predicted data quality rules. Automatically generate applicable data quality rules for unknown data source types for data quality inspection.

[0032] The data quality rule set corresponding to the target inspection data set is determined by the predicted data quality rule and the preset data quality rule, which specifically includes: applying the preset data quality rule of the specified data source type to the corresponding target inspection data, and establishing a mapping relationship between the preset data quality rule and the specified rule of the specified data source type; applying the predicted data quality rule to the corresponding target inspection data of other data source types, and establishing a mapping relationship between the predicted data quality rule and other rules of the other data source types; determining the data quality rule set corresponding to the target inspection data set based on multiple preset data quality rules, the specified rule mapping relationship, the predicted data quality rule and the other rule mapping relationship.

[0033] In one embodiment of the present specification, a data quality rule set corresponding to a target inspection data set is constructed by combining preset data quality rules for a specified data source type and predicted data quality rules for other data source types obtained by prediction. First, the preset data quality rules for the specified data source type are directly applied to the corresponding target inspection data. For each specified data source type, all target inspection data under this type are checked by the preset data quality rules to ensure that the data quality of the specified data source type meets the established standards and requirements. While applying the preset data quality rules, a mapping relationship between these rules and the specified data source type is established. The mapping relationship can be in the form of a simple key-value pair, where the key is the identifier of the data source type and the value is the preset data quality rule set corresponding to the type, which is convenient for subsequent management and query, ensuring that each data source type can quickly find its corresponding data quality rules.

[0034] Next, the predicted data quality rules for other data source types are applied to the corresponding target inspection data. For each data source type that is unknown or does not directly specify rules, all target inspection data under this type are checked using the predicted data quality rules. The coverage of data quality checks is expanded to ensure that a certain degree of data quality checks can be performed even for unknown data source types. Similar to applying preset data quality rules, a mapping relationship between predicted data quality rules and other data source types is established. The mapping relationship can also be in the form of a key-value pair, but the key is the unknown or predicted data source type identifier, and the value is the set of predicted data quality rules corresponding to the type, which provides a basis for data quality checks for unknown data source types and ensures that these rules can be correctly applied and managed.

[0035] Finally, based on multiple preset data quality rules, specified rule mapping relationships, predicted data quality rules and other rule mapping relationships, the data quality rule set corresponding to the target inspection data set is integrated and determined. The set includes data quality rules for all data source types (including specified and predicted), as well as the mapping relationships between them and the data source types, providing a comprehensive and unified data quality inspection standard for the entire target inspection data set to ensure the comprehensiveness and accuracy of data quality inspection.

[0036] Step S103: Perform data quality inspection on the target inspection data set through a data quality rule set to determine a quality inspection result set.

[0037] The quality inspection result set includes a plurality of problem data and a quality problem type corresponding to each of the problem data;

[0038] Through the data quality rule set, the target inspection data set is subjected to data quality inspection to determine a quality inspection result set, which specifically includes: determining multiple preset data quality rules, specified rule mapping relationships, multiple predicted data quality rules and other rule mapping relationships in the data quality rule set; traversing each target inspection data in the target inspection data set, and checking the target inspection data corresponding to the specified data source type item by item according to the multiple preset data quality rules and the specified rule mapping relationships in the data quality rule set; checking the target inspection data corresponding to other data source types item by item according to the multiple predicted data quality rules and the other rule mapping relationships; judging whether each target inspection data complies with the corresponding data quality rules, and recording the problem data that does not comply with the corresponding data quality rules and the corresponding quality problem type, so as to generate the quality inspection result set.

[0039] In one embodiment of the present specification, first, it is necessary to clarify all elements contained in the data quality rule set, including multiple preset data quality rules, specified rule mapping relationships, multiple predicted data quality rules, and other rule mapping relationships. These elements together constitute the basis for quality detection of the data set. Ensure that these rules can be accurately referenced and applied in the subsequent data quality detection process, so as to ensure the accuracy and comprehensiveness of the detection results.

[0040] Next, each target inspection data in the target inspection data set needs to be traversed. This means that each data item in the data set will be checked one by one to ensure that there is no omission. By comprehensively traversing the data set, all potential data quality issues can be discovered, providing a basis for subsequent rule application and problem recording. For the target inspection data corresponding to the specified data source type, each item is checked according to the preset data quality rules and the specified rule mapping relationship in the data quality rule set, including verifying the integrity, accuracy, consistency and other aspects of the data to ensure that the data quality meets the established standards. For the target inspection data corresponding to other data source types, each item is checked according to the predicted data quality rules and other rule mapping relationships. The predicted rules are inferred based on the historical data and business logic of similar data sources to improve the coverage of data quality detection and avoid data omissions during the data quality inspection process. In the process of applying data quality rules, it is necessary to determine whether each target inspection data complies with the corresponding data quality rules. For data that does not comply with the rules, it is considered as problem data, and its relevant information needs to be recorded, including the data itself and the corresponding quality problem type. By recording the problem data and its quality problem type, you can clearly understand the quality problems and their distribution in the data set, providing guidance for subsequent data cleaning and correction. Finally, all recorded problem data and their quality problem types are summarized to generate a quality inspection result set. The result set is the direct output of the data quality inspection process. By generating a quality inspection result set, the results of data quality inspection can be displayed intuitively, helping relevant personnel to quickly understand the data quality status and take corresponding measures to improve it.

[0041] Step S104: Analyze the quality inspection result set according to the data source type corresponding to each target inspection data to generate data quality inspection display information.

[0042] According to the data source type corresponding to each target inspection data, the quality inspection result set is analyzed to generate data quality inspection display information, specifically including: according to the data source type corresponding to each target inspection data source, the quality inspection result set is classified and counted, and a plurality of quality problem statistical indicators corresponding to each data source type are determined, wherein the quality problem statistical indicators include the total number of quality problems and the proportion data corresponding to each quality problem type; a data quality inspection report is generated through the plurality of quality problem statistical indicators corresponding to each data source type, so as to generate the data quality inspection display information based on the data quality inspection report.

[0043] In one embodiment of the present specification, the type of data source is determined, including databases (such as MySQL, Oracle), files (such as CSV, Excel), data returned by API interfaces, etc. Data sources are classified according to different data storage and acquisition methods. According to the data source type corresponding to each target inspection data, the problem data in the quality inspection result set is classified, and the problem data belonging to the same data source type are grouped together for subsequent statistics and analysis. Through classification statistics, the problems and distribution of data quality of each data source type can be clearly seen, and the dictionary or database table in the programming language (such as Python) can be used to record the number and type distribution of quality problems of each data source type. For each data source type, multiple quality problem statistical indicators are determined, including the total number of quality problems, that is, the number of all problem data under the data source type; and the proportion data corresponding to each quality problem type, that is, the proportion of a specific problem type in the total problem data. Through these problem statistical indicators, the data quality status of each data source type can be quantitatively evaluated and the main problem types can be identified. Based on the results of the above classification statistics and indicator calculations, a data quality inspection report is generated. This report should contain detailed quality problem statistics for each data source type, including the total number of problems, the proportion of each problem type, possible cause analysis, and recommended improvement measures. The data quality check report is an important document that shows the data quality check results to relevant personnel, helping users to quickly understand the data quality status. Finally, based on the data quality check report, generate data quality check display information. This information can be presented in the form of charts, dashboards, etc., so that users can understand the data quality status more intuitively. Through the data quality check display information, users can quickly obtain key data quality indicators and compare the differences between different data source types, so as to make more informed decisions.

[0044] Based on the data quality inspection report, the data quality inspection display information is generated, specifically including: determining a plurality of quality problem statistical indicators corresponding to each data source type in the data quality inspection report; generating a plurality of quality problem data distribution thermal display information corresponding to the data source type according to the problem data corresponding to each data source type and the quality problem type corresponding to each problem data; generating a problem type proportion pie chart through the proportion data corresponding to each quality problem type in each data source type; generating a plurality of problem quantity bar charts corresponding to the data source type based on the total number of quality problems in each data source type; determining the data quality inspection display information through the quality problem data distribution thermal display information, the problem type proportion pie chart and the problem quantity bar chart.

[0045] In one embodiment of the present specification, multiple quality problem statistical indicators corresponding to each data source type are determined, including the total number of quality problems, the number of each quality problem type, and their respective proportions. A heat display diagram is generated based on the problem data corresponding to each data source type and the quality problem type corresponding to each problem data. This kind of chart usually represents the difference in data density by color depth, where the darker the color, the denser the data points in the area, that is, a certain type of quality problem is more common in the data source type. Through the heat display diagram, the user can intuitively see the distribution of various types of quality problems in different data source types, so as to quickly identify the main problem areas. For each data source type, a pie chart is generated based on the proportion data corresponding to each quality problem type, and each sector area in the pie chart represents a type of quality problem, and its size is proportional to the proportion of the type in the total problem data. Through the pie chart, the user can clearly see the proportion of various types of quality problems in each data source type, and understand which problems are primary and which are secondary. Based on the total number of quality problems in each data source type, a bar chart is generated. Each bar in the bar chart represents a data source type, and its height is proportional to the total number of problems in the type. Through the bar chart, users can intuitively compare the difference in the number of problems between different data source types and identify which data source types have poor data quality. The above-generated quality problem data distribution thermal display information, problem type ratio pie chart and problem number bar chart are integrated to form a complete data quality inspection display information, which can be presented to users in the form of dashboards, reports or online platforms. By integrating multiple visual charts, users are provided with a comprehensive and intuitive data quality inspection display interface to help users quickly understand the data quality status.

[0046] By converting complex data quality check results into intuitive visual charts, such as heat maps, pie charts and bar charts, users can understand and grasp the data quality status more quickly. The intuitiveness not only improves the efficiency of information transmission, but also reduces the difficulty of understanding, allowing people with non-technical backgrounds to easily obtain key information; data quality check display information helps users quickly identify the main sources and types of data quality problems by clearly displaying the problem distribution, problem type ratio and number of problems of various data source types, which helps to focus limited resources on solving the most critical problems and improve the efficiency and pertinence of problem solving.

[0047] Through the above technical solution, by obtaining the quality inspection requirement information pre-set by the user, including the preset data quality rules for specific data source types, the inspection process can be more accurately focused on the data quality issues that the user is concerned about, avoiding the blindness problems that may exist in traditional quality inspections, and improving the pertinence and effectiveness of the inspection; it can automatically identify the attribution data source information corresponding to each target inspection data, including the data source type. This automatic identification function reduces manual intervention and speeds up data integration; based on the quality inspection requirement information pre-set by the user and the attribution data source information corresponding to each target inspection data, the corresponding data quality rule set is accurately matched and applied. The accurate matching ensures the pertinence and effectiveness of the inspection process, fully considers the characteristics of different data source types, and determines the corresponding data quality rules for each data source type, so that the inspection process is more in line with actual needs and improves the accuracy and reliability of the inspection. In addition, it not only focuses on the quality of a single data point, but also forms a target inspection data set by integrating multiple target inspection data, and performs inspections based on a comprehensive set of data quality rules. The comprehensive and systematic inspection method helps to discover potential data quality issues and ensure the overall quality of the data. The final generated data quality inspection display information is presented in the form of charts, reports, etc., making the inspection results more intuitive and easy to understand. Through these display information, you can quickly understand the data quality status, identify the problem type and distribution, and provide support for subsequent data processing and decision-making.

[0048] The embodiment of this specification also provides a data quality inspection device based on a data source, such as Figure 2 As shown, the device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above method.

[0049] The embodiments of the present specification also provide a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured to execute the above method.

[0050] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and non-volatile computer storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0051] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0052] The devices and media provided in the embodiments of this specification correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0053] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0054] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0055] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0056] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0057] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0058] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0059] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0060] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0061] The above description is only one or more embodiments of this specification and is not intended to limit this specification. For those skilled in the art, one or more embodiments of this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of this specification shall be included in the scope of the claims of this specification.

Claims

1. A data quality inspection method based on data source, It is characterized in that The method comprises: Acquire multiple target inspection data and quality inspection requirement information preset by a user, and determine the attribution data source information corresponding to each of the target inspection data, wherein the attribution data source information includes the data source type corresponding to the attribution data source, and the quality inspection requirement information includes at least one preset data quality rule corresponding to the data source type; Integrate the multiple target inspection data to generate a target inspection data set, and determine a data quality rule set corresponding to the target inspection data set based on the quality inspection requirement information and the attribution data source information corresponding to each of the target inspection data; Performing data quality inspection on the target inspection data set through the data quality rule set to determine a quality inspection result set, wherein the quality inspection result set includes a plurality of problem data and a quality problem type corresponding to each of the problem data; According to the data source type corresponding to each target inspection data, the quality inspection result set is analyzed to generate data quality inspection display information.

2. A data quality inspection method based on a data source according to claim 1, It is characterized in that Integrating the plurality of target inspection data to generate a target inspection data set specifically includes: Performing a data cleaning operation on the plurality of target inspection data to determine a plurality of target inspection data after processing; Determining data information of each of the target inspection data after processing, wherein the data information includes data type information and data unit information; According to the data information of each of the processed target inspection data, the target inspection data is standardized to generate multiple standard target inspection data and construct a target inspection data set, wherein the standardization includes unified data format, unified data unit, classification data coding standardization and numerical standard or normalization.

3. A data quality inspection method based on a data source according to claim 1, It is characterized in that Determining a data quality rule set corresponding to the target inspection data set based on the quality inspection requirement information and the attribution data source information corresponding to each target inspection data set specifically includes: Acquire the attribution data source identification information corresponding to each target inspection data in the target inspection data set, and classify according to the attribution data source identification information to generate an attribution data source list corresponding to the target inspection data set; By using the preset data quality rules corresponding to each specified data source type in the quality inspection requirement information and the belonging data source list, the preset data quality rules are set for a plurality of specified target inspection data corresponding to the specified data source type in the belonging data source list, and a quality rule identifier is generated; Based on the quality rule identifier, updating a rule state in the homed data source list to determine at least one other data source type other than the specified data source type; According to the preset data quality rules corresponding to each specified data source type, the quality rules of the other data source types are predicted to generate predicted data quality rules corresponding to each of the other data source types; A data quality rule set corresponding to the target inspection data set is determined through the predicted data quality rule and the preset data quality rule.

4. A data quality inspection method based on a data source according to claim 3, It is characterized in that According to the preset data quality rules corresponding to each specified data source type, the quality rules of the other data source types are predicted to generate the predicted data quality rules corresponding to each other data source type, specifically including: Converting the plurality of preset data quality rules to determine rule feature information corresponding to each of the preset data quality rules; Constructing a rule training data set according to the rule feature information corresponding to each of the preset data quality rules and the multiple specified data source types; The predetermined classification model is trained by using the rule training data set to adjust the model parameters and determine the rule prediction model; The quality rules of the other data source types are predicted according to the rule prediction model to generate predicted data quality rules corresponding to each of the other data source types.

5. A data quality inspection method based on a data source according to claim 3, It is characterized in that Determining a data quality rule set corresponding to the target inspection data set by using the predicted data quality rule and the preset data quality rule, specifically includes: Applying the preset data quality rules of the specified data source type to the corresponding target inspection data, and establishing a mapping relationship between the preset data quality rules and the specified rules of the specified data source type; Applying the predicted data quality rule to the target inspection data of the corresponding other data source types, and establishing a mapping relationship between the predicted data quality rule and other rules of the other data source types; A data quality rule set corresponding to the target inspection data set is determined according to the plurality of preset data quality rules, the specified rule mapping relationship, the predicted data quality rule and the other rule mapping relationship.

6. A data quality inspection method based on a data source according to claim 1, It is characterized in that Performing data quality detection on the target inspection data set by using the data quality rule set to determine a quality detection result set specifically includes: Determine a plurality of preset data quality rules, a specified rule mapping relationship, a plurality of predicted data quality rules and other rule mapping relationships in the data quality rule set; Traversing each of the target inspection data in the target inspection data set, and checking the target inspection data corresponding to the specified data source type item by item according to the mapping relationship between the multiple preset data quality rules in the data quality rule set and the specified rule; According to the mapping relationship between the multiple prediction data quality rules and the other rules, the target inspection data corresponding to other data source types are inspected item by item; Determine whether each of the target inspection data complies with the corresponding data quality rules, record the problem data that does not comply with the corresponding data quality rules and the corresponding quality problem type, so as to generate the quality inspection result set.

7. A data quality inspection method based on a data source according to claim 1, It is characterized in that According to the data source type corresponding to each target inspection data, the quality inspection result set is analyzed to generate data quality inspection display information, specifically including: According to the data source type corresponding to each of the target inspection data sources, the quality inspection result set is classified and counted to determine a plurality of quality problem statistical indicators corresponding to each data source type, wherein the quality problem statistical indicators include the total number of quality problems and the proportion data corresponding to each quality problem type; A data quality inspection report is generated by using a plurality of quality problem statistical indicators corresponding to each data source type, and the data quality inspection display information is generated based on the data quality inspection report.

8. A data quality inspection method based on a data source according to claim 7, It is characterized in that Based on the data quality inspection report, the data quality inspection display information is generated, specifically including: Determine a plurality of quality problem statistical indicators corresponding to each data source type in the data quality inspection report; Generating thermal display information of quality problem data distribution corresponding to a plurality of data source types according to the problem data corresponding to each data source type and the quality problem type corresponding to each problem data; Generate a pie chart of problem type proportions based on the proportion data corresponding to each quality problem type in each data source type; Based on the total number of quality problems in each type of data source, generate a bar chart of the number of problems corresponding to multiple types of data sources; The data quality inspection display information is determined through the thermal display information of the quality problem data distribution, the pie chart of the problem type proportion and the bar chart of the problem quantity.

9. A data quality inspection device based on a data source, It is characterized in that The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the method according to any one of claims 1 to 8.

10. A non-volatile computer storage medium storing computer executable instructions, It is characterized in that The computer executable instructions are configured to execute the method according to any one of claims 1 to 8.

Images