Data quality evaluation method and device, electronic equipment and computer readable storage medium
Through pre-configured data quality evaluation templates and rules sets, the detection results of data quality indicators are automatically obtained, which solves the problem of insufficient efficiency and accuracy of data quality evaluation in the prior art, and achieves efficient and objective data quality evaluation.
Patent Information
- Application Number
- CN202411028349.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-11-22
AI Technical Summary
Existing data quality assessment methods rely on manual audits or simple automation tools, lack systematicity and intelligence, making it difficult to efficiently and accurately evaluate the overall health and value of data assets.
Provide a data quality assessment method, through pre-configured data quality assessment templates and rule sets, use rules to associate configuration information and quality assessment strategies, automatically obtain the detection results of data quality indicators, and conduct multi-dimensional evaluations to reduce manual intervention and improve Assess efficiency and objectivity.
It realizes automated, rapid and comprehensive evaluation of data quality, supports rapid decision-making and business optimization, ensures the objectivity and comprehensiveness of evaluation, and reduces the need for manual writing of SQL statements.
Smart Images

Figure CN119003499A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of intelligent transportation technology, and in particular to a data quality assessment method, device, electronic device and computer-readable storage medium. Background Art
[0002] With the acceleration of digital transformation, data has become an indispensable strategic asset for enterprises. In the process of data-driven decision-making, data quality directly affects the accuracy and reliability of analysis results. Before entering the enterprise data warehouse, a large amount of data often faces problems such as data inconsistency, missing, error and redundancy, which not only increases the complexity of data management, but also may lead to business decision-making errors, affecting the operational efficiency and competitiveness of the enterprise.
[0003] Traditional data quality assessment methods mostly rely on manual review or simple automated tools. They lack a systematic and intelligent data quality scoring mechanism and make it difficult to efficiently and accurately assess the overall health and value of data assets. Summary of the invention
[0004] The embodiments of the present application provide a data quality assessment method, device, electronic device and computer-readable storage medium to improve the efficiency and accuracy of data quality assessment.
[0005] The present application embodiment adopts the following technical solutions:
[0006] In a first aspect, an embodiment of the present application provides a data quality assessment method, the method comprising:
[0007] Acquire a data quality assessment template, wherein the data quality assessment template includes a quality assessment strategy and rule association configuration information configured based on the data information to be assessed, wherein the rule association configuration information is used to match a target rule from a pre-configured rule set, wherein the rule set includes rules corresponding to different data quality indicators, and each rule includes a quality detection strategy constructed based on the rule description information;
[0008] Acquire a target rule corresponding to the data information to be evaluated from the rule set according to the rule association configuration information;
[0009] According to the quality detection strategy corresponding to the target rule, obtain the quality detection result corresponding to each data quality indicator of the data to be evaluated;
[0010] According to the quality assessment strategy and the quality detection results corresponding to each data quality indicator of the data to be assessed, a multi-dimensional data quality assessment result corresponding to the data information to be assessed is obtained.
[0011] Optionally, the data information to be evaluated includes data table information and field information corresponding to the data to be evaluated, and the rule association configuration information is used to indicate the correspondence between the data information to be evaluated and the rules in the rule set, and the rule association configuration information includes at least table-level configuration information and field-level configuration information;
[0012] The table-level configuration information at least includes data table information and rule information overview;
[0013] The field-level configuration information at least includes data table information, field information, and rule information corresponding to each field information.
[0014] Optionally, the quality detection strategy includes a database query statement expression, and each rule also includes indicator classification information and rule description information.
[0015] Optionally, acquiring a target rule corresponding to the data information to be evaluated from the rule set according to the rule association configuration information includes:
[0016] Acquire the target rule according to the rule information overview in the table-level configuration information;
[0017] According to the rule information corresponding to each field information in the field-level configuration information, the field information corresponding to the target rule is acquired.
[0018] Optionally, obtaining the quality detection result corresponding to each data quality indicator of the data to be evaluated according to the quality detection strategy corresponding to the target rule includes:
[0019] Generate a database query statement according to the data table information and field information of the data information to be evaluated and the database query statement expression;
[0020] According to the data table information of the data information to be evaluated and the database query statement, a quality detection result corresponding to each data quality indicator is obtained.
[0021] The data table information includes a table query condition, and the quality detection result corresponding to each data quality indicator is obtained according to the data table information of the data information to be evaluated and the database query statement, including:
[0022] Determine whether to establish a database link according to the table query condition;
[0023] When the database link is successfully established, the amount of data that complies with the corresponding rules is obtained through the return result of the database query statement;
[0024] According to the amount of data that complies with the corresponding rules and the total amount of data in the data table, a quality detection result corresponding to each data quality indicator is obtained.
[0025] Optionally, the quality assessment strategy includes a single indicator assessment strategy and a comprehensive indicator assessment strategy, and obtaining a multi-dimensional data quality assessment result corresponding to the data information to be assessed based on the quality assessment strategy and the quality detection result corresponding to each data quality indicator of the data to be assessed, includes:
[0026] Performing standardized calculation on the quality detection results corresponding to each data quality indicator according to the single indicator evaluation strategy to obtain the single evaluation results corresponding to each data quality indicator;
[0027] Perform weighted calculation on the individual evaluation results corresponding to each data quality indicator according to the comprehensive indicator evaluation strategy to obtain the comprehensive evaluation results corresponding to all data quality indicators;
[0028] The multi-dimensional data quality assessment result corresponding to the data information to be assessed is obtained through the single assessment result and the comprehensive assessment result.
[0029] In a second aspect, an embodiment of the present application further provides a data quality assessment device, the device comprising:
[0030] An evaluation template acquisition unit, used to acquire a data quality evaluation template, wherein the data quality evaluation template includes a quality evaluation strategy and rule association configuration information configured based on the data information to be evaluated, wherein the rule association configuration information is used to match a target rule from a pre-configured rule set, wherein the rule set includes rules corresponding to different data quality indicators, and each rule includes a quality detection strategy constructed based on the rule description information;
[0031] A rule acquisition unit, configured to acquire a target rule corresponding to the data information to be evaluated from the rule set according to the rule association configuration information;
[0032] A detection result acquisition unit, used to obtain a quality detection result corresponding to each data quality indicator of the data to be evaluated according to the quality detection strategy corresponding to the target rule;
[0033] The evaluation result acquisition unit is used to acquire the multi-dimensional data quality evaluation result corresponding to the data information to be evaluated according to the quality evaluation strategy and the quality detection result corresponding to each data quality indicator of the data to be evaluated.
[0034] In a third aspect, an embodiment of the present application further provides an electronic device, including:
[0035] A processor; and a memory arranged to store computer executable instructions, which, when executed, cause the processor to perform the aforementioned data quality assessment method.
[0036] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores one or more programs, and when the one or more programs are executed by an electronic device including multiple applications, the electronic device executes the aforementioned data quality assessment method.
[0037] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:
[0038] The embodiment of the present application pre-configures a data quality assessment template and a rule set, configures a corresponding quality detection strategy through each rule in the rule set, and respectively configures the correspondence between the data to be evaluated and the rules and the data quality assessment algorithm of the data to be evaluated through the rule association configuration information of the data quality assessment template and the quality assessment strategy; in this way, when data quality assessment is needed, the target rule corresponding to the data information to be evaluated can be obtained according to the acquired data quality assessment template, and the quality detection result of the data to be evaluated for each data quality indicator can be obtained according to the quality detection strategy corresponding to the target rule, and the quality detection result corresponding to each data quality indicator is evaluated and calculated based on the quality assessment strategy, so as to obtain a multi-dimensional data quality assessment result of the data to be evaluated.
[0039] The embodiment of the present application can construct an evaluation system that comprehensively covers various data quality indicators through simple rule configuration. It does not require manual writing or modification of SQL statements, can reduce manual intervention, greatly improve evaluation efficiency, and ensure the objectivity and comprehensiveness of quality evaluation by setting reasonable quality evaluation strategies. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0041] Figure 1 A flowchart of a data quality assessment method in an embodiment of the present application;
[0042] Figure 2 This is a schematic diagram of the structure of a data quality assessment template and rules in an embodiment of the present application;
[0043] Figure 3 This is a schematic diagram of a quality inspection result in an embodiment of the present application;
[0044] Figure 4 This is a schematic diagram of the structure of a data quality assessment device in an embodiment of the present application;
[0045] Figure 5 This is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION
[0046] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.
[0047] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.
[0048] As mentioned above, current data quality assessment solutions generally perform quality checks on specified fields, and require manual writing of corresponding database query statements, such as manually writing SQL (Structured Query Language) statements for specified fields. This solution cannot be quickly applied to massive, heterogeneous data resources, nor can it intuitively reflect the quality of data resources.
[0049] Based on this, the embodiment of the present application provides a technical solution that can automatically, efficiently and comprehensively evaluate data quality to support rapid decision-making and business optimization. The data quality evaluation solution of the embodiment of the present application comprehensively considers multiple data quality indicators of the data and automatically gives an intuitive comprehensive score to provide support for guiding the adjustment of data governance strategies and continuous improvement of data quality.
[0050] The following embodiments of the present application are illustrated by using a computing device as the execution subject of the data quality assessment method. It is understandable that the execution subject of the embodiments of the present application may also be other devices or systems, such as a data management device, a server or a chip system, the chip system includes at least one processor, and a communication interface or memory, wherein the memory is used to store programs or instructions, and the communication interface is used to provide the processor with information or data input and output functions. The execution subject of the embodiments of the present application may also be the cloud. Alternatively, the execution subject of the data quality assessment method of the embodiments of the present application may also be software or hardware, and different execution steps may be centralized or distributed.
[0051] To facilitate understanding of the following embodiments of the present application, the terms in the embodiments of the present application are first explained.
[0052] Data quality indicators, also known as data quality dimensions. In practical applications, data quality indicators can be determined according to business needs. For example, in this embodiment, data quality indicators include multiple indicators such as standardization, integrity, uniqueness, accuracy, consistency, accessibility, etc. In other embodiments, data quality indicators can be graded, for example, indicators such as standardization, integrity, uniqueness, accuracy, consistency, accessibility, etc. are used as first-level indicators, each of which can include multiple second-level indicators, and each second-level indicator can also include third-level indicators, etc.
[0053] like Figure 1 As shown, a flow chart of a data quality assessment method in an embodiment of the present application is provided, and the method at least includes the following steps S110 to S140:
[0054] Step S110, obtain a data quality assessment template, which includes a quality assessment strategy and rule association configuration information configured based on the data information to be evaluated, and the rule association configuration information is used to match the target rule from a pre-configured rule set. The rule set includes rules corresponding to different data quality indicators, and each rule includes a quality detection strategy constructed based on the rule description information.
[0055] Before performing data quality assessment, the embodiment of the present application pre-configures a data quality assessment template and a rule set, and the rule set includes rules corresponding to all data quality indicators, such as rules corresponding to standardization, rules corresponding to integrity, rules corresponding to uniqueness, rules corresponding to accuracy, rules corresponding to consistency, and rules corresponding to accessibility. Each rule includes a quality detection strategy constructed based on the rule description information, through which the quality detection results that meet the corresponding rules in the data to be evaluated can be obtained.
[0056] The data quality assessment template of this embodiment is mainly used to configure the correspondence between the data information to be evaluated and the rules. The data quality assessment template includes rule association configuration information and quality assessment strategy. The rule association configuration information is used to indicate the rules corresponding to the data information to be evaluated, and the quality assessment strategy is used to indicate how to calculate the multi-dimensional data quality assessment results of the data to be evaluated based on the quality detection results.
[0057] Step S120: acquiring a target rule corresponding to the to-be-evaluated data information from the rule set according to the rule association configuration information.
[0058] In practical applications, the data information to be evaluated may be multi-source, heterogeneous data. Therefore, in most cases, the data composition of the data to be evaluated is different. Therefore, different data contents of the data to be evaluated may require quality assessment of different indicators. For example, data content A of the data to be evaluated needs to be evaluated for normativeness and integrity, while data content B of the data to be evaluated needs to be evaluated for uniqueness and accuracy.
[0059] The rule association configuration information in the embodiment of the present application includes rules corresponding to each data content of the data to be evaluated. Therefore, according to the rule association configuration information, it can be obtained which data content needs to be evaluated for the quality of the corresponding indicators based on the target rules.
[0060] For example, assuming that the rule association configuration information stipulates that data content A needs to undergo normative evaluation and integrity evaluation, data content B needs to undergo uniqueness evaluation and accuracy evaluation, and data content C needs to undergo normative evaluation and uniqueness evaluation; it can be determined that both data contents A and C need to undergo normative evaluation based on normative rules, and at this time the normative rules are the target rules described in this embodiment.
[0061] Step S130: obtaining quality detection results corresponding to each data quality indicator of the data to be evaluated according to the quality detection strategy corresponding to the target rule.
[0062] Continuing with the previous example, when it is determined that both data contents A and C need to be subjected to normative evaluation based on normative rules, quality inspections are performed on data contents A and C in turn according to the quality inspection strategies corresponding to the normative rules to obtain quality inspection results corresponding to data content A and quality inspection results corresponding to data content C. The quality inspection results at least include the amount of data that meets the corresponding rules.
[0063] Step S140: obtaining a multi-dimensional data quality assessment result corresponding to the data to be assessed according to the quality assessment strategy and the quality detection result corresponding to each data quality indicator of the data to be assessed.
[0064] like Figure 1It can be seen from the data quality assessment method shown that the embodiment of the present application pre-configures a data quality assessment template and a rule set, configures a corresponding quality detection strategy through each rule in the rule set, and respectively configures the correspondence between the data to be evaluated and the rules and the data quality assessment algorithm of the data to be evaluated through the rule association configuration information of the data quality assessment template and the quality assessment strategy; in this way, when data quality assessment is needed, the target rule corresponding to the data information to be evaluated can be obtained according to the acquired data quality assessment template, and the quality detection result of the data to be evaluated for each data quality indicator can be obtained according to the quality detection strategy corresponding to the target rule, and the quality detection result corresponding to each data quality indicator is evaluated and calculated based on the quality assessment strategy, so as to obtain a multi-dimensional data quality assessment result of the data to be evaluated.
[0065] The embodiment of the present application can construct an evaluation system that comprehensively covers various data quality indicators through simple rule configuration. It does not require manual writing or modification of SQL statements, can reduce manual intervention, greatly improve evaluation efficiency, and ensure the objectivity and comprehensiveness of quality evaluation by setting reasonable quality evaluation strategies.
[0066] In some embodiments of the present application, the data information to be evaluated includes data table information and field information corresponding to the data to be evaluated, and the rule association configuration information is used to indicate the correspondence between the data information to be evaluated and the rules in the rule set, and the rule association configuration information includes at least table-level configuration information and field-level configuration information;
[0067] The table-level configuration information at least includes data table information and rule information overview;
[0068] The field-level configuration information at least includes data table information, field information, and rule information corresponding to each field information.
[0069] The data table information includes the data table ID and the data table description information. The data table description information includes, for example, the data table name, table query conditions, and Schema. Schema is a logical structure defined in a database management system, which describes objects in the database, such as tables, views, stored procedures, etc. In a relational database, a Schema can contain multiple tables and other objects, which together constitute part of the database.
[0070] The field information includes the field ID and field description information. The field description information includes, for example, the field name, field type, whether it is a primary key, etc.
[0071] In this embodiment, the data structure of the rule association configuration information may be a table structure, wherein the table-level configuration information is described by a table-level configuration table, and the field-level configuration information is described by a field-level configuration table, such as Figure 2 As shown, the table-level configuration table and the field-level configuration table can be associated according to the primary key PK+foreign key FK (foreign key FK includes FK1, FK2 and FK3) of the data table of the data information to be evaluated.
[0072] It should be noted that the rule information overview in the table-level configuration information can be understood as all the rules included in the field-level configuration information. If the rules corresponding to all field information of the field-level configuration information include standardization, completeness, uniqueness, accuracy and consistency, then the rule information overview in the table-level configuration information records the rule IDs or numbers of these five categories.
[0073] In some embodiments of the present application, the quality detection strategy includes a database query statement expression, such as an SQL statement expression; each rule includes indicator classification information, rule description information and an SQL statement expression.
[0074] The data structure of each rule can also be a table structure, for example Figure 2 The rule configuration table is shown.
[0075] Below Figure 2 The rule configuration table shown in the figure details the rules of this embodiment:
[0076] like Figure 2 As shown, the rule configuration table includes rule identification information, indicator classification information, rule description information and SQL statement expression;
[0077] The rule identification information can be understood as identification information such as rule ID and number;
[0078] The indicator classification information includes the first-level indicator information corresponding to the current rule and all the lower-level indicator information of the first-level indicator. Here, the table information includes, for example, the indicator ID, number and indicator name; taking the subdivision into second-level indicators as an example, if the first-level indicator corresponding to the current rule is a normative indicator, then its second-level indicators include, for example, business norms and content norms;
[0079] The indicator description information is used to describe the quality detection purpose of the lowest-level indicator. Each lowest-level indicator is configured with indicator description information. Taking the normative indicator as an example, the business normative indicator and the content normative indicator are configured with indicator description information respectively. In actual applications, the indicator description information corresponding to each indicator may include one or more description fields, and each description field corresponds to an SQL statement expression. For example, when the description field of an indicator includes "non-empty check", "value range check" and "uniqueness verification", the indicator is configured with three SQL statement expressions.
[0080] The SQL statement expression includes the logic code corresponding to the quality inspection purpose. For example, the SQL expression for verifying the non-null check may be similar to: sum(case when $is null then 1else 0end).
[0081] In addition, in other embodiments of the application, each rule may also include indicator weight information. The indicator weight information may include weight allocation information for indicators at all levels. Assuming that the six indicators of standardization, integrity, uniqueness, accuracy, consistency, and accessibility are taken as first-level indicators, these six first-level indicators correspond to a set of indicator weight allocation results, and the next-level indicators of each first-level indicator may also be configured with a set of indicator weight allocation results. The embodiments of the present application do not limit the weight allocation scheme, and those skilled in the art can flexibly set it according to the importance of the indicators.
[0082] In some embodiments of the present application, the step S120 of acquiring the target rule corresponding to the data information to be evaluated from the rule set according to the rule association configuration information specifically includes:
[0083] Acquire the target rule according to the rule information overview in the table-level configuration information;
[0084] According to the rule information corresponding to each field information in the field-level configuration information, the field information corresponding to the target rule is acquired.
[0085] As mentioned above, the rule information overview in the table-level configuration information is an overview of all the rules in the field-level configuration information. The rule information overview in the table-level configuration information can be used to determine which data quality indicators the data information to be evaluated needs to be based on for quality assessment. In this embodiment, each rule in the rule information overview of the table-level configuration information is used as the current target rule in turn; and then, based on the field-level configuration information, it is determined which field information corresponds to data content that needs to be quality-checked based on the current target rule.
[0086] Through this embodiment, the data content in the data table that can be quality-checked based on the same rule can be determined, and corresponding SQL statements can be generated for these data contents for quality checking. The quality check results corresponding to each rule can be obtained according to the return value of the SQL statement, which facilitates the management of the quality check results.
[0087] In some embodiments of the present application, in the above step S130, according to the quality detection strategy corresponding to the target rule, obtaining the quality detection result corresponding to each data quality indicator of the data to be evaluated specifically includes:
[0088] Generate a database query statement according to the data table information and field information of the data information to be evaluated and the database query statement expression;
[0089] According to the data table information of the data information to be evaluated and the database query statement, a quality detection result corresponding to each data quality indicator is obtained.
[0090] In the rule configuration stage of this embodiment, an SQL statement expression is configured in advance based on the indicator description information of each lowest-level indicator of each rule. The SQL statement expression already includes the logical code for quality detection. After the SQL engine obtains the data table information and field information of the data information to be evaluated and the SQL statement expression, it can automatically generate an SQL statement, and then perform quality detection by executing the SQL statement.
[0091] In practical applications, data tables of the data to be evaluated may have different types or restrictions. For example, the data tables may be snapshot tables or zipper tables. Different types of data tables may affect the query time of the data tables.
[0092] In view of this situation, in the rule association configuration information of the data quality assessment template, the data table information of its table-level configuration information includes table query conditions, and the table query time, the data query object of the data table, etc. can be determined through the table query conditions.
[0093] Therefore, in some possible implementation schemes of this embodiment, obtaining the quality detection result corresponding to each data quality indicator according to the data table information of the data information to be evaluated and the database query statement specifically includes:
[0094] Determine whether to establish a database link according to the table query condition;
[0095] When the database link is successfully established, the amount of data that complies with the corresponding rules is obtained through the return result of the database query statement;
[0096] According to the amount of data that complies with the corresponding rules and the total amount of data in the data table, a quality detection result corresponding to each data quality indicator is obtained.
[0097] As shown above, the target rule of this embodiment may correspond to multiple SQL statements. When the SQL engine executes the SQL statements, it can execute the SQL statements in parallel or in sequence. This embodiment does not impose specific restrictions on this. The return value of each SQL statement represents the amount of data in the data table that meets the corresponding indicator description field. This embodiment can record the quality detection results corresponding to each data quality indicator through a table structure.
[0098] like Figure 3As shown, the quality inspection result includes the data table targeted by the quality inspection, the rules based on which the quality inspection is based, the total amount of data for the quality inspection (the total amount of data can be all the data in the data table), and the amount of data that complies with the rules.
[0099] The amount of data that complies with the rules refers to the amount of data required by each indicator description field of each lowest-level indicator in the data table that complies with the current rules.
[0100] In some embodiments of the present application, the quality assessment strategy includes a single indicator evaluation strategy and a comprehensive indicator evaluation strategy. The single indicator evaluation strategy is used to perform quality assessment on the quality detection results corresponding to each data quality indicator, and the comprehensive indicator evaluation strategy is used to perform comprehensive quality assessment on the single evaluation results obtained based on the single indicator evaluation strategy.
[0101] In the above step S140, according to the quality assessment strategy and the quality detection results corresponding to each data quality indicator of the data to be assessed, obtaining the multi-dimensional data quality assessment results corresponding to the data to be assessed, specifically includes:
[0102] Performing standardized calculation on the quality detection results corresponding to each data quality indicator according to the single indicator evaluation strategy to obtain the single evaluation results corresponding to each data quality indicator;
[0103] Perform weighted calculation on the individual evaluation results corresponding to each data quality indicator according to the comprehensive indicator evaluation strategy to obtain the comprehensive evaluation results corresponding to all data quality indicators;
[0104] The multi-dimensional data quality assessment result corresponding to the data information to be assessed is obtained through the single assessment result and the comprehensive assessment result.
[0105] In some possible implementation schemes of this embodiment, the quality detection results corresponding to each data quality indicator are standardized and calculated according to the single indicator evaluation strategy to obtain the single evaluation results corresponding to each data quality indicator, specifically including:
[0106] Calculate the data ratio according to the amount of data that meets the rules in the quality inspection results and the total amount of quality inspection data, and obtain the data ratio result corresponding to each data quality indicator;
[0107] The data proportion results corresponding to each data quality indicator are converted into a unified scoring scale, such as a percentage system, to obtain the standardized calculation results corresponding to each data quality indicator.
[0108] When calculating the data proportion in this embodiment, the proportion calculation should be performed step by step from the lower-level indicators to the upper-level indicators. For the lowest-level indicators, it should be determined which SQL statements return values that correspond to the data volume corresponding to the lowest-level indicators based on their indicator description information. Assuming that a certain secondary indicator corresponds to three SQL statements, the return values of the three SQL statements and the total amount of data should be calculated for proportion, and the average of the proportion calculation results should be used as the data proportion result corresponding to the secondary indicator. In this way, the data proportion results corresponding to all secondary indicators under the current rules can be calculated. Then, according to the weight distribution information corresponding to the secondary indicator (if the secondary indicator is not configured with weight distribution information, the weighted value of the proportion of the secondary indicator can be used as the data proportion result of the corresponding primary indicator).
[0109] In some other possible implementation schemes of this embodiment, weighted calculation is performed on the single evaluation results corresponding to each data quality indicator according to the comprehensive indicator evaluation strategy to obtain the comprehensive evaluation results corresponding to all data quality indicators, specifically including:
[0110] The comprehensive evaluation results are obtained according to the comprehensive indicator evaluation formula.
[0111] Exemplarily, the comprehensive indicator evaluation formula is S=0.3I+0.4A+0.2C+0.1T, where I, A, C, and T respectively represent the individual evaluation results of completeness, accuracy, consistency, and timeliness, and S is the comprehensive evaluation result. The weight values 0.3, 0.4, 0.2, and 0.1 can be configured in advance.
[0112] Based on the above embodiments, it can be seen that the data quality assessment method of the embodiments of the present application has at least the following advantages:
[0113] First, the data quality assessment template and rule set are pre-configured, and there is no need to manually write SQL code directly. The configuration content covers the association between data quality indicators and indicator weight configuration, rule definition and SQL expression, and realizes the automatic mapping from rules to SQL. In addition, the configuration method of this embodiment supports fine management at the table level and field level, so that the data quality assessment solution of this application can be highly customized and easy to maintain;
[0114] Second, the data quality assessment template is automatically read and parsed through SQL parsing technology, and the corresponding database query statement is dynamically generated according to the template content. This mechanism eliminates the need to manually write and modify SQL statements, significantly improving the flexibility and execution efficiency of the data quality assessment solution of this application;
[0115] Third, the quality detection results are weightedly calculated through the configured quality assessment strategy. The quality assessment scheme of this application not only supports multiple data quality indicators, but also allows users to adjust the weight values of various data quality indicators according to actual conditions to adapt to the needs of different business scenarios. The calculation process includes result standardization, single indicator evaluation calculation and comprehensive indicator evaluation calculation, which ensures the objectivity and comprehensiveness of the quality assessment.
[0116] The present application also provides a data quality assessment device 400, such as Figure 4 As shown, a schematic diagram of the structure of a data quality assessment device in an embodiment of the present application is provided, wherein the device 400 comprises: an assessment template acquisition unit 410, a rule acquisition unit 420, a detection result acquisition unit 430 and an assessment result acquisition unit 440, wherein:
[0117] An evaluation template acquisition unit 410 is used to acquire a data quality evaluation template, wherein the data quality evaluation template includes a quality evaluation strategy and rule association configuration information configured based on the data information to be evaluated, wherein the rule association configuration information is used to match a target rule from a pre-configured rule set, wherein the rule set includes rules corresponding to different data quality indicators, and each rule includes a quality detection strategy constructed based on the rule description information;
[0118] A rule acquisition unit 420, configured to acquire a target rule corresponding to the data information to be evaluated from the rule set according to the rule association configuration information;
[0119] A detection result acquisition unit 430 is used to acquire a quality detection result corresponding to each data quality indicator of the data to be evaluated according to the quality detection strategy corresponding to the target rule;
[0120] The evaluation result acquisition unit 440 is used to acquire the multi-dimensional data quality evaluation result corresponding to the data information to be evaluated according to the quality evaluation strategy and the quality detection result corresponding to each data quality indicator of the data to be evaluated.
[0121] In some embodiments of the present application, the data information to be evaluated includes data table information and field information corresponding to the data to be evaluated, and the rule association configuration information is used to indicate the correspondence between the data information to be evaluated and the rules in the rule set, and the rule association configuration information includes at least table-level configuration information and field-level configuration information;
[0122] The table-level configuration information at least includes data table information and rule information overview;
[0123] The field-level configuration information at least includes data table information, field information, and rule information corresponding to each field information.
[0124] In some embodiments of the present application, the quality detection strategy includes a database query statement expression, and each rule also includes indicator classification information and rule description information.
[0125] In some embodiments of the present application, the rule acquisition unit 420 is specifically used to acquire the target rule according to the rule information overview in the table-level configuration information; and acquire the field information corresponding to the target rule according to the rule information corresponding to each field information in the field-level configuration information.
[0126] In some embodiments of the present application, the detection result acquisition unit 430 is specifically used to generate a database query statement based on the data table information and field information of the data information to be evaluated and the database query statement expression; and obtain the quality detection result corresponding to each data quality indicator based on the data table information of the data information to be evaluated and the database query statement.
[0127] In some embodiments of the present application, the data table information includes table query conditions, and the detection result acquisition unit 430 is also used to determine whether to establish a database link based on the table query conditions; when the database link is successfully established, the amount of data that complies with the corresponding rules is obtained through the return result of the database query statement; based on the amount of data that complies with the corresponding rules and the total amount of data in the data table, the quality detection result corresponding to each data quality indicator is obtained.
[0128] In some embodiments of the present application, the quality assessment strategy includes a single indicator assessment strategy and a comprehensive indicator assessment strategy. The assessment result acquisition unit 440 is specifically used to perform standardized calculations on the quality detection results corresponding to each data quality indicator according to the single indicator assessment strategy to obtain a single assessment result corresponding to each data quality indicator; perform weighted calculations on the single assessment results corresponding to each data quality indicator according to the comprehensive indicator assessment strategy to obtain a comprehensive assessment result corresponding to all data quality indicators; and obtain a multi-dimensional data quality assessment result corresponding to the data information to be evaluated through the single assessment results and the comprehensive assessment results.
[0129] It can be understood that the above-mentioned data quality assessment device can implement each step of the data quality assessment method provided in the above-mentioned embodiment, and the relevant explanations about the data quality assessment method are applicable to the data quality assessment device and will not be repeated here.
[0130] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 5At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.
[0131] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0132] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.
[0133] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a data quality assessment device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:
[0134] Acquire a data quality assessment template, wherein the data quality assessment template includes a quality assessment strategy and rule association configuration information configured based on the data information to be assessed, wherein the rule association configuration information is used to match a target rule from a pre-configured rule set, wherein the rule set includes rules corresponding to different data quality indicators, and each rule includes a quality detection strategy constructed based on the rule description information;
[0135] Acquire a target rule corresponding to the data information to be evaluated from the rule set according to the rule association configuration information;
[0136] According to the quality detection strategy corresponding to the target rule, obtain the quality detection result corresponding to each data quality indicator of the data to be evaluated;
[0137] According to the quality assessment strategy and the quality detection results corresponding to each data quality indicator of the data to be assessed, a multi-dimensional data quality assessment result corresponding to the data information to be assessed is obtained.
[0138] The above application Figure 1 The method performed by the data quality assessment device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in a decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above data quality assessment method in combination with its hardware.
[0139] The electronic device may also perform Figure 1 The method is performed by the data quality assessment device in Figure 1 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.
[0140] The present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by an electronic device including multiple application programs, enable the electronic device to execute Figure 1 The method performed by the data quality assessment device in the illustrated embodiment is specifically used to perform:
[0141] Acquire a data quality assessment template, wherein the data quality assessment template includes a quality assessment strategy and rule association configuration information configured based on the data information to be assessed, wherein the rule association configuration information is used to match a target rule from a pre-configured rule set, wherein the rule set includes rules corresponding to different data quality indicators, and each rule includes a quality detection strategy constructed based on the rule description information;
[0142] Acquire a target rule corresponding to the data information to be evaluated from the rule set according to the rule association configuration information;
[0143] According to the quality detection strategy corresponding to the target rule, obtain the quality detection result corresponding to each data quality indicator of the data to be evaluated;
[0144] According to the quality assessment strategy and the quality detection results corresponding to each data quality indicator of the data to be assessed, a multi-dimensional data quality assessment result corresponding to the data information to be assessed is obtained.
[0145] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0146] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0147] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0148] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0149] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0150] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0151] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0152] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0153] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0154] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims
1. A data quality assessment method, characterized in that: The data quality assessment method comprises: Acquire a data quality assessment template, wherein the data quality assessment template includes a quality assessment strategy and rule association configuration information configured based on the data information to be assessed, wherein the rule association configuration information is used to match a target rule from a pre-configured rule set, wherein the rule set includes rules corresponding to different data quality indicators, and each rule includes a quality detection strategy constructed based on the rule description information; Acquire a target rule corresponding to the data information to be evaluated from the rule set according to the rule association configuration information; According to the quality detection strategy corresponding to the target rule, obtain the quality detection result corresponding to each data quality indicator of the data to be evaluated; According to the quality assessment strategy and the quality detection results corresponding to each data quality indicator of the data to be assessed, a multi-dimensional data quality assessment result corresponding to the data information to be assessed is obtained.
2. The data quality assessment method according to claim 1, characterized in that: The data information to be evaluated includes data table information and field information corresponding to the data to be evaluated, and the rule association configuration information is used to indicate the corresponding relationship between the data information to be evaluated and the rules in the rule set, and the rule association configuration information at least includes table-level configuration information and field-level configuration information; The table-level configuration information at least includes data table information and rule information overview; The field-level configuration information at least includes data table information, field information, and rule information corresponding to each field information.
3. The data quality assessment method according to claim 2, characterized in that: The quality detection strategy includes a database query statement expression, and each rule also includes indicator classification information and rule description information.
4. The data quality assessment method according to claim 3, characterized in that: The acquiring, from the rule set according to the rule association configuration information, a target rule corresponding to the data information to be evaluated includes: Acquire the target rule according to the rule information overview in the table-level configuration information; According to the rule information corresponding to each field information in the field-level configuration information, the field information corresponding to the target rule is acquired.
5. The data quality assessment method according to claim 3, characterized in that: The step of obtaining the quality detection result corresponding to each data quality indicator of the data to be evaluated according to the quality detection strategy corresponding to the target rule includes: Generate a database query statement according to the data table information and field information of the data information to be evaluated and the database query statement expression; According to the data table information of the data information to be evaluated and the database query statement, a quality detection result corresponding to each data quality indicator is obtained.
6. The data quality assessment method according to claim 5, characterized in that: The data table information includes a table query condition, and the quality detection result corresponding to each data quality indicator is obtained according to the data table information of the data information to be evaluated and the database query statement, including: Determine whether to establish a database link according to the table query condition; When the database link is successfully established, the amount of data that complies with the corresponding rules is obtained through the return result of the database query statement; According to the amount of data that complies with the corresponding rules and the total amount of data in the data table, a quality detection result corresponding to each data quality indicator is obtained.
7. The data quality assessment method according to claim 1, characterized in that: The quality assessment strategy includes a single indicator assessment strategy and a comprehensive indicator assessment strategy. The multi-dimensional data quality assessment result corresponding to the data information to be assessed is obtained according to the quality assessment strategy and the quality detection result corresponding to each data quality indicator of the data to be assessed, including: Performing standardized calculation on the quality detection results corresponding to each data quality indicator according to the single indicator evaluation strategy to obtain the single evaluation results corresponding to each data quality indicator; Perform weighted calculation on the individual evaluation results corresponding to each data quality indicator according to the comprehensive indicator evaluation strategy to obtain the comprehensive evaluation results corresponding to all data quality indicators; The multi-dimensional data quality assessment result corresponding to the data information to be assessed is obtained through the single assessment result and the comprehensive assessment result.
8. A data quality assessment device, characterized in that: The data quality assessment device comprises: An evaluation template acquisition unit, used to acquire a data quality evaluation template, wherein the data quality evaluation template includes a quality evaluation strategy and rule association configuration information configured based on the data information to be evaluated, wherein the rule association configuration information is used to match a target rule from a pre-configured rule set, wherein the rule set includes rules corresponding to different data quality indicators, and each rule includes a quality detection strategy constructed based on the rule description information; A rule acquisition unit, configured to acquire a target rule corresponding to the data information to be evaluated from the rule set according to the rule association configuration information; A detection result acquisition unit, used to obtain a quality detection result corresponding to each data quality indicator of the data to be evaluated according to the quality detection strategy corresponding to the target rule; The evaluation result acquisition unit is used to acquire the multi-dimensional data quality evaluation result corresponding to the data information to be evaluated according to the quality evaluation strategy and the quality detection result corresponding to each data quality indicator of the data to be evaluated.
9. An electronic device, comprising: processor; as well as A memory arranged to store computer executable instructions, which, when executed, cause the processor to perform the data quality assessment method according to any one of claims 1 to 7.
10. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of application programs, enables the electronic device to execute the data quality assessment method according to any one of claims 1 to 7.
Citation Information
Cited By
Data evaluation method and device for building simulation data, equipment and medium
CN120596914A
Data set quality evaluation method and device based on knowledge graph, and electronic equipment
CN120974130A
