Data quality evaluation method and device, equipment and storage medium

Abstract

The invention relates to the technical field of data processing, and provides a data quality evaluation method, device and equipment and a storage medium, and the data quality evaluation method comprises the steps: obtaining technical metadata, carrying out the digital processing of the technical metadata from a plurality of evaluation dimensions, and obtaining the evaluation processing data corresponding to each evaluation dimension; calculating a data quality standard value corresponding to each evaluation dimension based on the evaluation processing data; and performing data quality evaluation based on the data quality standard value. Through the technical scheme provided by the invention, data quality evaluation on different dimensions is realized.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data quality assessment method, apparatus, device, and storage medium. Background Technology

[0002] Data quality management covers the entire data lifecycle and requires different quality control measures to be implemented at different stages to ensure that data quality is guaranteed in each processing step, thereby achieving the goal of end-to-end data quality management.

[0003] Existing data quality audits compare, analyze, and issue warnings on a single dimension, resulting in poor effectiveness of data quality audits across different dimensions. Summary of the Invention

[0004] This invention provides a data quality assessment method, apparatus, device, and storage medium to address the deficiencies in the prior art.

[0005] This invention provides a data quality assessment method, comprising: Acquire technical metadata, and digitize the technical metadata from multiple evaluation dimensions to obtain evaluation processing data corresponding to each evaluation dimension; Based on the evaluation data, calculate the data quality specification value corresponding to each of the evaluation dimensions; Data quality assessment is performed based on the aforementioned data quality specification values.

[0006] According to a data quality assessment method provided by the present invention, the step of digitizing the technical metadata from multiple assessment dimensions to obtain assessment processing data corresponding to each assessment dimension includes: The technical metadata is preprocessed from multiple evaluation dimensions to obtain dimension processing data corresponding to each evaluation dimension; wherein, the dimension processing data includes multiple corresponding specification items; Obtain the standard requirements conditions corresponding to each of the standard items, and determine the detection identifier of each of the standard items based on the standard items and the standard requirements conditions; Based on the detection identifier, the evaluation processing data corresponding to each of the evaluation dimensions is obtained.

[0007] According to a data quality assessment method provided by the present invention, the step of calculating the data quality specification value corresponding to each assessment dimension based on the assessment processed data includes: The presence of target elements is detected in each of the evaluation and processing data, and the corresponding detection results are obtained. Based on the detection results, the evaluation processing data containing the target element is used as anomaly evaluation processing data; The number of target elements in the anomaly assessment and processing data is determined, and the anomaly coefficient corresponding to each anomaly assessment and processing data is calculated based on the number of target elements. Obtain the quality classification value corresponding to each of the aforementioned evaluation dimensions; Based on the anomaly coefficient and the quality classification value, calculate the data quality specification value corresponding to each of the evaluation dimensions.

[0008] According to a data quality assessment method provided by the present invention, the data quality assessment based on the data quality specification value includes: If it is determined based on the data quality specification value that there is target technology metadata that requires local quality assessment, calculate the local audit status value corresponding to the target technology metadata; A local data quality assessment is performed based on the aforementioned local audit status values.

[0009] According to a data quality assessment method provided by the present invention, the step of calculating the local audit status value corresponding to the target technical metadata includes: Obtain the historical evaluation count, historical anomaly count, and historical data quality specification value of the target technology metadata in the target evaluation dimension; The local audit status value of the target evaluation dimension is calculated based on the historical evaluation frequency, historical anomaly frequency, and historical data quality specification value of the target evaluation dimension.

[0010] According to a data quality assessment method provided by the present invention, the step of performing a local data quality assessment based on the local audit status value includes: If the local audit status value is greater than the local audit status standard value, the target technical metadata will be subject to data quality optimization management in the target evaluation dimension.

[0011] According to a data quality assessment method provided by the present invention, the assessment dimensions include at least two of the following dimensions: consistency, integrity, compliance, timeliness, and lineage.

[0012] The present invention also provides a data quality assessment device, comprising: The digital processing module is configured to acquire technical metadata and perform digital processing on the technical metadata from multiple evaluation dimensions to obtain evaluation processing data corresponding to each evaluation dimension. The calculation module is configured to calculate the data quality specification value corresponding to each of the evaluation dimensions based on the evaluation processing data. The evaluation module is configured to perform data quality evaluation based on the data quality specification values.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the data quality assessment method as described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data quality assessment method as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the data quality assessment method as described above.

[0016] The data quality assessment method, apparatus, device, and storage medium provided by this invention perform standardization checks on technical metadata from multiple assessment dimensions to obtain assessment processing data corresponding to each assessment dimension. Based on the assessment processing data, a data quality standardization value corresponding to each assessment dimension is calculated. The data quality standardization value characterizes the overall standardization level of the technical metadata. Data quality assessment is performed on the technical metadata based on the calculated data quality standardization value, realizing data quality assessment on different dimensions. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the data quality assessment method provided by the present invention.

[0019] Figure 2 This is a schematic diagram of the data quality assessment device provided by the present invention.

[0020] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0022] Figure 1 This is a flowchart illustrating a data quality assessment method according to an exemplary embodiment. For example... Figure 1 As shown in an exemplary embodiment, the data quality assessment method includes steps 110 to 130, which are described in detail below.

[0023] Step 110: Obtain technical metadata, and digitize the technical metadata from multiple evaluation dimensions to obtain evaluation processing data corresponding to each evaluation dimension.

[0024] In this embodiment of the invention, the technical metadata includes characteristic descriptions of data structure and data processing, covering all data processing stages such as data platform data source interfaces, data computation, and data sharing. The technical metadata is digitized from multiple evaluation dimensions to obtain evaluation processing data corresponding to each evaluation dimension, facilitating subsequent evaluation of data quality.

[0025] Step 120: Based on the evaluation processing data, calculate the data quality specification value corresponding to each of the evaluation dimensions.

[0026] In this embodiment of the invention, data quality specification values ​​corresponding to each evaluation dimension are calculated based on the evaluation processing data. The data quality specification values ​​characterize the overall standardization level of the technical metadata.

[0027] Step 130: Perform a data quality assessment based on the data quality specification values.

[0028] In this embodiment of the invention, the technical metadata is evaluated for data quality based on the calculated data quality specification value.

[0029] In an exemplary embodiment of the present invention, the evaluation dimensions include at least two of the following dimensions: consistency dimension, integrity dimension, compliance dimension, timeliness dimension, and lineage dimension.

[0030] In this embodiment of the invention, consistency refers to following unified data standards in recording and transmitting data and information; following unified enterprise data specifications, the corresponding information attributes of the same information subject are the same in different systems; and information subjects with the same business definition should, in principle, have the same value across professional systems, and the reasons for differences should be explainable and traceable.

[0031] Integrity refers to ensuring that no data information is missing during the collection, processing, and sharing process, including: no missing data files, no missing data tables, and no missing data fields.

[0032] Compliance refers to the rules that data should follow during modeling and development to avoid ambiguity, including but not limited to naming conventions, metadata descriptions, and reasonable ranges of data values.

[0033] Timeliness refers to the fact that the data transmission frequency and transmission delay meet the real-time requirements of the relevant system.

[0034] Lineage refers to whether the source table corresponding to a non-original layer table is empty, and whether it exists.

[0035] The technical metadata is subjected to corresponding standardization checks based on at least two of the following dimensions: consistency, integrity, compliance, timeliness, and lineage.

[0036] In an exemplary embodiment of the present invention, the step of digitizing the technical metadata from multiple evaluation dimensions to obtain evaluation processing data corresponding to each evaluation dimension includes: The technical metadata is preprocessed from multiple evaluation dimensions to obtain dimension processing data corresponding to each evaluation dimension; wherein, the dimension processing data includes multiple corresponding specification items; Obtain the standard requirements conditions corresponding to each of the standard items, and determine the detection identifier of each of the standard items based on the standard items and the standard requirements conditions; Based on the detection identifier, the evaluation processing data corresponding to each of the evaluation dimensions is obtained.

[0037] In this embodiment of the invention, the acquired technical metadata is preprocessed to obtain a target processing dataset, and the target processing dataset is input into a data quality identification model for multi-dimensional norm detection. The norm detection results of different dimensions are combined to obtain a data quality analysis set.

[0038] Specifically, technical metadata undergoes preprocessing, which includes at least two of the following: consistency processing, integrity processing, compliance processing, timeliness processing, and lineage processing. Preprocessing corresponds to the evaluation dimensions. Preprocessing is implemented based on existing data processing rules to obtain the corresponding dimension-processed data, including consistency-processed data, integrity-processed data, compliance-processed data, timeliness-processed data, and lineage-processed data.

[0039] Each dimension of data processing includes multiple specification items, namely, consistent data processing, complete data processing, compliant data processing, timely data processing, and lineage-processed data processing, each containing a corresponding number of consistent specification items, complete specification items, compliant specification items, timely specification items, and lineage specification items, as shown in Table 1 below: Table 1 It is important to note that the various specification items included in the data processing for different dimensions can be added, deleted, or modified according to the actual needs of the application scenario.

[0040] Different dimensions of data processing are labeled C. i Let i represent the i-th dimension of data processing, i = 1, 2, 3, 4, 5; label the normalized terms in the data processing for different dimensions as C. ij , j represents the j-th specification term, j=1,2,3,...,m; m is a positive integer.

[0041] The target processing dataset is composed of labeled consistent processing data, complete processing data, compliant processing data, timely processing data, and lineage processing data.

[0042] In this embodiment of the invention, the target processing dataset is obtained by preprocessing the technical metadata, which can provide reliable multi-dimensional local data support for subsequent multi-dimensional audit data digitization processing and analysis.

[0043] The target processing dataset is input into the data quality identification model for multi-dimensional compliance detection. Consistent processing data, complete processing data, compliant processing data, timely processing data, and lineage processing data in the target processing dataset are matched and digitized through the consistency processing module, complete processing module, compliant processing module, timely processing module, and lineage processing module in the data quality identification model.

[0044] Specifically, the data quality identification model includes a consistency processing module, a completeness processing module, a compliance processing module, a timely processing module, and a lineage processing module. Different processing modules contain pre-stored normative requirements corresponding to several normative items.

[0045] When different processing modules perform data matching on consistent data, complete data, compliant data, timely data, and lineage data, they use a standard item identification segmentation function to perform data analysis on several standard items of data processed in different dimensions and output the corresponding detection identifiers. The expression for the piecewise function that identifies the canonical term is: ; In the above formula, The normative requirements for different normative items are determined based on the existing normative requirements for those items. The detection identifier can be represented by a value of 0 or 1, indicating whether the corresponding specification item meets or does not meet the corresponding specification requirements. In other embodiments of the present invention, the detection identifier may also be represented by other distinguishable characters.

[0046] It should be noted that by using the segmented function for identifying standard items to perform data analysis and digital representation of the standard status of different standard items in data processed from different dimensions, it is possible to not only digitally represent different standard items, but also provide reliable data item analysis support for subsequent local and overall audit status analysis of data processed from different dimensions. Compared with the existing technical solutions that only focus on the comparison analysis and early warning of standard items, the embodiments of this invention can achieve the overall effect of standard item processing analysis and extended utilization.

[0047] All detection identifiers belonging to the same dimension of data processing are sorted and combined to obtain the evaluation processing data of the corresponding dimension of data processing. The evaluation processing data corresponds to the corresponding evaluation dimension. The evaluation processing data includes consistent processing sequence, complete processing sequence, compliant processing sequence, timely processing sequence, and lineage processing sequence. Several processing sequences are combined to form a data quality analysis set.

[0048] In this embodiment of the invention, by digitizing and combining different specification items of data processed in different dimensions to obtain the corresponding processing sequence, reliable local processing data support can be provided for the overall status analysis of data quality specifications corresponding to subsequent technical metadata and the local data quality specification evaluation of different audit dimensions, thereby improving the depth and breadth of target technical metadata processing and extended utilization.

[0049] In an exemplary embodiment of the present invention, the generation path, generation rule basis, modification evolution of each data table are recorded, and an inference chain is formed, which is then associated with the corresponding data table.

[0050] In this embodiment of the invention, to ensure the traceability of data tables, an inference chain structure is constructed during the generation process of each data table. This inference chain records the entire process of generating a data table from its initial source table, and each data table has a corresponding inference chain. This inference chain is displayed to the user in a visual format on the terminal. Based on the inference chain, the generation process of each data table can be understood, thus clearly determining the data table's lineage. Through the inference chain, the existence of the source table for a given data table can be quickly determined; if it exists, it is checked whether the source table is empty.

[0051] In an exemplary embodiment of the present invention, calculating the data quality specification value corresponding to each of the evaluation dimensions based on the evaluation processing data includes: The presence of target elements is detected in each of the evaluation and processing data, and the corresponding detection results are obtained. Based on the detection results, the evaluation processing data containing the target element is used as anomaly evaluation processing data; The number of target elements in the anomaly assessment and processing data is determined, and the anomaly coefficient corresponding to each anomaly assessment and processing data is calculated based on the number of target elements. Obtain the quality classification value corresponding to each of the aforementioned evaluation dimensions; Based on the anomaly coefficient and the quality classification value, calculate the data quality specification value corresponding to each of the evaluation dimensions.

[0052] In this embodiment of the invention, the analysis results of different evaluation dimensions in the data quality analysis set are digitally processed and calculated to obtain the corresponding data quality specification values. Based on the data quality specification values, the overall data quality specification status corresponding to the relevant technical metadata is determined.

[0053] Specifically, several processing sequences are obtained from the data quality analysis set, and the elements of different processing sequences are traversed, analyzed, and dynamically labeled sequentially. The presence of target elements in each processing sequence is detected. Target elements are indicators that indicate that a specification item in the evaluation processing data does not meet the corresponding specification requirements. For example, if the target element is set to 1, and there are no elements with a value of 1 in the processing sequence, a local normal label is generated, and the corresponding evaluation processing data is marked as normal evaluation processing data. If there are elements with a value of 1 in the processing sequence, a local abnormal label is generated, and the corresponding evaluation processing data is marked as abnormal evaluation processing data. The number m of target elements in the abnormal evaluation processing data is then obtained. i The number of elements M i Through the formula The anomaly coefficients corresponding to each anomaly assessment and processing data were calculated. .

[0054] Count the number N of outlier data points in each evaluation dimension, and obtain the quality classification value corresponding to each evaluation dimension. The data quality specification values ​​for each evaluation dimension are calculated using the data quality specification identification function. ; The expression for the data quality specification identification function is as follows: ; The quality classification value can be determined based on the median of the data quality specification value corresponding to the technical metadata in the historical data quality assessment process.

[0055] It should be explained that the data quality specification value is used to integrate and calculate the analysis results of different evaluation dimensions to digitally represent the overall status of the data quality specification of the technical metadata; the larger the data quality specification value, the worse the overall status of the data quality specification of the corresponding technical metadata.

[0056] In this embodiment of the invention, the corresponding data quality specification value is obtained by integrating and calculating the evaluation processing data of different evaluation dimensions. The overall status of the data quality specification corresponding to the technical metadata is determined by data analysis of the data quality specification value. At the same time, it can also provide a reliable evaluation basis for the subsequent local data quality specification evaluation of technical metadata under different evaluation dimensions, thereby improving the overall effect of different evaluation dimensions in the analysis, processing and integration of data.

[0057] In an exemplary embodiment of the present invention, the data quality assessment based on the data quality specification value includes: If it is determined based on the data quality specification value that there is target technology metadata that requires local quality assessment, calculate the local audit status value corresponding to the target technology metadata; A local data quality assessment is performed based on the aforementioned local audit status values.

[0058] In this embodiment of the invention, the overall data quality specification status of the technical metadata is determined based on the data quality specification value. Specifically, the value of the data quality specification value is analyzed. If the value of the data quality specification value is 0, the overall data quality specification status of the technical metadata in the corresponding evaluation dimension is marked as normal. If the value of the data quality specification value is 1, the overall data quality specification status of the technical metadata in the corresponding evaluation dimension is marked as slightly abnormal. If the value of the data quality specification value is greater than 1, the overall data quality specification status of the technical metadata in the corresponding evaluation dimension is marked as severely abnormal.

[0059] When the overall data quality specification status is either slightly abnormal or severely abnormal (i.e., the data quality specification value is greater than or equal to 0), it is determined that target technical metadata requires local quality assessment. The technical metadata corresponding to the data quality specification value greater than or equal to 0 is taken as the target technical metadata. Simultaneously, the assessment dimension corresponding to the data quality specification value greater than or equal to 0 is taken as the target assessment dimension. The local audit status value corresponding to the target technical metadata is calculated, and then a local data quality assessment is performed based on the local audit status value.

[0060] In an exemplary embodiment of the present invention, calculating the local audit status value corresponding to the target technology metadata includes: Obtain the historical evaluation count, historical anomaly count, and historical data quality specification value of the target technology metadata in the target evaluation dimension; The local audit status value of the target evaluation dimension is calculated based on the historical evaluation frequency, historical anomaly frequency, and historical data quality specification value of the target evaluation dimension.

[0061] In this embodiment of the invention, the historical number of evaluations of the target technology metadata in the target evaluation dimension is obtained. Number of historical anomalies Historical data quality specification values. Number of historical assessments. This refers to the number of times the target technology metadata has undergone data quality assessment. (Number of historical anomalies) This refers to the number of times anomalies were identified based on data quality specification values ​​during the data quality assessment of the target technology metadata. Specifically, it represents the number of times the overall data quality specification status was determined to be either slightly or severely abnormal. Historical data quality specification values ​​are the data quality specification values ​​calculated by the target technology metadata during the data quality assessment process.

[0062] Through local evaluation formula Calculate the corresponding local audit status value ;in, This represents the historical data quality specification value.

[0063] In an exemplary embodiment of the present invention, the step of performing a local data quality assessment based on the local audit status value includes: If the local audit status value is greater than the local audit status standard value, the target technical metadata will be subject to data quality optimization management in the target evaluation dimension.

[0064] In this embodiment of the invention, the local data quality specifications of the corresponding target evaluation dimension are evaluated based on the calculated local audit status value. The local audit status value is compared with the local audit status standard value. The local audit status standard value can be determined based on the median of the historical local audit status values ​​of the corresponding target evaluation dimension in the historical evaluation process.

[0065] If the local audit status value is less than or equal to the local audit status standard value, a local audit specification instruction will be generated, and the corresponding target evaluation dimension will be marked as a normal evaluation dimension. If the local audit status value is greater than the local audit status standard value, a local audit non-standard instruction will be generated, and the corresponding target evaluation dimension will be marked as an abnormal evaluation dimension. Targeted data quality optimization management will be implemented on the target technical metadata in the target evaluation dimension.

[0066] This includes optimizing data quality management, including but not limited to providing regular training to relevant personnel on data management and anomaly detection skills to improve the team's data sensitivity and processing capabilities.

[0067] In another embodiment of the present invention, after implementing targeted data quality optimization management on the target technology metadata in the target evaluation dimension, the method further includes: Random numbers are generated using a random number generation algorithm; The target data quality specification value is determined from the data quality specification values ​​of each evaluation dimension of the technical metadata after data quality optimization management, based on the random number. The dimensions to be optimized are determined based on the target data quality specification value, and an optimization scheme is generated based on the dimensions to be optimized.

[0068] In this embodiment of the invention, after multiple data quality optimization management processes, a random number is generated according to a random algorithm. The generated random number is greater than 2 and less than the natural number of times data quality optimization management has been performed. A random number of data quality specification values ​​are randomly selected from the data quality specification values ​​of each evaluation dimension of the technical metadata after data quality optimization management as target data quality specification values. The target data quality specification values ​​for each evaluation dimension are sorted in chronological order. Based on the sorting, it is calculated whether the latter target data quality specification value is greater than or equal to the former target data quality specification value between two adjacent target data quality specification values. The number of times the latter target data quality specification value is greater than or equal to the former target data quality specification value is counted, and it is checked whether this number is less than the corresponding threshold. If it is less, the corresponding evaluation dimension is determined as the dimension to be optimized, and an optimization scheme corresponding to the dimension to be optimized is generated. The generated optimization scheme further improves the team's data sensitivity and processing capabilities.

[0069] In this embodiment of the invention, by digitizing and combining different specification items of technical metadata under different evaluation dimensions to obtain the corresponding processing sequence, reliable local processing data support can be provided for the overall status analysis of data quality specifications corresponding to technical metadata and the local data quality specification evaluation of different audit dimensions, thereby improving the depth and breadth of technical metadata processing and extended utilization.

[0070] In this embodiment of the invention, by integrating and calculating the dimensional processing data of all previous evaluation dimensions, a data quality specification value is obtained. Based on the data quality specification value, a local data quality assessment of the target technical metadata of the corresponding evaluation dimension is carried out. This improves the adaptability and flexibility of different evaluation dimensions in the local data quality assessment, so that targeted optimization management of abnormal evaluation dimensions can be carried out in a timely and efficient manner, thereby improving the proactive monitoring and proactive processing effect of abnormal evaluation dimensions.

[0071] The data quality assessment apparatus provided by this invention will be described below. The data quality assessment apparatus described below can be referred to in correspondence with the data quality assessment method described above. It should be noted that the apparatus provided in the embodiments below and the method provided in the embodiments above belong to the same concept, and the specific way in which each module and unit performs its operation has been described in detail in the method embodiments, and will not be repeated here.

[0072] In one exemplary embodiment of the present invention, please refer to Figure 2 , Figure 2 This is a data quality assessment apparatus according to an exemplary embodiment, comprising the following modules.

[0073] The digitization processing module 210 is configured to acquire technical metadata and perform digitization processing on the technical metadata from multiple evaluation dimensions to obtain evaluation processing data corresponding to each evaluation dimension. The calculation module 220 is configured to calculate the data quality specification value corresponding to each of the evaluation dimensions based on the evaluation processing data. The evaluation module 230 is configured to perform data quality evaluation based on the data quality specification value.

[0074] In an exemplary embodiment of the present invention, the digitization processing module 210 includes: The preprocessing submodule is configured to preprocess the technical metadata from multiple evaluation dimensions to obtain dimension processing data corresponding to each evaluation dimension; wherein, the dimension processing data includes multiple corresponding specification items; The first determining submodule is configured to obtain the standard requirements conditions corresponding to each of the standard items, and determine the detection identifier of each of the standard items based on the standard items and the standard requirements conditions; The numerical submodule is configured to obtain the evaluation processing data corresponding to each of the evaluation dimensions based on the detection identifier.

[0075] In an exemplary embodiment of the present invention, the computing module 220 includes: The detection submodule is configured to detect whether a target element exists in each of the evaluation and processing data, and obtain the corresponding detection result; As a submodule, it is configured to treat the evaluation processing data containing the target element as anomaly evaluation processing data based on the detection results; The second determining submodule is configured to determine the number of target elements in the anomaly assessment and processing data, and calculate the anomaly coefficient corresponding to each anomaly assessment and processing data based on the number of target elements. The acquisition submodule is configured to acquire the quality classification values ​​corresponding to each of the evaluation dimensions. The first calculation submodule is configured to calculate the data quality specification value corresponding to each of the evaluation dimensions based on the anomaly coefficient and the quality classification value.

[0076] In an exemplary embodiment of the present invention, the evaluation module 230 includes: The second calculation submodule is configured to calculate the local audit status value corresponding to the target technology metadata if it is determined based on the data quality specification value that there is target technology metadata that needs to be locally evaluated for quality. The local evaluation submodule is configured to perform local data quality evaluation based on the local audit status value.

[0077] In one exemplary embodiment of the present invention, the second computing submodule includes: The acquisition unit is configured to acquire the target technology metadata in terms of the number of historical assessments, the number of historical anomalies, and the historical data quality specification value in the target assessment dimension; The calculation unit is configured to calculate the local audit status value of the target evaluation dimension based on the historical evaluation count, historical anomaly count, and historical data quality specification value of the target evaluation dimension.

[0078] In an exemplary embodiment of the present invention, the local evaluation submodule includes: The optimization management unit is configured to perform data quality optimization management on the target technical metadata in the target evaluation dimension if the local audit status value is greater than the local audit status standard value.

[0079] In an exemplary embodiment of the present invention, the evaluation dimensions include at least two of the following dimensions: consistency dimension, integrity dimension, compliance dimension, timeliness dimension, and lineage dimension.

[0080] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3 As shown, the electronic device may include a processor 310, a communications interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communications interface 320, and the memory 330 communicate with each other via the communication bus 340. The processor 310 can invoke logical instructions stored in the memory 330 to execute a data quality assessment method, which includes: acquiring technical metadata, digitizing the technical metadata from multiple assessment dimensions, and obtaining assessment processing data corresponding to each assessment dimension. Based on the evaluation data, calculate the data quality specification value corresponding to each of the evaluation dimensions; Data quality assessment is performed based on the aforementioned data quality specification values.

[0081] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0082] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, the computer program being executed by a processor, the computer being able to execute the data quality assessment method provided by the above methods, the method including: acquiring technical metadata, digitizing the technical metadata from multiple assessment dimensions respectively, and obtaining assessment processing data corresponding to each assessment dimension; Based on the evaluation data, calculate the data quality specification value corresponding to each of the evaluation dimensions; Data quality assessment is performed based on the aforementioned data quality specification values.

[0083] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data quality assessment method provided by the above methods, the method comprising: acquiring technical metadata, and digitizing the technical metadata from multiple assessment dimensions to obtain assessment processing data corresponding to each assessment dimension; Based on the evaluation data, calculate the data quality specification value corresponding to each of the evaluation dimensions; Data quality assessment is performed based on the aforementioned data quality specification values.

[0084] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0085] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method of data quality assessment, characterized by, The method comprises the following steps: obtaining technical metadata, and respectively digitally processing the technical metadata from multiple evaluation dimensions to obtain evaluation processing data corresponding to each evaluation dimension; based on the evaluation processing data, calculating data quality specification values corresponding to each evaluation dimension; based on the data quality specification values, performing data quality evaluation.

2. The data quality assessment method of claim 1, wherein, The digital processing of the technical metadata from multiple evaluation dimensions to obtain evaluation processing data corresponding to each evaluation dimension comprises: respectively pre-processing the technical metadata from multiple evaluation dimensions to obtain dimension processing data corresponding to each evaluation dimension; wherein the dimension processing data comprises multiple specification items corresponding thereto; obtaining specification requirement conditions corresponding to each specification item, and determining the detection identifier of each specification item according to the specification item and the specification requirement condition; obtaining the evaluation processing data corresponding to each evaluation dimension according to the detection identifier.

3. The data quality assessment method of claim 1, wherein, The calculation of the data quality specification values corresponding to each evaluation dimension based on the evaluation processing data comprises: detecting whether there is a target element in each evaluation processing data to obtain a corresponding detection result; based on the detection result, regarding the evaluation processing data containing the target element as abnormal evaluation processing data; determining the number of target elements in the abnormal evaluation processing data, and calculating an abnormal coefficient corresponding to each abnormal evaluation processing data based on the number of target elements; obtaining quality classification values corresponding to each evaluation dimension; calculating the data quality specification values corresponding to each evaluation dimension according to the abnormal coefficient and the quality classification value.

4. The data quality assessment method of claim 1, wherein, The data quality evaluation based on the data quality specification values comprises: if it is determined based on the data quality specification values that there is target technical metadata that needs to be evaluated locally, calculating a local audit state value corresponding to the target technical metadata; based on the local audit state value, performing data quality local evaluation.

5. The data quality assessment method of claim 4, wherein, The calculation of the local audit state value corresponding to the target technical metadata comprises: obtaining the historical evaluation times, historical abnormal times and historical data quality specification values of the target evaluation dimension of the target technical metadata; calculating the local audit state value of the target evaluation dimension according to the historical evaluation times, historical abnormal times and historical data quality specification values of the target evaluation dimension.

6. The data quality assessment method of claim 4, wherein, The data quality local evaluation based on the local audit state value comprises: if the local audit state value is greater than a local audit state standard value, performing data quality optimization management on the target technical metadata in the target evaluation dimension.

7. The data quality evaluation method according to any one of claims 1 to 6, characterized in that, The evaluation dimensions include at least two of consistency dimension, integrity dimension, compliance dimension, timeliness dimension and blood relationship dimension.

8. A data quality assessment apparatus, characterized by comprising: The method comprises the following steps: a digital processing module configured to obtain technical metadata, and respectively digitally process the technical metadata from multiple evaluation dimensions to obtain evaluation processing data corresponding to each evaluation dimension; a calculation module configured to calculate data quality specification values corresponding to each evaluation dimension based on the evaluation processing data; an evaluation module configured to perform a data quality evaluation based on the data quality specification value.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor implements the data quality evaluation method according to any one of claims 1 to 7 when executing the computer program.

10. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, implements the data quality evaluation method according to any one of claims 1 to 7.

Images