Data resource directory guarantee method based on data quality monitoring

Abstract

The invention discloses a data resource directory guarantee method based on data quality monitoring. The method comprises the steps that S1, a data resource directory compilation standard is arranged; s2, designing a data resource directory quality evaluation system; s3, metadata information and data information of the data resource catalog are collected, and use information of the data resource catalog is collected; s4, executing quality evaluation of the data resource directory, and establishing a monitoring rule according to the metadata information and the data information of the data resource directory; s5, labeling data resource directories which do not meet requirements, and performing quality problem analysis and correction measures according to a monitoring result; and S6, comprehensively evaluating the quality of the data resource directory to form a data resource directory quality report. According to the method, normative, standard and accurate data resource directories are guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data resource directory assurance method based on data quality monitoring. Background Art

[0002] A data resource catalog organizes data resources according to specific categories and directory structures. It helps organizations better manage and organize their data resources, playing a vital role in data management, organization, retrieval, sharing, collaboration, security, access control, governance, and compliance. The compilation of a data resource catalog requires clear management mechanisms and the establishment of unified standards to ensure its quality and subsequent dynamic maintenance and updates. Initially, due to the diverse and volatile nature of organizational data, the compilation of data resource catalogs was plagued by inconsistent standards, inadequate updates, and inaccurate catalog elements, severely hindering the sharing and application of data resources.

[0003] Therefore, a data resource directory assurance method based on data quality monitoring is provided. Summary of the Invention

[0004] The purpose of the present invention is to overcome the existing defects and provide a data resource directory assurance method based on data quality monitoring, which uses data quality monitoring to evaluate the data resource directory and ensure the data resource directory is standardized, standard and accurate.

[0005] The technical solution to achieve the above purpose is: A data resource directory assurance method based on data quality monitoring, comprising: Step S1, sorting out the data resource catalog compilation standards; Step S2, designing a data resource catalog quality assessment system; Step S3, collecting metadata information and data information of the data resource directory and collecting usage information of the data resource directory; Step S4: perform data resource directory quality assessment and establish monitoring rules based on the metadata information and data information of the data resource directory; Step S5: Mark the data resource directory that does not meet the requirements, and perform quality problem analysis and corrective measures based on the monitoring results; Step S6: Comprehensively evaluate the quality of the data resource directory and generate a data resource directory quality report.

[0006] Preferably, in step S1, the data resource directory compilation standard stipulates the core metadata of the data resource directory and the corresponding filling specifications, specifically including: Data resource catalog classification: required, indicating the classification information of data resources; Data resource directory name: required, a title that describes the content of the data resource; Data resource directory code: required, a unique and unchanging identification code for the data resource; Data resource provider: required, the department providing the data resources; Data resource provider code: required, the code of the department providing the data resource; Data resource directory summary: required, a brief description of the data resource content; Data resource format: required, the form in which the data resource exists; Information item information: optional, detailed description of the structured data resource, including the information item name and data type; Sharing attributes: Description of the data resource sharing type and conditions, including sharing type, sharing conditions, and sharing method; Open attributes: Required, a description of whether the data resource is open to the public and the conditions for opening it, including whether it is open to the public and the conditions for opening it; Update cycle: required, the frequency of data resource updates, which can be divided into real-time, daily, weekly, monthly, quarterly, and annual; Related data resource catalog code: Optional. If the data resource belongs to other categories, the code of other resource categories or other data resources is required.

[0007] Preferably, in step S2, the evaluation indicators include quantitative indicators and qualitative indicators, wherein: Quantitative indicators include: data accuracy , data integrity , data consistency , data standardization , data timeliness and data availability ; Qualitative indicators include: rationality of classification , description clarity , semantic accuracy Scalability and user experience .

[0008] Preferably, in step S3, metadata information of the data resource directory is collected through an automated tool, including but not limited to physical table information and field information.

[0009] Preferably, in step S3, usage information of the data resource directory is collected through a questionnaire or a survey form.

[0010] Preferably, in step S4, quality assessment is performed according to the quantitative indicators and qualitative indicators set by the assessment system.

[0011] Preferably, in step S5, records that do not comply with quality rules are marked according to data quality monitoring rules to form a data resource directory monitoring result, that is, a list of problematic data in the data resource directory.

[0012] Preferably, in step S6, comprehensively evaluating the quality of the data resource directory includes: Step S61: Calculate quantitative indicators based on the data resource catalog quality assessment system The value of Step S62: Calculate qualitative indicators based on the data resource catalog quality assessment system The value of Step S63: Set the weight of each quantitative indicator and the weights of qualitative indicators ; Step S64: Calculate the comprehensive evaluation score based on the indicators and weights ; The calculation formula is as follows: .

[0013] The beneficial effects of the present invention are: the present invention can monitor the data resource directory at irregular intervals, discover problems with the data resource directory in a timely manner, and continuously ensure the quality of the data resource directory; provide assistance for data resource rectification based on the quality analysis report of the data resource directory, and gradually improve the quality of the data resource directory; construct a quality monitoring model for multiple data resource directories, and the data quality monitoring rules constructed based on the model can promptly discover problems such as non-standard, incomplete, inconsistent, and repeated compilation of the data resource directory, thereby greatly improving the accuracy, completeness, standardization, and timeliness of the data resource directory. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a flow chart of a data resource directory assurance method based on data quality monitoring of the present invention; Figure 2 This is a schematic diagram of the data resource directory quality assessment system in the present invention; Figure 3 It is a flow chart for comprehensively evaluating the quality of a data resource directory in the present invention. DETAILED DESCRIPTION

[0015] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. In the description of the present invention, it should be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0016] The present invention will be further described below with reference to the accompanying drawings.

[0017] like Figure 1 As shown, a data resource directory assurance method based on data quality monitoring includes: Step S1: Arrange the data resource directory compilation standards.

[0018] In the embodiment, the data resource directory compilation standard stipulates the core metadata of the data resource directory and the corresponding filling specifications, specifically including: Data resource catalog classification: required, indicating the classification information of data resources; Data resource directory name: required, a title that describes the content of the data resource; Data resource directory code: required, a unique and unchanging identification code for the data resource; Data resource provider: required, the department providing the data resources; Data resource provider code: required, the code of the department providing the data resource; Data resource directory summary: required, a brief description of the data resource content; Data resource format: required, the form in which the data resource exists; Information item information: optional, detailed description of the structured data resource, including the information item name and data type; Sharing attributes: Description of the data resource sharing type and conditions, including sharing type, sharing conditions, and sharing method; Open attributes: Required, a description of whether the data resource is open to the public and the conditions for opening it, including whether it is open to the public and the conditions for opening it; Update cycle: required, the frequency of data resource updates, which can be divided into real-time, daily, weekly, monthly, quarterly, and annual; Related data resource catalog code: Optional. If the data resource belongs to other categories, the code of other resource categories or other data resources is required.

[0019] Step S2: Design a data resource catalog quality assessment system.

[0020] In the embodiment, the evaluation indicators include quantitative indicators and qualitative indicators, wherein: Quantitative indicators include: data accuracy , data integrity , data consistency , data standardization , data timeliness and data availability ; Qualitative indicators include: rationality of classification , description clarity , semantic accuracy Scalability and user experience .

[0021] like Figure 2 As shown: Data Accuracy ( ), refers to the proportion of accurate data in the data resource directory.

[0022] Evaluation Methodology: Data accuracy = (number of accurate data entries / total number of data entries) × 100%. For example, in a resource directory containing 1,000 data items, if the classification, naming, metadata, and other information of 980 data items are verified to be accurate, the data accuracy is (980 / 1,000) × 100% = 98%.

[0023] Data integrity ( ), an indicator to measure the completeness of metadata and key information in cataloging data.

[0024] Evaluation Methodology: Data completeness = (number of records with complete metadata / total number of records) × 100%. Assume there are 500 data records, of which 450 have all required metadata fields completed. The data completeness is (450 / 500) × 100% = 90%.

[0025] Data consistency ( ), refers to the degree to which cataloged data remains consistent in terms of format, coding, naming conventions, data logic, etc.

[0026] Evaluation Methodology: This can be calculated by checking the number of records that violate consistency rules. Data consistency = (Total number of records - Number of inconsistent records) / Total number of records × 100%. For example, if 30 records are found to have format or encoding inconsistencies, and there are 1000 records in total, the data consistency is (1000 - 30) / 1000 × 100% = 97%.

[0027] Data standardization ( ), which is used to assess the extent to which catalogued data adheres to established cataloguing rules and standards.

[0028] Evaluation Methodology: Data standardization = (number of data items that meet the standard / total number of data items) × 100%. If, according to the cataloging rules, 1,000 data items are checked and 950 are found to fully meet the standard requirements, the data standardization is (950 / 1,000) × 100% = 95%.

[0029] Data timeliness ( ), reflects whether the cataloging data is updated in a timely manner to ensure that it is consistent with the latest status of the actual data resources.

[0030] Evaluation Methodology: This can be calculated based on the interval between data updates. For example, if data is updated monthly and 80% of the data is updated within a month, the data timeliness is 80%. Alternatively, the average data update cycle can be used to measure data timeliness. For example, an average update cycle of 30 days indicates good data timeliness.

[0031] Data availability ( ), refers to the extent to which cataloged data can be accessed and used easily and quickly by users.

[0032] Evaluation Methodology: This can be measured by indicators such as the user access success rate and retrieval success rate of the data resource directory. For example, within a certain period of time, a user made 1,000 data access requests, of which 900 were successful. The data availability is (900 / 1,000) × 100% = 90%.

[0033] Rationality of classification ( ), examine whether the classification system of data resources meets business needs and data characteristics, whether the classification is logical and hierarchical, and whether it can facilitate users to quickly locate the required data.

[0034] Evaluation Methodology: There are five levels of classification plausibility as follows: 5 points: The classification system fully meets business needs and data characteristics, is highly logical and hierarchical, and allows users to accurately locate the required data in a very short time, with almost no thinking or searching required.

[0035] 4 points: The classification system basically meets business needs and data characteristics, has clear logic and hierarchy, and users can find data quickly through simple browsing or trial and error.

[0036] 3 points: The classification system partially meets business needs, with average logic and hierarchy. Users need to spend some time and effort to find data, but they will eventually find it.

[0037] 2 points: The classification system does not meet business needs, the logic is not clear enough, the hierarchy is not clear, and it is difficult for users to find data, requiring multiple attempts and exploration.

[0038] 1 point: The classification system does not meet business needs at all. It lacks logic and hierarchy, and users cannot find the data they need through classification.

[0039] An evaluation team composed of personnel from relevant business departments and data management experts can be organized to score the data resource catalog classifications based on the above criteria, with the average score used as the quantitative value for this indicator. For example, within an enterprise's data resource catalog, customer data can be categorized by multiple dimensions, such as customer type, business area, and data generation time. This allows employees from different departments to quickly find relevant customer data based on their needs. This categorization method is highly rational.

[0040] Clarity of description ( ), to see whether the description of the data resources is clear and accurate, and whether it can enable users to accurately understand the meaning, source, purpose and other key information of the data through metadata description without understanding the specific content of the data.

[0041] Evaluation Methodology: There are five levels of description clarity, as follows: 5 points: The description of the data resource is extremely clear and accurate. The metadata includes all key information, such as the meaning, source, purpose, data format, and update frequency of the data. Users can fully understand the relevant information of the data without additional questions.

[0042] 4 points: The description is clear and accurate. The metadata covers the main key information. Only some less important information is missing. Users can basically understand the core content of the data.

[0043] 3 points: The description is generally clear. There are some ambiguities or incompleteness in key information, but this does not affect the user's general understanding of the data. Further inquiries or reference to other materials may be needed to clarify certain details.

[0044] 2 points: The description is poorly clear, with much key information missing or many ambiguous statements. It is difficult for users to accurately understand the meaning and purpose of the data, and it takes a lot of time to verify and confirm.

[0045] 1 point: The description is confusing or seriously lacks key information. Users cannot obtain effective information from the metadata description, and it is of little help in understanding the data.

[0046] Similarly, the evaluation team scores a number of randomly selected data resource descriptions based on the above criteria, and then calculates the average score to quantify the metric. For a sales data table, the metadata details the meaning of each field. For example, the "Sales" field clearly states that it refers to the total amount of product sales during a specific time period, including actual receipts after various promotional activities. This provides a clear description.

[0047] Semantic accuracy ( ), check whether the terms and concepts used in the cataloging process are consistent with the semantics of the business field, and whether there are ambiguous or unclear expressions, so as to ensure consistency in understanding of the data among different users.

[0048] Evaluation Methodology: There are five levels of semantic accuracy as follows: 5 points: The terms and concepts used in the cataloging process are completely consistent with the semantics of the business domain, strictly follow the relevant standards and specifications, and there are no ambiguous or unclear expressions. Different users have exactly the same understanding of the data.

[0049] 4 points: Most terms and concepts are consistent with business semantics, with only a few minor inconsistencies or ambiguities in less critical terms, which do not affect overall data understanding and usage.

[0050] 3 points: Some terms and concepts deviate from business semantics, but their meaning can be understood through context or simple explanation, which has a certain impact on data usage, but it is not serious.

[0051] 2 points: Many terms and concepts do not conform to business semantics, and there is obvious ambiguity or vagueness. Users are prone to misunderstanding when understanding and using data, requiring frequent communication and confirmation.

[0052] 1 point: Terminology and concepts are seriously inconsistent with business semantics, and there are numerous ambiguous or unclear situations, which prevents users from accurately understanding the meaning of the data, seriously affecting the use and sharing of data.

[0053] An evaluation team composed of experts and data management personnel familiar with the business domain reviews the terms and concepts in the data resource catalog, assigns scores according to the aforementioned grading standards, and averages the scores to generate quantitative results. For example, in the medical data catalog, the "Disease Diagnosis" field uses internationally recognized disease classification codes and standard terminology, such as ICD-10 codes, to ensure accurate and consistent understanding of disease diagnoses among medical staff in different hospitals and departments.

[0054] Scalability ( ), consider whether the data resource directory has good scalability and can adapt to future data growth, the addition of new data types, and changes in business needs.

[0055] Evaluation Methodology: There are five levels of scalability as follows: 5 points: The data resource directory is highly scalable and can easily handle significant future data growth, the frequent addition of new data types, and complex changes in business needs. No major adjustments to the directory structure are required; only simple configuration or additions are required.

[0056] 4 points: Good scalability. When faced with a certain degree of data growth, new data types, and changes in business needs, the directory structure can adapt through appropriate adjustments and optimizations without causing a significant impact on existing systems and businesses.

[0057] 3 points: The scalability is average and can cope with some common data volume growth and small changes in business needs. However, for the large-scale addition of new data types or major adjustments to business needs, a certain degree of reconstruction of the directory structure may be required.

[0058] 2 points: Poor scalability. When faced with data growth, new data types, or changes in business needs, it is difficult to adjust the directory structure, which may have a certain impact on existing systems and businesses.

[0059] 1 point: There is little scalability. Once the data volume, data type, or business needs change, the directory structure will be difficult to adapt and require a complete redesign and reconstruction.

[0060] By analyzing technical documentation such as the data resource directory's architectural design and metadata model, combined with forecasts for future business development, technical experts and data management personnel jointly evaluate and score the system to determine a quantitative value for scalability. For example, by adopting a flexible metadata model and classification system, when an enterprise introduces new business lines and generates new types of data, it can simply add the corresponding classification and metadata descriptions to the existing system without requiring a large-scale adjustment to the entire directory structure, demonstrating good scalability.

[0061] User experience ( ), by collecting user feedback on the use of data resource catalogs, understanding the convenience and satisfaction of users in finding, accessing and using data, and evaluating the impact of cataloging quality on users' actual use.

[0062] Evaluation Methodology: User experience is divided into five levels, as follows: 5 points: Users are extremely satisfied with the data resource catalog, generally reflecting that it is very convenient to find, access, and use data, the search function is powerful and accurate, the interface design is user-friendly, and the operation process is simple and easy to understand, which can greatly improve work efficiency.

[0063] 4 points: Users are generally satisfied. Most users think that the app is convenient to use and that the search function and interface design perform well. Only a few users have some minor suggestions for improvement.

[0064] 3 points: The user experience is average. Some users find that there are some inconveniences in searching and using data, such as inaccurate search results and unreasonable interface layout, but this does not affect normal use.

[0065] 2 points: The user experience is poor. Most users report encountering many problems during use, such as difficulty finding data, impractical search functions, and an unfriendly interface, which have a certain impact on work efficiency.

[0066] 1 point: The user experience is very poor. Almost all users are dissatisfied with the data resource directory. There are serious usage barriers, such as the inability to retrieve data normally and complex and confusing interface operations, which seriously affect the progress of work.

[0067] By distributing user questionnaires, conducting user interviews, and collecting user feedback, we can compile statistics on user evaluations and satisfaction with various aspects, and then quantify and grade them according to the aforementioned grading standards. We can assign different weights, such as 40% for questionnaires, 30% for user interviews, and 30% for user feedback, to comprehensively calculate and quantify the user experience. For example, through user questionnaires and interviews, we can collect user evaluations of the data resource catalog, such as whether users can easily find the data they need, whether the search function is convenient and efficient, and whether the interface design is user-friendly, to comprehensively evaluate the user experience. If users generally report that they can quickly and accurately obtain the data they need and are generally satisfied with the catalog's operational processes, this indicates a good user experience and that the cataloging quality has been recognized by users.

[0068] Step S3: Collect metadata information and data information of the data resource directory and collect usage information of the data resource directory.

[0069] In the embodiment, metadata and data information of the data resource directory are collected, and database information of the data resource directory management system is configured, including database name, IP address, port, user with read permission and access password. Through the automation adapter, metadata information of different types of databases is collected, including but not limited to physical table information and field information.

[0070] In the embodiment, the usage information of the data resource directory is collected manually, and the usage information of the data resource directory is collected through a questionnaire, a questionnaire, or the like.

[0071] Step S4: perform data resource directory quality assessment and establish monitoring rules based on the metadata information and data information of the data resource directory.

[0072] In the embodiment, quality assessment is performed based on quantitative and qualitative indicators set by the assessment system.

[0073] In the embodiment, monitoring rules are established based on metadata information and data information of the data resource directory.

[0074] Examples of monitoring rules are as follows: Integrity indicator null value check: Checks whether the required items of the data resource directory are missing, such as the data resource directory name, data resource directory code, data resource directory providing department, and other key attributes. If they are empty, it is marked as erroneous data.

[0075] Data resource directory duplication check for integrity indicators: Check whether there are identical or similar data resource directories, set fields for duplication checking, such as name and summary, and determine whether data resources are duplicated based on similarity; Format normative check of normative indicators: check whether the resource directory code complies with the established specifications according to the generation rules of the data resource directory code; Business normativeness check of normative indicators: Check whether the sharing attributes and open attributes comply with the established business dictionary. For example, the sharing type in the sharing attribute can only have the three values of "unconditional sharing", "conditional sharing" and "not sharing"; the data resource provider can only be an institution within the organization; Check the association relationship of consistency indicators: For example, the associated data resource code should exist in the existing data resource directory.

[0076] Step S5: Mark the data resource directory that does not meet the requirements, and perform quality problem analysis and corrective measures based on the monitoring results.

[0077] In the embodiment, records that do not comply with the quality rules are marked according to the data quality monitoring rules to form a data resource directory monitoring result, that is, a list of problematic data in the data resource directory.

[0078] Step S6: Comprehensively evaluate the quality of the data resource directory and generate a data resource directory quality report.

[0079] like Figure 3 As shown, in step S6, the quality of the data resource directory is comprehensively evaluated, including: Step S61: Calculate quantitative indicators based on the data resource catalog quality assessment system The value of Step S62: Calculate qualitative indicators based on the data resource catalog quality assessment system The value of Step S63, setting the weights of quantitative indicators and qualitative indicators, such as setting the quantitative indicators The weights are , qualitative indicators The weights are The value of Step S64: Calculate the comprehensive evaluation score based on the indicators and weights ; The calculation formula is as follows: .

[0080] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data resource directory assurance method based on data quality monitoring, characterized in that: include: Step S1, sorting out the data resource catalog compilation standards; Step S2, designing a data resource catalog quality assessment system; Step S3, collecting metadata information and data information of the data resource directory and collecting usage information of the data resource directory; Step S4: perform data resource directory quality assessment and establish monitoring rules based on the metadata information and data information of the data resource directory; Step S5: Mark the data resource directory that does not meet the requirements, and perform quality problem analysis and corrective measures based on the monitoring results; Step S6: Comprehensively evaluate the quality of the data resource directory and generate a data resource directory quality report.

2. A data resource directory assurance method based on data quality monitoring according to claim 1, characterized in that: In step S1, the data resource directory compilation standard stipulates the core metadata of the data resource directory and the corresponding filling specifications, specifically including: Data resource catalog classification: required, indicating the classification information of data resources; Data resource directory name: required, a title that describes the content of the data resource; Data resource directory code: required, a unique and unchanging identification code for the data resource; Data resource provider: required, the department providing the data resources; Data resource provider code: required, the code of the department providing the data resource; Data resource directory summary: required, a brief description of the data resource content; Data resource format: required, the form in which the data resource exists; Information item information: optional, detailed description of the structured data resource, including the information item name and data type; Sharing attributes: Description of the data resource sharing type and conditions, including sharing type, sharing conditions, and sharing method; Open attributes: Required, a description of whether the data resource is open to the public and the conditions for opening it, including whether it is open to the public and the conditions for opening it; Update cycle: required, the frequency of data resource updates, which can be divided into real-time, daily, weekly, monthly, quarterly, and annual; Related data resource catalog code: Optional. If the data resource belongs to other categories, the code of other resource categories or other data resources is required.

3. The data resource directory guarantee method based on data quality monitoring according to claim 1 is characterized in that: In step S2, the evaluation indicators include quantitative indicators and qualitative indicators, wherein: Quantitative indicators include: data accuracy , data integrity , data consistency , data standardization , data timeliness and data availability ; Qualitative indicators include: rationality of classification , description clarity , semantic accuracy Scalability and user experience .

4. A data resource directory assurance method based on data quality monitoring according to claim 1, characterized in that: In step S3, metadata information of the data resource directory is collected through an automated tool, including but not limited to physical table information and field information.

5. The data resource directory guarantee method based on data quality monitoring according to claim 1 is characterized in that: In step S3, usage information of the data resource directory is collected through a questionnaire or a survey form.

6. A data resource directory assurance method based on data quality monitoring according to claim 1, characterized in that: In step S4, quality assessment is performed according to the quantitative indicators and qualitative indicators set by the assessment system.

7. The data resource directory guarantee method based on data quality monitoring according to claim 1 is characterized in that: In step S5, records that do not meet the quality rules are marked according to the data quality monitoring rules to form a data resource directory monitoring result, that is, a list of problematic data in the data resource directory.

8. The data resource directory guarantee method based on data quality monitoring according to claim 1 is characterized in that: In step S6, the quality of the data resource directory is comprehensively evaluated, including: Step S61: Calculate quantitative indicators based on the data resource catalog quality assessment system The value of Step S62: Calculate qualitative indicators based on the data resource catalog quality assessment system The value of Step S63: Set the weight of each quantitative indicator and the weights of qualitative indicators ; Step S64: Calculate the comprehensive evaluation score based on the indicators and weights ; The calculation formula is as follows: 。

Images