Method and system for labeling high-quality directory

Abstract

The invention relates to the technical field of Java Web application development, in particular to a high-quality catalogue labeling method and system, and the method comprises the following steps: formulating a quality detection rule, comprising a directory compiling quality rule, a resource hooking condition rule, a resource data quality rule, a data updating timeliness rule, a data security grading condition rule and a user-defined detection rule; integrating directory data, defining a directory data model and a unified standard, and extracting the data to an intermediate database to ensure the total amount and consistency of the data; the high-quality catalogue labeling method and system have the beneficial effects that catalogue quality detection rules including catalogue compiling quality, hitching resource conditions, data security levels and other rules are formulated by analyzing influence factors of catalogue used by a user and data and security regulations in policies and regulations, the catalogue quality is improved, and the catalogue labeling efficiency is improved. Normativity and data security guarantee are enhanced, and better-quality data resources are provided for users.

Description

Technical Field

[0001] The present invention relates to the technical field of Java Web application development, and in particular to a method and system for high-quality catalog annotation. Background Art

[0002] In the current information environment, various data directories, as indexes and navigation of information resources, play a vital role in the organization, management and utilization of data. However, with the rapid growth of data volume and the diversification of data sources, the quality issues of departmental basic directory data are becoming increasingly prominent. These issues include but are not limited to irregular directory compilation, incorrect or missing resource attachment, uneven data quality, untimely updates, and lack of data security classification. These issues not only affect the availability and credibility of data, but also increase the difficulty and cost of data management and utilization.

[0003] Traditional catalog management methods often rely on manual review and annotation, which is not only inefficient but also error-prone and difficult to meet the needs of large-scale data catalog management. Therefore, there is an urgent need for an automated, standardized, quantifiable, high-quality catalog annotation system to achieve comprehensive detection, evaluation, and continuous improvement of catalog quality. Summary of the invention

[0004] The object of the present invention is to provide a method and system for high-quality catalog annotation to solve the problems raised in the above-mentioned background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: a method for high-quality catalog annotation, the method comprising the following steps:

[0006] Formulate quality inspection rules, including catalog compilation quality rules, resource attachment rules, resource data quality rules, data update timeliness rules, data security classification rules, and user-defined inspection rules;

[0007] Integrate catalog data, define catalog data models and unified standards, and extract data to an intermediate database to ensure the full volume and consistency of data;

[0008] Preprocess the data, including deleting duplicate values, excluding special characters, and cleaning missing information items, and establish a preprocessing detection mechanism to distinguish normal from abnormal data;

[0009] Use a scheduled task scheduling tool to regularly test the catalog quality. According to the preset test cycle and test rules, perform batch query and test on the catalog to generate test results.

[0010] Calculate the catalog quality score based on the detection results. The score is calculated based on the indicator weights set by the user and the score of each detection rule;

[0011] Generate quality reports, including department inspection reports and catalog inspection reports, to visually display catalog quality status, problem distribution, inspection failure rules, and improvement suggestions.

[0012] Preferably, the catalog compilation quality rules in the quality inspection rules include basic inspection items and other inspection items. The basic inspection items involve the clarity, accuracy and standardization of the catalog name, description, and data item name. Other inspection items include the rationality of the catalog structure and classification accuracy.

[0013] Preferably, the data preprocessing step is performed using the OpenRefine tool, including deleting duplicate value data, excluding special character data, processing missing data for information items, and dividing the preprocessing results into normal data and abnormal data, and the abnormal data is pushed to the data providing department for manual inspection.

[0014] Preferably, the directory quality score is calculated by taking the weighted sum of resource data quality, data security classification, directory compilation quality, data update timeliness and resource attachment status according to the weights set by the user to obtain a total quality score.

[0015] Preferably, the quality report is displayed in a visual manner, including line graphs, pie charts, etc., showing the catalog quality status, problem distribution, the top 5 rules that failed detection, and providing a comprehensive analysis report for each catalog that does not meet quality standards, including the rules for failing detection, analysis suggestions and improvement measures.

[0016] A high-quality catalog annotation system, comprising:

[0017] The rule-making module is used to formulate the detection rules and quality score details that affect the catalog quality, including catalog compilation quality rules, resource attachment rules, resource data quality rules, data update timeliness rules, data security classification rules, and user-defined detection rules;

[0018] The data integration module is used to define the catalog data model and unified standards, and extract the integrated catalog data to the intermediate database to ensure the full amount and consistency of the data;

[0019] The data preprocessing module uses data processing tools to preprocess the data, including deleting duplicate values, excluding special characters, processing missing information items, and distinguishing between normal and abnormal data. Abnormal data is pushed to the data provider department for manual inspection;

[0020] The quality inspection module regularly executes quality inspection tasks through the scheduled task scheduling tool, batch queries and inspections the catalog according to the preset inspection rules and cycles, and generates inspection results;

[0021] The quality assessment and reporting module calculates the catalog quality score based on the test results and generates department test reports and catalog test reports to visually display the catalog quality status, problem distribution, test failure rules and improvement suggestions.

[0022] Preferably, the quality detection module further includes:

[0023] Sensitive data detection function, which uses similarity-based algorithms and semantic analysis technology to build a fingerprint model for sensitive information documents, and performs fingerprint capture and comparison on the document or content under test to determine whether it contains sensitive information;

[0024] Resource attachment status detection function, through the interface query directory mounted resources to determine the resource and interface attachment status;

[0025] The resource security and quality detection function determines the resource security level and update timeliness based on the provided interface, and calculates the quality score based on the preset quality rules and weights.

[0026] Preferably, the quality assessment and reporting module also includes:

[0027] The weight configuration function allows users to customize the weights of quality indicators according to actual needs;

[0028] The quality score calculation function calculates the total quality score based on the indicator weights set by the user and the scores of each test item;

[0029] Report generation function generates department inspection reports and catalog inspection reports. The department inspection report displays the inspection status, problem distribution and TOP5 inspection rules of all catalogs. The catalog inspection report provides a comprehensive analysis report for each catalog that does not meet quality standards.

[0030] Preferably, the directory data model defined by the data integration module includes:

[0031] Basic directory information, such as directory name, directory ID, department, directory description, information item name, and information item length;

[0032] Resource information, such as the attached resource name, resource ID, and update status;

[0033] Service information, such as service name and service ID, ensures that all directory data is integrated into the intermediate database according to unified standards.

[0034] Preferably, the system supports custom detection cycles. Users can set detection cycles through CRON expressions according to their own data update cycles or catalog compilation cycles to achieve regular detection and continuous improvement of catalog quality.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] The method and system for high-quality catalog annotation proposed in the present invention formulate catalog quality detection rules by analyzing the factors affecting user use of catalogs and data and the security provisions in policies and regulations. The rules include the quality of catalog compilation, the status of attached resources, the data security level, etc., to improve the quality of the catalog, enhance standardization and data security, and provide users with better quality data resources.

[0037] By formulating strict catalog compilation quality rules and standardizing catalog names, descriptions, data item names, etc., we can effectively avoid non-standard phenomena in the catalog compilation process and improve the accuracy and readability of the catalog.

[0038] Through the resource attachment rules, we can ensure that the resources attached to the directory are correct, without missing or wrong links, thus improving the availability and reliability of resources.

[0039] According to the detection rules of resource data quality, a comprehensive assessment is conducted on the data quality of library tables, data service interfaces and proxy interfaces to ensure the high quality of resource data.

[0040] Through the detection rules for the timeliness of data updates, the data providing departments are urged to update the data in a timely manner, ensuring the timeliness and accuracy of the data.

[0041] Through the detection rules of data security classification, directory data items and resources are required to undergo security classification, which improves data security and compliance.

[0042] The use of automated and standardized testing processes combined with data processing tools for preprocessing and testing has greatly improved testing efficiency and accuracy, and reduced the workload and error rate of manual review.

[0043] Department inspection reports and catalog inspection reports are generated regularly to visually display the catalog quality status, problem distribution and improvement suggestions, providing data provision departments with intuitive and comprehensive quality feedback to facilitate targeted improvements and optimizations. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose and technical solution of the present invention clearly and completely described, and the advantages more clearly understood, the embodiments of the present invention are further described in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are part of the embodiments of the present invention, rather than all of the embodiments, and are only used to explain the embodiments of the present invention, and are not used to limit the embodiments of the present invention. All other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] For example, see Figure 1 The present invention provides a technical solution: a method for high-quality catalog annotation, the method comprising the following steps:

[0047] 1. Formulate quality inspection rules

[0048] Cataloging Quality Rules

[0049] Basic testing: Directory names, descriptions, data item names, etc. that contain sensitive information will be directly scored 0 points. Directory names and descriptions must be clear, accurate, and unambiguous, and data item names must be standardized. 5 points will be deducted for each non-compliance, until all points are deducted.

[0050] Other inspection items: including but not limited to the rationality of the catalog structure, classification accuracy, etc. 10 points will be deducted for each quality problem until all points are deducted.

[0051] Resource attachment rules

[0052] Mounting resources: If resources are mounted correctly without missing or incorrect links, 100 points will be awarded.

[0053] No resources are attached: No resources are attached or the attachment is incorrect, and 0 points are awarded.

[0054] Resource data quality

[0055] No resources: 0 points for this item.

[0056] Only proxy interface: 100 points for this option

[0057] Only library tables and data service interfaces: convert the average score of data quality inspection (including library tables and data services) into percentage * score 100.

[0058] There are both library tables / data services and proxy interfaces: library tables / data services and proxy interfaces each account for 50 points. The average score of data quality inspection (including library tables and data services) is converted into a percentage * score 50+50 for calculation.

[0059] Average quality inspection score of the directory = average quality inspection score of all database tables / data services under the directory Timely data update

[0060] No resources: 0 points for this item.

[0061] Only proxy interface: 100 points for this option

[0062] Only database table / data service: timely update will get 100 points, and failure to update in time will get 0 points

[0063] There are both database tables / data services and proxy interfaces: 50 points for proxy interfaces and 50 points for database tables and data services. The score for this item is 50 + the timeliness score for database tables / data services. Database tables and data service data are updated in a timely manner: 50 points for timeliness, and 0 points for failure to update in a timely manner.

[0064] Calculation rules: Score = (timely resource update / total number of resources) * 10

[0065] Data security rating

[0066] Directory without resources: 10 points if the directory data item maintains the security level, otherwise 0 points

[0067] The directory has resources: 10 points if the directory data items and resources have been security graded, otherwise 0 points

[0068] Security level information items / total information items*5

[0069] User-defined detection rules

[0070] Allow users to customize quality indicators and weights according to actual needs to more flexibly evaluate catalog quality

[0071] 2. Catalog data integration

[0072] In order to make the detection rules applicable to all directory data, we will define the directory data model and unified standards, including directory name, directory ID, department, directory description, information item name, information item length; attached resource name, resource ID, update status; service name, service ID, and extract data to the MYSQL intermediate database according to the unified standards of basic information to ensure the full amount and consistency of data.

[0073] 3. Data preprocessing

[0074] In order to ensure the accuracy of data and the integrity of quality inspection, and to prevent abnormalities in quality inspection, the data must first be preprocessed, and a preprocessing inspection normal and abnormal data table must be established. Normal data will be entered into the inspection table, and abnormal data will be pushed separately to the data provider department, which will manually check the data to determine the catalog compilation problems and improve the catalog quality. The data inspection uses the OpenRefine tool for preprocessing, and the data is first pre-inspected and processed according to the following rules:

[0075] (1) Delete duplicate data

[0076] (2) Exclude special character data

[0077] (3) Information items with missing data

[0078] 4. Regular testing of quality indicators

[0079] The xxl-job scheduled task scheduling tool is used to test the catalog quality in the form of scheduled tasks. Users can also customize the detection cycle according to their own data update cycle or catalog compilation cycle. The detection cycle is set in the form of CRON expressions. The general cycle is daily, monthly, and quarterly.

[0080] The detection tasks are divided into three types: all, by department, and by directory, which can meet the user's detection needs for directories of different granularity.

[0081] Testing process:

[0082] 1) Batch query the directory list that meets the detection conditions

[0083] 2) Query the configured detection rules, complete the detection of the target directory according to the rules and return the results;

[0084] Sensitive data detection: Use similarity-based algorithms to create sensitive data files, extract features of detection data, learn and train sensitive files, use semantic analysis technology for word segmentation, and build fingerprint models for sensitive information documents. Finally, use the same method to capture fingerprints of the document or content being tested, and compare them with the fingerprints obtained through training, and determine whether the document being tested is a sensitive information document based on the preset similarity threshold.

[0085] Resource attachment status detection: query the number of resources attached in the directory through the dubbo interface to determine the resource and interface attachment status

[0086] Resource security and quality inspection: The resource security level and whether it is updated in a timely manner are determined based on the provided interface to determine whether the inspection rules are met; quality inspection calculates the quality score based on the quality rules and rule weights set in the quality system to determine which type of inspection rules are met.

[0087] 3) Calculate the score of each detection rule based on the detection results

[0088] 5. Calculate the catalog quality score

[0089] The total quality score is calculated by multiplying the indicator weight set by the user by the score of each detection item.

[0090] Quality score = resource data quality * 20% + data security classification * 10% + catalog compilation quality * 20% + data update timeliness * 20% + resource attachment * 30%

[0091] 6. Quality Report

[0092] Departmental inspection report: Analyze the inspection status of all directories according to the departments to which the directories belong, and display the distribution of all problems in a visual way such as line graphs and pie charts; display the top 5 inspection rules with the most problems, so that the data provider department can focus on such problems and effectively improve the compilation quality; provide comprehensive inspection reports for directories with problems, including inspection categories and rules under the categories.

[0093] Catalog Inspection Report: Provide a comprehensive analysis report for each catalog that does not meet quality standards, including the rules for detection failure, analysis suggestions and improvement measures.

[0094] The above process is executed regularly to generate a catalog quality report, including scores, problem details, improvement suggestions, etc., so that users can understand the catalog quality status and make continuous improvements.

[0095] Embodiment 2, based on embodiment 1, proposes a high-quality catalog annotation system, the system comprising:

[0096] The rule-making module is used to formulate the detection rules and quality score details that affect the catalog quality, including catalog compilation quality rules, resource attachment rules, resource data quality rules, data update timeliness rules, data security classification rules, and user-defined detection rules;

[0097] The data integration module is used to define the catalog data model and unified standards, and extract the integrated catalog data to the intermediate database to ensure the full amount and consistency of the data;

[0098] The data preprocessing module uses data processing tools to preprocess the data, including deleting duplicate values, excluding special characters, processing missing information items, and distinguishing between normal and abnormal data. Abnormal data is pushed to the data provider department for manual inspection;

[0099] The quality inspection module regularly executes quality inspection tasks through a scheduled task scheduling tool, batch queries and inspects the catalog according to preset inspection rules and cycles, and generates inspection results; further includes:

[0100] Sensitive data detection function, which uses similarity-based algorithms and semantic analysis technology to build a fingerprint model for sensitive information documents, and performs fingerprint capture and comparison on the document or content under test to determine whether it contains sensitive information;

[0101] Resource attachment status detection function, through the interface query directory mounted resources to determine the resource and interface attachment status;

[0102] The resource security and quality detection function determines the resource security level and update timeliness based on the provided interface, and calculates the quality score based on the preset quality rules and weights.

[0103] The quality assessment and reporting module calculates the catalog quality score based on the test results and generates department test reports and catalog test reports to visually display the catalog quality status, problem distribution, test failure rules and improvement suggestions; it also includes:

[0104] The weight configuration function allows users to customize the weights of quality indicators according to actual needs;

[0105] The quality score calculation function calculates the total quality score based on the indicator weights set by the user and the scores of each test item;

[0106] Report generation function generates department inspection reports and catalog inspection reports. The department inspection report displays the inspection status, problem distribution and TOP5 inspection rules of all catalogs. The catalog inspection report provides a comprehensive analysis report for each catalog that does not meet quality standards.

[0107] The directory data model defined by the data integration module includes:

[0108] Basic directory information, such as directory name, directory ID, department, directory description, information item name, and information item length;

[0109] Resource information, such as the attached resource name, resource ID, and update status;

[0110] Service information, such as service name and service ID, ensures that all directory data is integrated into the intermediate database according to unified standards.

[0111] The system supports custom detection cycles. Users can set detection cycles through CRON expressions based on their own data update cycle or catalog compilation cycle to achieve regular detection and continuous improvement of catalog quality.

[0112] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for high-quality catalog annotation, characterized by: The method comprises the following steps: Formulate quality inspection rules, including catalog compilation quality rules, resource attachment rules, resource data quality rules, data update timeliness rules, data security classification rules, and user-defined inspection rules; Integrate catalog data, define catalog data models and unified standards, and extract data to an intermediate database to ensure the full volume and consistency of data; Preprocess the data, including deleting duplicate values, excluding special characters, and cleaning missing information items, and establish a preprocessing detection mechanism to distinguish normal from abnormal data; Use a scheduled task scheduling tool to regularly test the catalog quality. According to the preset test cycle and test rules, perform batch query and test on the catalog to generate test results. Calculate the catalog quality score based on the detection results. The score is calculated based on the indicator weights set by the user and the score of each detection rule; Generate quality reports, including department inspection reports and catalog inspection reports, to visually display catalog quality status, problem distribution, inspection failure rules, and improvement suggestions.

2. A high-quality catalog annotation method according to claim 1, characterized in that: The catalog compilation quality rules in the quality inspection rules include basic inspection items and other inspection items. The basic inspection items involve the clarity, accuracy and standardization of the catalog name, description, and data item name. Other inspection items include the rationality of the catalog structure and classification accuracy.

3. A high-quality catalog annotation method according to claim 1, characterized in that: The data preprocessing step is performed using the OpenRefine tool, including deleting duplicate value data, excluding special character data, processing missing data for information items, and dividing the preprocessing results into normal data and abnormal data. The abnormal data is pushed to the data provider department for manual inspection.

4. A high-quality catalog annotation method according to claim 1, characterized in that: The calculation method of the catalog quality score includes weighted summing up the resource data quality, data security classification, catalog compilation quality, data update timeliness and resource attachment status according to the weights set by the user to obtain the total quality score.

5. A high-quality catalog annotation method according to claim 1, characterized in that: The quality report is displayed in a visual way, including line graphs, pie charts, etc., showing the catalog quality status, problem distribution, the top 5 rules that failed detection, and providing a comprehensive analysis report for each catalog that does not meet quality standards, including the rules for failing detection, analysis suggestions and improvement measures.

6. A high-quality catalog annotation system according to the high-quality catalog annotation method according to any one of claims 1 to 5, characterized in that: The system comprises: The rule-making module is used to formulate the detection rules and quality score details that affect the catalog quality, including catalog compilation quality rules, resource attachment rules, resource data quality rules, data update timeliness rules, data security classification rules, and user-defined detection rules; The data integration module is used to define the catalog data model and unified standards, and extract the integrated catalog data to the intermediate database to ensure the full amount and consistency of the data; The data preprocessing module uses data processing tools to preprocess the data, including deleting duplicate values, excluding special characters, processing missing information items, and distinguishing between normal and abnormal data. Abnormal data is pushed to the data provider department for manual inspection; The quality inspection module regularly executes quality inspection tasks through the scheduled task scheduling tool, batch queries and inspections the catalog according to the preset inspection rules and cycles, and generates inspection results; The quality assessment and reporting module calculates the catalog quality score based on the test results and generates department test reports and catalog test reports to visually display the catalog quality status, problem distribution, test failure rules and improvement suggestions.

7. A high-quality catalog annotation system according to claim 6, characterized in that: The quality inspection module further includes: Sensitive data detection function, which uses similarity-based algorithms and semantic analysis technology to build a fingerprint model for sensitive information documents, and performs fingerprint capture and comparison on the document or content under test to determine whether it contains sensitive information; Resource attachment status detection function, through the interface query directory mounted resources to determine the resource and interface attachment status; The resource security and quality detection function determines the resource security level and update timeliness based on the provided interface, and calculates the quality score based on the preset quality rules and weights.

8. A high-quality catalog annotation system according to claim 6, characterized in that: The Quality Assessment and Reporting module also includes: The weight configuration function allows users to customize the weights of quality indicators according to actual needs; The quality score calculation function calculates the total quality score based on the indicator weights set by the user and the scores of each test item; Report generation function generates department inspection reports and catalog inspection reports. The department inspection report displays the inspection status, problem distribution and TOP5 inspection rules of all catalogs. The catalog inspection report provides a comprehensive analysis report for each catalog that does not meet quality standards.

9. A high-quality catalog annotation system according to claim 6, characterized in that: The directory data model defined by the data integration module includes: Basic directory information, such as directory name, directory ID, department, directory description, information item name, and information item length; Resource information, such as the attached resource name, resource ID, and update status; Service information, such as service name and service ID, ensures that all directory data is integrated into the intermediate database according to unified standards.

10. A high-quality catalog annotation system according to claim 6, characterized in that: The system supports custom detection cycles. Users can set detection cycles through CRON expressions based on their own data update cycle or catalog compilation cycle to achieve regular detection and continuous improvement of catalog quality.

Images