Method for generating quality report based on directory detection rule

Abstract

The invention particularly relates to a method for generating a quality report based on a directory detection rule. According to the method for generating the quality report based on the directory detection rule, a rule template is preset in a configuration interface, and the detection rule is automatically identified and configured according to user requirements; a preset time interval is self-defined according to requirements, a detection task is automatically triggered through a timed task, and comprehensive detection is conducted on the data catalog; a detection report is displayed and exported through a Web interface, detailed detection results are displayed according to directory dimensions, and directory quality conditions are summarized from a department perspective; a user checks a detection result through a Web interface, customizes an export format according to needs, and exports a report, so that subsequent analysis and processing are facilitated. According to the method for generating the quality report based on the directory detection rule, a management department can be helped to manage and utilize data resources more effectively, the efficiency of data management is improved, more accurate data support is provided for decision making of the management department, and therefore scientificity and effectiveness of management decision making are promoted.

Description

Technical Field

[0001] The present invention relates to the field of data governance technology, and in particular to a method for generating a quality report based on directory detection rules. Background Art

[0002] With the deepening of digital transformation and the accelerating pace of informatization construction, the shared exchange platform has become a key infrastructure for data interconnection and interoperability among management departments. This change not only improves the efficiency of collaboration among departments, but also promotes the openness and sharing of data, providing the public with more transparent and efficient services. However, as an index and entry point for data resources, the quality of the data catalog directly affects the efficiency and value of data utilization.

[0003] Currently, data catalog management faces many challenges. Inaccurate data is common, which may cause decision makers to make decisions based on wrong information, thus affecting the accuracy and effectiveness of policies. In addition, incomplete data is also a serious problem, which may lead to the omission of important information and bias the results of data analysis. At the same time, the problem of non-standard data cannot be ignored. The differences in data standards and formats between different departments and systems increase the difficulty of data integration and reduce the availability of data.

[0004] What is more serious is that the risk of sensitive information leakage always exists. In the process of data sharing, if there is a lack of effective security measures, sensitive information may be obtained by unauthorized personnel, which not only violates personal privacy but also poses a threat to national security. Therefore, it is particularly important to establish an efficient and automated data quality detection method.

[0005] Based on the above situation, the present invention proposes a method for generating a quality report based on directory detection rules. Summary of the invention

[0006] In order to make up for the defects of the prior art, the present invention provides a simple and efficient method for generating a quality report based on directory detection rules.

[0007] The present invention is achieved through the following technical solutions:

[0008] A method for generating a quality report based on a directory detection rule comprises the following steps:

[0009] Step S1, configuring automatic detection rules;

[0010] Customize the addition, modification or deletion of detection rules through the configuration interface, including rules for sensitive word detection of directory names, data item names, descriptions and information items, and detection of the standardization, uniqueness, integrity and confidentiality of directories and data items;

[0011] Step S2, setting a scheduled task to automatically detect the quality of the directory;

[0012] Customize preset time intervals according to needs, automatically trigger detection tasks through scheduled tasks, and conduct comprehensive detection of data directories;

[0013] Step S3, generating a quality inspection report;

[0014] Display and export test reports through the web interface, display detailed test results by catalog dimension, and summarize catalog quality status from the department's perspective;

[0015] Users can view the test results through the web interface and customize the export format as needed to export the report for subsequent analysis and processing.

[0016] In step S1, the standardization check of the directory and data items includes checking whether a directory has and has only one data item, and whether the directory name and the data item are repeated, so as to avoid confusion in the directory structure and naming;

[0017] Uniqueness detection is to check whether there are duplications in directories and data items to ensure that each directory and data item is unique and avoid data redundancy and confusion;

[0018] Completeness check is to confirm whether the core metadata is fully filled in. The core metadata includes but is not limited to the data source and update cycle to ensure the integrity and accuracy of the data;

[0019] Confidentiality testing includes three sub-items: sensitive information detection, resource attachment and data security classification. It comprehensively evaluates the security compliance of the directory to protect the security of sensitive information and prevent unauthorized access and leakage.

[0020] In step S2, the implementation steps are as follows:

[0021] S2.1. Build detection tasks through thread pool, and allocate, schedule and process detection tasks in parallel;

[0022] S2.2, construct a data detection executor according to the detection range type to perform the detection task;

[0023] S2.3, the data detection executor obtains detection rules based on the detection task, performs specific detection operations, and obtains detection results;

[0024] S2.4. Implement detection tasks and detection rules based on specific rules, traverse the objects to be detected, collect the generated detection results, and classify, summarize and record them.

[0025] In step S2.1, the implementation steps are as follows:

[0026] S2.1.1. First, query the data detection tasks whose status is on (status=1), and obtain the list of tasks to be processed from the database by constructing query conditions;

[0027] If the task to be detected is not obtained, the corresponding log information is recorded; otherwise, the task list is initialized and the data detection task is prepared to be executed;

[0028] S2.1.2, use multi-threading technology to process each task in the task list in parallel;

[0029] Specifically, a new thread is created for each task, and the data detection task is executed in the thread;

[0030] S2.1.3. Create a DataDetectJob object for each task and set the task attributes according to the task information, including but not limited to the task ID, task name, task operation type, detection range type and detection range object;

[0031] S2.1.4. Submit the constructed DataDetectJob object to the thread pool for execution. The thread pool allocates and schedules tasks based on task priority and existing resources.

[0032] S2.1.5. While executing the task, update the status of the corresponding task in the database to running (status=2), so that other systems or users can understand the current status of the task by querying the database.

[0033] In step S2.2, the implementation steps are as follows:

[0034] S2.2.1. Receive data detection tasks and initialize task parameters, including but not limited to task ID, task name and operation type, detection range type and detection range object; before the task starts to execute, record the log of the start of the task execution, the log information includes task ID, task name and start time, and insert the log information into the task operation log table;

[0035] S2.2.2. Determine the specific scope of detection based on the detection scope type (detectScopeType) and detection scope object (detectScopeObject) in the task parameters; the detection scope types include specific resources, all resources, resources and directories under the specified department;

[0036] According to different detection scope types, select the corresponding data detection executor (such as CatalogDetectExecutor) to perform the detection task;

[0037] S2.2.3, the data detection executor performs the detection task according to the selected detection range and data detection parameters;

[0038] The data detection executor records detailed information during the detection process, including but not limited to the number of detections and the number of detected problems, and returns the recorded information to the system;

[0039] S2.2.4. Update the task parameters according to the results returned by the data detection executor, including the number of detections and the number of problem rows; record the log of task execution completion, including the end time and execution results, and insert the log information into the task operation log table.

[0040] In step S2.3, the implementation steps are as follows:

[0041] S2.3.1. According to the detection scope type, obtain the corresponding parser IParseDetectScope from the detection scope parser factory DetectScopeParserFactory; the parser IParseDetectScope parses the detection scope object and generates a list of objects to be detected List <detectobject>;

[0042] S2.3.2. Obtain the list of applicable detection rules from the catalog quality rule service catalogQualityRuleService according to the detection scope type and detection scope object. <Map<String,Object> >

[0043] For each object to be detected, call the doDetect method with the object to be detected and the list of detection rules as parameters, perform specific detection operations, and obtain the list of detection results List <Map<String,Object> >

[0044] S2.3.3. Traverse the test result list List <Map<String,Object> >, add the detection objects with problems to the problem directory set Set <object>In the process, call the batchAddCatalogQualityTaskResultZD (batch add catalog quality task results) method of the catalogQualityTaskResultService to save the test results in batches;

[0045] S2.3.4. During the task execution process, the job log recorder XxlJobLogger records log information, including the start of the task execution, the object to be detected, the applicable rules and the detection results.

[0046] In step S2.4, the implementation steps are as follows:

[0047] S2.4.1. The detection execution module receives a list of data objects to be detected and a list of detection rules, traverses the list of detection rules, and applies a corresponding detection rule to each data object;

[0048] S2.4.2, the rule loading and parsing module obtains the specific detection rule implementation object through Java reflection technology according to the detection rule configuration, and converts it into a rule entity that can be recognized by the system;

[0049] S2.4.3. The result processing and recording module collects the test results generated by the test execution module, classifies and summarizes the test results, and records the test results in a database or log for subsequent analysis and processing.

[0050] A system for generating a quality report based on a directory detection rule, comprising:

[0051] Detection rule configuration module, users can customize detection rules according to their needs; add, modify or delete detection rules through the configuration interface, including rules for sensitive word detection of directory names, data item names, descriptions and information items, and detect the standardization, uniqueness, integrity and confidentiality of directories and data items;

[0052] The scheduled task module is responsible for customizing the preset time interval according to needs, automatically triggering the detection task through the scheduled task, and performing a comprehensive detection of the data directory;

[0053] The detection execution module is responsible for receiving the list of data objects to be detected and the list of detection rules, traversing the list of detection rules, and applying the corresponding detection rules to each data object;

[0054] The rule loading and parsing module is responsible for obtaining specific detection rule implementation objects through Java reflection technology according to the detection rule configuration, and converting them into rule entities that can be recognized by the system;

[0055] The result processing and recording module is responsible for collecting the test results generated by the test execution module, classifying and summarizing the test results, and recording the test results in a database or log for subsequent analysis and processing;

[0056] The quality inspection report module is responsible for generating quality inspection reports, displaying and exporting inspection reports through the web interface, displaying detailed inspection results by directory dimension, and summarizing directory quality status from a departmental perspective; users view inspection results through the web interface, and customize the export format as needed to export the report for subsequent analysis and processing.

[0057] A device for generating a quality report based on directory detection rules, characterized in that it includes a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the above method steps when executing the computer program.

[0058] A readable storage medium, characterized in that: a computer program is stored on the readable storage medium, and the computer program implements the above method steps when executed by a processor.

[0059] The beneficial effect of the present invention is that the method of generating quality reports based on directory detection rules can help management departments to manage and utilize data resources more effectively, which not only improves the efficiency of data management, but also provides more accurate data support for management department decision-making, thereby promoting the scientificity and effectiveness of management decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0061] Attached Figure 1 A schematic diagram of a method for generating a quality report based on directory detection rules according to the present invention.

[0062] Attached Figure 2 A schematic diagram of the system architecture for generating a quality report based on directory detection rules in the present invention. DETAILED DESCRIPTION

[0063] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0064] The method for generating a quality report based on a directory detection rule comprises the following steps:

[0065] Step S1, configuring automatic detection rules;

[0066] Customize the addition, modification or deletion of detection rules through the configuration interface, including rules for sensitive word detection of directory names, data item names, descriptions and information items, and detection of the standardization, uniqueness, integrity and confidentiality of directories and data items;

[0067] In step S1, the standardization check of the directory and data items includes checking whether a directory has and has only one data item, and whether the directory name and the data item are repeated, so as to avoid confusion in the directory structure and naming;

[0068] S1.1. One directory has only one data item: Ensure that the directory structure is clear and avoids confusion. This rule helps maintain the organization and readability of data, allowing users to quickly find the information they need.

[0069] S1.2. Duplicate directory names and data items: Prevent naming confusion and facilitate retrieval. By avoiding duplicate names, we can improve the searchability and usability of data.

[0070] Uniqueness detection is to check whether there are duplications in directories and data items to ensure that each directory and data item is unique and avoid data redundancy and confusion;

[0071] Integrity check is to confirm whether the core metadata is fully filled in. The core metadata includes but is not limited to the data source and update cycle to ensure the integrity and accuracy of the data so that users can trust and use the data;

[0072] Confidentiality testing includes three sub-items: sensitive information detection, resource attachment and data security classification. It comprehensively evaluates the security compliance of the directory to protect the security of sensitive information and prevent unauthorized access and leakage.

[0073] Step S2, setting a scheduled task to automatically detect the quality of the directory;

[0074] Customize preset time intervals according to needs, automatically trigger detection tasks through scheduled tasks, and conduct comprehensive detection of data directories. This scheduled task-driven approach can ensure that the data directory is always in good condition and potential problems can be discovered and resolved in a timely manner.

[0075] In step S2, the implementation steps are as follows:

[0076] S2.1. Build detection tasks through thread pool, and allocate, schedule and process detection tasks in parallel;

[0077] S2.2, construct a data detection executor according to the detection range type to perform the detection task;

[0078] S2.3, the data detection executor obtains detection rules based on the detection task, performs specific detection operations, and obtains detection results;

[0079] S2.4. Implement detection tasks and detection rules based on specific rules, traverse the objects to be detected, collect the generated detection results, and classify, summarize and record them.

[0080] In step S2.1, the implementation steps are as follows:

[0081] S2.1.1. First, query the data detection tasks whose status is on (status=1), and obtain the list of tasks to be processed from the database by constructing query conditions;

[0082] If the task to be detected is not obtained, the corresponding log information is recorded; otherwise, the task list is initialized and the data detection task is prepared to be executed;

[0083] S2.1.2, use multi-threading technology to process each task in the task list in parallel;

[0084] Specifically, a new thread is created for each task, and the data detection task is executed in the thread;

[0085] S2.1.3. Create a DataDetectJob object for each task and set the task attributes according to the task information, including but not limited to the task ID, task name, task operation type, detection range type and detection range object;

[0086] S2.1.4. Submit the constructed DataDetectJob object to the thread pool for execution. The thread pool allocates and schedules tasks based on their priorities and available resources.

[0087] S2.1.5. While executing the task, update the status of the corresponding task in the database to running (status=2), so that other systems or users can understand the current status of the task by querying the database.

[0088] In step S2.2, the implementation steps are as follows:

[0089] S2.2.1. Receive data detection tasks and initialize task parameters, including but not limited to task ID, task name and operation type, detection range type and detection range object; before the task starts to execute, record the log of the start of the task execution, the log information includes task ID, task name and start time, and insert the log information into the task operation log table;

[0090] S2.2.2. Determine the specific scope of detection based on the detection scope type (detectScopeType) and detection scope object (detectScopeObject) in the task parameters; the detection scope types include specific resources, all resources, resources and directories under the specified department;

[0091] According to different detection scope types, select the corresponding data detection executor (such as CatalogDetectExecutor) to perform the detection task;

[0092] S2.2.3, the data detection executor performs the detection task according to the selected detection range and data detection parameters;

[0093] The data detection executor records detailed information during the detection process, including but not limited to the number of detections and the number of detected problems, and returns the recorded information to the system;

[0094] S2.2.4. Update the task parameters according to the results returned by the data detection executor, including the number of detections and the number of problem rows; record the log of task execution completion, including the end time and execution results, and insert the log information into the task operation log table.

[0095] In step S2.3, the implementation steps are as follows:

[0096] S2.3.1. According to the detection scope type, obtain the corresponding parser IParseDetectScope from the detection scope parser factory DetectScopeParserFactory; the parser IParseDetectScope parses the detection scope object and generates a list of objects to be detected List <detectobject>;

[0097] S2.3.2. Obtain the list of applicable detection rules from the catalog quality rule service catalogQualityRuleService according to the detection scope type and detection scope object. <Map<String,Object> >

[0098] For each object to be detected, call the doDetect method with the object to be detected and the list of detection rules as parameters, perform specific detection operations, and obtain the list of detection results List <Map<String,Object> >

[0099] S2.3.3. Traverse the test result list List <Map<String,Object> >, add the detection objects with problems to the problem directory set Set <object>In the process, call the batchAddCatalogQualityTaskResultZD (batch add catalog quality task results) method of the catalogQualityTaskResultService to save the test results in batches;

[0100] S2.3.4. During the task execution process, the job log recorder XxlJobLogger records log information, including the start of the task execution, the object to be detected, the applicable rules and the detection results.

[0101] In step S2.4, the implementation steps are as follows:

[0102] S2.4.1. The detection execution module receives a list of data objects to be detected and a list of detection rules, traverses the list of detection rules, and applies a corresponding detection rule to each data object;

[0103] S2.4.2, the rule loading and parsing module obtains the specific detection rule implementation object through Java reflection technology according to the detection rule configuration, and converts it into a rule entity that can be recognized by the system;

[0104] S2.4.3. The result processing and recording module collects the test results generated by the test execution module, classifies and summarizes the test results, and records the test results in a database or log for subsequent analysis and processing.

[0105] Step S3, generating a quality inspection report;

[0106] The test report is displayed and exported through the web interface. Users can view the test results through the web interface and customize the export format as needed to export the report for subsequent analysis and processing.

[0107] In step S3, when the web page displays the quality report, detailed test results are displayed by directory dimension, and the directory quality status is summarized from the departmental perspective.

[0108] Detailed inspection results are displayed by directory dimension, allowing administrators to clearly see the quality status of each directory and make corresponding decisions and adjustments.

[0109] Summarizing the catalog quality status from a departmental perspective allows administrators to understand the overall data quality status from the department's perspective, thereby better managing and optimizing data resources.

[0110] Supports exporting quality reports to document formats for easy archiving and sharing. This function allows administrators to save quality reports in common document formats for easy subsequent review and sharing.

[0111] In terms of generating quality reports, we use a variety of visualization methods to intuitively display the test results. For example, we use charts, bar charts, etc. to display the scores of various indicators to help users quickly understand the overall quality status of the data directory. In addition, we can also provide customized report templates according to user needs to meet the display needs in different scenarios.

[0112] The system for generating quality reports based on directory detection rules includes:

[0113] Detection rule configuration module, users can customize detection rules according to their needs; add, modify or delete detection rules through the configuration interface, including rules for sensitive word detection of directory names, data item names, descriptions and information items, and detect the standardization, uniqueness, integrity and confidentiality of directories and data items;

[0114] The scheduled task module is responsible for customizing the preset time interval according to needs, automatically triggering the detection task through the scheduled task, and performing a comprehensive detection of the data directory;

[0115] The detection execution module is responsible for receiving the list of data objects to be detected and the list of detection rules, traversing the list of detection rules, and applying the corresponding detection rules to each data object;

[0116] The rule loading and parsing module is responsible for obtaining specific detection rule implementation objects through Java reflection technology according to the detection rule configuration, and converting them into rule entities that can be recognized by the system;

[0117] The result processing and recording module is responsible for collecting the test results generated by the test execution module, classifying and summarizing the test results, and recording the test results in a database or log for subsequent analysis and processing;

[0118] The quality inspection report module is responsible for generating quality inspection reports, displaying and exporting inspection reports through the web interface, displaying detailed inspection results by directory dimension, and summarizing directory quality status from a departmental perspective; users view inspection results through the web interface, and customize the export format as needed to export the report for subsequent analysis and processing.

[0119] The device for generating a quality report based on a catalog detection rule comprises a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the above method steps when executing the computer program.

[0120] The readable storage medium stores a computer program, and when the computer program is executed by a processor, the above method steps are implemented.

[0121] This method of generating quality reports based on directory detection rules realizes comprehensive, efficient and accurate detection of data directories. It also has the function of displaying and exporting detection reports through a Web interface, providing users with a convenient operation experience. Compared with the existing technology, this method has the following characteristics:

[0122] First, by introducing an automated detection process, the efficiency of data directory management has been greatly optimized, which not only reduces the need for manual operations, but also significantly reduces errors caused by human factors. With the help of automation, the data processing speed has been improved, and the error rate has been effectively controlled. In addition, the introduction of automated processes has also enhanced data security, ensuring the security of sensitive data through strict permission management and access control.

[0123] Second, in order to further improve the transparency and efficiency of data management, this method also provides multi-dimensional quality reports. These reports record every step of data processing in detail, including the source of data, processing process, and final output results. Management departments can use these reports to quickly identify problems that may arise in the data management process, so as to take effective measures to optimize it in a timely manner.

[0124] This multi-dimensional quality report not only improves the transparency of data management, but also provides an effective tool for management departments to help them better understand and grasp the quality of data. By analyzing these reports, management departments can more accurately understand the current status of data and how to further optimize data quality.

[0125] In summary, this method provides a powerful tool for management departments through automated detection processes and multi-dimensional quality reports, helping them to manage and utilize data resources more effectively. It not only improves the efficiency of data management, but also provides more accurate data support for management department decision-making, thereby promoting the scientificity and effectiveness of management decisions.

[0126] The embodiment described above is only one specific implementation of the present invention. Common changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included in the protection scope of the present invention.< / object> < / detectobject> < / object> < / detectobject>

Claims

1. A method for generating quality reports based on catalog detection rules, Features: The following steps are involved: Step S1, configuring automatic detection rules; Customize the addition, modification or deletion of detection rules through the configuration interface, including rules for sensitive word detection of directory names, data item names, descriptions and information items, and detection of the standardization, uniqueness, integrity and confidentiality of directories and data items; Step S2, setting a scheduled task to automatically detect the quality of the directory; Customize preset time intervals according to needs, automatically trigger detection tasks through scheduled tasks, and conduct comprehensive detection of data directories; Step S3, generating a quality inspection report; Display and export test reports through the web interface, display detailed test results by catalog dimension, and summarize catalog quality status from the department's perspective; Users can view the test results through the web interface and customize the export format as needed to export the report for subsequent analysis and processing.

2. The method for generating a quality report based on directory detection rules according to claim 1, Features: In step S1, the standardization check of the directory and data items includes checking whether a directory has and has only one data item, and whether the directory name and the data item are repeated, so as to avoid confusion in the directory structure and naming; Uniqueness detection is to check whether there are duplications in directories and data items to ensure that each directory and data item is unique and avoid data redundancy and confusion; Completeness check is to confirm whether the core metadata is fully filled in. The core metadata includes but is not limited to the data source and update cycle to ensure the integrity and accuracy of the data; Confidentiality testing includes three sub-items: sensitive information detection, resource attachment and data security classification. It comprehensively evaluates the security compliance of the directory to protect the security of sensitive information and prevent unauthorized access and leakage.

3. The method for generating a quality report based on directory detection rules according to claim 1, Features: In step S2, the implementation steps are as follows: S2.

1. Build detection tasks through thread pool, and allocate, schedule and process detection tasks in parallel; S2.2, construct a data detection executor according to the detection range type to perform the detection task; S2.3, the data detection executor obtains detection rules based on the detection task, performs specific detection operations, and obtains detection results; S2.

4. Implement detection tasks and detection rules based on specific rules, traverse the detection objects, collect the generated detection results, and classify, summarize and record them.

4. The method for generating a quality report based on directory detection rules according to claim 3, Features: In step S2.1, the implementation steps are as follows: S2.1.

1. First, query the data detection tasks whose status is on, that is, the status parameter is 1, and obtain the list of tasks to be processed from the database by constructing query conditions; If the task to be detected is not obtained, the corresponding log information is recorded; otherwise, the task list is initialized and the data detection task is prepared to be executed; S2.1.2, use multi-threading technology to process each task in the task list in parallel; Specifically, a new thread is created for each task, and the data detection task is executed in the thread; S2.1.

3. Create a DataDetectJob object for each task and set the task attributes according to the task information, including but not limited to the task ID, task name, task operation type, detection range type and detection range object; S2.1.

4. Submit the constructed DataDetectJob object to the thread pool for execution. The thread pool allocates and schedules tasks based on their priorities and available resources. S2.1.

5. While executing the task, update the status of the corresponding task in the database to running, and change the status parameter to 2 so that other systems or users can understand the current status of the task by querying the database.

5. The method for generating a quality report based on directory detection rules according to claim 3, Features: In step S2.2, the implementation steps are as follows: S2.2.

1. Receive data detection tasks and initialize task parameters, including but not limited to task ID, task name and operation type, detection range type and detection range object; before the task starts to execute, record the log of the start of the task execution, the log information includes task ID, task name and start time, and insert the log information into the task operation log table; S2.2.

2. Determine the specific scope of detection based on the detection scope type and detection scope object in the task parameters; the detection scope types include specific resources, all resources, and resources and directories under designated departments; According to different detection range types, select the corresponding data detection executor to perform the detection task; S2.2.3, the data detection executor performs the detection task according to the selected detection range and data detection parameters; The data detection executor records detailed information during the detection process, including but not limited to the number of detections and the number of detected problems, and returns the recorded information to the system; S2.2.4, update the task parameters according to the results returned by the data detection executor, including the number of detections and the number of problem rows; Record the task execution completion log, including the end time and execution result, and insert the log information into the task operation log table.

6. The method for generating a quality report based on directory detection rules according to claim 3, Features: In step S2.3, the implementation steps are as follows: S2.3.

1. According to the detection scope type, obtain the corresponding parser IParseDetectScope from the detection scope parser factory DetectScopeParserFactory; the parser IParseDetectScope parses the detection scope object and generates a list of objects to be detected List <detectobject> ；< / detectobject> S2.3.

2. Obtain the list of applicable detection rules from the catalog quality rule service catalogQualityRuleService according to the detection scope type and detection scope object. <Map<String,Object> > For each object to be detected, call the doDetect method with the object to be detected and the detection rule list as parameters, perform specific detection operations, and obtain the detection result list List <Map<String,Object> > S2.3.

3. Traverse the test result list List <Map<String,Object> >, add the detection objects with problems to the problem directory set Set <object>In the process, call the batchAddCatalogQualityTaskResultZD method of the catalogQualityTaskResultService to save the test results in batches. S2.3.

4. During the task execution process, the job log recorder XxlJobLogger records log information, including the start of the task execution, the object to be detected, the applicable rules and the detection results.

7. The method for generating a quality report based on directory detection rules according to claim 3, Features: In step S2.4, the implementation steps are as follows: S2.4.

1. The detection execution module receives a list of data objects to be detected and a list of detection rules, traverses the list of detection rules, and applies a corresponding detection rule to each data object; S2.4.2, the rule loading and parsing module obtains the specific detection rule implementation object through Java reflection technology according to the detection rule configuration, and converts it into a rule entity that can be recognized by the system; S2.4.

3. The result processing and recording module collects the test results generated by the test execution module, classifies and summarizes the test results, and records the test results in a database or log for subsequent analysis and processing.

8. A system for generating quality reports based on catalog detection rules, It is characterized in that include: Detection rule configuration module, users can customize detection rules according to their needs; add, modify or delete detection rules through the configuration interface, including rules for sensitive word detection of directory names, data item names, descriptions and information items, and detect the standardization, uniqueness, integrity and confidentiality of directories and data items; The scheduled task module is responsible for customizing the preset time interval according to needs, automatically triggering the detection task through the scheduled task, and performing a comprehensive detection of the data directory; The detection execution module is responsible for receiving the list of data objects to be detected and the list of detection rules, traversing the list of detection rules, and applying the corresponding detection rules to each data object; The rule loading and parsing module is responsible for obtaining specific detection rule implementation objects through Java reflection technology according to the detection rule configuration, and converting them into rule entities that can be recognized by the system; The result processing and recording module is responsible for collecting the test results generated by the test execution module, classifying and summarizing the test results, and recording the test results in a database or log for subsequent analysis and processing; The quality inspection report module is responsible for generating quality inspection reports, displaying and exporting inspection reports through the web interface, displaying detailed inspection results by catalog dimension, and summarizing catalog quality status from the department's perspective; Users can view the test results through the web interface and customize the export format as needed to export the report for subsequent analysis and processing.

9. A device for generating a quality report based on catalog detection rules, Features: The method comprises a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the method according to any one of claims 1 to 7 when executing the computer program.

10. A readable storage medium, Features: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented. < / object>

Images