Data quality monitoring system and method under data security and privacy protection

Abstract

The invention relates to a data quality monitoring system and method under data security and privacy protection. The system comprises a basic information management module, a metadata integration Agent module, an intelligent classification and grading Agent module, a desensitization rule generation Agent module, a desensitization rule execution Agent module, a monitoring rule generation Agent module, a monitoring rule execution Agent module, a monitoring and abnormity early warning Agent module, a system optimization Agent module and a visualization module. According to the method, the classification and grading technology, the desensitization technology and the large model technology are combined for data quality monitoring, data differential protection can be achieved, quality monitoring rules can be intelligently generated, differential monitoring strategy configuration based on data service importance and sensitivity can be achieved, safe and explainable anomaly analysis and reports are provided, and the safety and reliability of data quality monitoring are improved. Therefore, the pertinence, the safety and the self-adaptive capability of monitoring are improved.

Description

Technical Field

[0001] This invention relates to the field of data quality monitoring, and in particular to a data quality monitoring system and method under the protection of data security and privacy. Background Technology

[0002] Data quality monitoring refers to the continuous monitoring and evaluation of data quality indicators such as accuracy, completeness, consistency, uniqueness, and timeliness through a series of technical means and tools to ensure that the data meets business needs and standards. By monitoring data quality, enterprises can promptly identify and resolve quality issues in their data, thereby improving data reliability and value.

[0003] Existing data quality monitoring technologies suffer from a lack of specificity, low security, and adaptability, leading to significant monitoring difficulties, low efficiency, and delayed or distorted results. Furthermore, they are susceptible to leakage or tampering during data transmission, storage, and processing, struggle to cope with changing data and business environments, and increase labor costs and subjective decision-making risks. Therefore, existing data quality monitoring technologies have the following drawbacks:

[0004] 1) Existing data quality monitoring technologies lack specificity. Current technologies do not classify and categorize data according to its importance, leading to increased monitoring difficulty and poor efficiency and effectiveness.

[0005] 2) Existing data quality monitoring technologies have low security. Current data quality monitoring technologies do not address encryption and privacy protection measures during data transmission, storage, and processing, leading to data interception during transmission, tampering during storage, or leakage during processing, thus causing serious data security and privacy issues.

[0006] 3) Existing data quality monitoring technologies lack adaptability. Current data quality monitoring rules heavily rely on expert experience, lacking adaptability and dynamic learning capabilities. This makes it difficult to cope with changing data and business environments, leading to delayed or distorted monitoring results and increasing labor costs and subjective risks. Summary of the Invention

[0007] To address the technical problems existing in the background art, this invention provides a data quality monitoring system and method under data security and privacy protection. It combines classification and grading technology, de-identification technology and large model technology for data quality monitoring, which can realize differentiated data protection, intelligent generation of quality monitoring rules, and differentiated monitoring strategy configuration based on the importance and sensitivity of data business. It also provides secure and interpretable anomaly analysis and reporting, thereby improving the targeting, security and adaptability of monitoring.

[0008] The technical solution of this invention is as follows: This invention is a data quality monitoring system under data security and privacy protection, characterized in that: the data quality monitoring system under data security and privacy protection includes a basic information management module, a metadata integration agent module, an intelligent classification and grading agent module, a de-identification rule generation agent module, a de-identification rule execution agent module, a monitoring rule generation agent module, a monitoring rule execution agent module, a monitoring and anomaly warning agent module, a system optimization agent module, and a visualization module. The basic information management module serves as the system entry point and is responsible for uniformly managing the connection parameters and access permissions of heterogeneous data sources; the metadata integration agent... The module acts as the data perception center of the system, receiving connection information for the data to be monitored from the basic information management module. The intelligent classification and grading agent module receives metadata information obtained from the metadata collection agent module. Using a classification and grading model and knowledge base, it identifies and labels the metadata information of the data to be monitored based on business importance and sensitivity, providing a basis for subsequent differentiated strategy formulation. The desensitization rule generation agent module receives the business importance and sensitivity classification and grading results of the data to be monitored from the intelligent classification and grading agent module. It preprocesses these results to obtain sensitivity identification results and automatically generates rules corresponding to the sensitivity levels using a desensitization rule generation model and built-in desensitization rule templates. The data anonymization rule scripts ensure that data at different sensitivity levels is properly protected. The anonymization rule execution agent module is responsible for automatically establishing real-time connections to different data sources and performing real-time anonymization of the data to be monitored based on the connection information configured by the basic information management module, including the database address, authentication credentials, and connection parameters, as well as the data anonymization rule script set generated by the anonymization rule generation agent module. Under the unified scheduling of the anonymization rule execution model, it calls the multi-source data real-time anonymization tool to automatically establish real-time connections to different data sources and perform real-time anonymization of the data to be monitored. The monitoring rule generation agent module preprocesses the data based on the business importance and sensitivity classification and grading results of the data to be monitored output by the intelligent classification and grading agent module to obtain business importance labels. As a result, the monitoring rule generation model and monitoring rule knowledge base are used to automatically generate a matching monitoring rule set. The monitoring rule execution agent module receives the connection information of the data to be monitored configured by the basic information management module, including the database address, authentication credentials, connection parameters, and the data quality monitoring rule set generated by the monitoring rule generation agent module, including Cron scheduling expressions, monitoring SQL, and threshold configurations. The monitoring and anomaly warning agent module periodically obtains the most recent monitoring execution result generated by the monitoring rule execution agent module, and uses the monitoring analysis model to perform multi-dimensional aggregation analysis on the structured quality inspection results produced by the monitoring rule execution agent module to generate a monitoring report.

[0009] Furthermore, the basic information management module supports multiple mainstream database types such as MySQL, Oracle, and MongoDB through standardized configuration templates. By defining key parameters such as host address, port, authentication method, database and table mapping, and contact information of the person in charge, it establishes a precise and secure data channel foundation for subsequent metadata collection, data anonymization, monitoring and analysis, and task scheduling.

[0010] Furthermore, the metadata integration agent module establishes connections with various heterogeneous data sources through JDBC and ODBC connectors, automatically discovering and collecting basic metadata including table structure, field type, field description, nullability, primary key, and uniqueness constraints, thereby achieving comprehensive perception and unified management of metadata for the data to be monitored.

[0011] Furthermore, in the intelligent classification and grading agent module, the specific criteria for classifying business importance and sensitivity are as follows: 1) Business Importance Classification Criteria: 1.1) Core Data: Data that has a direct impact on a company's core business and strategic decisions; 1.2) Key Data: Data that plays a vital role in business operations and risk management; 1.3) General data: Data that is valuable for daily operations and supporting business processes; 2) Sensitivity classification criteria: 2.1) Public level: Data that can be publicly shared or released to the public; 2.2) Secret level: Data for internal use only, requiring basic access control; 2.3) Confidential: Data involving personal privacy, corporate trade secrets, or contract information; 2.4) Top Secret: Data that is highly sensitive to the core interests or compliance requirements of the enterprise.

[0012] Furthermore, in the de-identification rule execution Agent module, the real-time connection will adopt different real-time monitoring mechanisms according to the data source type: MySQL, MongoDB, Oracle, in order to achieve continuous and automated dynamic data de-identification.

[0013] Furthermore, in the monitoring rule generation Agent module, the monitoring rule set includes monitoring frequency, monitoring rule scripts, and thresholds. The monitoring rules cover multiple dimensions such as completeness, accuracy, consistency, timeliness, and availability. The monitoring frequency and threshold settings are determined based on business importance and business characteristics.

[0014] Furthermore, under the scheduling of the monitoring rule execution model, the monitoring rule execution agent module calls the monitoring rule execution tool to transform static rules into executable timed calculation tasks, producing quality check results including pass / fail status and detailed indicators, so as to realize real-time monitoring of the data to be monitored, and storing the execution results of each scheduled task corresponding to each monitoring indicator item locally, providing a high-quality, low-latency decision data foundation for the real-time monitoring and early warning agent module.

[0015] Furthermore, once the monitoring and anomaly warning agent module detects through rule engine analysis that the actual measured value of a data quality monitoring indicator exceeds a preset threshold, it immediately triggers an warning mechanism and sends the monitoring report to the relevant task manager via a messaging tool. This messaging tool supports multiple notification methods, including email and DingTalk. In the formula, refer to Time of the first Zhang Biao's first The results of the inspection of each monitoring indicator refer to Time of the first Zhang Biao's first Thresholds for each monitoring indicator.

[0016] Furthermore, the data quality monitoring system under the protection of data security and privacy also includes a system optimization agent module and a visualization result display module. The system optimization agent module establishes a closed-loop optimization mechanism. Through the system optimization model, it stores the classification and grading rules generated in each classification and grading task and the key information of the monitoring rules generated in each monitoring task into the classification and grading knowledge base and the monitoring rule knowledge base, respectively. Through the continuous accumulation of knowledge from different business scenarios, the classification, grading and monitoring effects are continuously optimized, realizing the self-evolution of the system. The visualization result display module analyzes the monitoring results generated by the monitoring rule execution agent module in real time through a visualization dashboard and displays them in the form of visualization charts, providing data managers with a global and intuitive view of the data security and quality situation.

[0017] This invention also provides a data quality monitoring method based on the above-mentioned data security and privacy protection data quality monitoring system, characterized in that the method includes the following steps: Step 1: In the basic information management module, based on the pre-set standardized template for data source connection parameters, configure the connection information for the data to be monitored. This establishes a precise and secure data channel foundation for subsequent metadata collection, data anonymization, monitoring and analysis, and task scheduling. The connection information includes host address, port, authentication method, database name, table name, and contact information for the task manager. The module supports multiple mainstream database types, including MySQL, Oracle, and MongoDB. Step 2: The metadata integration agent module loads and parses the connection information of the data to be monitored configured in the basic information management module. Under the unified scheduling of the metadata integration model, the module uses its built-in collection tools to collect and integrate the metadata of the data to be monitored. Step 3: The intelligent classification and grading agent module receives the metadata information obtained by the metadata integration agent module. Through the classification and grading model and classification and grading knowledge base, it identifies and labels the metadata information of the data to be monitored based on business importance and sensitivity, providing a basis for the formulation of subsequent differentiated strategies. The specific criteria for classifying business importance and sensitivity are as follows: Business Importance Classification Criteria: Core data: Data that has a direct impact on a company's core business and strategic decisions; Key data: Data that plays a vital role in business operations and risk management; General data: Data that is valuable for reference in daily operations and supporting business processes; Sensitivity classification criteria: Public level: Data that can be publicly shared or released to the public; Secret level: Data for internal use only, requiring basic access control; Confidential: Data involving personal privacy, corporate trade secrets, or contract information; Top Secret: Data that is highly sensitive to the company's core interests or compliance requirements; Step 4: The data anonymization rule generation agent module receives the business importance and sensitivity classification results of the data to be monitored from the intelligent classification and grading agent module, preprocesses them to obtain sensitivity identification results, and automatically generates data anonymization rule scripts corresponding to the sensitivity level through the data anonymization rule generation model and built-in data anonymization rule templates to ensure that data of different sensitivity levels are properly protected. Step 5: The data masking rule execution agent module is responsible for establishing real-time connections to different data sources based on the connection information of the data to be monitored configured by the basic information management module, including the database address, authentication credentials, connection parameters, and the data masking rule script set generated by the data masking rule generation agent module. Under the unified scheduling of the data masking rule execution model, it calls the multi-source data real-time masking tool to automatically establish real-time connections to different data sources and execute real-time masking of the data to be monitored. Among them, the real-time connection will adopt different real-time monitoring mechanisms according to the data source type: MySQL, MongoDB, Oracle, so as to achieve continuous and automated dynamic data masking. Step 6: The monitoring rule generation agent module preprocesses the business importance and sensitivity classification results of the data to be monitored, based on the output of the intelligent classification and grading agent module, to obtain business importance labeling results. Then, using the monitoring rule generation model and monitoring rule knowledge base, it automatically generates a set of monitoring rules that match the data. The monitoring rule set includes monitoring frequency, monitoring rule scripts, and thresholds. The monitoring rules cover multiple dimensions such as completeness, accuracy, consistency, timeliness, and availability. The monitoring frequency and thresholds are determined based on business importance and business characteristics. Step 7: The monitoring rule execution agent module receives the connection information of the data to be monitored configured by the basic information management module, including the database address, authentication credentials, connection parameters, and the monitoring rule set generated by the monitoring rule generation agent module, which includes Cron scheduling expressions, monitoring SQL, and threshold configuration. Under the scheduling of the monitoring rule execution model, it calls the monitoring rule execution tool to transform the static rules into executable scheduled calculation tasks, and produces quality check results including pass / fail status and detailed indicators to achieve real-time monitoring of the data to be monitored. It also stores the monitoring-related information locally, providing a high-quality, low-latency decision data foundation for the real-time monitoring and early warning agent module. Step 8: The monitoring and anomaly warning agent module periodically obtains the most recent monitoring execution results generated by the monitoring rule execution agent module, and uses the monitoring analysis model to perform multi-dimensional aggregation analysis on the structured quality inspection results produced by the monitoring rule execution agent module to generate a monitoring report. Once the rule engine analysis finds that the actual measured value of the data quality monitoring indicator exceeds the preset threshold, the warning mechanism is immediately triggered, and the monitoring report is sent to the corresponding task manager through a message sending tool, which supports multiple notification methods such as email and DingTalk. In the formula, refer to Timetable The The results of the inspection of each monitoring indicator refer to Timetable The Thresholds for each monitoring metric; Step 9: The system optimization agent module establishes a closed-loop optimization mechanism. Through the system optimization model, key information such as classification and grading rules generated in each classification and grading task and monitoring rules generated in each monitoring task are stored in the classification and grading knowledge base and the monitoring rule knowledge base, respectively. Through the continuous accumulation of knowledge from different business scenarios, the classification and grading and monitoring effects are continuously optimized, and the system achieves self-evolution. Step 10: The results visualization module analyzes the monitoring results generated by the monitoring rule execution agent module in real time through a visualization dashboard, and displays them in the form of visual charts, providing data managers with a global and intuitive view of the data security and quality situation.

[0018] This invention provides a data quality monitoring system and method under the protection of data security and privacy. It combines classification and grading technology, de-identification technology, and large-scale model technology for data quality monitoring. This enables differentiated data protection, intelligent generation of quality monitoring rules, and configuration of differentiated monitoring strategies based on the importance and sensitivity of data services. It also provides secure and interpretable anomaly analysis and reporting, thereby improving the targeting, security, and adaptability of monitoring. Therefore, it has the following advantages:

[0019] 1) This invention uses classification and grading technology to classify and grade the monitored data according to their importance, ensuring that in actual monitoring, data with high importance is given a higher monitoring priority and data with low importance is given a lower monitoring priority.

[0020] 2) This invention adopts differentiated data protection strategies for data with different sensitivity levels to ensure that data is not leaked or tampered with during the monitoring process, thereby improving the overall data security and privacy protection capabilities.

[0021] 3) This invention introduces large-scale model technology and knowledge base technology to perform semantic analysis on data table fields, automatically generating classification and grading rules and monitoring rules that conform to the current business scenario, and storing these rules. This not only improves the efficiency and accuracy of subsequent rule generation, but also reduces subjective risks and operational costs. Attached Figure Description

[0022] Figure 1 Functional architecture diagram of a data quality monitoring system under data security and privacy protection provided in a specific embodiment of the present invention; Figure 2 A flowchart illustrating data quality monitoring under data security and privacy protection, provided for a specific embodiment of the present invention; Figure 3This is the data source connection information of the data to be monitored in a data quality monitoring system under data security and privacy protection, as provided in a specific embodiment of the present invention. Detailed Implementation

[0023] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments: Example 1 See Figure 1 The data quality monitoring system for data security and privacy protection of this invention combines intelligent classification and grading agent modules, de-identification rule generation agent modules, de-identification rule execution agent modules, monitoring rule generation agent modules, monitoring rule execution agent modules, and monitoring and anomaly early warning agent modules to achieve differentiated protection, intelligent quality monitoring, and anomaly interpretability analysis of data. This addresses the problems of insufficient targeting, low security, and lack of adaptability in existing technologies. The system can improve monitoring efficiency, ensure data security, dynamically adapt to business changes, and reduce the cost of manual intervention.

[0024] The data quality monitoring system under the protection of data security and privacy of this invention comprises the following modules: a basic information management module, a metadata integration agent module, an intelligent classification and grading agent module, a de-identification rule generation agent module, a de-identification rule execution agent module, a monitoring rule generation agent module, a monitoring rule execution agent module, a monitoring and anomaly warning agent module, a system optimization agent module, and a visualization module. This system can intelligently classify and monitor the quality of multi-source data while ensuring data security and privacy, achieving secure management and dynamic evaluation of the entire data flow process, promptly detecting anomalies and notifying relevant personnel, thereby improving the intelligence and adaptability of data quality monitoring.

[0025] The basic information management module serves as the system's entry point, responsible for the unified management of connection parameters and access permissions for heterogeneous data sources. This module supports multiple mainstream database types, including MySQL, Oracle, and MongoDB, through standardized configuration templates. By defining key parameters such as host address, port, authentication method, database / table mapping, and contact information for responsible personnel, it establishes a precise and secure data channel foundation for subsequent metadata collection, data anonymization, monitoring and analysis, and task scheduling.

[0026] The metadata integration agent module serves as the data perception hub of the system. It receives connection information for the data to be monitored from the basic information management module, establishes connections with various heterogeneous data sources through the JDBC / ODBC connector, and automatically discovers and collects basic metadata, including table structure, field type, field description, nullability, primary key, and uniqueness constraints, to achieve comprehensive perception and unified management of the metadata of the data to be monitored.

[0027] The intelligent classification and grading agent module receives metadata information from the metadata collection agent module. Using a classification and grading model and knowledge base, it identifies and labels the metadata information of the data to be monitored based on its business importance and sensitivity, providing a basis for subsequent differentiated strategy formulation. The specific criteria for classifying business importance and sensitivity are as follows:

[0028] 1) Business Importance Classification Criteria: i) Core data: Data that has a direct impact on a company’s core business and strategic decisions.

[0029] ii) Important data: Data that plays an important role in business operations and risk management.

[0030] iii) General data: Data that is valuable for reference in daily operations and auxiliary business processes.

[0031] 2) Sensitivity classification criteria i) Public level: Data that can be publicly shared or released to the public.

[0032] ii) Secret level: Data for internal use only, requiring basic access control.

[0033] iii) Confidential: Data involving personal privacy, corporate trade secrets, or contract information.

[0034] iv) Top Secret: Data that is highly sensitive to the company's core interests or compliance requirements.

[0035] The data anonymization rule generation agent module receives the business importance and sensitivity classification results of the data to be monitored from the intelligent classification and grading agent module, preprocesses them to obtain sensitivity identification results, and automatically generates data anonymization rule scripts corresponding to the sensitivity level through the data anonymization rule generation model and built-in data anonymization rule templates, ensuring that data of different sensitivity levels are properly protected.

[0036] The data masking rule execution agent module is responsible for automatically establishing real-time connections to different data sources and performing real-time data masking on the data to be monitored, based on the connection information (including database address, authentication credentials, connection parameters, etc.) configured by the basic information management module and the data masking rule generation agent module's set of data masking rule scripts. Under the unified scheduling of the data masking rule execution model, it calls the multi-source data real-time masking tool to automatically establish real-time connections to different data sources and execute real-time data masking on the data to be monitored. The real-time connection employs different real-time monitoring mechanisms depending on the data source type (MySQL, MongoDB, Oracle, etc.) to achieve continuous and automated dynamic data masking.

[0037] The monitoring rule generation agent module preprocesses the business importance and sensitivity classification results of the data to be monitored, output by the intelligent classification and grading agent module, to obtain business importance labeling results. Then, using the monitoring rule generation model and monitoring rule knowledge base, it automatically generates a matching set of monitoring rules. This set includes monitoring frequency, monitoring rule scripts, and thresholds. The monitoring rules cover multiple dimensions, including completeness, accuracy, consistency, timeliness, and availability. The monitoring frequency and thresholds are determined based on business importance and characteristics.

[0038] The monitoring rule execution agent module receives connection information (including database address, authentication credentials, connection parameters, etc.) configured by the basic information management module for the data to be monitored, as well as the data quality monitoring rule set (including Cron scheduling expressions, monitoring SQL, and threshold configurations) generated by the monitoring rule generation agent module. Under the scheduling of the monitoring rule execution model, it calls the monitoring rule execution tool to transform static rules into executable scheduled calculation tasks, producing quality check results including pass / fail status and detailed indicators to achieve real-time monitoring of the data to be monitored. It also stores the execution results of each scheduled task corresponding to each monitoring indicator item locally, providing a high-quality, low-latency decision data foundation for the real-time monitoring and early warning agent module.

[0039] The monitoring and anomaly alert agent module periodically retrieves the most recent monitoring execution results from the monitoring rule execution agent module. Using a monitoring analysis model, it performs multi-dimensional aggregation and analysis on the structured quality check results produced by the monitoring rule execution agent module, generating a monitoring report. Once the rule engine analysis detects that the actual measured value of a data quality monitoring indicator exceeds a preset threshold, an alert mechanism is immediately triggered, and the monitoring report is sent to the relevant task manager via a messaging tool. This messaging tool supports multiple notification methods, including email and DingTalk.

[0040] In the formula, refer to Time of the first Zhang Biao's first The results of the inspection of each monitoring indicator refer to Time of the first Zhang Biao's first Thresholds for each monitoring indicator.

[0041] The system optimization agent module establishes a closed-loop optimization mechanism. Through the system optimization model, key information such as classification and grading rules generated in each classification and grading task and monitoring rules generated in each monitoring task are stored in the classification and grading knowledge base and the monitoring rule knowledge base, respectively. Through the continuous accumulation of knowledge from different business scenarios, the classification, grading and monitoring effects are continuously optimized, realizing the self-evolution of the system.

[0042] The visualization results display module uses a visual dashboard to analyze the monitoring results generated by the monitoring rule execution agent module in real time and display them in the form of visual charts, providing data managers with a global and intuitive view of the data security and quality situation.

[0043] Example 2 See Figure 2 This invention also provides a data quality monitoring method for a data quality monitoring system based on data security and privacy protection, which completes metadata collection from heterogeneous data sources, intelligent classification and grading, data anonymization protection, monitoring rule generation and execution, real-time early warning, and system adaptive optimization. Specifically, the execution process is as follows:

[0044] Step 1: In the basic information management module, based on the pre-defined standardized template for data source connection parameters, configure the connection information for the data to be monitored. This establishes a precise and secure data channel foundation for subsequent metadata collection, data anonymization, monitoring analysis, and task scheduling. The connection information includes the host address, port, authentication method, database name, table name, and contact information for the task manager. The module supports multiple mainstream database types such as MySQL, Oracle, and MongoDB. The connection information for the data to be monitored in this embodiment is as follows: Figure 3 As shown.

[0045] Step 2: The metadata integration agent module loads and parses the connection information of the data to be monitored configured in the basic information management module. Under the unified scheduling of the metadata integration model, the module uses its built-in collection tools to collect and integrate the metadata of the data to be monitored.

[0046] Furthermore, the specific execution process of step 2 is as follows: 2.1) The metadata integration agent module reads the connection information and authentication information of the data to be monitored configured in the basic information management module, and parses it into the parameters required by the metadata collection tool through the metadata integration model.

[0047] 2.2 2) The metadata integration model calls the corresponding metadata acquisition tool to obtain data.

[0048] Furthermore, the specific data collection process of each metadata collection tool is as follows: 2.2.1) MySQL metadata collection tool: Connect to the order_db database and collect complete metadata (including database name, table name, field name, field type, whether it is nullable, default value, comments, etc.) of the user_info and order_info tables under it.

[0049] 2.2.2) Oracle Metadata Acquisition Tool: Connect to the HR_PROD database and collect complete metadata (including database name, table name, field name, field type, whether it is nullable, default value, comments, etc.) of the employees, department, and manager tables.

[0050] 2.2.3) Mongo metadata collection tool: Connects to the system_db database and collects metadata information from the access_logs collection under it.

[0051] 2.3) The metadata integration model formats and merges the collected data to be monitored, and outputs it in a unified format.

[0052] Step 3) The intelligent classification and grading agent module receives the metadata information obtained by the metadata integration agent module. Using the classification and grading model and knowledge base, it identifies and labels the metadata information of the data to be monitored based on its business importance and sensitivity, providing a basis for subsequent differentiated strategy formulation. The specific criteria for classifying business importance and sensitivity are as follows:

[0053] 1) Business Importance Classification Standards i) Core data: Data that has a direct impact on a company’s core business and strategic decisions.

[0054] ii) Important data: Data that plays an important role in business operations and risk management.

[0055] iii) General data: Data that is valuable for reference in daily operations and auxiliary business processes.

[0056] 2) Sensitivity classification criteria i) Public level: Data that can be publicly shared or released to the public.

[0057] ii) Secret level: Data for internal use only, requiring basic access control.

[0058] iii) Confidential: Data involving personal privacy, corporate trade secrets, or contract information.

[0059] iv) Top Secret: Data that is highly sensitive to the company's core interests or compliance requirements.

[0060] Furthermore, the specific execution process of step 3 is as follows: 3.1) The intelligent classification and grading agent module reads metadata information obtained by the integrated agent module, including the database name, table name, field name, field type, whether it is empty, default value, comments, etc. of the data to be monitored.

[0061] 3.2) Using a classification and grading model, semantic analysis is performed on the metadata information of the data to be monitored. Combined with historical templates from the classification and grading knowledge base, the business importance and sensitivity of the metadata information are classified and graded, resulting in the classification and grading results of the data to be monitored. The classification and grading results for business importance and sensitivity include: database name, table name, field name, field type, field description, business importance, and sensitivity.

[0062] Step 4: The Desensitization Rule Generation Agent module receives the importance and sensitivity classification results of the data to be monitored from the intelligent classification and grading Agent module, preprocesses them to obtain sensitivity identification results, and automatically generates data desensitization rule scripts corresponding to the sensitivity levels through the desensitization rule generation model and built-in desensitization rule templates to ensure that data of different sensitivity levels are properly protected.

[0063] Furthermore, the specific execution process of step 4 is as follows: 4.1) The desensitization rule generation agent module reads the business importance and sensitivity classification and grading results of the data to be monitored output by the intelligent classification and grading agent module, preprocesses them, and obtains sensitivity labeling results.

[0064] 4.2) Based on the built-in desensitization rule template, the desensitization rule generation model is used to perform semantic recognition on the sensitivity labeling results obtained in Step 1, and automatically generate SQL scripts that match the sensitivity level.

[0065] Step 5: The Agent module, responsible for executing data masking rules, is responsible for establishing real-time connections to different data sources and performing real-time data masking on the data to be monitored, based on the connection information (including database address, authentication credentials, connection parameters, etc.) configured in the basic information management module and the data masking rule generation Agent module's set of data masking rule scripts. Under the unified scheduling of the data masking rule execution model, it calls the multi-source real-time data masking tool to automatically establish real-time connections to different data sources and execute real-time data masking on the data to be monitored. The real-time connection employs different real-time monitoring mechanisms depending on the data source type (MySQL, MongoDB, Oracle, etc.) to achieve continuous and automated dynamic data masking.

[0066] Furthermore, the specific execution process of step 5 is as follows: 5.1) The Agent module reads the connection information of the data to be monitored configured by the Basic Information Management Module and the desensitization rule generation Agent module generates a set of SQL scripts for data desensitization rules.

[0067] 5.2) The data masking rule execution model parses the connection information and data masking rule SQL script set of the data to be monitored read in step 5.1), and calls the multi-source data real-time data masking tool to perform real-time data masking on the dataset to be monitored. The real-time data masking mechanism varies depending on the data source type. In this embodiment, the MySQL / Oracle database uses database triggers (BEFORE INSERT / BEFORE UPDATE) to achieve immediate data masking, ensuring that newly added or updated data is processed immediately. MongoDB uses Change Streams or watch() to monitor collection changes in real time and performs data masking operations on newly added or updated documents.

[0068] Step 6: The monitoring rule generation agent module, based on the business importance and sensitivity classification results of the data to be monitored output by the intelligent classification and grading agent module, preprocesses the data to obtain business importance labeling results. Then, using the monitoring rule generation model and monitoring rule knowledge base, it automatically generates a matching set of monitoring rules. This set includes monitoring frequency, monitoring rule scripts, and thresholds. The monitoring rules cover multiple dimensions such as completeness, accuracy, consistency, timeliness, and availability. The monitoring frequency and thresholds are determined based on business importance and characteristics. Further, the specific execution process of step 6 is as follows:

[0069] 6.1) The monitoring rule generation agent module reads the business importance and sensitivity classification results of the data to be monitored from the output of the intelligent classification and grading agent module, and preprocesses them to obtain the business importance classification results of the data to be monitored.

[0070] 6.2) The monitoring rule generation model, combining historical rule templates from the monitoring rule knowledge base, performs semantic recognition on the business importance classification and grading results of the data to be monitored obtained in Step 1, and automatically generates a monitoring frequency (cron expression), monitoring rule SQL script, and threshold that match the business importance. The specific configuration of the monitoring frequency and threshold in this embodiment is as follows:

[0071] i) Core data: Monitored every 1 minute (* / 1 * * * *), null value rate threshold <0.1% ii) Important data: Monitored every 5 minutes (* / 5 * * * *), null value rate threshold <1%.

[0072] iii) General data: Monitored once per hour (0 * / 1 * * *), with a null value rate threshold of <5%.

[0073] Step 7: The monitoring rule execution agent module receives the connection information (including database address, authentication credentials, and connection parameters) of the data to be monitored configured by the basic information management module, as well as the monitoring rule set (including Cron scheduling expressions, monitoring SQL, and threshold configuration) generated by the monitoring rule generation agent module. Under the scheduling of the monitoring rule execution model, it calls the monitoring rule execution tool to transform the static rules into executable scheduled calculation tasks, producing quality check results including pass / fail status and detailed indicators, so as to realize real-time monitoring of the data to be monitored, and store the monitoring-related information locally, providing a high-quality, low-latency decision data foundation for the real-time monitoring and early warning agent module.

[0074] Furthermore, the specific execution process of step 7 is as follows: 7.1) The monitoring rule execution Agent module reads the connection information and monitoring rules configured by the basic information management module to generate the monitoring rule set generated by the Agent module.

[0075] 7.2) The monitoring rule execution model calls the built-in monitoring rule execution tool of the module, passes in the connection information and monitoring rule set read in Step 17.1, performs timed monitoring on the anonymized data to be monitored according to the pre-generated Cron scheduling cycle, and persists the monitoring execution results for each time.

[0076] The further step 7.2) is executed as follows: 7.2.1) Monitoring rule execution model: Call the built-in monitoring rule execution tool of the module and pass in the connection information and monitoring rule set read in step 7.1.

[0077] 7.2.2) The monitoring rule execution tool creates and stores the basic task information and monitoring metrics under the current dataset to be monitored. A CronTrigger scheduled task is created for each monitoring metric. When the scheduled task is due, it is automatically triggered and executed. The execution results of each scheduled task for each monitoring metric are stored locally, providing a data foundation for subsequent monitoring and alerts. The basic task information includes: task code (task_id), creation time (create_time), status, data source connection information (data_sources), and monitoring rule set (monitoring_rules). The monitoring metrics include: monitoring metric ID, task ID, table name (table_name), field name (field_name), metric rule (sql_rule), monitoring frequency (frequency_cron), status, threshold, creation time (create_time), and update time (update_time). The monitoring task execution results include: executed task ID, monitoring metric ID (monitor_item_id), etc. The execution metrics include the execution_code, start_time, end_time, execution status, actual_value, threshold, and create_time.

[0078] Step 8: The Monitoring and Anomaly Warning Agent module periodically retrieves the most recent monitoring execution results generated by the Monitoring Rule Execution Agent module. Using a monitoring analysis model, it performs multi-dimensional aggregation analysis on the structured quality check results produced by the Monitoring Rule Execution Agent module, generating a monitoring report. Once the rule engine analysis detects that the actual measured value of a data quality monitoring indicator exceeds a preset threshold, an early warning mechanism is immediately triggered, and the monitoring report is sent to the relevant task manager via a messaging tool. This messaging tool supports multiple notification methods, including email and DingTalk.

[0079] In the formula, refer to Timetable The The results of the inspection of each monitoring indicator refer to Timetable The Thresholds for each monitoring indicator.

[0080] Furthermore, the specific execution process of step 8 is as follows: 8.1) Monitoring and Anomaly Warning Agent module receives monitoring rules, obtains monitoring rule execution, and executes the task code generated by the Agent module.

[0081] 8.2) The monitoring and analysis model is uniformly scheduled, and the database operation tool is called to obtain the monitoring frequency of all monitoring indicators under the task code obtained in 8.1, and the minimum value is taken.

[0082] 8.3) The monitoring analysis model calls the monitoring report generation tool. The monitoring report generation tool uses the minimum monitoring frequency obtained in step 8.2) as the report generation cycle. It periodically queries the monitoring task execution result table for the most recent execution result of all monitoring execution items under 8.1. Based on the query results, it generates periodic reports. When the actual value of a monitoring indicator item exceeds the set threshold, an early warning is immediately triggered, and the periodic report is sent to the designated person in charge through the message sending tool.

[0083] Step 9: The system optimization agent module establishes a closed-loop optimization mechanism. Through the system optimization model, key information such as classification and grading rules generated in each classification and grading task and monitoring rules generated in each monitoring task are stored in the classification and grading knowledge base and the monitoring rule knowledge base, respectively. Through the continuous accumulation of knowledge from different business scenarios, the classification and grading and monitoring effects are continuously optimized, realizing the self-evolution of the system.

[0084] Furthermore, the specific execution process of step 9 is as follows: 9.1) The system optimization agent module obtains the classification and grading rules and monitoring rule script sets from the intelligent classification and grading agent module and the monitoring rule generation agent module, respectively.

[0085] 9.2) The system optimization model performs intelligent identification to determine whether it is a classification and grading rule or a monitoring rule script set. Based on the rule type, the corresponding knowledge base management tool is called to store the classification and grading rules in the classification and grading knowledge base and the monitoring rules in the monitoring rule knowledge base. Meta-information such as the rule generation time, source task, and application scenario is recorded. Through continuous accumulation and optimization, historical experience support is provided for subsequent rule generation, scheduling execution, and decision-making, realizing a closed loop of rule optimization.

[0086] Step 10: The results visualization module analyzes the monitoring results generated by the monitoring rule execution agent module in real time through a visualization dashboard, and displays them in the form of visual charts, providing data managers with a global and intuitive view of the data security and quality situation.

[0087] This invention provides the following technical solutions: a classification and grading monitoring mechanism based on the importance of data services, including a classification and grading model and a dynamic mapping relationship between the importance of data services and monitoring strategies; a differentiated security protection system for sensitive data, encompassing technical implementation schemes for multi-layered security measures such as data processing and desensitization; an intelligent generation and adaptive optimization method for monitoring rules based on a large language model, including a closed-loop technical process of semantic analysis, rule generation, strategy adjustment, and effect feedback; and an intelligent decision-making and automated execution mechanism for the entire data quality monitoring process, achieving integrated collaborative processing from data access, classification and grading, security protection to dynamic monitoring.

[0088] The key points of this invention's protective technology, based on the aforementioned technologies, are: protecting the technical solution of a data classification and grading quality monitoring system based on a large model, including data importance assessment methods, multi-level monitoring frequency division mechanisms, and their correlation with monitoring resource scheduling; protecting differentiated security protection methods for data quality monitoring, including sensitive data identification, security level division, and corresponding encryption technology solutions; protecting intelligent generation and dynamic optimization technology for monitoring rules based on a large language model, including semantic feature extraction, automatic rule generation, and historical feedback fusion mechanisms; and protecting the intelligent architecture of the entire data quality monitoring system, encompassing integrated technical solutions for data identification, classification and grading, security protection, rule generation, and adaptive execution.

[0089] The following beneficial effects can be achieved by implementing the technical solution of this invention: (1) Realize data classification and hierarchical monitoring, and improve the efficiency of monitoring resource utilization. In view of the problem of resource waste and low efficiency caused by the failure of traditional data quality monitoring technology to distinguish the importance of data, this invention introduces a data classification and hierarchical mechanism to automatically allocate the monitoring frequency according to the importance and business value of the data, so as to ensure that key data receives more intensive and strict monitoring, and non-key data adopts an appropriate monitoring strategy, thereby optimizing the configuration of monitoring resources and significantly improving the monitoring efficiency and effect. (2) Realize differentiated data security protection and enhance data security and privacy protection capabilities. In view of the problem that existing monitoring technologies lack security protection measures, this invention establishes a differentiated protection strategy that matches the data sensitivity level, effectively prevents data leakage, tampering and abuse, and significantly improves the security and compliance of the data quality monitoring process. (3) Realize adaptive optimization of monitoring rules and improve the accuracy and real-time performance of monitoring. To address the issues of traditional monitoring rules relying on expert experience and lacking flexibility, this invention introduces the semantic analysis capabilities of a large language model, combined with historical monitoring logs and business context, to achieve automatic generation and dynamic adjustment of monitoring rules. This enables monitoring strategies to adaptively optimize with changes in data characteristics and business needs, significantly improving monitoring accuracy and real-time performance while reducing labor costs and subjective risks. (4) Achieving intelligent monitoring decision-making reduces operational complexity and costs. By constructing a full-link intelligent monitoring system from data identification, classification and grading, security protection to rule adaptation, the need for manual configuration and intervention is significantly reduced, lowering operational complexity and labor costs. At the same time, through continuous learning and optimization mechanisms, the system possesses long-term evolution capabilities to adapt to increasingly complex data environments.

[0090] The technical contents of this invention and those not specifically described in the above embodiments are the same as those in the prior art.

[0091] The above are merely specific embodiments disclosed in this invention, but the scope of protection disclosed in this invention is not limited thereto. The scope of protection disclosed in this invention should be determined by the scope of the claims.

Claims

1. A data quality monitoring system for data security and privacy protection, characterized in that: The data quality monitoring system under the protection of data security and privacy includes a basic information management module, a metadata integration agent module, an intelligent classification and grading agent module, a de-identification rule generation agent module, a de-identification rule execution agent module, a monitoring rule generation agent module, a monitoring rule execution agent module, a monitoring and anomaly warning agent module, a system optimization agent module, and a visualization module. The basic information management module serves as the system's entry point, responsible for uniformly managing the connection parameters and access permissions of heterogeneous data sources. The metadata integration agent module acts as the system's data perception center, receiving connection parameters for the data to be monitored configured by the basic information management module. The intelligent classification and grading agent module receives metadata information obtained from the metadata collection agent module. Using a classification and grading model and a classification and grading knowledge base, it identifies and labels the metadata information of the data to be monitored based on business importance and sensitivity, providing a basis for subsequent differentiated strategy formulation. The de-identification rule generation agent module receives the business importance and sensitivity classification and grading results of the data to be monitored from the intelligent classification and grading agent module. It preprocesses these results to obtain sensitivity identification results and automatically generates data de-identification rule scripts corresponding to the sensitivity levels using a de-identification rule generation model and built-in de-identification rule templates, ensuring that data at different sensitivity levels is de-identified. Appropriate protection; the de-identification rule execution agent module is responsible for automatically establishing real-time connections to different data sources and performing real-time de-identification of the data to be monitored based on the connection information of the data to be monitored configured by the basic information management module, including the database address, authentication credentials, connection parameters, and the data de-identification rule script set generated by the de-identification rule generation agent module, under the unified scheduling of the de-identification rule execution model, and calling the multi-source data real-time de-identification tool; the monitoring rule generation agent module preprocesses the business importance and sensitivity classification and grading results of the data to be monitored based on the output of the intelligent classification and grading agent module to obtain business importance labeling results, and uses the monitoring rule generation... The model and monitoring rule knowledge base automatically generates a matching set of monitoring rules. The monitoring rule execution agent module receives connection information of the data to be monitored configured by the basic information management module, including database address, authentication credentials, connection parameters, and the data quality monitoring rule set generated by the monitoring rule generation agent module, including Cron scheduling expressions, monitoring SQL, and threshold configurations. The monitoring and anomaly warning agent module periodically obtains the most recent monitoring execution result generated by the monitoring rule execution agent module and uses the monitoring analysis model to perform multi-dimensional aggregation analysis on the structured quality inspection results produced by the monitoring rule execution agent module to generate a monitoring report.

2. The data quality monitoring system under data security and privacy protection according to claim 1, characterized in that: The basic information management module supports multiple mainstream database types such as MySQL, Oracle, and MongoDB through standardized configuration templates. By defining key parameters such as host address, port, authentication method, database and table mapping, and contact information of the person in charge, it establishes a precise and secure data channel foundation for subsequent metadata collection, data anonymization, monitoring and analysis, and task scheduling.

3. The data quality monitoring system under data security and privacy protection according to claim 2, characterized in that: The metadata integration agent module establishes connections with various heterogeneous data sources through JDBC and ODBC connectors, automatically discovers and collects basic metadata including table structure, field type, field description, nullability, primary key and uniqueness constraints, and realizes comprehensive perception and unified management of metadata of the data to be monitored.

4. The data quality monitoring system under data security and privacy protection according to claim 3, characterized in that: The specific criteria for classifying business importance and sensitivity in the intelligent classification and grading agent module are as follows: 1) Business Importance Classification Criteria: 1.1) Core Data: Data that has a direct impact on a company's core business and strategic decisions; 1.2) Key Data: Data that plays a vital role in business operations and risk management; 1.3) General data: Data that is valuable for daily operations and supporting business processes; 2) Sensitivity classification criteria: 2.1) Public level: Data that can be publicly shared or released to the public; 2.2) Secret level: Data for internal use only, requiring basic access control; 2.3) Confidential: Data involving personal privacy, corporate trade secrets, or contract information; 2.4) Top Secret: Data that is highly sensitive to the core interests or compliance requirements of the enterprise.

5. The data quality monitoring system under data security and privacy protection according to claim 4, characterized in that: In the Agent module for executing the data masking rules, the real-time connection will use different real-time monitoring mechanisms depending on the data source type: MySQL, MongoDB, or Oracle, in order to achieve continuous and automated dynamic data masking.

6. The data quality monitoring system under data security and privacy protection according to claim 5, characterized in that: In the monitoring rule generation Agent module, the monitoring rule set includes monitoring frequency, monitoring rule scripts, and thresholds. The monitoring rules cover multiple dimensions such as completeness, accuracy, consistency, timeliness, and availability. The monitoring frequency and thresholds are set according to the importance and characteristics of the business.

7. The data quality monitoring system under data security and privacy protection according to claim 6, characterized in that: Under the scheduling of the monitoring rule execution model, the monitoring rule execution agent module calls the monitoring rule execution tool to transform static rules into executable timed calculation tasks, producing quality check results including pass / fail status and detailed indicators, so as to realize real-time monitoring of the data to be monitored, and storing the execution results of each scheduled task corresponding to each monitoring indicator item locally, providing a high-quality, low-latency decision data foundation for the real-time monitoring and early warning agent module.

8. The data quality monitoring system under data security and privacy protection according to claim 7, characterized in that: Once the monitoring and anomaly warning agent module detects through rule engine analysis that the actual measured value of a data quality monitoring indicator exceeds a preset threshold, it immediately triggers an warning mechanism and sends a monitoring report to the relevant task manager via a messaging tool. This messaging tool supports multiple notification methods, including email and DingTalk. In the formula, refer to Time of the first Zhang Biao's first The results of the inspection of each monitoring indicator refer to Time of the first Zhang Biao's first Thresholds for each monitoring indicator.

9. The data quality monitoring system for data security and privacy protection according to any one of claims 1 to 8, characterized in that: The data quality monitoring system under the protection of data security and privacy also includes a system optimization agent module and a visualization result display module. The system optimization agent module establishes a closed-loop optimization mechanism. Through the system optimization model, it stores the classification and grading rules generated in each classification and grading task and the key information of the monitoring rules generated in each monitoring task into the classification and grading knowledge base and the monitoring rule knowledge base, respectively. Through the continuous accumulation of knowledge from different business scenarios, the classification, grading and monitoring effects are continuously optimized, realizing the self-evolution of the system. The visualization result display module analyzes the monitoring results generated by the monitoring rule execution agent module in real time through a visualization dashboard and displays them in the form of visualization charts, providing data managers with a global and intuitive view of the data security and quality situation.

10. A data quality monitoring method based on the data quality monitoring system for data security and privacy protection as described in claim 1, characterized in that: The method includes the following steps: Step 1: In the basic information management module, based on the pre-set standardized template for data source connection parameters, configure the connection information for the data to be monitored. This establishes a precise and secure data channel foundation for subsequent metadata collection, data anonymization, monitoring and analysis, and task scheduling. The connection information includes host address, port, authentication method, database name, table name, and contact information for the task manager. The module supports multiple mainstream database types, including MySQL, Oracle, and MongoDB. Step 2: The metadata integration agent module loads and parses the connection information of the data to be monitored configured in the basic information management module. Under the unified scheduling of the metadata integration model, the module uses its built-in collection tools to collect and integrate the metadata of the data to be monitored. Step 3: The intelligent classification and grading agent module receives the metadata information obtained by the metadata integration agent module. Through the classification and grading model and classification and grading knowledge base, it identifies and labels the metadata information of the data to be monitored based on business importance and sensitivity, providing a basis for the formulation of subsequent differentiated strategies. The specific criteria for classifying business importance and sensitivity are as follows: Business Importance Classification Criteria: Core data: Data that has a direct impact on a company's core business and strategic decisions; Key data: Data that plays a vital role in business operations and risk management; General data: Data that is valuable for reference in daily operations and supporting business processes; Sensitivity classification criteria: Public level: Data that can be publicly shared or released to the public; Secret level: Data for internal use only, requiring basic access control; Confidential: Data involving personal privacy, corporate trade secrets, or contract information; Top Secret: Data that is highly sensitive to the company's core interests or compliance requirements; Step 4: The data anonymization rule generation agent module receives the business importance and sensitivity classification results of the data to be monitored from the intelligent classification and grading agent module, preprocesses them to obtain sensitivity identification results, and automatically generates data anonymization rule scripts corresponding to the sensitivity level through the data anonymization rule generation model and built-in data anonymization rule templates to ensure that data of different sensitivity levels are properly protected. Step 5: The data masking rule execution agent module is responsible for establishing real-time connections to different data sources based on the connection information of the data to be monitored configured by the basic information management module, including the database address, authentication credentials, connection parameters, and the data masking rule script set generated by the data masking rule generation agent module. Under the unified scheduling of the data masking rule execution model, it calls the multi-source data real-time masking tool to automatically establish real-time connections to different data sources and execute real-time masking of the data to be monitored. Among them, the real-time connection will adopt different real-time monitoring mechanisms according to the data source type: MySQL, MongoDB, Oracle, so as to achieve continuous and automated dynamic data masking. Step 6: The monitoring rule generation agent module preprocesses the business importance and sensitivity classification results of the data to be monitored, based on the output of the intelligent classification and grading agent module, to obtain business importance labeling results. Then, using the monitoring rule generation model and monitoring rule knowledge base, it automatically generates a set of monitoring rules that match the data. The monitoring rule set includes monitoring frequency, monitoring rule scripts, and thresholds. The monitoring rules cover multiple dimensions such as completeness, accuracy, consistency, timeliness, and availability. The monitoring frequency and thresholds are determined based on business importance and business characteristics. Step 7: The monitoring rule execution agent module receives the connection information of the data to be monitored configured by the basic information management module, including the database address, authentication credentials, connection parameters, and the monitoring rule set generated by the monitoring rule generation agent module, which includes Cron scheduling expressions, monitoring SQL, and threshold configuration. Under the scheduling of the monitoring rule execution model, it calls the monitoring rule execution tool to transform the static rules into executable scheduled calculation tasks, and produces quality check results including pass / fail status and detailed indicators to achieve real-time monitoring of the data to be monitored. It also stores the monitoring-related information locally, providing a high-quality, low-latency decision data foundation for the real-time monitoring and early warning agent module. Step 8: The monitoring and anomaly warning agent module periodically obtains the most recent monitoring execution results generated by the monitoring rule execution agent module, and uses the monitoring analysis model to perform multi-dimensional aggregation analysis on the structured quality inspection results produced by the monitoring rule execution agent module to generate a monitoring report; Once the rule engine analysis reveals that the actual measured value of the data quality monitoring indicator exceeds the preset threshold, an early warning mechanism is immediately triggered, and the monitoring report is sent to the relevant task manager via a messaging tool. The messaging tool supports multiple notification methods such as email and DingTalk. In the formula, refer to Timetable The The results of the inspection of each monitoring indicator refer to Timetable The Thresholds for each monitoring metric; Step 9: The system optimization agent module establishes a closed-loop optimization mechanism. Through the system optimization model, key information such as classification and grading rules generated in each classification and grading task and monitoring rules generated in each monitoring task are stored in the classification and grading knowledge base and the monitoring rule knowledge base, respectively. Through the continuous accumulation of knowledge from different business scenarios, the classification and grading and monitoring effects are continuously optimized, and the system achieves self-evolution. Step 10: The results visualization module analyzes the monitoring results generated by the monitoring rule execution agent module in real time through a visualization dashboard, and displays them in the form of visual charts, providing data managers with a global and intuitive view of the data security and quality situation.

Images