Method and system for quality monitoring of data assets based on secure intersection

Abstract

The application relates to the field of data quality monitoring, and discloses a data asset quality monitoring method and system based on secure intersection, which comprises the following steps: submitting a metadata collection query request to a metadata query service; judging whether the metadata collection query request meets metadata requirements; if yes, responding to the metadata collection query request, querying metadata based on an SQL analysis engine, generating data quality monitoring SQL information based on the result of the metadata; the SQL analysis engine sends the data quality monitoring SQL information to a secure intersection service of a server; the secure intersection service responds to the data quality monitoring SQL information, reads corresponding metadata from a database, and forms a data file; homomorphic encryption decryption calculation is performed on the to-be-monitored data and the data file; and a quality monitoring result is output and fed back to a monitoring party. The application has the advantages of improving the efficiency of data quality problem monitoring, reducing data transmission cost and storage cost, eliminating data security risks and potential compliance problems.

Description

Technical Field

[0001] This application relates to the technical field of data quality monitoring, and in particular to a method and system for quality monitoring of data assets based on secure intersection. Background Technology

[0002] Multi-party data quality monitoring between regulatory bodies and regulated entities, between enterprises, and among member institutions of group companies is a common requirement. Implementing multi-party data quality monitoring can identify issues such as missing data, inconsistencies, and errors, thereby accelerating the resolution of data problems. Currently, common methods include two approaches: first, conducting manual, offline data comparison and verification by multiple parties; and second, establishing a data platform to centrally collect, aggregate, and integrate detailed data, define monitoring rules, systematically operate data quality monitoring rules, and identify and report data quality issues.

[0003] These methods generally suffer from low efficiency, high cost, and compliance issues: manual labor is inefficient; data aggregation, integration, and monitoring through traditional data platforms pose data security risks, and personal privacy data involves compliance issues; with large data volumes, data transmission costs, storage costs, and data quality monitoring costs remain high. Summary of the Invention

[0004] To improve the efficiency of data quality problem monitoring, reduce data transmission and storage costs, and eliminate data security risks and potential compliance issues, this application provides a data asset quality monitoring method and system based on secure intersection.

[0005] On the one hand, this application provides a method for quality monitoring of data assets based on secure intersection, which adopts the following technical solution:

[0006] A method for quality monitoring of data assets based on secure intersection includes the following steps:

[0007] The monitoring party submits a metadata collection query request to the metadata query service on the server based on the data to be monitored;

[0008] The server determines whether the metadata collection query request meets the metadata requirements based on the metadata adaptation engine. If it does, it responds to the metadata collection query request by querying the metadata from the metadata database, mapping the metadata to the SQL database to form metadata, querying the metadata corresponding to the metadata collection query request based on the SQL analysis engine, and generating data quality monitoring SQL information based on the query results of the metadata.

[0009] The SQL analysis engine sends the data quality monitoring SQL information to the secure intersection service of the server.

[0010] The secure intersection service responds to the data quality monitoring SQL information by reading the corresponding metadata from the SQL database and forming a data file.

[0011] The monitored data and the data file are homomorphically encrypted, and the data in the data file and the monitored data are compressed and valued to obtain a first value result and a second value result. The first value result and the second value result are then subjected to a secure intersection calculation to obtain an intersection result. The intersection result is then homomorphically decrypted to obtain a decryption result.

[0012] Based on the decryption results, output quality monitoring results and feed them back to the monitoring party.

[0013] By adopting the above technical solutions, using technologies such as metadata sharing, homomorphic encryption, and secure intersection, the quality of data assets can be monitored. By implementing multi-party data quality monitoring, it is possible to discover whether there are missing, inconsistent, or erroneous data, thereby accelerating the resolution of data problems. Moreover, without obtaining additional information from the other party, it improves the efficiency of monitoring data quality problems for both parties, reduces data transmission and storage costs, and eliminates data security risks and potential compliance issues.

[0014] Optionally, the metadata adaptation engine is used for the adaptation and transformation of metadata, which includes datasets, data models, and data structures.

[0015] By adopting the above technical solutions, metadata can come from different sources and have different formats and structures, so it is necessary to adapt and convert it for effective use.

[0016] Optionally, the step of outputting quality monitoring results based on the decryption results includes:

[0017] The decryption result is compared with the data to be monitored to determine whether the decryption result is consistent with the data to be monitored, whether there are any missing or incorrect results, and the quality monitoring result is output.

[0018] By adopting the above technical solution, the data quality can be monitored and verified by comparing the decryption results with the data to be monitored. The error types can be reflected by the results of the three monitoring methods, which facilitates the matching of subsequent data analysis and solutions.

[0019] Optionally, comparing the decryption result with the data to be monitored includes:

[0020] Extend SQL functionality in the SQL database by creating user-defined functions and embedding them in the SQL database for query calls. When a query contains a user-defined function, the SQL database system executes the function code corresponding to the query and returns the execution result to the query.

[0021] By adopting the above technical solutions, user-defined functions enhance the flexibility of the database system, enabling users to define and execute specific calculations or operations according to their own needs.

[0022] Optionally, the homomorphic encryption function is f, which transforms plaintext A into ciphertext A' and plaintext B into ciphertext B', i.e., f(A) = A' and f(B) = B'; f is an encryption function capable of homomorphic encryption, and f is applied to C'. -1 Decryption yields result C; C = A + B.

[0023] By adopting the above technical solutions, the right to process data can be separated from the right to own data. This allows enterprises to prevent their own data from being leaked while utilizing the computing power of cloud services.

[0024] Optionally, the method further includes: based on the quality monitoring results, calculating the accuracy rate 'a' within a previously set time period at the current moment, and adjusting the length of the data to be monitored inversely according to the magnitude of the accuracy rate 'a'.

[0025] By adopting the above technical solutions, the higher the accuracy rate 'a' of the monitoring data, the higher the quality of the data, thereby shortening the length of the data to be monitored, optimizing the monitoring method, and improving the efficiency of monitoring.

[0026] Optionally, the method further includes: based on the quality monitoring results, calculating the cumulative correct character count b within a pre-set data length, and adjusting the monitoring period length of the monitoring data according to the positive correlation between the size of the character count b and the size of the character count b.

[0027] By adopting the above technical solution, the more correct characters (b) in the monitored data, the higher the data quality, thereby shortening the monitoring cycle, optimizing the monitoring method, and improving the monitoring efficiency.

[0028] Optionally, the method further includes: calculating an adjustment value c based on the accuracy a and the number of characters b, c = m × accuracy a + n × number of characters, where m and n are both adjustment parameters with values ​​greater than 0; and adjusting the length of the data packets split during the data transmission process to be monitored based on the magnitude of the adjustment value c.

[0029] By adopting the above technical solution, the higher the accuracy rate 'a' of the monitoring data and the more correct characters 'b' of the monitoring data, the higher the quality of the data. This shortens the length of the split data packets, optimizes the monitoring method, and further improves the efficiency of monitoring.

[0030] On the other hand, the multi-party secure and efficient data asset quality monitoring system provided in this application adopts the following technical solution:

[0031] A multi-party secure and efficient data asset quality monitoring system includes a processor, wherein the processor runs a program for the aforementioned data asset quality monitoring method based on secure intersection.

[0032] In summary, this application includes at least one of the following beneficial technical effects: implementing data quality monitoring through a multi-party data quality monitoring tool. The participating parties in data quality monitoring employ technologies such as metadata sharing, homomorphic encryption, and secure intersection to improve the efficiency of monitoring data quality issues, reduce data transmission and storage costs, and eliminate data security risks and potential compliance issues without obtaining additional information from each other. This adapts to the future needs of continuously leveraging the value of data resources in my country's digital economy and enables efficient and sustainable monitoring of the quality of data resources from increasingly complex enterprises. Attached Figure Description

[0033] Figure 1 This is a module diagram of the server side in the data asset quality monitoring method based on secure intersection in this application.

[0034] Figure 2 This is a flowchart illustrating the method steps of the data asset quality monitoring method based on secure intersection in this application.

[0035] Figure 3 This is a flowchart of the data asset quality monitoring method based on secure intersection in this application. Detailed Implementation

[0036] The present application will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the application and are not intended to limit the scope of the application.

[0037] This application discloses a method for quality monitoring of data assets based on secure intersection. (Refer to...) Figure 1 and Figure 2 A method for quality monitoring of data assets based on secure intersection includes the following steps:

[0038] The monitoring party submits a metadata collection query request to the metadata query service on the server based on the data to be monitored;

[0039] The server uses a metadata adaptation engine to determine whether a metadata collection query request meets the metadata requirements. If it does, it responds by retrieving metadata from the metadata database, mapping the metadata to the SQL database to form metadata, and then using an SQL analysis engine to query the metadata corresponding to the metadata collection query request. Based on the results of the metadata query, it generates data quality monitoring SQL information. The metadata requirements serve as the basis for determining whether a metadata collection query request is valid.

[0040] The SQL analysis engine sends data quality monitoring SQL information to the secure request service on the server.

[0041] The secure intersection service responds to the SQL information for data quality monitoring by reading the corresponding metadata from the SQL database and forming a data file.

[0042] The monitoring data and the data file are homomorphically encrypted, and the data in the data file and the monitoring data are compressed and valued to obtain a first value result and a second value result. The first value result and the second value result are then securely intersected to obtain the intersection result. The intersection result is then homomorphically decrypted to obtain the decryption result.

[0043] The quality monitoring results are output based on the decryption results and fed back to the monitoring party.

[0044] By employing technologies such as metadata sharing, homomorphic encryption, and secure intersection, the quality of data assets can be monitored. Through multi-party data quality monitoring, issues such as missing, inconsistent, and erroneous data can be identified, thereby accelerating the resolution of data problems. Furthermore, without obtaining additional information from the other party, the efficiency of monitoring data quality issues for both parties can be improved, data transmission and storage costs can be reduced, and data security risks and potential compliance issues can be eliminated.

[0045] Reference Figure 1 and Figure 3 Before submitting a metadata collection query request to the server-side metadata query service, the metadata query tool must first be updated and deployed, followed by configuring local and target connection information and authentication information. The tool update and deployment steps mainly include checking the current version information and ensuring it matches the latest version from official channels. If not, the latest version is downloaded, installed locally, and then tested to ensure it functions correctly.

[0046] The steps for configuring a local connection include: opening the tool's configuration or settings menu; finding options related to local connection, such as database connection and API key; filling in relevant information according to the actual situation, such as hostname, port number, username, and password; saving the configuration and testing the connection to ensure a successful connection to local resources.

[0047] The steps for configuring the target connection include: Locate the options related to the target (i.e., server) connection in the configuration or settings menu; fill in the target connection information, which may include the server's IP address, port number, protocol type (such as HTTP, HTTPS), etc.; if the target requires authentication, provide the correct authentication information, such as an API key, access token, or username / password combination; save the configuration and test the connection to ensure a successful connection to the target's metadata query service.

[0048] The server provides API interface services, including a metadata query service interface and a secure intersection service interface. The API interface services provide necessary security authentication services.

[0049] An API (Application Programming Interface) service is a service that enables different software applications to communicate and exchange data. An API defines how one application can call (or provide) another application's service. It allows developers to call or provide API services through predefined interface specifications, enabling data transfer and function calls between different systems. The working principle of an API service is as follows: When an application needs to call a service from another application, it sends a request to the target system. The target system provides data and services to the sender by providing the corresponding API interface. This interaction method allows developers to interact solely through the API interface without needing to understand the specific implementation details of their program, thus improving software reusability and interoperability. To access an API service, an API key or token is typically required for authentication and usage tracking purposes.

[0050] The metadata adaptation engine is primarily responsible for collecting, parsing, extracting, storing, and managing metadata. Metadata is data about data, describing information such as the definition, structure, attributes, and relationships of the data. By adapting to different data sources and formats, the metadata engine provides a unified and standardized data view for other engines or services. Therefore, the metadata adaptation engine can work collaboratively with API interface services, providing access to and exchange of metadata through APIs.

[0051] The SQL analysis engine is primarily responsible for parsing and optimizing incoming SQL queries, as well as controlling user access permissions to data in the database. It is tightly integrated with the database, providing robust support for data querying and processing. The API service can invoke the SQL analysis engine's functions to provide data querying and processing services.

[0052] File processing engine. API interface services enable large-scale data processing, rapid network communication, and quick request responses by calling different functional modules. File processing engines typically offer high efficiency and automation when handling large numbers of files or complex file operations. Combining with API interface services can further improve file processing efficiency, automate workflows, and reduce manual intervention and errors.

[0053] Homomorphic encryption / decryption engines provide a function for processing encrypted data, allowing users to perform calculations on ciphertext without decryption, and the result obtained after decryption is the same as the result obtained by performing the same calculation on the plaintext. This feature ensures the security of data processing. API interface services can be integrated with homomorphic encryption / decryption engines to provide secure data processing services.

[0054] Secure intersection engines involve calculating the intersection of two or more datasets while protecting privacy. This is particularly useful in scenarios where multiple parties need to share information but do not want to reveal their complete datasets. Although the direct connection to API services may not be as obvious as with other engines, API services can still serve as a bridge, allowing different applications or systems to securely invoke the functionality of secure intersection engines via APIs.

[0055] The steps for outputting quality monitoring results based on the decryption results include:

[0056] The decryption result is compared with the data to be monitored to determine whether the result is consistent with the data, whether there are any missing or incorrect parts, and the quality monitoring result is output. By comparing the decryption result with the data to be monitored, data quality can be monitored and verified. The error types are reflected through the three monitoring results, which facilitates the matching of subsequent data analysis and solutions.

[0057] in ， The steps for comparing the decryption result with the data to be monitored include:

[0058] Extending SQL functionality within an SQL database involves creating user-defined functions. These functions are embedded within the SQL database for queries to invoke. When a query includes a user-defined function, the SQL database system executes the corresponding function code and returns the result. User-defined functions enhance the flexibility of the database system, allowing users to define and execute specific calculations or operations according to their needs.

[0059] The homomorphic encryption function is f, which transforms plaintext A into ciphertext A' and plaintext B into ciphertext B', i.e., f(A) = A' and f(B) = B'. f is an encryption function capable of homomorphic encryption; applying f to C'... -1Decryption yields result C; C = A + B. Data processing rights and data ownership can be separated, allowing companies to prevent data leaks while leveraging the computing power of cloud services.

[0060] For details, please refer to Table 1, which provides examples of three different monitoring results: inconsistent, incorrect, and correct.

[0061] Table 1:

[0062]

[0063] User-defined functions (UDFs) are a way to extend SQL functionality in a database system. They allow users or developers to create custom functions and use them in SQL queries just like built-in functions. UDFs enhance the flexibility of database systems, enabling users to define and perform specific calculations or operations according to their needs.

[0064] User-defined functions (UDFs) are typically written in a programming language (such as Python, Java, C++, etc.) and registered in the database system so that they can be invoked in SQL queries. When a query contains a UDF, the database system executes the corresponding function code and returns the result to the query.

[0065] The benefits of using UDFs include:

[0066] Code reuse: You can create reusable functions to avoid writing the same logic repeatedly in multiple queries.

[0067] Performance optimization: By encapsulating complex computational logic in UDFs, query performance can be improved.

[0068] Extensibility: UDFs allow users to define custom functions according to their needs, thereby extending the functionality of the database system.

[0069] To further optimize monitoring efficiency, the following three methods are also included:

[0070] The first method involves: based on quality monitoring results, calculating the accuracy rate *a* over a predetermined time period at the current moment, and adjusting the length of the monitored data inversely based on the magnitude of the accuracy rate *a*. A higher accuracy rate *a* indicates higher data quality, thus shortening the length of the monitored data to optimize the monitoring method and improve efficiency. For example, a time period (e.g., the most recent hour, day, or week) is set, and monitoring data and labels are collected within that time period. Then, the accuracy rate *a* is calculated using the `calculate_accuracy` function. A rule or function is then set to adjust the length of the monitored data. For example, a positive correlation function can be used: length of monitored data = k × accuracy rate *a*, where k is an adjustment parameter greater than 0. Alternatively, if the accuracy rate *a* is higher than a certain threshold (e.g., 90%), the length of the monitored data can be reduced; if the accuracy rate *a* is lower than another threshold (e.g., 70%), the length of the monitored data can be increased.

[0071] The second method involves: based on quality monitoring results, calculating the cumulative number of correct characters *b* within a pre-defined data length, and adjusting the monitoring period based on the magnitude of *b*. A higher number of correct characters *b* indicates higher data quality, thus shortening the monitoring period to optimize the monitoring method and improve efficiency. The calculation of the number of correct characters *b* is as follows: defined as the total number of correctly identified characters within a pre-defined data length. This value is calculated using a function, such as `calculate_correct_chars(data, correct_data)`, where `data` is the monitored data and `correct_data` is the corresponding correct data or standard answer. For example, the number of characters in the most recent 100 characters, 1000 characters, or a specific time window can be used to collect monitoring data and corresponding correct data within this length. The correct character count *b* is then calculated using the aforementioned function. Based on the calculated number of correct characters *b*, a rule can be set to adjust the monitoring period. For example, if *b* exceeds a certain threshold, it indicates high data quality, and the monitoring period can be shortened; if *b* is below the threshold, the period can be extended.

[0072] The third method involves: calculating an adjustment value c based on the accuracy rate *a* and the number of characters *b*, where c = m × accuracy rate *a* + n × number of characters, and m and n are adjustment parameters with values ​​greater than 0; and adjusting the length of the data packets segmented during the monitoring data transmission process based on the magnitude of the adjustment value c. A higher accuracy rate *a* and a greater number of correct characters *b* indicate higher data quality, allowing for shorter segmented data packet lengths to optimize the monitoring method and further improve monitoring efficiency. Specifically, the adjustment value c is a weighted sum of accuracy rate *a* and the number of characters *b*, where m and n are adjustment parameters used to control the relative importance of *a* and *b* in c. A larger adjustment value c indicates higher data quality, thus allowing for shorter segmented data packet lengths to reduce transmission overhead and improve efficiency. Conversely, a smaller c may require increasing the data packet length to ensure data integrity and accuracy. Therefore, a function or rule can be set to dynamically adjust the data packet length based on the value of c. For example, a maximum and minimum data packet length can be set, and interpolation or mapping can be performed within this range based on the value of c. Furthermore, to more accurately determine the values ​​of m and n, and to design more complex mapping functions to adjust the packet length based on the value of c, other factors such as network bandwidth, latency, and packet loss rate also need to be considered when adjusting the packet length. Therefore, thorough testing and verification are required before actual deployment.

[0073] The implementation principle of the data asset quality monitoring method based on secure intersection in this application embodiment is as follows: data quality monitoring is carried out through multiple data quality monitoring tools. The participating parties employ technologies such as metadata sharing, homomorphic encryption, and secure intersection to improve the efficiency of monitoring data quality issues, reduce data transmission and storage costs, and eliminate data security risks and potential compliance issues without obtaining additional information from each other. This method adapts to the future needs of continuously leveraging the value of data resources in my country's digital economy and can sustainably and efficiently monitor the quality of data resources from increasingly complex enterprise entities.

[0074] A multi-party secure and efficient data asset quality monitoring system includes a processor, in which a program based on the aforementioned secure intersection method for monitoring data asset quality runs.

[0075] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A method for quality monitoring of data assets based on secure intersection, characterized in that, Includes the following steps: The monitoring party submits a metadata collection query request to the metadata query service on the server based on the data to be monitored; The server uses a metadata adaptation engine to determine whether the metadata collection query request meets the metadata requirements. If the conditions are met, in response to the metadata collection query request, metadata is retrieved from the metadata database, the metadata is mapped to the SQL database to form metadata, the metadata corresponding to the metadata collection query request is queried based on the SQL analysis engine, and data quality monitoring SQL information is generated based on the result of the metadata query. The SQL analysis engine sends the data quality monitoring SQL information to the secure intersection service of the server. The secure intersection service responds to the data quality monitoring SQL information by reading the corresponding metadata from the SQL database and forming a data file. The data to be monitored and the data file are homomorphically encrypted, and the data in the data file and the data to be monitored are compressed and valued to obtain a first value result and a second value result. The first value result and the second value result are then subjected to a secure intersection calculation to obtain an intersection result. The decryption result is obtained by homomorphic decryption of the intersection result; Based on the decryption results, output quality monitoring results and feed them back to the monitoring party.

2. The method for quality monitoring of data assets based on secure intersection as described in claim 1, characterized in that, The metadata adaptation engine is used for the adaptation and transformation of metadata, which includes datasets, data models, and data structures.

3. The method for quality monitoring of data assets based on secure intersection as described in claim 2, characterized in that, The step of outputting quality monitoring results based on the decryption results includes: The decryption result is compared with the data to be monitored to determine whether the decryption result is consistent with the data to be monitored, whether there are any missing or incorrect results, and the quality monitoring result is output.

4. The method for quality monitoring of data assets based on secure intersection as described in claim 3, characterized in that, The step of comparing the decryption result with the data to be monitored includes: Extend SQL functionality in the SQL database by creating user-defined functions and embedding them in the SQL database for query calls. When a query contains a user-defined function, the SQL database system executes the function code corresponding to the query and returns the execution result to the query.

5. The method for quality monitoring of data assets based on secure intersection as described in claim 4, characterized in that, The homomorphic encryption function is f, which transforms plaintext A into ciphertext A' and plaintext B into ciphertext B', i.e., f(A) = A' and f(B) = B'; f is an encryption function capable of homomorphic encryption, and f is applied to C'. -1 Decryption yields result C; C = A + B.

6. The method for quality monitoring of data assets based on secure intersection as described in claim 1, characterized in that, The method also includes: based on the quality monitoring results, calculating the accuracy rate 'a' within the previous set time period at the current moment, and adjusting the length of the data to be monitored inversely according to the magnitude of the accuracy rate 'a'.

7. The method for quality monitoring of data assets based on secure intersection as described in claim 6, characterized in that, The method also includes: based on the quality monitoring results, calculating the cumulative correct character count b within the previously set data length, and adjusting the monitoring period length of the monitoring data according to the positive correlation between the size of the character count b and the data.

8. The method for quality monitoring of data assets based on secure intersection as described in claim 7, characterized in that, The method also includes: calculating an adjustment value c based on the accuracy a and the number of characters b, c = m × accuracy a + n × number of characters, where m and n are both adjustment parameters with values ​​greater than 0; and adjusting the length of the data packets split during the data transmission process to be monitored based on the magnitude of the adjustment value c.

9. A multi-party secure and efficient data asset quality monitoring system, characterized in that, Includes a processor, wherein the processor runs a program of the data asset quality monitoring method based on secure intersection as described in any one of claims 1-8.

Images