Data governance method and apparatus, electronic device, and computer-readable storage medium

Abstract

Embodiments of the present application disclose a data governance method and device, electronic equipment and computer readable storage medium, relating to the technical field of data processing, applied to electronic equipment including data governance plug-ins, the data governance plug-ins including a memory pre-verification layer, a SQL accurate verification layer and a business conversion layer, the method comprising: performing complex verification through the memory pre-verification layer, and generating an intermediate verification result based on multi-data source data; performing targeted verification through the SQL accurate verification layer, and generating a rule violation data ID list carrying a rule violation type identifier based on the intermediate verification result, the rule violation data ID list being used for accurately locating and classifying abnormal data in the multi-data source data; performing hierarchical conversion through the business conversion layer, and performing field conversion on the multi-data source data based on the rule violation data ID list to obtain converted data. The embodiments of the present application can improve the accuracy, efficiency and flexibility of data processing.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically, to a data governance method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] In data processing, routine validation and transformation are fundamental steps in ensuring data quality. Common routine validations include validating named entity data such as ID cards, mobile phone numbers, IP addresses, and email addresses. These validations primarily use established rules to determine the correctness of the data format. Routine transformations mainly involve simple format adjustments and processing of the data itself, used to transform the raw data into a uniform, standardized, and analyzable format. Their core function is to eliminate data noise, improve data consistency, and provide high-quality input for subsequent data analysis, machine learning, and other scenarios.

[0003] Currently, while existing technologies provide basic solutions for routine validation and transformation, their limitations are becoming increasingly apparent as business scenarios become more complex. Most existing validation technologies only analyze the data itself and cannot meet the complex validation needs across data sources and tables. For complex validation scenarios, such as those involving multiple table joins and complex query conditions, existing technologies lack effective processing methods and cannot efficiently obtain a list of data IDs that do not meet the conditions, thus affecting the efficiency and accuracy of data validation. Regarding data transformation, existing technologies can only perform simple data transformations themselves and are insufficient to meet the data transformation needs under complex business logic.

[0004] No effective solutions have yet been proposed to address the problems in the relevant technologies. Summary of the Invention

[0005] To address the problems in related technologies, this application proposes a data governance method, apparatus, electronic device, and computer-readable storage medium. The aim is to solve the problem that existing data verification and transformation technologies cannot meet the needs of complex business applications. It provides a technical solution capable of cross-data source and cross-table verification, handling complex SQL verification, and meeting complex business transformation requirements, thereby improving the accuracy, efficiency, and flexibility of data processing to adapt to diverse business scenarios.

[0006] The technical solution of this application embodiment is implemented as follows:

[0007] on the one hand:

[0008] This application provides a data governance method applied to an electronic device including a data governance plugin. The data governance plugin includes a memory pre-verification layer, an SQL precise verification layer, and a business transformation layer. The method includes:

[0009] Complex verification is performed through the memory pre-verification layer, and intermediate verification results are generated based on data from multiple data sources.

[0010] The SQL precision verification layer performs targeted verification, and generates a list of violation data IDs carrying violation type identifiers based on the intermediate verification results. The list of violation data IDs is used to accurately locate and classify abnormal data in the multi-data source data.

[0011] The business conversion layer performs hierarchical conversion, and the multi-source data is converted based on the list of violation data IDs to obtain the converted data.

[0012] Specifically, the step of performing complex verification through the memory pre-verification layer and generating intermediate verification results based on data from multiple data sources includes:

[0013] Data from multiple data sources is loaded into memory using data preloading technology;

[0014] A graph-structured dataset of relationships across data sources is constructed in the memory.

[0015] Distributed cross-table validation is performed on the graph structure relationship dataset to obtain intermediate validation results.

[0016] Specifically, the intermediate verification results include data fingerprints and data source topology relationships. The data fingerprint is a unique identifier for verifying data characteristics, and the data source topology relationships are the logical and physical connection relationships between multiple data sources. The distributed cross-table verification performed on the graph structure association dataset to obtain intermediate verification results includes:

[0017] The data fingerprint is generated by extracting features from the field combinations in the graph structure association dataset using a hash algorithm; and the data source topology is constructed based on the connection protocol and data lineage dependency of the multi-data source data.

[0018] Based on the data source topology, locate the set of cross-database tables with primary and foreign key relationships;

[0019] Perform referential integrity checks and value range compatibility checks on the cross-database table set to obtain check error data;

[0020] The verification anomaly data, the data fingerprint of the verification anomaly data, and the topology path that triggered the verification anomaly data are used as intermediate verification results.

[0021] Specifically, the step of performing targeted verification through the SQL precision verification layer and generating a list of violation data IDs carrying violation type identifiers based on the intermediate verification results includes:

[0022] The intermediate verification results are parsed to obtain the data fingerprint of the verification anomaly data and the topological path that triggered the verification anomaly data; and a mapping table is established with the verification rule base according to the preset violation type identifier, wherein the mapping table records the SQL verification template corresponding to each violation type;

[0023] Fill the data fingerprint of the verification anomaly data and the topological path that triggered the verification anomaly data into the SQL verification template to construct a targeted SQL query statement;

[0024] The targeted SQL query statement is used to perform a query, and a list of violation data IDs carrying violation type identifiers is obtained.

[0025] Specifically, the step of performing hierarchical conversion through the business conversion layer, and converting the fields of the multi-source data based on the list of violation data IDs to obtain the converted data, includes:

[0026] Based on the violation type identifier, predefined transformation rules are loaded from the policy library, and target data is determined from the multi-data source data based on the violation data ID list. The target data includes the violation data and associated data of the violation data determined according to the data source topology.

[0027] Based on the confidence level of the target data, the target data is subjected to hierarchical transformation using the transformation rules to obtain the transformed data.

[0028] Specifically, the step of performing hierarchical transformation on the target data according to the transformation rules based on the confidence level of the target data to obtain the transformed data includes:

[0029] If the confidence level of the target data is greater than or equal to the first threshold, a real-time conversion thread is started, and the target data is directly modified using the conversion rules to obtain the converted data.

[0030] If the confidence level of the target data is greater than the first threshold and less than or equal to the second threshold, a queue to be converted is generated, and the target data is processed in batches by asynchronous tasks using the conversion rules to obtain the converted data.

[0031] If the confidence level of the target data is less than the second threshold, a manual review work order is generated, and the target data is modified according to the review result of the manual review work order.

[0032] Furthermore, before performing hierarchical transformation on the target data according to the transformation rule based on the confidence level of the target data to obtain the transformed data, the method further includes:

[0033] The basic confidence level is determined based on the identifier of the violation type.

[0034] The confidence level of the target data is determined based on the baseline confidence level and the number of times the primary key of the target data is triggered when the target SQL query is performed.

[0035] on the other hand:

[0036] This application provides a data governance device applied to an electronic device including a data governance plugin. The data governance plugin includes a memory pre-verification layer, an SQL precise verification layer, and a business transformation layer. The device includes:

[0037] A complex verification unit is used to perform complex verification through the memory pre-verification layer and generate intermediate verification results based on data from multiple data sources.

[0038] The targeted verification unit is used to perform targeted verification through the SQL precise verification layer and generate a list of violation data IDs carrying violation type identifiers based on the intermediate verification results. The list of violation data IDs is used to accurately locate and classify abnormal data in the multi-data source data.

[0039] The hierarchical conversion unit is used to perform hierarchical conversion through the business conversion layer, and to perform field conversion on the multi-source data based on the list of violation data IDs to obtain the converted data.

[0040] This application also provides an electronic device, including: a processor and a memory;

[0041] The processor is connected to a memory, wherein the memory is used to store computer programs and the processor is used to invoke the computer programs to execute the methods as described in one aspect of the embodiments of this application.

[0042] This application also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, perform the method as described in one aspect of this application.

[0043] This application's embodiments utilize a layered processing architecture, combining in-memory computing and native database capabilities, to achieve end-to-end data governance, from coarse-grained pre-validation to precise SQL validation, ultimately completing business transformation. Each module employs a loosely coupled design, supporting horizontal scalability and adaptable to TB-level data governance scenarios. Using this application's embodiments offers the following advantages:

[0044] 1. The embodiments of this application can improve verification capabilities and efficiency, break through the limitations of traditional verification, perform complex verification through a memory pre-verification layer, generate intermediate verification results based on data from multiple data sources, and realize complex verification across data sources and tables. In a memory environment, frequent database queries are avoided, greatly improving verification efficiency. Compared with traditional verification methods, the time overhead of data transmission and processing is greatly reduced.

[0045] 2. The embodiments of this application can enhance the efficiency and accuracy of verification. For complex multi-table relationships and complex conditions, it is not necessary to verify all original multi-data metadata, which greatly reduces the amount of data that needs to be verified, avoids full table scans, and enhances accuracy by performing targeted verification through the SQL precise verification layer.

[0046] 3. The embodiments of this application can enhance the consistency and flexibility of data. Based on the list of non-compliant data IDs, field transformation is performed on data from multiple data sources, enabling complex data transformation to be flexibly implemented according to business rules. At the same time, the transformed data is standardized and highly consistent. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 A schematic diagram of the runtime environment of a data governance application provided in one embodiment of this application is shown;

[0049] Figure 2 A flowchart illustrating a data governance method provided in one embodiment of this application is shown;

[0050] Figure 3 A schematic diagram of the structure of a data governance apparatus provided in one embodiment of this application is shown;

[0051] Figure 4 A schematic diagram of the structure of a computer device provided in one embodiment of this application is shown. Detailed Implementation

[0052] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the protection scope of this application.

[0053] In the data processing process, routine verification and transformation are fundamental steps to ensure data quality.

[0054] Common routine validations include validating named entity data such as ID cards, mobile phone numbers, IP addresses, and email addresses. These validations primarily use established rules to determine the correctness of the data format. For example, ID card numbers must conform to specific encoding rules, including regional codes, birth date codes, sequence codes, and check codes. Regular expressions are used to match the format, and the length is checked to ensure data validity. Non-empty validation checks whether data fields contain empty values, null values, or special characters representing emptiness, preventing data loss from affecting subsequent processing. Latitude and longitude determination aims to verify the accuracy of latitude and longitude information in the data. Traditionally, this is done using the geo search function in Elasticsearch to find the county / district to which the latitude and longitude belong and compare it with the corresponding field to verify accuracy.

[0055] Routine data conversion primarily involves simple format adjustments and processing of the data itself, transforming the raw data into a unified, standardized, and analyzable format. Its core function is to eliminate data noise, improve data consistency, and provide high-quality input for subsequent data analysis, machine learning, and other scenarios. For example, whitespace removal removes leading and trailing whitespace characters from a string to standardize the data format, facilitating subsequent data matching and analysis. Case conversion standardizes the case of data, avoiding data recognition problems caused by case differences. Specific character removal eliminates unnecessary special characters from the data based on business needs, making the data more standardized. String truncation extracts portions of content from the original string according to specified positions and lengths to meet specific business display or processing requirements. Date and time standardization typically uses Simple Date Formatter to convert date and time data into a unified format, facilitating data storage, comparison, and display. Regarding numerical precision standards, Big Decimal is commonly used for rounding operations to control the number of decimal places, ensuring the accuracy of data in numerical calculations and displays.

[0056] Currently, while existing technologies provide basic solutions for routine verification and transformation, their limitations are becoming increasingly apparent as business scenarios become more complex. Most existing verification technologies only analyze the data itself and cannot meet the complex verification needs across data sources and tables. In practical business scenarios, such as in a company's customer information management system, it may be necessary to verify whether a customer's department code is in the company's organizational structure table. This involves data association verification between different data sources and tables, which existing technologies struggle to handle. For complex verification scenarios, such as those involving multiple table joins and complex query conditions, existing technologies lack effective processing methods and cannot efficiently obtain a list of data IDs that do not meet the conditions, thus affecting the efficiency and accuracy of data verification. Regarding data transformation, existing technologies can only perform simple data transformations themselves, making it difficult to meet the data transformation needs under complex business logic. In some financial business scenarios, multi-condition transformations of data are required based on complex business rules, and existing technologies cannot provide flexible and powerful transformation functions.

[0057] To address the problems in the aforementioned related technologies, this application proposes a data governance method, apparatus, electronic device, and computer-readable storage medium. The aim is to solve the problem that existing data verification and transformation technologies cannot meet the needs of complex business operations. It provides a technical solution capable of cross-data source and cross-table verification, handling complex SQL verification, and meeting complex business transformation requirements, thereby improving the accuracy, efficiency, and flexibility of data processing to adapt to diverse business scenarios.

[0058] Please refer to Figure 1 This diagram illustrates a data governance application runtime environment provided in one embodiment of this application. The application runtime environment may include: terminal 10 and server 20.

[0059] Terminal 10 includes, but is not limited to, electronic devices such as mobile phones, computers, smart voice interaction devices, smart home appliances, in-vehicle terminals, game consoles, e-book readers, multimedia playback devices, and wearable devices. Application clients can be installed on terminal 10.

[0060] In this embodiment, the application described above can be any application capable of providing data governance services. Typically, this application is an industrial application. Of course, other types of applications besides industrial applications can also provide services dependent on data governance. For example, research applications, browser applications, virtual reality (VR) applications, augmented reality (AR) applications, etc., are not limited in this embodiment. Optionally, a client of the aforementioned application runs on terminal 10.

[0061] Server 20 provides background services to clients of applications in terminal 10. For example, server 20 can be a background server for the aforementioned applications. Server 20 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, server 20 can simultaneously provide background services to applications in multiple terminals 10.

[0062] Optionally, terminal 10 and server 20 can communicate with each other via network 30. Terminal 10 and server 20 can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0063] Please refer to Figure 2 This document illustrates a flowchart of a data governance method provided in one embodiment of this application. This method can be applied to electronic devices including data governance plugins. The aforementioned electronic devices refer to those with data computing and processing capabilities. For example, the executing entity for each step can be... Figure 1 The application runtime environment shown is either terminal 10 or server 20. The method may include the following steps:

[0064] Step 201: Perform complex verification through the memory pre-verification layer to generate intermediate verification results based on data from multiple data sources.

[0065] The intermediate verification result is a transitional verification result generated after performing complex verification on data from multiple data sources through a memory pre-verification layer.

[0066] In step 201, namely, performing complex verification through the memory pre-verification layer and generating intermediate verification results based on data from multiple data sources, the following is included:

[0067] Data from multiple data sources is loaded into memory using data preloading technology;

[0068] A graph-structured dataset of relationships across data sources is constructed in the memory.

[0069] Distributed cross-table validation is performed on the graph structure relationship dataset to obtain intermediate validation results.

[0070] Among these technologies, data preloading employs efficient data extraction, transformation, and memory management strategies to rapidly load heterogeneous data from multiple sources into memory, providing a highly available dataset for subsequent verification and analysis. For example, this can be achieved by connecting to different data sources through an adapter pattern. These data sources may include, but are not limited to, MySQL, Hive, and Kafka.

[0071] For example, after loading data from multiple data sources into memory based on data preloading technology, the method may further include: performing preliminary cleaning on the data from multiple data sources to obtain a cleaned data table, and preliminary extraction including at least one of the following: performing null value filling, format standardization, and outlier filtering.

[0072] For example, constructing a graph structure relationship dataset across data sources in memory includes: parsing the primary and foreign key relationships of the pre-cleaned data tables to generate vertices and edges, wherein vertices are represented by tables or fields, and edges are represented by the relationships between tables or fields; then constructing a graph structure relationship dataset across data sources based on the vertices and edges, wherein the dataset can be stored in a graph database (such as Neo4j) to support fast lineage tracing.

[0073] Furthermore, a relationship strength matrix can be constructed based on the graph-structured relationship dataset to determine the priority ordering during subsequent distributed cross-table validation. The relationship strength matrix can be sorted according to the business importance of tables or fields. For example, if a table or field is only divided into primary keys and ordinary fields, its business importance can be assigned values ​​of 0.8 and 0.2 respectively; if it is divided into primary keys, intermediate fields, and ordinary fields, its business importance can be assigned values ​​of 0.5, 0.3, and 0.2 respectively. It should be noted that the above assignments are merely illustrative examples, and specific values ​​are not limited.

[0074] Specifically, the intermediate verification results include data fingerprints and data source topology relationships. The data fingerprint is a unique identifier for verifying data characteristics, and the data source topology relationships are the logical and physical connection relationships between multiple data sources. The distributed cross-table verification performed on the graph structure association dataset to obtain intermediate verification results includes:

[0075] The data fingerprint is generated by extracting features from the field combinations in the graph structure association dataset using a hash algorithm; and the data source topology is constructed based on the connection protocol and data lineage dependency of the multi-data source data.

[0076] Based on the data source topology, locate the set of cross-database tables with primary and foreign key relationships;

[0077] Perform referential integrity checks and value range compatibility checks on the cross-database table set to obtain check error data;

[0078] The verification anomaly data, the data fingerprint of the verification anomaly data, and the topology path that triggered the verification anomaly data are used as intermediate verification results.

[0079] For example, the data fingerprint is generated by extracting features from the field combinations in the graph structure association dataset using a hash algorithm. This includes: if the field combinations in the graph structure association dataset include multiple key fields, multi-dimensional feature fusion can be performed on these key fields first, followed by calculating the hash value using a hash algorithm, which is then used as the data fingerprint. The hash algorithm could be, for example, SHA-256. Alternatively, the multiple key fields can be divided into blocks, for example, by primary key range, by fixed size, and / or by time window; then the hash value of each block of data is calculated; a Merkle Tree is constructed based on the hash value of each block of data, and the root hash value of the Merkle Tree is used as the data fingerprint.

[0080] In another embodiment provided in this application, a data fingerprint can be generated by calculating a hash value using a hash algorithm based on the data type, format constraints, and value range of the field combination. When calculating the hash value, different weights can be assigned to the data type, format constraints, and value range of the field combination. The data type is the storage format defined by the field metadata, including numeric, character, and date types; the format constraints are predefined validation rules, including regular expressions and enumerated value lists; and the value range is the minimum and maximum values ​​of the field and its statistical distribution histogram. By adding a value range to the hash value calculation to limit the input parameter range of the hash algorithm, overly broad concepts are prevented. Furthermore, dynamic value range features can be bound using a statistical distribution histogram method to cover complex scenarios such as numerical drift. For example, the initial data type feature weight is 0.4, the format constraint weight is 0.3, and the value range weight is 0.3. When the kurtosis of the field value range histogram is >3, the value range weight is automatically increased to 0.5.

[0081] It can be seen that traditional data identification methods use auto-incrementing IDs or timestamps as data fingerprints, which rely on database transaction logs. When locating data by fingerprint, full comparison is inefficient and cannot detect content tampering. The data fingerprint method in this application is generated based on data content, decoupled from business logic, independent of storage engine, and supports heterogeneous systems. When locating data by fingerprint, it can improve the efficiency of data consistency checks, and the data fingerprint will change as the fields are modified.

[0082] For example, the data source topology can include node attributes, connection paths, and data lineage. Therefore, the data source topology can be determined based on the connection protocols and data lineage dependencies of multiple data sources. Node attributes can be represented by data source type, version, IP address, access credentials, etc. Connection paths can be represented by cross-data source access protocols, gateway configurations, etc. Data lineage can be represented by upstream and downstream dependencies at the table or field level.

[0083] Among them, referential integrity verification verifies the existence of foreign key fields in the associated master table; value range compatibility verification compares the value range histograms of the fields in the associated tables to detect out-of-bounds data (such as values ​​in the child table exceeding the defined range of the master table).

[0084] Furthermore, historical false alarm rates can be introduced to correct the confidence level of data in intermediate verification results. Then, the intermediate verification results can be further filtered based on this confidence level. The confidence level of data in intermediate verification results is calculated as: (Rule weight × Number of hits) / (Total rule weight × Total data volume). This embodiment of the application, by introducing historical false alarm rates to correct the confidence level, allows the subsequent business conversion layer to choose between automatic repair or manual intervention based on a scoring threshold, forming a technical closed loop.

[0085] Step 202: Perform targeted verification through the SQL precision verification layer, and generate a list of violation data IDs carrying violation type identifiers based on the intermediate verification results. The list of violation data IDs is used to accurately locate and classify abnormal data in the multi-source data.

[0086] In step 202, namely, performing targeted verification through the SQL precision verification layer and generating a list of violation data IDs carrying violation type identifiers based on the intermediate verification results, the following steps are included:

[0087] The intermediate verification results are parsed to obtain the data fingerprint of the verification anomaly data and the topological path that triggered the verification anomaly data; and a mapping table is established with the verification rule base according to the preset violation type identifier, wherein the mapping table records the SQL verification template corresponding to each violation type;

[0088] Fill the data fingerprint of the verification anomaly data and the topological path that triggered the verification anomaly data into the SQL verification template to construct a targeted SQL query statement;

[0089] The targeted SQL query statement is used to perform a query, and a list of violation data IDs carrying violation type identifiers is obtained.

[0090] The preset violation types include at least one of the following: data duplication, privacy leakage, and format abnormality; the violation type ID can be, for example, A to represent data duplication, B to privacy leakage, and C to format abnormality, or 1 to represent data duplication, 2 to privacy leakage, and 3 to format abnormality; the SQL validation template can be a triple representation of violation type identifier, field constraint conditions, and related data tables.

[0091] For example, the data fingerprint of the verification anomaly and the topology path that triggered the verification anomaly are filled into the SQL verification template to construct a targeted SQL query statement. For example, the data fingerprint can be injected as the primary key into the WHERE clause of the SQL template to generate a filtering condition in the form of WHERE fingerprint IN (fingerprint list) AND field constraint condition. The cross-database query statement is dynamically concatenated according to the topology path (for example, by associating heterogeneous data tables through database link aliases).

[0092] When performing a query using the targeted SQL query statement, multiple violation types may be triggered. These violation types can be marked according to a preset priority, for example, the priority is as follows: privacy leakage > format abnormality > data duplication.

[0093] Step 203: Perform hierarchical conversion through the business conversion layer, and perform field conversion on the multi-source data based on the list of violation data IDs to obtain the converted data.

[0094] In step 203, namely, the hierarchical conversion performed through the business conversion layer, the multi-data source data is converted based on the list of violation data IDs to obtain the converted data, including:

[0095] Based on the violation type identifier, predefined transformation rules are loaded from the policy library, and target data is determined from the multi-data source data based on the violation data ID list. The target data includes the violation data and associated data of the violation data determined according to the data source topology.

[0096] Based on the confidence level of the target data, the target data is subjected to hierarchical transformation using the transformation rules to obtain the transformed data.

[0097] The predefined conversion rules are as follows:

[0098] If the violation type is format error, a type cast will be performed (e.g., the string "2023 / 02 / 30" will be converted to NULL and an error log will be recorded); if the violation type is privacy leakage, the field value will be encrypted using an encryption algorithm to generate ciphertext and key management credentials; if the violation type is data duplication, the first record will be retained and a blockchain notarization hash (e.g., SHA-3 (original value + timestamp)) will be added to the duplicate data.

[0099] Specifically, the step of performing hierarchical transformation on the target data according to the transformation rules based on the confidence level of the target data to obtain the transformed data includes:

[0100] If the confidence level of the target data is greater than or equal to the first threshold, a real-time conversion thread is started, and the target data is directly modified using the conversion rules to obtain the converted data.

[0101] If the confidence level of the target data is greater than the first threshold and less than or equal to the second threshold, a queue to be converted is generated, and the target data is processed in batches by asynchronous tasks using the conversion rules to obtain the converted data.

[0102] If the confidence level of the target data is less than the second threshold, a manual review work order is generated, and the target data is modified according to the review result of the manual review work order.

[0103] As can be seen, in this embodiment of the application, the confidence-driven hierarchical processing mechanism can not only ensure the ability to block high-risk data in real time, but also achieve efficient governance of massive amounts of low- and medium-risk data, thus balancing security, performance and resource costs.

[0104] Furthermore, before performing hierarchical transformation on the target data according to the transformation rule based on the confidence level of the target data to obtain the transformed data, the method further includes:

[0105] The basic confidence level is determined based on the identifier of the violation type.

[0106] The confidence level of the target data is determined based on the baseline confidence level and the number of times the primary key of the target data is triggered when the target SQL query is performed.

[0107] The Base Confidence (BC) is calculated using the following formula: BC = (Rule Weight × Historical Accuracy) + Data Quality Factor. Here, the rule weight is a predefined rule importance coefficient, historical accuracy is the proportion of correct judgments made by the rule in the past N validations (e.g., accuracy = number of correct triggers / total number of triggers), and the data quality factor is a correction coefficient based on the reliability of the data source. In this embodiment, the linkage between the rule weight and the data quality factor allows the confidence to automatically adjust with changes in the data environment (e.g., automatically reducing the confidence of associated rules when the quality of external API data declines). The introduction of historical accuracy can suppress the impact of high-frequency false alarm rules (e.g., when the false alarm rate of a certain format rule exceeds 30%, its base confidence automatically drops to 0.6 × 0.7 = 0.42).

[0108] For example, the confidence level of the target data can be calculated as: Confidence level = Σ(base confidence level × number of triggers) / maximum possible weight value; or as: Confidence level = base confidence level × (1 + log10 number of triggers).

[0109] In summary, this application's embodiments, through a layered processing architecture combining in-memory computing and native database capabilities, achieve end-to-end data governance, from coarse-grained pre-validation to precise SQL validation, ultimately completing business transformation. Each module employs a loosely coupled design, supports horizontal scaling, and can adapt to TB-level data governance scenarios. Using this application's embodiments offers the following advantages:

[0110] 1. The embodiments of this application can improve verification capabilities and efficiency, break through the limitations of traditional verification, perform complex verification through a memory pre-verification layer, generate intermediate verification results based on data from multiple data sources, and realize complex verification across data sources and tables. In a memory environment, frequent database queries are avoided, greatly improving verification efficiency. Compared with traditional verification methods, the time overhead of data transmission and processing is greatly reduced.

[0111] 2. The embodiments of this application can enhance the efficiency and accuracy of verification. For complex multi-table relationships and complex conditions, it is not necessary to verify all original multi-data metadata, which greatly reduces the amount of data that needs to be verified, avoids full table scans, and enhances accuracy by performing targeted verification through the SQL precise verification layer.

[0112] 3. The embodiments of this application can enhance the consistency and flexibility of data. Based on the list of non-compliant data IDs, field transformation is performed on data from multiple data sources, enabling complex data transformation to be flexibly implemented according to business rules. At the same time, the transformed data is standardized and highly consistent.

[0113] Figure 3A schematic diagram of a data governance apparatus according to an embodiment of this application is shown. Applied to an electronic device including a data governance plugin, the data governance plugin includes a memory pre-verification layer, an SQL precise verification layer, and a business transformation layer. The apparatus includes:

[0114] The complex verification unit 301 is used to perform complex verification through the memory pre-verification layer and generate intermediate verification results based on data from multiple data sources.

[0115] The targeted verification unit 302 is used to perform targeted verification through the SQL precise verification layer and generate a list of violation data IDs carrying violation type identifiers based on the intermediate verification results. The list of violation data IDs is used to accurately locate and classify abnormal data in the multi-source data.

[0116] The hierarchical conversion unit 303 is used to perform hierarchical conversion through the business conversion layer, and to perform field conversion on the multi-source data based on the list of violation data IDs to obtain the converted data.

[0117] Figure 4 A schematic diagram of the structure of a computer device provided in one embodiment of this application is shown, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the functions of the computer system of the data governance method in any of the above embodiments.

[0118] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer, causes the computer to perform the functions of the computer system of the data governance method in any of the above embodiments.

[0119] This application also provides a computer program product containing instructions that, when executed by a computer, cause the computer to perform the functions of the computer system of the data governance method described in any of the above embodiments.

[0120] It is understood that the specific examples in this application are only intended to help those skilled in the art better understand the implementation methods of this application, and are not intended to limit the scope of the invention.

[0121] It is understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not limit the implementation process of the embodiments of this application in any way.

[0122] It is understood that the various implementation methods described in this application can be implemented individually or in combination, and the implementation methods in this application are not limited in this respect.

[0123] Unless otherwise stated, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The term "and / or" as used in this application includes any and all combinations of one or more of the associated listed items. The singular forms "a," "the," and "the" as used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0124] It is understood that the processor in the embodiments of this application can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiments can be completed by the integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0125] It is understood that the memory in the embodiments of this application may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Specifically, non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM). It should be noted that the memory in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0126] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0127] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the aforementioned method implementations, and will not be repeated here.

[0128] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0129] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0130] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0131] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0132] The above are merely specific embodiments of this application, but the scope of protection of this invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this invention should be determined by the scope of the claims.

Claims

1. A data governance method, characterized in that, An electronic device including a data governance plugin, the data governance plugin comprising a memory pre-validation layer, an SQL precision validation layer, and a business transformation layer, the method comprising: Data from multiple data sources is loaded into memory using data preloading technology; A graph-structured dataset of relationships across data sources is constructed in the memory. Feature extraction is performed on field combinations in the graph structure association dataset using a hash algorithm to generate data fingerprints; and data source topology is constructed based on the connection protocols and data lineage dependencies of the multi-data source data. Based on the data source topology, locate the set of cross-database tables with primary and foreign key relationships; Perform referential integrity checks and value range compatibility checks on the cross-database table set to obtain check error data; The verification anomaly data, the data fingerprint of the verification anomaly data, and the topology path that triggered the verification anomaly data are used as intermediate verification results; Wherein, the data fingerprint is a unique identifier for verifying data characteristics, and the data source topology relationship is the logical and physical connection relationship between multiple data sources; The SQL precision verification layer performs targeted verification, and generates a list of violation data IDs carrying violation type identifiers based on the intermediate verification results. The list of violation data IDs is used to accurately locate and classify abnormal data in the multi-data source data. The business conversion layer performs hierarchical conversion, and the multi-source data is converted based on the list of violation data IDs to obtain the converted data.

2. The data governance method according to claim 1, characterized in that, The step of performing targeted verification through the SQL precision verification layer and generating a list of violation data IDs carrying violation type identifiers based on the intermediate verification results includes: The intermediate verification results are parsed to obtain the data fingerprint of the verification anomaly data and the topological path that triggered the verification anomaly data; and a mapping table is established with the verification rule base according to the preset violation type identifier, the mapping table recording the SQL verification template corresponding to each violation type; Fill the data fingerprint of the verification anomaly data and the topological path that triggered the verification anomaly data into the SQL verification template to construct a targeted SQL query statement; The targeted SQL query statement is used to perform a query, and a list of violation data IDs carrying violation type identifiers is obtained.

3. The data governance method according to claim 2, characterized in that, The step of performing hierarchical transformation through the business transformation layer, and transforming the fields of the multi-source data based on the list of violation data IDs to obtain the transformed data includes: Based on the violation type identifier, predefined transformation rules are loaded from the policy library, and target data is determined from the multi-data source data based on the violation data ID list. The target data includes the violation data and associated data of the violation data determined according to the data source topology. Based on the confidence level of the target data, the target data is subjected to hierarchical transformation using the transformation rules to obtain the transformed data.

4. The data governance method according to claim 3, characterized in that, The step of performing hierarchical transformation on the target data according to the transformation rules based on the confidence level of the target data to obtain the transformed data includes: If the confidence level of the target data is greater than or equal to the first threshold, a real-time conversion thread is started, and the target data is directly modified using the conversion rules to obtain the converted data. If the confidence level of the target data is greater than the first threshold and less than or equal to the second threshold, a queue to be converted is generated, and the target data is processed in batches by asynchronous tasks using the conversion rules to obtain the converted data. If the confidence level of the target data is less than the second threshold, a manual review work order is generated, and the target data is modified according to the review result of the manual review work order.

5. The data governance method according to claim 3 or 4, characterized in that, Before performing hierarchical transformation on the target data according to the transformation rule based on the confidence level of the target data to obtain the transformed data, the method further includes: The basic confidence level is determined based on the identifier of the violation type. The confidence level of the target data is determined based on the baseline confidence level and the number of times the primary key of the target data is triggered when the target SQL query is performed.

6. A data governance device, characterized in that, An electronic device including a data governance plugin, the data governance plugin comprising a memory pre-validation layer, an SQL precision validation layer, and a business transformation layer, the device comprising: A complex verification unit is used to load data from multiple data sources into memory based on data preloading technology; construct a graph-structured relational dataset across data sources in the memory; extract features from field combinations in the graph-structured relational dataset using a hash algorithm to generate data fingerprints; construct data source topology relationships based on the connection protocols and data lineage dependencies of the multiple data sources; locate cross-database table sets with primary and foreign key relationships according to the data source topology relationships; perform referential integrity verification and value range compatibility verification on the cross-database table sets to obtain verification anomaly data; and use the verification anomaly data, the data fingerprint of the verification anomaly data, and the topology path that triggered the verification anomaly data as intermediate verification results; wherein, the data fingerprint is a unique identifier for verifying data characteristics, and the data source topology relationship is the logical and physical connection relationship between multiple data sources; The targeted verification unit is used to perform targeted verification through the SQL precise verification layer and generate a list of violation data IDs carrying violation type identifiers based on the intermediate verification results. The list of violation data IDs is used to accurately locate and classify abnormal data in the multi-data source data. The hierarchical conversion unit is used to perform hierarchical conversion through the business conversion layer, and to perform field conversion on the multi-source data based on the list of violation data IDs to obtain the converted data.

7. An electronic device, characterized in that, include: Processor and memory; The processor is connected to a memory, wherein the memory is used to store a computer program, and the processor is used to invoke the computer program to execute the data governance method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions, which, when executed by a processor, perform the data governance method as described in any one of claims 1-5.

Images