Visual display method for data governance

Abstract

The invention relates to the technical field of data governance, in particular to a visual display method for data governance, which comprises the following steps: S1, data acquisition and cleaning: acquiring data; s2, data classification and storage: classifying according to a preset classification rule; s3, data dependency analysis: generating a data dependency graph; s4, defining and optimizing data governance rules: defining the data governance rules, and dynamically optimizing the data governance rules; s5, executing a data governance process: automatically executing the data governance process; s6, visual display: using a data visualization tool to carry out graphical display on key indexes, execution results and a data dependence graph in the data governance process; and S7, feedback and optimization: optimizing a data governance rule according to a visualized display result. According to the invention, it is ensured that the data governance rule can accurately reflect a complex data dependence structure, and the accuracy and applicability of a governance decision are improved.

Description

Technical Field

[0001] The present invention relates to the field of data management technology, and in particular to a visual display method for data management. Background Art

[0002] With the rapid growth of data scale and the diversification of data types, enterprises and organizations are facing increasingly complex data governance challenges. Data governance not only needs to manage large amounts of structured and unstructured data, but also needs to ensure the high quality, security and compliance of data. In addition, with the complexity of data dependencies, how to effectively manage and utilize this data to support corporate decision-making has become a key issue in modern enterprise operations.

[0003] Current data governance methods often rely on static rules and manual configuration, which makes it difficult to cope with dynamically changing data environments and complex dependencies. These traditional methods lack intelligent analysis tools and find it difficult to accurately capture the dependencies between multi-level data, resulting in inaccurate data governance rules. In addition, the existing governance process lacks automation and dynamic adjustment mechanisms, and data governance is inefficient and difficult to adapt to changes in data structure and business needs in real time. Especially when processing massive and complex data, data quality problems, insufficient compliance, and weakened decision-making support capabilities are prone to occur.

[0004] The present invention aims to address the deficiencies in the prior art, ensure the accuracy and adaptability of data governance rules through intelligent analysis and optimization, and provide managers with intuitive and reliable decision support through visualization tools, thereby achieving efficient governance in complex data environments. Summary of the invention

[0005] Based on the above objectives, the present invention provides a visual display method for data governance.

[0006] A visual display method for data governance includes the following steps:

[0007] S1, data collection and cleaning: collect data from different data sources through multiple data interfaces, including structured data and unstructured data, and clean the collected data;

[0008] S2, data classification and storage: classify the cleaned data according to the predetermined classification rules, and store the classified data in a distributed database;

[0009] S3, data dependency analysis: Use the graph neural network (GNN) model to analyze and model the multi-level dependencies in the data and generate a data dependency graph, including:

[0010] S31, data dependency graph construction: Use the graph neural network (GNN) model to analyze the data stored in the distributed database and construct a dependency graph between data entities. The nodes represent data entities and the edges represent the dependency relationships between data.

[0011] S32, graph neural network training: train the constructed data dependency graph, learn the weights and features of data dependency relationships through iterative optimization, and extract complex dependency structures between multi-level data;

[0012] S33, dependency optimization and output: Based on the training results, optimize the data dependency, eliminate redundant dependencies and strengthen the key dependency chain to generate the final data dependency graph;

[0013] S4, data governance rule definition and optimization: Define data governance rules based on the final data dependency graph and the governance requirements of the enterprise or organization, including data security policies, access control policies, data quality standards, and data lifecycle management, and dynamically optimize data governance rules based on the results of data dependency analysis;

[0014] S5, data governance process execution: Based on the optimized data governance rules, the data governance process is automatically executed, including data access control, quality inspection, redundant data cleaning and compliance inspection, and the execution log of data governance is recorded in real time;

[0015] S6, Visualization: Use data visualization tools to graphically display key indicators, execution results, and data dependency maps in the data governance process. The display content includes the implementation of data governance rules, the improvement of data quality, the compliance of data compliance, and the changes in data dependency relationships.

[0016] S7, Feedback and Optimization: Evaluate the effectiveness of data governance based on the results of the visualization, and optimize data governance rules based on dynamic changes in the evaluation results.

[0017] Optionally, the data collection and cleaning in S1 includes:

[0018] S11, data collection: collect data from different data sources (such as relational databases, document databases, log files, sensor data streams, etc.) by integrating RESTful API, JDBC, FTP, and MQTT. Structured data is extracted from relational databases through SQL queries, and unstructured data is obtained from document databases through text parsers and file transfer protocols;

[0019] S12, cleaning of structured data: using linear interpolation to process missing values, calculating hash values ​​to identify whether data entries are repeated, and deduplicating the data;

[0020] S13, cleaning of unstructured data: For text data, use regular expressions (Regex) to filter out irrelevant characters and noise data, and convert unstructured data from different sources into a specified format.

[0021] Optionally, the data classification and storage in S2 includes:

[0022] S21, Classification rule definition: Based on business needs, data characteristics and usage, pre-define data classification rules, including those based on data source (such as sensors, user logs), data type (such as text, images, videos), time dimension, geographic location or business department;

[0023] S22, classification of structured data: For structured data, a classification algorithm based on a decision tree is used. By analyzing the field characteristics of the structured data, the data set D is segmented according to the field f. The decision tree is constructed by calculating the information gain IG to determine the optimal classification feature;

[0024] S23, classification of unstructured data: For unstructured data, feature extraction (using TF-IDF) is performed on unstructured data (such as text and images), and then the data is classified using the K-means algorithm;

[0025] S24, labeling of classification labels: according to the classification rules and classification results, label each type of data with a classification label, the label includes a predefined category name (such as "customer data", "sensor data") or a dynamically generated identifier (such as a time-based batch ID or a geographic location code);

[0026] S25, distributed database selection: based on the data classification results, select a distributed database for storage, structured data is stored in a distributed relational database (such as MySQL Cluster), and unstructured data is stored in a distributed document database (such as MongoDB or Cassandra);

[0027] S26, data sharding and copy management: For structured data, the data is sharded and stored according to classification tags and hash sharding rules. For unstructured data, the data is sharded according to data features or file types, and copies are saved on multiple nodes.

[0028] S27, index creation: create an index for the stored data based on the classification label and query field.

[0029] Optionally, the data dependency graph construction in S31 includes:

[0030] S311, definition of data entity nodes: extract data entities from the distributed database. Data entities include business data (such as order information, customer records), log data (such as access logs, error logs), and configuration data (such as system configuration, parameter settings). Data entities are represented as nodes in the graph neural network (GNN) model. Each node v i Represents a data entity;

[0031] S312, establishment of dependency edges: according to the logical relationship between data entities (such as the relationship between orders and customers, the relationship between logs and configurations), dependency relationships are established between data entities, which are represented as edges e in the dependency graph. ij , to quantify the strength of the dependency, calculate the weight w of each edge ij , expressed as:

[0032] w ij =α×f ij +β×d ij +γ×a ij ;

[0033] Among them, w ij For node v i and node v j The edge weight between ij is the interaction frequency between data entities (such as the number of interactions between orders and customers), d ij is the degree of association or distance between data entities (such as geographical distance or logical distance), a ij is the numerical attribute between data entities (such as transaction amount, log severity, etc.), α, β, and γ are adjustment coefficients;

[0034] S313, graph structure initialization: The defined data entity nodes (such as order information, customer records, access logs, system configuration) and dependency edges are combined to form an initial dependency graph G = (V, E), where V is the data entity set, E is the dependency set, and each node v i Contains the feature vector x associated with the data entity i , each edge e ij Contains edge weight w ij .

[0035] Optionally, the graph neural network training in S32 includes:

[0036] S321, input of the initial graph structure: input the constructed data dependency graph G = (V, E) into the graph neural network (GNN) model;

[0037] S322, message passing and node feature update: The message passing mechanism is used to update the node representation. In each layer iteration, each node vi The feature is updated according to the information of its neighbor node N(i), which is expressed as:

[0038]

[0039] in, For node v i The feature vector at layer t, w ij is the edge weight, σ is the activation function, W (t) is the weight matrix of the tth layer, b (t) is the bias vector;

[0040] S323, Multi-layer iteration and global optimization: Through the multi-layer iterative optimization process, the graph neural network (GNN) model gradually updates the feature vector of each node Where T is the total number of iterations. Through the back propagation algorithm, the loss function L is calculated to optimize the model parameters W, which is expressed as:

[0041]

[0042] Where |E| is the size of the edge set E, (i,j)∈E is the set of all edges in the graph, is the edge weight predicted by the model, w ij is the actual edge weight.

[0043] Optionally, the dependency optimization and output in S33 includes:

[0044] S331, Detection and elimination of redundant dependencies: After training the graph neural network (GNN) model, by analyzing the weight of each edge in the graph Determine the strength of the dependency relationship. For edges e whose weight is lower than the preset threshold τ ij , it is considered as redundant dependency, and the detected redundant dependency edge is deleted from the dependency graph;

[0045] S332, Strengthening of key dependency chains: Based on the training results of the graph neural network (GNN) model, identify high-weight edges in the graph side Represents the key dependency chain between data entities. By strengthening the key dependency chain, the edge Optimize, expressed as:

[0046]

[0047] Among them, θ is the weight threshold of the key dependency, and γ>1 is the reinforcement coefficient;

[0048] S333, generation of optimized graph: after redundant dependencies are eliminated and key dependency chains are strengthened, the optimized data dependency graph is recalculated and generated. The final data dependency graph G final =(V,E final ) contains optimized nodes V and edges E final , where E final Contains optimized dependencies.

[0049] Optionally, the data governance rule definition and optimization in S4 includes:

[0050] S41, data security strategy: based on the final data dependency graph G final =(V,E final ) in the data structure, formulate data security policies, and set strict access control and encryption requirements for data entities (nodes) and key dependency chains (edges) with high dependency intensity, which can be expressed as:

[0051]

[0052] in, For node v i The edge weights between its related nodes, and the function f sets the corresponding security level according to the edge weights;

[0053] S42, access control strategy: Based on the dependencies and access frequencies between data entities, a role-based access control model (RBAC) is established. i , based on the node features h in the dependency graph i and edge weights Dynamically assign access rights, expressed as:

[0054]

[0055] Among them, A i For access rights, function g is based on the node’s feature vector h i The sum of the weights of all edges connected to it determines the level of access rights;

[0056] S43, data quality standard: Using the node features and edge weights in the data dependency graph, set the data quality standard, expressed as:

[0057]

[0058] Among them, Q i is the data quality criterion, α and β are adjustment parameters;

[0059] S44, Data lifecycle management: Set the data lifecycle strategy based on the edge weights and node features in the data dependency graph. i It is expressed as:

[0060]

[0061] Among them, γ is the adjustment coefficient;

[0062] S45 introduces a feedback mechanism to dynamically adjust the rules based on the actual execution effect of data governance rules and new dependencies, expressed as:

[0063]

[0064] Among them, R old is the existing rule, δ is the learning rate, and F(t) is the feedback function after the rule is executed.

[0065] Optionally, the data governance process execution in S5 includes:

[0066] S51, Data Access Control: Based on the optimized data governance rules, data access control is automatically implemented, and the role-based access control model (RBAC) is used to verify and authorize access requests from different users and applications in real time;

[0067] S52, Quality Check: Automates the data quality check process and continuously monitors the data based on optimized data quality standards. If quality issues are detected, corrective actions are automatically triggered according to preset rules, including missing value filling, error value correction, format conversion, duplicate data cleaning, and redundant data merging.

[0068] S53, redundant data cleaning: automatically identify and clean redundant data according to the redundant detection standards set in the data governance rules;

[0069] S54, Compliance Check: Automated check of each operation in the data processing process according to the defined compliance requirements;

[0070] S55, logging mechanism: During the data governance process, all relevant operations are automatically recorded, including data access requests, quality check results, redundant data cleanup operations, and the execution of compliance checks. The log content includes operation time, operation type, operator identity, and operation results.

[0071] Optionally, the visual display in S6 includes:

[0072] S61, Graphical display of key indicators: Display the execution of data governance rules through visual charts (such as bar charts, line charts or dashboards), including the execution frequency, success rate and instance statistics of rule violations;

[0073] S62, display of data compliance: based on the compliance check results, use pie charts, radar charts or heat maps to display the compliance rates of different data operations, and monitor data compliance in real time on a visual interface;

[0074] S63, graphical display of data dependency: Display the data dependency graph in the form of a node-edge graph, where nodes represent data entities and edges represent dependency relationships between data. Through graphical display, the display of nodes and edges can be dynamically adjusted;

[0075] S64, User Interaction and Customized Display: Provides user interaction capabilities, allowing users to customize visualization views and report content by clicking, dragging, or filtering, focusing on governance indicators or data dependencies.

[0076] Optionally, the feedback and optimization in S7 includes:

[0077] S71, Evaluation of data governance effectiveness: By analyzing the key indicators, execution results, compliance status and changes in data dependency graphs displayed by data visualization tools, the execution effect and governance quality of data governance rules are evaluated. The evaluation indicators include the improvement rate of data quality, the success rate of rule execution, the compliance rate and the stability of dependency relationships. An automated algorithm is used for quantitative evaluation, which is expressed as:

[0078] E(t)=w4×R s (t)+w5×Q p (t)+w6×C r (t);

[0079] Among them, E(t) is the data governance effect evaluation score, w4, w5, w6 are the weight coefficients of the evaluation indicators, R s is the average success rate of data governance rule execution, Q p is the percentage of data quality improvement, C r Compliance achievement rate;

[0080] S72, Optimization of data governance rules: If the data governance effectiveness evaluation score E(t) is lower than the adjustment threshold, the data governance rules are automatically adjusted, including resetting data security policies, adjusting access control permissions, optimizing data quality standards, and adjusting data lifecycle management.

[0081] Beneficial effects of the present invention:

[0082] The present invention introduces a graph neural network model to analyze and model the multi-level dependencies in the data, generate an accurate dependency graph, and combine it with an automated evaluation algorithm and optimization mechanism to make real-time adjustments based on the actual data governance effect, ensuring that data governance rules can accurately reflect complex data dependency structures, greatly improving the accuracy and applicability of governance decisions.

[0083] The present invention realizes an efficient governance process by automating key processes such as data access control, quality inspection, redundant data cleaning and compliance inspection, and recording operation logs in real time. The system can automatically adjust data governance rules based on dynamically changing data dependencies and governance effect evaluation results to ensure that the governance process is highly adaptable and can effectively cope with complex and changing data environments.

[0084] The present invention uses data visualization tools to intuitively present key indicators, execution results, data quality improvement, compliance status, and changes in data dependencies in the data governance process to managers. In addition, it provides an interactive customized display function, allowing users to adjust views and report content according to actual needs, thereby improving the transparency of the data governance process and enhancing the decision-making support capabilities of managers. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only for the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0086] Figure 1 It is a schematic diagram of the flow chart of the display method according to an embodiment of the present invention;

[0087] Figure 2 Schematic diagram of data dependency analysis according to an embodiment of the present invention. DETAILED DESCRIPTION

[0088] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. At the same time, it is explained here that in order to make the embodiments more detailed, the following embodiments are the best and preferred embodiments, and those skilled in the art may also adopt other alternatives to implement some known technologies; and the accompanying drawings are only for more specific description of the embodiments, and are not intended to specifically limit the present invention.

[0089] It should be noted that the references to "one embodiment", "an embodiment", "an exemplary embodiment", "some embodiments" and the like in the specification indicate that the embodiments described may include specific features, structures or characteristics, but not every embodiment may include the specific features, structures or characteristics. In addition, when a specific feature, structure or characteristic is described in conjunction with an embodiment, it should be within the knowledge of a person skilled in the art to implement such feature, structure or characteristic in conjunction with other embodiments (whether or not explicitly described).

[0090] In general, a term can be understood, at least in part, from its use in context. For example, depending, at least in part, on the context, the term "one or more" as used herein can be used to describe any feature, structure, or characteristic in the singular sense, or can be used to describe a combination of features, structures, or characteristics in the plural sense. Additionally, the term "based on" can be understood as not necessarily intended to convey an exclusive set of factors, but can instead, depending, at least in part, on the context, allow for the presence of other factors that are not necessarily explicitly described.

[0091] like Figure 1-Figure 2 As shown, a visual display method of data governance includes the following steps:

[0092] S1, data collection and cleaning: collect data from different data sources through multiple data interfaces, including structured data and unstructured data, clean the collected data, delete redundant data, correct erroneous data, and standardize the data format to ensure data consistency and accuracy;

[0093] S2, data classification and storage: classify the cleaned data according to the predetermined classification rules, and store the classified data in a distributed database to ensure high availability and fast retrieval capabilities of the data;

[0094] S3, data dependency analysis: Use the graph neural network (GNN) model to analyze and model the multi-level dependencies in the data and generate a data dependency graph, including:

[0095] S31, data dependency graph construction: Use the graph neural network (GNN) model to analyze the data stored in the distributed database and construct a dependency graph between data entities. The nodes represent data entities and the edges represent the dependency relationships between data.

[0096] S32, graph neural network training: train the constructed data dependency graph, learn the weights and features of data dependency relationships through iterative optimization, and extract complex dependency structures between multi-level data;

[0097] S33, dependency optimization and output: Based on the training results, optimize the data dependency, eliminate redundant dependencies and strengthen the key dependency chain, generate the final data dependency graph, and provide a basis for the definition and optimization of subsequent data governance rules;

[0098] S4, Data governance rule definition and optimization: Define data governance rules based on the final data dependency graph and the governance requirements of the enterprise or organization, including data security policies, access control policies, data quality standards, and data lifecycle management. Based on the results of data dependency analysis, dynamically optimize data governance rules to better adapt to complex data dependency structures and improve the accuracy and efficiency of governance.

[0099] S5, data governance process execution: Based on the optimized data governance rules, the data governance process is automatically executed, including data access control, quality inspection, redundant data cleaning and compliance inspection, and the data governance execution log is recorded in real time to ensure the transparency and auditability of the governance process;

[0100] S6, Visualization: Use data visualization tools to graphically display key indicators, execution results, and data dependency maps in the data governance process. The display content includes the implementation of data governance rules, the improvement of data quality, the compliance of data compliance, and the changes in data dependencies, so that managers can intuitively understand and evaluate the effectiveness of data governance.

[0101] S7, Feedback and Optimization: Evaluate the effectiveness of data governance based on the results of the visualization, and optimize data governance rules based on the dynamic changes in the evaluation results to form a closed-loop optimization process, thereby continuously improving the quality and efficiency of data governance;

[0102] Through the above steps, the accuracy, efficiency and adaptability of data governance are improved. At the same time, the system can dynamically adjust the governance strategy to ensure the high quality and consistency of data, provide managers with intuitive and reliable governance results display and optimization feedback, and effectively respond to governance challenges in complex data environments.

[0103] Data collection and cleaning in S1 include:

[0104] S11, data collection: collect data from different data sources (such as relational databases, document databases, log files, sensor data streams, etc.) by integrating RESTful API, JDBC, FTP, and MQTT. Structured data is extracted from relational databases through SQL queries, and unstructured data is obtained from document databases through text parsers and file transfer protocols;

[0105] S12, cleaning of structured data: using linear interpolation to process missing values, calculating hash values ​​to identify whether data entries are repeated, and deduplicating the data;

[0106] S13, cleaning of unstructured data: For text data, use regular expressions (Regex) to filter out irrelevant characters and noise data, and convert unstructured data from different sources into a specified format for subsequent data processing and analysis;

[0107] Through the above, the integrity and accuracy of the data can be preserved to the maximum extent, and the consistency and availability of the data in multi-level analysis can be ensured through standardization and unified format.

[0108] Data classification and storage in S2 include:

[0109] S21, Classification rule definition: Based on business needs, data characteristics and usage, pre-define data classification rules, including those based on data source (such as sensors, user logs), data type (such as text, images, videos), time dimension, geographic location or business department;

[0110] S22, classification of structured data: For structured data, a classification algorithm based on a decision tree is used. By analyzing the field characteristics of the structured data, the data set D is segmented according to the field F. The construction of the decision tree determines the optimal classification feature by calculating the information gain IG, which is expressed as:

[0111]

[0112] Among them, H(D) is the entropy of data set D, D v is the subset divided by the value v of feature F;

[0113] S23, classification of unstructured data: For unstructured data, feature extraction (using TF-IDF) is performed on unstructured data (such as text and images), and then the data is classified using the K-means algorithm, which is expressed as:

[0114]

[0115] Among them, μ i is the i-th cluster center, C i is the set of data points in the i-th category, and x is a data point;

[0116] S24, labeling of classification labels: according to the classification rules and classification results, each type of data is labeled with a classification label. The label includes a predefined category name (such as "customer data", "sensor data") or a dynamically generated identifier (such as a time-based batch ID or a geographic location code). For structured data, the label may include "table name" or "field type". For unstructured data, the label may include "file type" or "feature category".

[0117] S25, distributed database selection: based on the data classification results, select a distributed database for storage. Structured data is stored in a distributed relational database (such as MySQL Cluster), and unstructured data is stored in a distributed document database (such as MongoDB or Cassandra) to meet the storage requirements of different data types.

[0118] S26, Data Sharding and Copy Management: Data sharding and copy management technologies are used to ensure high availability and fast retrieval. For structured data, data is sharded and stored according to classification tags and hash sharding rules. For unstructured data, data is sharded according to data features or file types, and copies are saved on multiple nodes to ensure fault tolerance and load balancing of data in a distributed environment.

[0119] S27, index establishment: To improve data retrieval efficiency, index the stored data based on classification labels and query fields. For structured data, use B-tree index or hash index, and for unstructured data, use full-text index or inverted index to quickly query and access large-scale data;

[0120] Through the above content, high availability of data, fast retrieval and load balancing of the system are ensured, which provides a solid foundation for subsequent data governance work, and through labeling and sharding management, effective governance of complex data environments is achieved.

[0121] The data dependency graph construction in S31 includes:

[0122] S311, definition of data entity nodes: extract data entities from the distributed database. Data entities include business data (such as order information, customer records), log data (such as access logs, error logs), and configuration data (such as system configuration, parameter settings). Data entities are represented as nodes in the graph neural network (GNN) model. Each node v i Represents a data entity, such as an order record or a set of configuration parameters;

[0123] S312, establishment of dependency edges: according to the logical relationship between data entities (such as the relationship between orders and customers, the relationship between logs and configurations), dependency relationships are established between data entities, which are represented as edges e in the dependency graph. ij , such as e ij represents the dependency between order information and customer records, or the dependency between access logs and system configuration. To quantify the strength of the dependency, the weight w of each edge is calculated. ij , expressed as:

[0124] w ij =α×f ij +β×d ij +γ×a ij ;

[0125] Among them, w ij For node v i and node v j The edge weight between ij is the interaction frequency between data entities (such as the number of interactions between orders and customers), d ij is the degree of association or distance between data entities (such as geographical distance or logical distance), a ij is the numerical attribute between data entities (such as transaction amount, log severity, etc.), α, β, and γ are adjustment coefficients used to balance the importance of interaction frequency, correlation, and numerical attributes in weight calculation;

[0126] S313, graph structure initialization: The defined data entity nodes (such as order information, customer records, access logs, system configuration) and dependency edges are combined to form an initial dependency graph G = (V, E), where V is the data entity set, E is the dependency set, and each node v i Contains the feature vector x associated with the data entity i , each edge e ij Contains edge weight w ij ;

[0127] Through the above content, the dependencies between these entities can be accurately captured and quantified, thereby providing precise basic data support for rule definition and optimization in the data governance process, making data governance more intelligent and effective.

[0128] Graph neural network training in S32 includes:

[0129] S321, input of the initial graph structure: input the constructed data dependency graph G = (V, E) into the graph neural network (GNN) model;

[0130] S322, message passing and node feature update: The message passing mechanism is used to update the node representation. In each layer iteration, each node v i The feature is updated according to the information of its neighbor node N(i), which is expressed as:

[0131]

[0132] in, For node v i The feature vector at layer t, w ij is the edge weight, indicating that node v i and v j The dependence strength between them, σ is the activation function, W (t) is the weight matrix of the tth layer, b (t) is the bias vector;

[0133] S323, Multi-layer iteration and global optimization: Through the multi-layer iterative optimization process, the graph neural network (GNN) model gradually updates the feature vector of each node Where T is the total number of iterations. Through the back propagation algorithm, the loss function L is calculated to optimize the model parameters W, which is expressed as:

[0134]

[0135] Where |E| is the size of the edge set E, (i,j)∈E is the set of all edges in the graph, is the edge weight predicted by the model, w ij is the actual edge weight;

[0136] Through the above content, the accuracy and applicability of data governance rules are effectively improved, enabling the system to better adapt to complex data environments and provide more accurate data governance decision support.

[0137] Dependency optimization and output in S33 include:

[0138] S331, Detection and elimination of redundant dependencies: After training the graph neural network (GNN) model, by analyzing the weight of each edge in the graph Determine the strength of the dependency relationship. For edges e whose weight is lower than the preset threshold τ ij , it is considered as redundant dependency. For the detected redundant dependency edges, they are deleted from the dependency graph to simplify the graph structure, reduce the computational complexity, and ensure the accuracy of subsequent analysis;

[0139] S332, Strengthening of key dependency chains: Based on the training results of the graph neural network (GNN) model, identify high-weight edges in the graph side Represents the key dependency chain between data entities. By strengthening the key dependency chain, the edge Optimize, expressed as:

[0140]

[0141] Among them, θ is the weight threshold of the key dependency, and γ>1 is the reinforcement coefficient, which ensures that the key dependency chain is further highlighted in the graph structure;

[0142] S333, generation of optimized graph: after redundant dependencies are eliminated and key dependency chains are strengthened, the optimized data dependency graph is recalculated and generated. The final data dependency graph G final =(V,E final ) contains optimized nodes V and edges E final , where E final Contains optimized dependencies;

[0143] Through the above content, not only the efficiency of data governance is improved, but also it is ensured that governance decisions are based on the most important and reliable dependencies, reducing the interference of misleading dependencies and providing strong support for the enterprise's data governance process.

[0144] The threshold τ is set based on the mean and standard deviation, expressed as:

[0145] τ=μ w -k×σ w ;

[0146] Where k is a constant, μ w is the average of all edge weights, σ w is the standard deviation of all edge weights;

[0147] The weight threshold θ of the critical dependency is based on the distribution of edge weights, and for all edge weights in the graph Sorting is performed and the top 25% or top 10% edges of the weight distribution are selected as the key dependency chains.

[0148] The definition and optimization of data governance rules in S4 include:

[0149] S41, data security strategy: based on the final data dependency graph G final =(V,F final ) in the data structure, formulate data security policies, and set strict access control and encryption requirements for data entities (nodes) and key dependency chains (edges) with high dependency intensity, which can be expressed as:

[0150]

[0151] in, For node vi The edge weights between the nodes are related to each other. The function f sets the corresponding security level according to the edge weight. The higher the weight, the higher the security level S. i The higher;

[0152] The function f is expressed as:

[0153]

[0154] S42, access control strategy: Based on the dependencies and access frequencies between data entities, a role-based access control model (RBAC) is established. i , based on the node features h in the dependency graph i and edge weights Dynamically assign access rights, expressed as:

[0155]

[0156] Among them, A i For access rights, function g is based on the node’s feature vector h i The sum of the weights of all edges connected to it determines the level of access rights;

[0157] The function g is expressed as:

[0158]

[0159] Among them, k1 and k2 are adjustment coefficients;

[0160] S43, data quality standard: Use the node features and edge weights in the data dependency graph to set data quality standards. For nodes (data entities) with strong dependencies and high weights, the data quality requirements are higher, expressed as:

[0161]

[0162] Among them, Q i is the data quality criterion, α and β are adjustment parameters to ensure that the data quality is proportional to the dependency;

[0163] S44, Data lifecycle management: Set the data lifecycle strategy based on the edge weights and node characteristics in the data dependency graph. For highly dependent data, lifecycle management should be more stringent, including regular backup, archiving, and auditing. i It is expressed as:

[0164]

[0165] Among them, γ is the adjustment coefficient to ensure that the setting of the life cycle is consistent with the dependency;

[0166] S45 introduces a feedback mechanism to dynamically adjust the rules based on the actual execution effect of data governance rules and new dependencies, expressed as:

[0167]

[0168] Among them, R old is the existing rule, δ is the learning rate, which is used to control the amplitude of rule optimization, and F(t) is the feedback function after the rule is executed;

[0169] The feedback function F(t) after rule execution is expressed as:

[0170] F(t)=w1×S(t)+w2×Q(t)+w3×A(t)+w4×L(t);

[0171] Among them, S(t) represents the evaluation result of data security at time t, Q(t) represents the evaluation result of data quality at time t, A(t) represents the evaluation result of access control at time t, L(t) represents the evaluation result of data lifecycle management at time t, w1, w2, w3, w4 are the weight coefficients of each indicator;

[0172] Through the above content, we ensure that the governance rules can accurately reflect the complex dependency structure between data, and through the dynamic optimization mechanism, continuously improve the accuracy and efficiency of data governance. Enterprises can adjust their data governance strategies in real time to ensure the security, integrity and availability of data.

[0173] Data governance process execution in S5 includes:

[0174] S51, Data Access Control: Based on the optimized data governance rules, data access control is automatically implemented, and the role-based access control model (RBAC) verifies and authorizes access requests from different users and applications in real time to ensure the security and compliance of data access;

[0175] S52, Quality Check: Automates the data quality check process and continuously monitors the integrity, consistency, and accuracy of data based on optimized data quality standards. If quality issues are detected, corrective measures are automatically triggered according to preset rules, including missing value filling, error value correction, format conversion, duplicate data cleaning, and redundant data merging, to ensure that data quality always meets requirements;

[0176] S53, Redundant Data Cleaning: Automatically identify and clean redundant data based on the redundancy detection standards set in the data governance rules, delete duplicate or invalid data records through periodic scanning and comparison of data sets, optimize data storage space, and improve data processing efficiency;

[0177] S54, Compliance Check: Automated check of each operation in the data processing process according to the defined compliance requirements;

[0178] S55, Logging mechanism: During the data governance process, all relevant operations are automatically recorded, including data access requests, quality inspection results, redundant data cleaning operations, and compliance inspection execution. The log content includes operation time, operation type, operator identity, and operation results to facilitate post-audit and analysis;

[0179] Through the above content, the strict implementation and real-time monitoring of data governance rules are ensured. Through the optimized rules, the efficiency and accuracy of data governance are greatly improved. At the same time, the system ensures the transparency and auditability of the governance process by recording all operation logs in real time, providing strong protection for the company's data compliance and security.

[0180] Visual displays in S6 include:

[0181] S61, Graphical display of key indicators: Display the execution of data governance rules through visual charts (such as bar charts, line charts or dashboards), including the execution frequency, success rate and instance statistics of rule violations;

[0182] S62, display of data compliance: based on the compliance check results, use pie charts, radar charts or heat maps to display the compliance rates of different data operations, and monitor data compliance in real time on a visual interface;

[0183] S63, graphical display of data dependency: Display the data dependency graph in the form of a node-edge graph, where nodes represent data entities and edges represent dependencies between data. Through graphical display, the display of nodes and edges is dynamically adjusted to highlight key dependency chains or high-risk dependency relationships;

[0184] S64, User Interaction and Customized Display: Provides user interaction functions, allowing users to customize visualization views and report content by clicking, dragging, or filtering operations, focusing on governance indicators or data dependencies;

[0185] Through the above content, managers can quickly understand the overall status of data governance, including the effectiveness of rule execution, improvement of data quality, compliance status, and changes in data dependencies. In addition, users can also customize display and interactive operations according to actual needs, which improves the transparency of the data governance process and decision-making support capabilities.

[0186] Feedback and optimization in S7 include:

[0187] S71, Evaluation of data governance effectiveness: By analyzing the key indicators, execution results, compliance status and changes in data dependency graphs displayed by data visualization tools, the execution effect and governance quality of data governance rules are evaluated. The evaluation indicators include the improvement rate of data quality, the success rate of rule execution, the compliance rate and the stability of dependency relationships. An automated algorithm is used for quantitative evaluation, which is expressed as:

[0188] E(t)=w4×R s (t)+w5×Q p (t)+w6×C r (t);

[0189] Among them, E(t) is the data governance effect evaluation score, w4, w5, and w6 are the weight coefficients of the evaluation indicators, which are adjusted according to the specific governance goals, and R s is the average success rate of data governance rule execution, Q p o is the percentage of data quality improvement, C r Compliance achievement rate;

[0190] S72, optimization of data governance rules: If the data governance effect evaluation score E(t) is lower than the adjustment threshold, the data governance rules are automatically adjusted, including resetting data security policies, adjusting access control permissions, optimizing data quality standards, and adjusting data lifecycle management to improve governance effectiveness;

[0191] The adjustment threshold is set based on percentiles and is expressed as:

[0192] T adjust =Q p ;

[0193] Among them, T adjust To adjust the threshold, Q p The 25th percentile of historical assessment scores;

[0194] Through the above content, the dynamic adaptability of the data governance process is ensured, and data governance rules can be adjusted and optimized in a timely manner to improve governance efficiency and effectiveness.

[0195] The present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention. In order to make the public have a thorough understanding of the present invention, specific details are described in detail in the following preferred embodiments of the present invention, but those skilled in the art can fully understand the present invention without the description of these details. In addition, in order to avoid unnecessary confusion about the essence of the present invention, well-known methods, processes, procedures, components and circuits are not described in detail.

[0196] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A visual display method for data governance, characterized in that: The following steps are involved: S1, data collection and cleaning: collect data from different data sources through multiple data interfaces, including structured data and unstructured data, and clean the collected data; S2, data classification and storage: classify the cleaned data according to the predetermined classification rules, and store the classified data in a distributed database; S3, data dependency analysis: Use the graph neural network model to analyze and model the multi-level dependencies in the data and generate a data dependency graph, including: S31, data dependency graph construction: Use the graph neural network model to analyze the data stored in the distributed database and construct a dependency graph between data entities. The nodes represent data entities and the edges represent the dependency relationships between data. S32, graph neural network training: train the constructed data dependency graph, learn the weights and features of data dependency relationships through iterative optimization, and extract complex dependency structures between multi-level data; S33, dependency optimization and output: Based on the training results, optimize the data dependency, eliminate redundant dependencies and strengthen the key dependency chain to generate the final data dependency graph; S4, data governance rule definition and optimization: Define data governance rules based on the final data dependency graph and the governance requirements of the enterprise or organization, including data security policies, access control policies, data quality standards, and data lifecycle management, and dynamically optimize data governance rules based on the results of data dependency analysis; S5, data governance process execution: Based on the optimized data governance rules, the data governance process is automatically executed, including data access control, quality inspection, redundant data cleaning and compliance inspection, and the execution log of data governance is recorded in real time; S6, Visualization: Use data visualization tools to graphically display key indicators, execution results, and data dependency maps in the data governance process. The display content includes the implementation of data governance rules, the improvement of data quality, the compliance of data compliance, and the changes in data dependency relationships. S7, Feedback and Optimization: Evaluate the effectiveness of data governance based on the results of the visualization, and optimize data governance rules based on dynamic changes in the evaluation results.

2. A visual display method for data governance according to claim 1, characterized in that: The data collection and cleaning in S1 includes: S11, Data Collection: Collect data from different data sources by integrating RESTful API, JDBC, FTP, and MQTT. Structured data is extracted from relational databases through SQL queries, and unstructured data is obtained from document databases through text parsers and file transfer protocols; S12, cleaning of structured data: using linear interpolation to process missing values, calculating hash values ​​to identify whether data entries are repeated, and deduplicating the data; S13, cleaning of unstructured data: For text data, use regular expressions to filter out irrelevant characters and noise data, and convert unstructured data from different sources into a specified format.

3. A visual display method for data governance according to claim 1, characterized in that: The data classification and storage in S2 include: S21, Classification rule definition: pre-define data classification rules based on business needs, data characteristics and usage, including based on data source, data type, time dimension, geographic location or business department; S22, classification of structured data: For structured data, a classification algorithm based on a decision tree is used. By analyzing the field characteristics of the structured data, the data set D is segmented according to the field F. The decision tree is constructed by calculating the information gain IG to determine the optimal classification feature; S23, classification of unstructured data: for unstructured data, feature extraction is performed on the unstructured data, and then the data is classified using the K-means algorithm; S24, labeling of classification labels: labeling each type of data with a classification label according to the classification rules and classification results, the label includes a predefined category name or a dynamically generated identifier; S25, distributed database selection: based on the data classification result, a distributed database is selected for storage, structured data is stored in a distributed relational database, and unstructured data is stored in a distributed document database; S26, data sharding and copy management: For structured data, the data is sharded and stored according to classification tags and hash sharding rules. For unstructured data, the data is sharded according to data features or file types, and copies are saved on multiple nodes. S27, index creation: create an index for the stored data based on the classification label and query field.

4. A visual display method for data governance according to claim 1, characterized in that: The data dependency graph construction in S31 includes: S311, definition of data entity node: extract data entities from the distributed database. Data entities include business data, log data, and configuration data. Data entities are represented as nodes in the graph neural network model. Each node v i Represents a data entity; S312, establishment of dependency edges: according to the logical relationship between data entities, dependency relationships are established between data entities, which are represented as edges e in the dependency graph. ij , to quantify the strength of the dependency, calculate the weight w of each edge i j, expressed as: In ij =α×f ij +β×d ij +γ×a ij ; Among them, w ij For node v i and node v j The edge weight between ij is the interaction frequency between data entities, d ij is the association or distance between data entities, a ij is the numerical attribute between data entities, α, β, and γ are adjustment coefficients; S313, graph structure initialization: combine the defined data entity nodes and dependency edges to form an initial dependency graph G = (V, E), where V is the data entity set, E is the dependency set, and each node v i Contains the feature vector x associated with the data entity i , each edge e ij Contains edge weight w ij .

5. A visual display method for data governance according to claim 4, characterized in that: The graph neural network training in S32 includes: S321, input of the initial graph structure: input the constructed data dependency graph G = (V, E) into the graph neural network model; S322, message passing and node feature update: The message passing mechanism is used to update the node representation. In each layer iteration, each node v i The feature is updated according to the information of its neighbor node N(i), which is expressed as: in, For node v i The feature vector at layer t, w ij is the edge weight, σ is the activation function, W(t) is the weight matrix of the tth layer, and b(t) is the bias vector; S323, Multi-layer iteration and global optimization: Through the multi-layer iterative optimization process, the graph neural network model gradually updates the feature vector of each node Where T is the total number of iterations. Through the back propagation algorithm, the loss function L is calculated to optimize the model parameters W, which is expressed as: Where |E| is the size of the edge set E, (i, j)∈E is the set of all edges in the graph, is the edge weight predicted by the model, w ij is the actual edge weight.

6. A visual display method for data governance according to claim 5, characterized in that: The dependency optimization and output in S33 include: S331, Detection and elimination of redundant dependencies: After training the graph neural network model, by analyzing the weight of each edge in the graph Determine the strength of the dependency relationship. For edges e whose weight is lower than the preset threshold τ ij , it is considered as redundant dependency, and the detected redundant dependency edge is deleted from the dependency graph; S332, Strengthening the key dependency chain: Based on the training results of the graph neural network model, identify the high-weight edges in the graph side Represents the key dependency chain between data entities. By strengthening the key dependency chain, the edge Optimize, expressed as: Among them, θ is the weight threshold of the key dependency, and γ>1 is the reinforcement coefficient; S333, generation of optimized graph: after redundant dependencies are eliminated and key dependency chains are strengthened, the optimized data dependency graph is recalculated and generated. The final data dependency graph G final =(V,E final ) contains optimized nodes V and edges E final , where E final Contains optimized dependencies.

7. A visual display method for data management according to claim 6, characterized in that: The data governance rule definition and optimization in S4 include: S41, data security strategy: based on the final data dependency graph G final =(V,E final ), formulate data security policies, and set strict access control and encryption requirements for data entities with high dependency intensity and key dependency chains, which can be expressed as: in, For node v i The edge weights between its related nodes, and the function f sets the corresponding security level according to the edge weights; S42, access control strategy: Based on the dependencies and access frequencies between data entities, a role-based access control model is established. i , based on the node features h in the dependency graph i and edge weights Dynamically assign access rights, expressed as: Among them, A i For access rights, function g is based on the node’s feature vector h i The sum of the weights of all edges connected to it determines the level of access rights; S43, data quality standard: Using the node features and edge weights in the data dependency graph, set the data quality standard, expressed as: Among them, Q i is the data quality criterion, α and β are adjustment parameters; S44, Data lifecycle management: Set the data lifecycle strategy based on the edge weights and node features in the data dependency graph. i It is expressed as: Among them, γ is the adjustment coefficient; S45 introduces a feedback mechanism to dynamically adjust the rules based on the actual execution effect of data governance rules and new dependencies, expressed as: Among them, R old is the existing rule, δ is the learning rate, and F(t) is the feedback function after the rule is executed.

8. A visual display method for data governance according to claim 7, characterized in that: The data governance process execution in S5 includes: S51, Data Access Control: Based on the optimized data governance rules, data access control is automatically implemented, and the role-based access control model verifies and authorizes access requests from different users and applications in real time; S52, Quality Check: Automates the data quality check process and continuously monitors the data based on optimized data quality standards. If quality issues are detected, corrective actions are automatically triggered according to preset rules, including missing value filling, error value correction, format conversion, duplicate data cleaning, and redundant data merging. S53, redundant data cleaning: automatically identify and clean redundant data according to the redundant detection standards set in the data governance rules; S54, Compliance Check: Automated check of each operation in the data processing process according to the defined compliance requirements; S55, logging mechanism: During the data governance process, all relevant operations are automatically recorded, including data access requests, quality check results, redundant data cleanup operations, and the execution of compliance checks. The log content includes operation time, operation type, operator identity, and operation results.

9. A visual display method for data governance according to claim 1, characterized in that: The visualization display in S6 includes: S61, Graphical display of key indicators: Display the execution of data governance rules through visual charts, including the execution frequency, success rate and instance statistics of rule violations; S62, display of data compliance: based on the compliance check results, use pie charts, radar charts or heat maps to display the compliance rates of different data operations, and monitor data compliance in real time on a visual interface; S63, graphical display of data dependency: Display the data dependency graph in the form of a node-edge graph, where nodes represent data entities and edges represent dependency relationships between data. Through graphical display, the display of nodes and edges can be dynamically adjusted; S64, User Interaction and Customized Display: Provides user interaction capabilities, allowing users to customize visualization views and report content by clicking, dragging, or filtering, focusing on governance indicators or data dependencies.

10. A visual display method for data governance according to claim 9, characterized in that: The feedback and optimization in S7 include: S71, Evaluation of data governance effectiveness: By analyzing the key indicators, execution results, compliance status and changes in data dependency graphs displayed by data visualization tools, the execution effect and governance quality of data governance rules are evaluated. The evaluation indicators include the improvement rate of data quality, the success rate of rule execution, the compliance rate and the stability of dependency relationships. An automated algorithm is used for quantitative evaluation, which is expressed as: E(t)=w4×R s (t)+w5×Q p (t)+w6×C r (t); Among them, E(t) is the data governance effect evaluation score, w4, w5, and W6 are the weight coefficients of the evaluation indicators, and R s is the average success rate of data governance rule execution, Q p is the percentage of data quality improvement, C r Compliance achievement rate; S72, Optimization of data governance rules: If the data governance effectiveness evaluation score E(t) is lower than the adjustment threshold, the data governance rules are automatically adjusted, including resetting data security policies, adjusting access control permissions, optimizing data quality standards, and adjusting data lifecycle management.

Images