Data management method and device, nonvolatile storage medium and electronic equipment

Abstract

The application discloses a data management method and device, a nonvolatile storage medium and an electronic device. The method comprises the following steps: determining an initial graph structure corresponding to an initial knowledge graph, wherein the graph structure comprises nodes and edges, the nodes correspond to data, the edges correspond to the association relationship between different data, and the data comprises structured data and unstructured data; determining the embedding vectors of the nodes, and clustering the nodes according to the similarity between the embedding vectors; updating the edge relationship in the initial knowledge graph according to the clustering result to obtain a target knowledge graph; and determining the access strategy of the target knowledge graph according to the node state in the target knowledge graph. The application solves the technical problem that the related art cannot comprehensively and efficiently manage data because some possible association relationships between data are missed.

Description

Technical Field

[0001] This application relates to the field of data processing, and more specifically, to a data management method, apparatus, non-volatile storage medium, and electronic device. Background Technology

[0002] When managing data, the knowledge graphs used in related technologies can usually only store static metadata and lack dynamic semantic discovery capabilities. This can lead to the omission of some relationships between data, resulting in the inability to manage the data comprehensively and efficiently.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a data management method, apparatus, non-volatile storage medium, and electronic device to at least solve the technical problem in the related art that the inability to comprehensively and efficiently manage data is caused by the omission of possible relationships between some data.

[0005] According to one aspect of the embodiments of this application, a data management method is provided, comprising: determining an initial graph structure corresponding to an initial knowledge graph, wherein the graph structure includes nodes and edges, nodes correspond to data, edges correspond to the association relationships between different data, and the data includes structured data and unstructured data; determining the embedding vector of each node, and clustering the nodes according to the similarity between the embedding vectors; updating the edge relationships in the initial knowledge graph according to the clustering results to obtain a target knowledge graph; and determining the access strategy of the target knowledge graph according to the node states in the target knowledge graph.

[0006] Optionally, updating the edge relationships in the initial knowledge graph based on the clustering results includes: determining knowledge graph nodes belonging to the same cluster based on the clustering results; establishing a target edge of semantic similarity between two knowledge graph nodes belonging to the same cluster, wherein the target edge is used to indicate that there is semantic similarity between the two connected knowledge graph nodes.

[0007] Optionally, the method further includes: upon receiving new data, determining a subgraph structure corresponding to the new data based on the new data and the target knowledge graph, wherein the subgraph structure includes subgraph nodes and subgraph edges, the subgraph nodes correspond to the new data, and the data in the data corresponding to the target knowledge graph that has a relationship with the new data; determining the embedding vectors of the subgraph nodes in the subgraph structure, and clustering the subgraph nodes based on the similarity between the embedding vectors of the subgraph nodes; updating the subgraph structure based on the clustering results obtained from the clustering of the subgraph nodes, and updating the target knowledge graph based on the updated subgraph structure.

[0008] Optionally, determining the access strategy for the target knowledge graph based on the node states in the target knowledge graph includes: determining a state space and an action space, wherein the state space includes the node states of each node in the target knowledge graph, and the action space includes the data processing strategies to be performed on the target knowledge graph; determining a reward function, wherein the reward function is used to compare the merits of different sets of node states, and the higher the reward function value, the better the set of node states; and determining the access strategy for the target knowledge graph based on the state space, action space, and reward function.

[0009] Optionally, node status includes node data quality score, node access frequency, and node label type.

[0010] Optionally, determining the access strategy for the target knowledge graph based on the node status in the target knowledge graph includes: identifying target nodes whose access frequency is higher than a preset frequency threshold when the node status includes node access frequency; identifying similar nodes of the target node based on clustering results when the target node is marked as a sensitive node, wherein similar nodes of the target node are nodes that are clustered in the same cluster as the target node; and generating data anonymization access strategies for the target node and its similar nodes, wherein the data anonymization access strategies include the access object types that need to be anonymized.

[0011] Optionally, after determining the access strategy of the target knowledge graph based on the node states in the target knowledge graph, the method further includes: obtaining user behavior simulation data, wherein the user behavior simulation data includes at least one of the following: historical query logs, custom test cases; replicating the target knowledge graph in a virtual runtime environment; determining the node states of the replicated target knowledge graph after executing the access strategy in the virtual runtime environment based on the user behavior simulation data and the replicated target knowledge graph, and determining a first evaluation result of the access strategy based on the node states of the target knowledge graph; performing compliance checks on the access strategy to obtain a second evaluation result; performing conflict checks on the access strategy to obtain a third evaluation result; determining a comprehensive verification result of the access strategy based on the first evaluation result, the second evaluation result, and the third evaluation result, and adjusting the access strategy based on the comprehensive verification result.

[0012] According to another aspect of the embodiments of this application, a data management device is also provided, comprising: a first processing module, configured to determine an initial graph structure corresponding to an initial knowledge graph, wherein the graph structure includes nodes and edges, nodes correspond to data, edges correspond to the association relationships between different data, and the data includes structured data and unstructured data; a second processing module, configured to determine the embedding vector of each node and cluster the nodes according to the similarity between the embedding vectors; a third processing module, configured to update the edge relationships in the initial knowledge graph according to the clustering results to obtain a target knowledge graph; and a fourth processing module, configured to determine the access strategy of the target knowledge graph according to the node states in the target knowledge graph.

[0013] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, wherein a program is stored in the non-volatile storage medium, and the program controls the device where the non-volatile storage medium is located to execute a data management method when it runs.

[0014] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory and a processor, the processor being configured to run a program stored in the memory, wherein the program executes a data management method during runtime.

[0015] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that implements a data management method when executed by a processor.

[0016] In this embodiment, an initial graph structure corresponding to the initial knowledge graph is determined. This graph structure includes nodes and edges, where nodes correspond to data and edges correspond to relationships between different data. The data includes structured and unstructured data. The embedding vectors of each node are determined, and the nodes are clustered based on the similarity between the embedding vectors. The edge relationships in the initial knowledge graph are updated based on the clustering results to obtain the target knowledge graph. The access strategy for the target knowledge graph is determined based on the node states in the target knowledge graph. By determining the graph structure corresponding to the initial knowledge graph, determining the embedding vectors of each node in the graph structure, and performing clustering based on the embedding vectors and updating the edges based on the clustering results, the aim of mining potential relationships between different data is achieved. This results in comprehensive and efficient data management, thus solving the technical problem in related technologies where potential relationships between some data are missed, leading to an inability to achieve comprehensive and efficient data management. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0018] Figure 1 This is a schematic diagram of the structure of a computer terminal (mobile terminal) according to an embodiment of this application;

[0019] Figure 2 This is a flowchart illustrating a data management method according to an embodiment of this application;

[0020] Figure 3 This is a schematic diagram illustrating the parameters of a reward function according to an embodiment of this application;

[0021] Figure 4 This is a schematic diagram of the structure of a data management system provided according to an embodiment of this application;

[0022] Figure 5 This is a schematic diagram of the structure of a data management device according to an embodiment of this application. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] To better understand the embodiments of this application, the technical terms involved in the embodiments of this application are explained below:

[0026] Data Fabric: is an architecture that enables seamless cross-platform data integration through proactive metadata, knowledge graphs, and automation technologies.

[0027] In the digital age, data has become one of the core assets of enterprises. Data Fabric is an architecture that enables seamless cross-platform data integration through proactive metadata, knowledge graphs and automation technologies, and it is one of the important research directions in the field of data governance.

[0028] The current application of Data Fabric faces numerous challenges. For example, enterprises suffer from severe data silos, with data scattered across multiple systems (such as CRM, ERP, and log systems) and lacking unified semantic understanding, making effective data integration and utilization difficult. Secondly, traditional static governance methods are ineffective; traditional metadata management relies on manual rule maintenance, failing to adapt to business changes, such as the inability to adjust governance strategies promptly when new data sources are added or regulatory policies are updated. Furthermore, governance costs are high; data lineage, quality rules, and other parameters require manual configuration by experts, which is not only time-consuming but also prone to errors.

[0029] However, traditional knowledge graphs in related technologies only store static metadata (such as table structure and field meanings), lacking dynamic relationship discovery capabilities and failing to meet enterprises' needs for dynamic data analysis. Rule engines, based on fixed logic (such as SQL quality checks), cannot self-optimize based on data behavior and struggle to cope with complex and ever-changing data environments. Furthermore, the lack of governance verification, coupled with the direct implementation of policies in the production environment, may lead to data service interruptions or compliance risks, resulting in significant potential losses for enterprises.

[0030] Current governance verification costs are high, and strategies are implemented directly in the production environment, which can easily lead to data service interruptions or compliance risks, and there is a lack of secure verification mechanisms.

[0031] In summary, the relevant technologies have the following technical shortcomings:

[0032] Traditional knowledge graphs only store static metadata (such as table structure and field comments) and lack dynamic semantic discovery capabilities (such as the inability to automatically associate "cust_id" with "client_no").

[0033] Rule engine: Based on fixed logic (such as predefined IF-THEN rules), it cannot self-optimize according to data usage behavior (such as dynamic desensitization when accessing sensitive fields frequently).

[0034] Limitations of governance tools: Existing tools (such as Collibra and Alation) rely on manual configuration by experts and cannot achieve the "automated and intelligent" governance required by Data Fabric.

[0035] Furthermore, the relevant technologies cannot solve the problem of coordinating the self-evolution of knowledge graphs with policy security verification. Therefore, an automated data asset governance method for Data Fabric architecture needs to be designed to address the above issues.

[0036] To address the aforementioned issues, this application provides relevant solutions, which are detailed below.

[0037] According to an embodiment of this application, a method embodiment for data management is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0038] The methods and embodiments provided in this application can be executed on mobile terminals, computer terminals, or similar computing devices. Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing a data management method is shown. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0039] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0040] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the data management method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the above-mentioned data management method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0041] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0042] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0043] Under the above operating environment, embodiments of this application provide a data management method, such as... Figure 2 As shown, the method includes the following steps:

[0044] Step S202: Determine the initial graph structure corresponding to the initial knowledge graph. The graph structure includes nodes and edges. Nodes correspond to data, and edges correspond to the relationships between different data. The data includes structured data and unstructured data.

[0045] In the technical solution provided in step S202, the aforementioned structured data is structured metadata, including data table structure, field comments, and ETL task logs. The aforementioned unstructured data is unstructured metadata, including a business terminology table, user query logs, and API call records.

[0046] In some embodiments of this application, the relationships corresponding to the above edges include data lineage (Table A → ETL → Table B), access relationship (User X queries Table Y), semantic similarity (Field A ≈ Business term B), etc.

[0047] Step S204: Determine the embedding vector of each node and cluster the nodes based on the similarity between the embedding vectors;

[0048] In the technical solution provided in step S204, the GraphSAGE algorithm can be used to generate the embedding vectors of each node. When clustering based on similarity, the fields corresponding to each node can be clustered based on the cosine similarity between different embedding vectors, thereby determining similar fields (such as "cust_id" and "client_no").

[0049] In some embodiments of this application, clustering results can be used for dynamic semantic association and as a reference when generating data access strategies. For example, clustering results can be directly used to update edge relationships in the knowledge graph. If two fields are clustered as semantically similar, the system will automatically establish a "semantic similarity" edge in the knowledge graph (e.g., field A ≈ business term B) without manual intervention.

[0050] In the data access policy generation process, the semantic relationships after clustering are used as part of the state space by the reinforcement learning policy generation module. For example, if a field is marked as a sensitive field (PII), its semantically related other fields will also be dynamically incorporated into the access control policy generation logic.

[0051] Step S206: Update the edge relationships in the initial knowledge graph based on the clustering results to obtain the target knowledge graph;

[0052] In the technical solution provided in step S206, the step of updating the edge relationships in the initial knowledge graph based on the clustering results includes: determining the knowledge graph nodes belonging to the same cluster based on the clustering results; establishing a target edge of semantic similarity between two knowledge graph nodes belonging to the same cluster, wherein the target edge is used to indicate that there is semantic similarity between the two connected knowledge graph nodes.

[0053] As an alternative implementation, the knowledge graph can use the Neo4j graph database to store dynamic relationships and support real-time querying and visualization.

[0054] Step S208: Determine the access strategy of the target knowledge graph based on the node status in the target knowledge graph.

[0055] In the technical solution provided in step S208, the step of determining the access strategy of the target knowledge graph based on the node states in the target knowledge graph includes: determining the state space and the action space, wherein the state space includes the node states of each node in the target knowledge graph, and the action space includes the data processing strategy to be executed on the target knowledge graph; determining the reward function, wherein the reward function is used to compare the merits of different sets of node states, and the higher the reward function value, the better the set of node states; and determining the access strategy of the target knowledge graph based on the state space, the action space, and the reward function.

[0056] As an optional implementation, node status includes node data quality score, node access frequency, and node label type. The data quality score includes a null value rate score, and the node label type includes whether it is a sensitive label.

[0057] In some embodiments of this application, when designing the dynamic space, it is necessary to define data quality rules, access control policies, and data lineage optimization methods. Data quality rules include control validation and format standardization; access control policies include field masking and row-level permission control; and data lineage optimization methods include ETL task scheduling priority adjustment methods.

[0058] In some embodiments of this application, the specific expression of the reward function may be as follows:

[0059]

[0060] In the above expression, such as Figure 3 As shown, α, β, and γ are weighting coefficients, which can be determined through training with historical data. R is the overall reward value, a quantitative indicator used to evaluate the effectiveness of the strategy. Q is the quality improvement rate, used to reflect the extent of data quality improvement, such as the decrease in the null value rate. The compliance score reflects the degree to which the strategy meets regulatory requirements. Performance overhead reflects the system latency or resource consumption caused by the strategy execution.

[0061] In some embodiments of this application, the state space, reward function, and action space are core components of building an automatic data access policy generation system based on reinforcement learning, each playing an indispensable role:

[0062] The function of the state space is to comprehensively capture the current state of data governance in a Data Fabric environment. The state space includes metrics such as field null value rate, access frequency, sensitivity tags, and data consistency, as well as dynamic information from user behavior logs, business glossaries, and data lineage graphs. This state information forms the basis for reinforcement learning model decisions, enabling the model to understand the current data governance context and make policy choices accordingly. Dynamic updates to the state space ensure the real-time nature and adaptability of model decisions, allowing it to quickly respond to changes in the data environment.

[0063] The reward function guides the direction of policy generation. It comprehensively evaluates the effectiveness of data access policies, including improvements in data quality (Q), enhanced compliance (C), and performance degradation (P). By adjusting the weighting coefficients α, β, and γ, the reward function balances the relationships between different governance objectives, prompting the model to prioritize both data quality and compliance while also considering system performance when generating policies. The setting of the reward function directly influences the model's learning objectives and behavioral patterns, and is a key mechanism by which reinforcement learning algorithms optimize policies.

[0064] The action space (policy space) provides all possible data access control strategies. This covers field-level anonymization, row-level access control, data quality rule setting, and data lineage optimization. Whenever the model is in a certain state, it selects one or more strategies from the action space to execute. By rehearsing the effects of these strategies in a digital twin validation sandbox, the model can observe the immediate feedback and state changes after strategy execution, thereby guiding the reinforcement learning algorithm to optimize the strategies. The flexibility and diversity of the action space ensures that the model can explore and generate the most suitable data access strategies for the current environment.

[0065] These three elements interact to form a closed-loop learning system. The state space describes the environment, the reward function guides the direction of action, and the action space provides possible action options. The reinforcement learning model selects an action based on the current state, observes the reward feedback after execution, and gradually optimizes the policy through iterative learning to achieve the goal of accumulating the maximum reward over a long period. In the scenario of this application, this process realizes the automated and intelligent generation of data access control policies under the Data Fabric architecture, while ensuring the security and effectiveness of the policies.

[0066] In some embodiments of this application, the state space serves to provide a real-time description of the data environment, including data quality, access patterns, and compliance status. By analyzing this space, data assets requiring governance and potential issues can be identified. The policy space provides a set of operations for policy generation, automatically selecting or generating the governance policy most suitable for the current environment based on the state space information, such as automatically adding field anonymization rules to respond to frequently accessed sensitive fields.

[0067] The reward function serves as a quantitative indicator for strategy optimization, guiding strategy generation and adjustment. Weight coefficients α, β, and γ, determined through training with historical data, ensure that strategy selection balances the needs for quality improvement, enhanced compliance, and performance maintenance.

[0068] At each decision-making moment, the current data governance state is retrieved from the state space, and an action policy is selected or generated from the action policy space based on this state. After execution, the reward value is calculated based on changes in data quality, compliance scores, and performance metrics. By learning from historical rewards, the algorithm automatically adjusts the probability of policy selection in the policy space, aiming to maximize long-term cumulative rewards, continuously optimizing policies, and improving data governance effectiveness.

[0069] Through this approach, the state space ensures the accuracy of the basic information for policy generation, the action policy space provides flexible operations for policy generation, and the reward function quantifies the policy effect, guiding the adaptive optimization of the policy. This mechanism requires no manual intervention, automatically iterating through algorithms to improve the efficiency and accuracy of data access policies, supporting intelligent governance under the Data Fabric architecture.

[0070] As an alternative implementation, a data access strategy can be generated using a Markov decision chain based on the aforementioned state space, action space, and reward function.

[0071] In some embodiments of this application, the state space can also be defined to include metrics such as field null value rate, access frequency, sensitive tags, and data consistency. Real-time updates are achieved by analyzing data lineage, user behavior logs, and business terminology using graph neural networks. The action space includes governance strategies such as field anonymization, access permission adjustments, and data quality rule formulation. A reward function is designed to evaluate the effectiveness of strategies by considering quality improvement, compliance, and performance degradation.

[0072] Then, in the digital twin environment, metadata and sampled data are cloned, historical query logs are injected, and the pre-execution of the simulated strategy is simulated. Indicators of data quality improvement, performance changes, and compliance risks are collected to generate real-time feedback. Using a reinforcement learning model, the probability of strategy selection is dynamically adjusted based on the feedback signals, with the goal of maximizing long-term rewards.

[0073] Then, an incremental training mechanism can be used to automatically identify synonymous fields (such as "cust_id" and "client_no"), avoiding full graph reconstruction and optimizing the accuracy of semantic association. Rule generation is automated, monitoring high-frequency access to sensitive fields and automatically triggering de-identification commands. New strategies are executed in a simulation verification sandbox to assess their impact on data quality, system performance, and compliance, detect policy conflicts, and ensure the security and feasibility of the governance solution.

[0074] Through multiple iterations, the model learns to select the optimal strategy in the action space given a state space. By optimizing the strategy through a feedback loop using a digital twin, it achieves adaptive generation of data access control strategies. This process requires no manual intervention; the intelligence level of the strategy is improved through algorithmic iteration, meeting the requirements of automated governance under the Data Fabric architecture.

[0075] It can be seen that by combining reinforcement learning with graph neural networks, dynamic data changes are captured, adaptive policies are automatically generated, and a digital twin verification sandbox is used to ensure the safety and feasibility of the policy before implementation. The whole mechanism forms a closed loop and promotes continuous policy optimization.

[0076] In some embodiments of this application, the step of determining the access strategy of the target knowledge graph based on the node status in the target knowledge graph includes: when the node status includes the node access frequency, determining the target node whose access frequency is higher than a preset frequency threshold; when the target node is marked as a sensitive node, determining the similar nodes of the target node based on the clustering results, wherein the similar nodes of the target node are nodes that are clustered in the same cluster as the target node; generating a data desensitization access strategy for the target node and the similar nodes of the target node, wherein the data desensitization access strategy includes the access object type that needs to be data desensitized.

[0077] As an optional implementation, dynamic desensitization rules can be automatically added when the PII tags in the knowledge graph are detected and real-time behavior analysis is dynamically triggered. For example, when the access frequency of the field user_identity reaches 120 times within 5 minutes and its attribute is marked as PII=TRUE in the knowledge graph, the system automatically generates dynamic desensitization instructions.

[0078] In some embodiments of this application, after determining the access strategy of the target knowledge graph based on the node states in the target knowledge graph, the method further includes the following steps to further verify and optimize the access strategy: obtaining user behavior simulation data, wherein the user behavior simulation data includes at least one of the following: historical query logs, custom test cases; copying the target knowledge graph in a virtual runtime environment; determining the node states of the copied target knowledge graph after executing the access strategy in the virtual runtime environment based on the user behavior simulation data and the copied target knowledge graph, and determining a first evaluation result of the access strategy based on the node states of the target knowledge graph; performing compliance checks on the access strategy to obtain a second evaluation result; performing conflict checks on the access strategy to obtain a third evaluation result; determining a comprehensive verification result of the access strategy based on the first evaluation result, the second evaluation result, and the third evaluation result, and adjusting the access strategy based on the comprehensive verification result.

[0079] As an optional implementation, metadata and 10% sampled data from the production environment can be cloned in a virtual runtime environment, and historical query logs or custom test cases can be injected to simulate user behavior. The following verification process can then be used to determine the comprehensive verification results of the access policy and to provide feedback for optimization:

[0080] The first step, strategy pre-execution: running the newly generated governance rules in the sandbox;

[0081] The second step is impact assessment, where specific assessment metrics can include data quality, performance metrics, and compliance risks. Data quality includes null value rate and consistency error; performance metrics include query latency and resource utilization; and compliance risks include the probability of sensitive fields being exposed.

[0082] The third step is conflict detection: checking the mutual exclusivity between the new rules and existing strategies (such as the conflict between the anonymization rules and the data requirements of the risk control model).

[0083] The fourth step is feedback optimization: the validation results are returned to the reinforcement learning model to iteratively optimize the strategy.

[0084] In some embodiments of this application, upon receiving new data, a subgraph structure corresponding to the new data is determined based on the new data and the target knowledge graph. The subgraph structure includes subgraph nodes and subgraph edges. The subgraph nodes correspond to the new data, as well as data in the target knowledge graph that has a relationship with the new data. The embedding vectors of the subgraph nodes in the subgraph structure are determined, and the subgraph nodes are clustered based on the similarity between their embedding vectors. The subgraph structure is updated based on the clustering results obtained from the subgraph node clustering, and the target knowledge graph is updated based on the updated subgraph structure.

[0085] Optionally, when a new data source (such as a newly integrated CRM system) or user behavior logs are updated, the system will extract the subgraph corresponding to the new data (such as a new data table and its associated fields). The GraphSAGE algorithm can then be used to perform node embedding calculations only on the new subgraph, rather than reconstructing the entire graph. For example, fields in the new table Table_C will be automatically associated with similar fields in the existing knowledge graph (such as "client_no"). The updated node embeddings and edge relationships are then synchronized to the Neo4j graph database, ensuring the dynamic nature of the knowledge graph. For instance, after the new field "user_identity" is clustered into a PII, its associated access frequency data will influence the policy generation module's decisions in real time.

[0086] As an optional implementation, data lineage tracing or behavior log analysis can be used to determine the data associated with the newly added data, thereby determining the data range covered by the subgraph structure corresponding to the new data, that is, determining which nodes should be included in the subgraph structure. Optionally, when a new data source is connected, the system uses ETL task logs and data lineage relationships (such as Table_A → ETL → Table_B) to determine its directly related upstream and downstream nodes. For example, if a new table Table_C is associated with Table_B through an ETL task, then Table_C and its directly connected nodes (such as fields, users) constitute a local subgraph. In addition, user query logs and API call records will identify frequently accessed nodes (such as frequently queried fields), and these nodes and their neighboring nodes (such as associated business terms) will be included in the local subgraph.

[0087] Once the local subgraph extent is determined, incremental training of the GNN can be performed only on the local subgraph, such as updating the embedding vectors of Table_C and its neighboring nodes, instead of reconstructing the entire graph. This process is achieved through the real-time query capabilities of the graph database, ensuring both efficiency and accuracy.

[0088] By determining the initial graph structure corresponding to the initial knowledge graph, where the graph structure includes nodes and edges, nodes correspond to data, and edges correspond to the relationships between different data, including structured and unstructured data; determining the embedding vectors of each node and clustering the nodes based on the similarity between the embedding vectors; updating the edge relationships in the initial knowledge graph based on the clustering results to obtain the target knowledge graph; and determining the access strategy of the target knowledge graph based on the node states in the target knowledge graph, this method achieves the goal of mining the possible relationships between different data by determining the graph structure corresponding to the initial knowledge graph, determining the embedding vectors of each node in the graph structure, and performing clustering based on the embedding vectors and updating the edges based on the clustering results. This achieves the technical effect of comprehensive and efficient data management, and solves the technical problem in related technologies that may miss some possible relationships between data, thus preventing comprehensive and efficient data management.

[0089] According to embodiments of this application, a method is also provided. Figure 4 The data management system shown is used to perform, for example... Figure 2The data management method is illustrated. The self-evolving knowledge graph module 40 generates a dynamically updatable knowledge graph based on metadata. The reinforcement learning policy generation module 42 generates a data access policy based on the state of the knowledge graph generated by the self-evolving knowledge graph module (e.g., quality score, access frequency). The digital twin verification module 44 clones metadata from the self-evolving knowledge graph module 40 and verifies the data access policy sent by the reinforcement learning policy generation module. The verification result is then fed back to the reinforcement learning policy generation module 42.

[0090] This application provides a data management device. Figure 5 This is a schematic diagram of the device. From Figure 5 As can be seen from the diagram, the device includes: a first processing module 50, used to determine the initial graph structure corresponding to the initial knowledge graph, wherein the graph structure includes nodes and edges, nodes correspond to data, edges correspond to the association relationships between different data, and the data includes structured data and unstructured data; a second processing module 52, used to determine the embedding vector of each node and cluster the nodes according to the similarity between the embedding vectors; a third processing module 54, used to update the edge relationships in the initial knowledge graph according to the clustering results to obtain the target knowledge graph; and a fourth processing module 56, used to determine the access strategy of the target knowledge graph according to the node state in the target knowledge graph.

[0091] In some embodiments of this application, the step of the third processing module 56 updating the edge relationships in the initial knowledge graph based on the clustering results includes: determining knowledge graph nodes belonging to the same cluster in the knowledge graph based on the clustering results; establishing a target edge of semantic similarity between two knowledge graph nodes belonging to the same cluster, wherein the target edge is used to indicate that there is semantic similarity between the two connected knowledge graph nodes.

[0092] In some embodiments of this application, the step of the fourth processing module 58 in determining the access strategy of the target knowledge graph based on the node states in the target knowledge graph includes: determining a state space and an action space, wherein the state space includes the node states of each node in the target knowledge graph, and the action space includes the data processing strategy to be performed on the target knowledge graph; determining a reward function, wherein the reward function is used to compare the merits of different sets of node states, and the higher the reward function value, the better the set of node states; and determining the access strategy of the target knowledge graph based on the state space, the action space, and the reward function.

[0093] In some embodiments of this application, node status includes node data quality score, node access frequency, and node tag type.

[0094] In some embodiments of this application, the step of the fourth processing module 58 determining the access strategy of the target knowledge graph based on the node status in the target knowledge graph includes: when the node status includes the node access frequency, determining the target node whose access frequency is higher than a preset frequency threshold; when the target node is marked as a sensitive node, determining the similar nodes of the target node based on the clustering results, wherein the similar nodes of the target node are nodes that are clustered in the same cluster as the target node; generating a data desensitization access strategy for the target node and the similar nodes of the target node, wherein the data desensitization access strategy includes the access object type that needs to be data desensitized.

[0095] In some embodiments of this application, after determining the access strategy of the target knowledge graph based on the node states in the target knowledge graph, the fourth processing module 58 is further configured to: acquire user behavior simulation data, wherein the user behavior simulation data includes at least one of the following: historical query logs, custom test cases; copy the target knowledge graph in a virtual runtime environment; in the virtual runtime environment, based on the user behavior simulation data and the copied target knowledge graph, determine the node states of the copied target knowledge graph after executing the access strategy, and determine the first evaluation result of the access strategy based on the node states of the target knowledge graph; perform compliance testing on the access strategy to obtain a second evaluation result; perform conflict testing on the access strategy to obtain a third evaluation result; determine the comprehensive verification result of the access strategy based on the first evaluation result, the second evaluation result, and the third evaluation result, and adjust the access strategy based on the comprehensive verification result.

[0096] In some embodiments of this application, the data management device is further configured to, upon receiving new data, determine a subgraph structure corresponding to the new data based on the new data and the target knowledge graph, wherein the subgraph structure includes subgraph nodes and subgraph edges, the subgraph nodes correspond to the new data, and the data in the data corresponding to the target knowledge graph that has a relationship with the new data; determine the embedding vectors of the subgraph nodes in the subgraph structure, and cluster the subgraph nodes based on the similarity between the embedding vectors of the subgraph nodes; update the subgraph structure based on the clustering results obtained from the clustering of the subgraph nodes, and update the target knowledge graph based on the updated subgraph structure.

[0097] It should be noted that each module in the above data management device can be a program module (e.g., a set of program instructions to implement a certain function) or a hardware module. For the latter, it can be manifested in the following forms, but is not limited to them: each of the above modules is manifested as a processor, or the functions of each of the above modules are implemented by a processor.

[0098] According to an embodiment of this application, a non-volatile storage medium is also provided. The non-volatile storage medium stores a program, wherein, during program execution, it controls the device where the non-volatile storage medium resides to execute the following data management method: determining an initial graph structure corresponding to an initial knowledge graph, wherein the graph structure includes nodes and edges, nodes correspond to data, edges correspond to the relationships between different data, and the data includes structured data and unstructured data; determining the embedding vector of each node, and clustering the nodes based on the similarity between the embedding vectors; updating the edge relationships in the initial knowledge graph based on the clustering results to obtain a target knowledge graph; and determining the access strategy of the target knowledge graph based on the node states in the target knowledge graph.

[0099] According to an embodiment of this application, an electronic device is also provided, including: a memory and a processor. The processor is used to run a program stored in the memory, wherein the program executes the following data management method during runtime: determining an initial graph structure corresponding to an initial knowledge graph, wherein the graph structure includes nodes and edges, nodes correspond to data, edges correspond to the association relationships between different data, and the data includes structured data and unstructured data; determining the embedding vector of each node, and clustering the nodes according to the similarity between the embedding vectors; updating the edge relationships in the initial knowledge graph according to the clustering results to obtain a target knowledge graph; and determining the access strategy of the target knowledge graph according to the node states in the target knowledge graph.

[0100] According to an embodiment of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the following data management method: determining an initial graph structure corresponding to an initial knowledge graph, wherein the graph structure includes nodes and edges, nodes correspond to data, edges correspond to the relationships between different data, and the data includes structured data and unstructured data; determining the embedding vector of each node and clustering the nodes based on the similarity between the embedding vectors; updating the edge relationships in the initial knowledge graph based on the clustering results to obtain a target knowledge graph; and determining the access strategy of the target knowledge graph based on the node states in the target knowledge graph.

[0101] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0102] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0103] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0104] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0105] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0106] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data management method, characterized in that, include: Determine the initial graph structure corresponding to the initial knowledge graph, wherein the graph structure includes nodes and edges, the nodes correspond to data, the edges correspond to the association relationships between different data, and the data includes structured data and unstructured data; Determine the embedding vector of each node, and cluster the nodes based on the similarity between the embedding vectors; The edge relationships in the initial knowledge graph are updated based on the clustering results to obtain the target knowledge graph; Determining the access strategy for the target knowledge graph based on the node status in the target knowledge graph includes: when the node status includes node access frequency, identifying target nodes whose access frequency is higher than a preset frequency threshold; when the target node is marked as a sensitive node, determining similar nodes of the target node based on the clustering results, wherein the similar nodes of the target node are nodes clustered in the same cluster as the target node; generating a data anonymization access strategy for the target node and the similar nodes of the target node, wherein the data anonymization access strategy includes the access object types that need to be anonymized.

2. The data management method according to claim 1, characterized in that, Updating the edge relationships in the initial knowledge graph based on the clustering results includes: Based on the clustering results, knowledge graph nodes belonging to the same cluster in the knowledge graph are determined; Establish a semantically similar target edge between two knowledge graph nodes belonging to the same cluster, wherein the target edge is used to indicate that there is semantic similarity between the two connected knowledge graph nodes.

3. The data management method according to claim 1, characterized in that, The method further includes: Upon receiving new data, a subgraph structure corresponding to the new data is determined based on the new data and the target knowledge graph. The subgraph structure includes subgraph nodes and subgraph edges. The subgraph nodes correspond to the new data, as well as data in the data corresponding to the target knowledge graph that have a relationship with the new data. Determine the embedding vectors of the subgraph nodes in the subgraph structure, and cluster the subgraph nodes based on the similarity between the embedding vectors of the subgraph nodes; The subgraph structure is updated based on the clustering results obtained from clustering the subgraph nodes, and the target knowledge graph is updated based on the updated subgraph structure.

4. The data management method according to claim 1, characterized in that, Determining the access strategy for the target knowledge graph based on the node states in the target knowledge graph includes: A state space and an action space are determined, wherein the state space includes the node states of each node in the target knowledge graph, and the action space includes the data processing strategies to be executed on the target knowledge graph; A reward function is determined, wherein the reward function is used to compare the merits of different sets of node states, and the higher the reward function value, the better the set of node states; The access strategy for the target knowledge graph is determined based on the state space, the action space, and the reward function.

5. The data management method according to claim 4, characterized in that, The node status includes node data quality score, node access frequency, and node tag type.

6. The data management method according to claim 1, characterized in that, After determining the access strategy of the target knowledge graph based on the node states in the target knowledge graph, the method further includes: Obtain simulated user behavior data, wherein the simulated user behavior data includes at least one of the following: historical query logs, custom test cases; Replicate the target knowledge graph in a virtual runtime environment; In the virtual operating environment, based on the user behavior simulation data and the copied target knowledge graph, the node status of the copied target knowledge graph after the access policy is executed is determined, and the first evaluation result of the access policy is determined based on the node status of the target knowledge graph. The access policy is subjected to compliance testing to obtain a second evaluation result; Conflict detection is performed on the access policy to obtain a third evaluation result; Based on the first evaluation result, the second evaluation result, and the third evaluation result, a comprehensive verification result of the access policy is determined, and the access policy is adjusted based on the comprehensive verification result.

7. A data management device, characterized in that, include: The first processing module is used to determine the initial graph structure corresponding to the initial knowledge graph, wherein the graph structure includes nodes and edges, the nodes correspond to data, the edges correspond to the association between different data, and the data includes structured data and unstructured data; The second processing module is used to determine the embedding vector of each node and cluster the nodes based on the similarity between the embedding vectors. The third processing module is used to update the edge relationships in the initial knowledge graph based on the clustering results to obtain the target knowledge graph. The fourth processing module is used to determine the access strategy of the target knowledge graph based on the node status in the target knowledge graph, including: when the node status includes node access frequency, determining the target node whose access frequency is higher than a preset frequency threshold; when the target node is marked as a sensitive node, determining the similar nodes of the target node based on the clustering results, wherein the similar nodes of the target node are nodes that are clustered in the same cluster as the target node; generating a data anonymization access strategy for the target node and the similar nodes of the target node, wherein the data anonymization access strategy includes the access object type that needs to be data anonymized.

8. A non-volatile storage medium, characterized in that, The non-volatile storage medium stores a program, wherein when the program is executed, it controls the device where the non-volatile storage medium is located to perform the data management method according to any one of claims 1 to 6.

9. An electronic device, characterized in that, include: A memory and a processor, the processor being configured to run a program stored in the memory, wherein the program, when running, performs the data management method according to any one of claims 1 to 6.

10. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the data management method according to any one of claims 1 to 6.

Images