Data management method and device, nonvolatile storage medium and electronic equipment

Abstract

The invention discloses a data management method and device, a nonvolatile storage medium and electronic equipment. The method comprises the steps that an initial graph structure corresponding to an initial knowledge graph is determined, the graph structure comprises nodes and edges, the nodes correspond to data, the edges correspond to the incidence relation between different data, and the data comprise structured data and unstructured data; determining an embedded vector of each node, and clustering the nodes according to the similarity between the embedded vectors; updating an edge relationship in the initial knowledge graph according to a clustering result to obtain a target knowledge graph; and determining an access strategy of the target knowledge graph according to the node state in the target knowledge graph. According to the data management method and device, the technical problem that the data cannot be comprehensively and efficiently managed due to the fact that the incidence relation possibly existing among partial data is missed in the related technology is solved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and more specifically, to a data management method, device, non-volatile storage medium, and electronic device. Background Art

[0002] The knowledge graphs used in related technologies can usually only store static metadata when managing data and lack dynamic semantic discovery capabilities, which leads to the omission of some associations between data and the inability to manage data comprehensively and efficiently.

[0003] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0004] The embodiments of the present application provide a data management method, device, non-volatile storage medium and electronic device to at least solve the technical problem in the related art that possible associations between some data are omitted, resulting in the inability to comprehensively and efficiently manage data.

[0005] According to one aspect of an embodiment of the present application, a data management method is provided, including: determining an initial graph structure corresponding to an initial knowledge graph, wherein the graph structure includes nodes and edges, the nodes correspond to data, the edges correspond to the association relationship between different data, and the data includes structured data and unstructured data; determining the embedding vector of each node, and clustering the nodes based on the similarity between the embedding vectors; updating the edge relationship in the initial knowledge graph based on the clustering results to obtain a target knowledge graph; and determining the access strategy of the target knowledge graph based on the node status in the target knowledge graph.

[0006] Optionally, updating the edge relationship in the initial knowledge graph based on the clustering results includes: determining the knowledge graph nodes in the knowledge graph belonging to the same cluster cluster based on the clustering results; establishing a target edge of a semantically similar type between two knowledge graph nodes belonging to the same cluster cluster, wherein the target edge is used to indicate that there is semantic similarity between the two connected knowledge graph nodes.

[0007] Optionally, the method also includes: when new data is received, determining a subgraph structure corresponding to the new data based on the new data and the target knowledge graph, wherein the subgraph structure includes subgraph nodes and subgraph edges, the subgraph nodes corresponding to the new data, and data corresponding to the target knowledge graph and having an association relationship with the new data; determining the embedding vectors of the subgraph nodes in the subgraph structure, and clustering the subgraph nodes based on the similarity between the embedding vectors of the subgraph nodes; updating the subgraph structure based on the clustering results obtained by clustering the subgraph nodes, and updating the target knowledge graph based on the updated subgraph structure.

[0008] Optionally, determining the access strategy of the target knowledge graph based on the node states in the target knowledge graph includes: determining a state space and an action space, wherein the state space includes the node states of each node in the target knowledge graph, and the action space includes the data processing strategy executed on the target indication graph; determining a reward function, wherein the reward function is used to compare the pros and cons of different node state sets, and the higher the reward function value, the better the node state set; determining the access strategy of the target knowledge graph based on the state space, the action space and the reward function.

[0009] Optionally, the node status includes a node data quality score, a node access frequency, and a node label type.

[0010] Optionally, determining the access strategy of the target knowledge graph based on the node status in the target knowledge graph includes: in a case where the node status includes the node access frequency, determining the target node whose access frequency is higher than a preset frequency threshold; in a case where the target node is marked as a sensitive node, determining the similar nodes of the target node based on the clustering result, wherein the similar nodes of the target node are nodes that are clustered into the same cluster as the target node; generating a data desensitizing access strategy for the target node and the similar nodes of the target node, wherein the data desensitizing strategy includes the type of access object that needs to be desensitized.

[0011] Optionally, after determining the access strategy of the target knowledge graph based on the node status in the target knowledge graph, the method also includes: obtaining user behavior simulation data, wherein the user behavior simulation data includes at least one of the following: historical query logs, custom test cases; copying the target knowledge graph in a virtual operating environment; determining the node status of the copied target knowledge graph after executing the access strategy in the virtual environment based on the user behavior simulation data and the copied target knowledge graph, and determining a first evaluation result of the access strategy based on the node status of the target knowledge graph; performing compliance detection on the access strategy to obtain a second evaluation result; performing conflict detection on the access strategy to obtain a third evaluation result; determining a comprehensive verification result of the access strategy based on the first evaluation result, the second evaluation result and the third evaluation result, and adjusting the access strategy based on the comprehensive verification result.

[0012] According to another aspect of an embodiment of the present application, a data management device is also provided, including: a first processing module for determining an initial graph structure corresponding to an initial knowledge graph, wherein the graph structure includes nodes and edges, the nodes correspond to data, the edges correspond to the association relationship between different data, and the data includes structured data and unstructured data; a second processing module for determining the embedding vector of each node, and clustering the nodes according to the similarity between the embedding vectors; a third processing module for updating the edge relationship in the initial knowledge graph based on the clustering results to obtain a target knowledge graph; a fourth processing module for determining the access strategy of the target knowledge graph based on the node status in the target knowledge graph.

[0013] According to another aspect of an embodiment of the present application, a non-volatile storage medium is provided, in which a program is stored. When the program is running, a device where the non-volatile storage medium is located is controlled to execute a data management method.

[0014] According to another aspect of an embodiment of the present application, an electronic device is provided, including: a memory and a processor, the processor being configured to run a program stored in the memory, wherein the data management method is executed when the program is run.

[0015] According to another aspect of an embodiment of the present application, a computer program product is provided, including a computer program, which implements the data management method when executed by a processor.

[0016] In an embodiment of the present application, an initial graph structure corresponding to an initial knowledge graph is determined, wherein the graph structure includes nodes and edges, the nodes correspond to data, the edges correspond to the association relationships between different data, and the data include structured data and unstructured data; the embedding vector of each node is determined, and the nodes are clustered according to the similarity between the embedding vectors; the edge relationships in the initial knowledge graph are updated according to the clustering results to obtain a target knowledge graph; the access strategy of the target knowledge graph is determined according to the node status in the target knowledge graph. By determining the graph structure corresponding to the initial knowledge graph, determining the embedding vector of each node in the graph structure, clustering according to the embedding vector and updating the edges according to the clustering results, the purpose of mining the possible association relationships between different data is achieved, thereby realizing the technical effect of comprehensive and efficient management of data, and further solving the technical problem that the possible association relationships between some data are omitted in the related technology and that the data cannot be comprehensively and efficiently managed. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 1 is a schematic diagram of the structure of a computer terminal (mobile terminal) provided according to an embodiment of the present application;

[0019] Figure 2 This is a flow chart of a data management method provided according to an embodiment of the present application;

[0020] Figure 3 This is a schematic diagram illustrating reward function parameters according to an embodiment of the present application;

[0021] Figure 4 This is a structural diagram of a data management system provided according to an embodiment of the present application;

[0022] Figure 5 It is a structural diagram of a data management device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0026] Data Fabric: An architecture that enables seamless cross-platform data integration through active metadata, knowledge graphs, and automation technologies.

[0027] In the digital age, data has become one of the core assets of an enterprise. Data Fabric is an architecture that achieves seamless integration of cross-platform data through active metadata, knowledge graphs, and automation technologies. It is one of the important research directions in the field of data governance.

[0028] Current Data Fabric applications face numerous challenges, including severe data silos within enterprises, fragmented data across multiple systems (such as CRM, ERP, and logging systems), and a lack of unified semantic understanding, making effective data integration and utilization difficult. Furthermore, traditional static governance approaches are ineffective. Traditional metadata management relies on manually maintained rules and is unable to adapt to business changes, such as the inability to adjust governance strategies in response to new data sources or regulatory policy updates. Furthermore, governance costs are high, requiring experts to manually configure data lineage and quality rules, which is time-consuming and error-prone.

[0029] However, traditional knowledge graphs in related technologies only store static metadata (such as table structure and field meaning), lack dynamic relationship discovery capabilities, and are unable to meet enterprises' needs for dynamic data analysis. Rule engines are based on fixed logic (such as SQL quality checks) and cannot self-optimize based on data behavior, making them difficult to cope with complex and changing data environments. Furthermore, the lack of governance verification and the direct implementation of policies in production environments can lead to data service interruptions and compliance risks, resulting in significant potential losses for enterprises.

[0030] The current governance verification cost is high, and policies are implemented directly in the production environment, which can easily lead to data service interruptions or compliance risks and lack a secure verification mechanism.

[0031] In summary, the relevant technologies have the following technical defects:

[0032] Traditional knowledge graphs: only store static metadata (such as table structure and field annotations) and lack dynamic semantic discovery capabilities (such as the inability to automatically associate "cust_id" and "client_no").

[0033] Rule engine: Based on fixed logic (such as predefined IF-THEN rules), it cannot self-optimize based on data usage behavior (such as dynamic desensitization when sensitive fields are frequently accessed).

[0034] Limitations of governance tools: Existing tools (such as Collibra and Alation) rely on manual configuration by experts and cannot achieve the "automated and intelligent" governance required by Data Fabric.

[0035] In addition, related technologies cannot solve the coordination problem between knowledge graph self-evolution and policy security verification. Therefore, to address the above problems, it is necessary to design an automated data asset governance method for the Data Fabric architecture.

[0036] In order to solve the above problems, relevant solutions are provided in the embodiments of the present application, which are described in detail below.

[0037] According to an embodiment of the present application, a method embodiment of a data management method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0038] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a data management method. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more (illustrated as 102a, 102b, ..., 102n) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0039] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0040] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the data management method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implementing the above-mentioned data management method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0041] The transmission device 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0042] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0043] In the above operating environment, the embodiment of the present application provides a data management method, such as Figure 2 As shown, the method includes the following steps:

[0044] Step S202: determining an initial graph structure corresponding to the initial knowledge graph, wherein the graph structure includes nodes and edges, nodes correspond to data, edges correspond to associations between different data, and data includes structured data and unstructured data;

[0045] In the technical solution provided in step S202, the structured data is structured metadata, including data table structure, field annotations, and ETL task logs. The unstructured data is unstructured metadata, including business glossary, user query logs, and API call records.

[0046] In some embodiments of the present application, the association relationships corresponding to the above edges include data lineage (Table A→ETL→Table B), access relationship (User X queries Table Y), semantic similarity (field A≈business term B), etc.

[0047] Step S204: determining the embedding vector of each node and clustering the nodes based on the similarity between the embedding vectors;

[0048] In the technical solution provided in step S204, the GraphSAGE algorithm can be used to generate the embedding vectors of each node. When clustering based on similarity, the fields corresponding to each node can be clustered based on the cosine similarity between different embedding vectors to determine similar fields (such as "cust_id" and "client_no").

[0049] In some embodiments of the present application, clustering results can be used for dynamic semantic association and provide reference when generating data access strategies. For example, clustering results are directly used to update edge relationships in the knowledge graph. If two fields are clustered as semantically similar, the system will automatically create a "semantic similarity" edge in the knowledge graph (such as field A ≈ business term B) without manual intervention.

[0050] When generating data access policies, the clustered semantic associations are used by the reinforcement learning strategy generation module as part of the state space. For example, if a field is marked as sensitive (PII), other semantically associated fields are dynamically incorporated into the access control policy generation logic.

[0051] Step S206: Update the edge relationships in the initial knowledge graph based on the clustering results to obtain the target knowledge graph;

[0052] In the technical solution provided in step S206, the step of updating the edge relationship in the initial knowledge graph based on the clustering results includes: determining the knowledge graph nodes in the knowledge graph belonging to the same cluster cluster based on the clustering results; establishing a target edge of a semantically similar type between two knowledge graph nodes belonging to the same cluster cluster, wherein the target edge is used to indicate that there is semantic similarity between the two connected knowledge graph nodes.

[0053] As an optional implementation, the Neo4j graph database can be used in the knowledge graph to store dynamic relationships and support real-time query and visualization.

[0054] Step S208: Determine the access strategy of the target knowledge graph based on the node status in the target knowledge graph.

[0055] In the technical solution provided in step S208, the step of determining the access strategy of the target knowledge graph based on the node status in the target knowledge graph includes: determining the state space and the action space, wherein the state space includes the node status of each node in the target knowledge graph, and the action space includes the data processing strategy executed on the target indication graph; determining the reward function, wherein the reward function is used to compare the advantages and disadvantages between different node state sets, and the higher the reward function value, the better the node state set; determining the access strategy of the target knowledge graph based on the state space, the action space and the reward function.

[0056] As an optional implementation, the node status includes the node data quality score, node access frequency, and node label type. The data quality score includes the null value rate score, and the node label type includes whether it is a sensitive label.

[0057] In some embodiments of this application, when designing a dynamic space, it is necessary to define data quality rules, access control policies, and data lineage optimization methods. Data quality rules include control validation and format standardization, and access control policies include field desensitization and row-level permission control. Data lineage optimization methods include ETL task scheduling priority adjustment methods.

[0058] In some embodiments of the present application, the specific expression of the reward function can be as follows:

[0059] R=αQ+βC-γP

[0060] In the above expression, if Figure 3 As shown in the figure, α, β, and γ are weight coefficients, which can be determined through historical data training. R is the comprehensive reward value, a quantitative indicator used to evaluate the quality of the strategy. Q is the quality improvement rate, which reflects the degree of improvement in data quality, such as the reduction in the null value rate. The compliance score reflects the strategy's satisfaction with regulatory provisions. The performance loss reflects the system delay or resource usage caused by the strategy execution.

[0061] In some embodiments of the present application, the state space, reward function, and action space are core components of building a reinforcement learning-based data access strategy automatic generation system, and each plays an indispensable role:

[0062] The state space comprehensively captures the current state of data governance within the Data Fabric environment. It includes metrics such as field null rate, access frequency, sensitivity labels, and data consistency, as well as dynamic information from user behavior logs, business glossaries, and data lineage graphs. This state information forms the foundation for reinforcement learning model decisions, enabling the model to understand the current data governance context and make strategic choices based on it. Dynamic updates to the state space ensure real-time and adaptable model decisions, enabling rapid response to changes in the data environment.

[0063] The reward function guides policy generation. It comprehensively evaluates the effectiveness of data access policies, including improvements in data quality (Q), enhanced compliance (C), and performance loss (P). By adjusting the three weighting coefficients α, β, and γ, the reward function balances the relationships between different governance objectives, prompting the model to prioritize both data quality and compliance while also considering system performance when generating policies. The reward function's setting directly influences the model's learning objectives and behavioral patterns, and is a key mechanism by which reinforcement learning algorithms optimize policies.

[0064] The action space (policy space) provides all possible data access control strategies. This covers field-level desensitization, row-level access control, data quality rule setting, data lineage optimization, and more. Whenever the model is in a certain state, it selects one or more strategies from the action space to execute. By rehearsing the effects of strategies in the digital twin verification sandbox, the model can observe immediate feedback and state changes after strategy execution, thereby guiding the reinforcement learning algorithm to optimize the strategy. The flexibility and diversity of the action space ensures that the model can explore and generate the data access strategy that best suits the current environment.

[0065] These three interact to form a closed-loop learning system. The state space describes the environment, the reward function guides the direction of action, and the action space provides possible action options. The reinforcement learning model selects an action based on the current state, observes the reward feedback after execution, and gradually optimizes the strategy through iterative learning to achieve the goal of maximizing the long-term reward. In the context of this application, this process enables the automated and intelligent generation of data access control policies within the Data Fabric architecture, while ensuring the security and effectiveness of the policies.

[0066] In some embodiments of this application, the state space provides a real-time description of the data environment, including data quality, access patterns, and compliance status. By analyzing this space, data assets requiring governance and potential issues can be identified. The policy space provides an action set for policy generation. Based on the state space information, it automatically selects or generates the governance policy most appropriate for the current environment, such as automatically adding field desensitization rules to address frequent access to sensitive fields.

[0067] The reward function serves as a quantitative indicator for policy optimization, guiding policy generation and adjustment. The weight coefficients α, β, and γ, determined through historical data training, ensure that policy selection balances the needs of improving quality, enhancing compliance, and maintaining performance.

[0068] At each decision moment, the current data governance state is retrieved from the state space, and an action strategy is selected or generated in the action strategy space based on this state. After execution, a reward value is calculated based on changes in data quality, compliance scores, and performance indicators. By learning from historical rewards, the algorithm automatically adjusts the probability of selecting a strategy in the strategy space, continuously optimizing strategies to maximize long-term cumulative rewards and improve data governance effectiveness.

[0069] Through this approach, the state space ensures the accuracy of the underlying information for policy generation, the action policy space provides flexible operations for policy generation, and the reward function quantifies the effectiveness of the policy and guides adaptive optimization. This mechanism, without manual intervention, improves the efficiency and accuracy of data access policies through automatic algorithmic iteration, supporting intelligent governance within the Data Fabric architecture.

[0070] As an optional implementation, a data access strategy may be generated using a Markov decision chain approach based on the above-mentioned state space, action space, and reward function.

[0071] In some embodiments of this application, the state space can also be defined to include metrics such as field null rate, access frequency, sensitivity labels, and data consistency. Graph neural networks can be used to analyze data lineage, user behavior logs, and business terminology relationships to achieve real-time updates. The action space includes governance policies such as field desensitization, access rights adjustment, and data quality rule development. Reward functions are designed to evaluate the effectiveness of strategies based on quality improvement, compliance, and performance loss.

[0072] Then, within the digital twin environment, metadata and sampled data are cloned, historical query logs are injected, and policy pre-execution is simulated. Data quality improvements, performance changes, and compliance risk indicators are collected to generate immediate feedback. Leveraging a reinforcement learning model, policy selection probabilities are dynamically adjusted based on these feedback signals, with the goal of maximizing long-term rewards.

[0073] Incremental training can then be used to automatically identify synonymous fields (such as "cust_id" and "client_no"), avoiding full graph reconstruction and optimizing semantic association accuracy. Rule generation and automated decision-making monitor frequent access to sensitive fields and automatically trigger desensitization instructions. New policies can be simulated and validated in a sandbox to assess their impact on data quality, system performance, and compliance, detect policy conflicts, and ensure the security and feasibility of governance solutions.

[0074] After multiple iterations, the model learns to select the optimal strategy for the action space within a given state space. This strategy is then optimized through a digital twin feedback loop, enabling the adaptive generation of data access control policies. This process requires no human intervention, and algorithmic iteration enhances the intelligence of the policy, meeting the requirements of automated governance within the Data Fabric architecture.

[0075] It can be seen that by combining reinforcement learning with graph neural networks, dynamically capturing data changes, automatically generating adaptive strategies, and using a digital twin verification sandbox to ensure the safety and feasibility of strategies before implementation, the entire mechanism forms a closed loop and promotes continuous optimization of strategies.

[0076] In some embodiments of the present application, the step of determining the access strategy of the target knowledge graph based on the node status in the target knowledge graph includes: in a case where the node status includes the node access frequency, determining the target node whose access frequency is higher than a preset frequency threshold; in a case where the target node is marked as a sensitive node, determining the similar nodes of the target node based on the clustering result, wherein the similar nodes of the target node are nodes that are clustered into the same cluster as the target node; generating a data desensitizing access strategy for the target node and the similar nodes of the target node, wherein the data desensitizing strategy includes the type of access object that needs to be desensitized.

[0077] As an optional implementation, dynamic desensitization rules can be automatically added when frequent accesses to fields containing PII are detected, using PII tags in the knowledge graph and real-time behavioral analysis. For example, if the user_identity field is accessed 120 times within 5 minutes and its attribute is marked as PII=TRUE in the knowledge graph, the system will automatically generate a dynamic desensitization instruction.

[0078] In some embodiments of the present application, after determining the access strategy of the target knowledge graph based on the node status in the target knowledge graph, in order to further verify and optimize the access strategy, the method also includes: obtaining user behavior simulation data, wherein the user behavior simulation data includes at least one of the following: historical query logs, custom test cases; copying the target knowledge graph in a virtual operating environment; determining the node status of the copied target knowledge graph after executing the access strategy in the virtual environment based on the user behavior simulation data and the copied target knowledge graph, and determining a first evaluation result of the access strategy based on the node status of the target knowledge graph; performing compliance detection on the access strategy to obtain a second evaluation result; performing conflict detection on the access strategy to obtain a third evaluation result; determining a comprehensive verification result of the access strategy based on the first evaluation result, the second evaluation result and the third evaluation result, and adjusting the access strategy based on the comprehensive verification result.

[0079] As an optional implementation, you can clone the production environment's metadata and 10% sampled data in a virtual runtime environment and inject historical query logs or custom test cases to simulate user behavior. You can then use the following verification process to determine the comprehensive verification results of the access policy and provide feedback for optimization:

[0080] The first step is policy pre-execution: running the newly generated governance rules in the sandbox;

[0081] The second step is impact assessment, where specific assessment indicators may include data quality, performance indicators, and compliance risks. Data quality includes null value rate and consistency error; performance indicators include query latency and resource utilization; and compliance risks include the probability of sensitive field exposure.

[0082] The third step is conflict detection: check the mutual exclusivity between new rules and existing policies (such as conflicts between desensitization rules and data requirements of risk control models).

[0083] Step 4: Feedback optimization: return the verification results to the reinforcement learning model and iteratively optimize the strategy.

[0084] In some embodiments of the present application, when new data is received, a subgraph structure corresponding to the new data is determined based on the new data and the target knowledge graph, wherein the subgraph structure includes subgraph nodes and subgraph edges, and the subgraph nodes correspond to the new data, as well as data corresponding to the target knowledge graph and having an association relationship with the new data; the embedding vectors of the subgraph nodes in the subgraph structure are determined, and the subgraph nodes are clustered based on the similarity between the embedding vectors of the subgraph nodes; the subgraph structure is updated based on the clustering results obtained by clustering the subgraph nodes, and the target knowledge graph is updated based on the updated subgraph structure.

[0085] Optionally, when a new data source (such as a newly connected CRM system) or a user behavior log is updated, the system will extract the subgraph corresponding to the newly added data (such as a new data table and its associated fields). The GraphSAGE algorithm can then be used to calculate node embeddings only for the newly added subgraph, rather than reconstructing the entire graph. For example, the fields of the newly added table Table_C will be automatically associated with similar fields in the existing knowledge graph (such as "client_no"). The updated node embeddings and edge relationships are then synchronized to the Neo4j graph database to ensure the dynamic nature of the knowledge graph. For example, after the newly added field "user_identity" is clustered as PII, its associated access frequency data will affect the decision-making of the policy generation module in real time.

[0086] As an optional implementation method, data lineage tracking or behavior log analysis can be used to determine the data associated with the newly added data, thereby determining the data range covered by the subgraph structure corresponding to the newly added data, that is, determining which nodes should be included in the subgraph structure. Optionally, when a new data source is connected, the system determines its directly associated upstream and downstream nodes through the ETL task log and data lineage relationship (such as Table_A→ETL→Table_B). For example, if a new table Table_C is associated with Table_B through an ETL task, Table_C and its directly connected nodes (such as fields, users) constitute a local subgraph. In addition, user query logs and API call records will identify frequently accessed nodes (such as frequently queried fields), and these nodes and their neighboring nodes (such as associated business terms) will be included in the local subgraph.

[0087] After determining the scope of a local subgraph, incremental training of the GNN can be performed on that subgraph, for example, updating the embedding vectors of Table_C and its adjacent nodes, rather than reconstructing the entire graph. This process is enabled by the real-time query capabilities of the graph database, ensuring efficiency and accuracy.

[0088] By determining an initial graph structure corresponding to an initial knowledge graph, wherein the graph structure includes nodes and edges, nodes correspond to data, edges correspond to associations between different data, and data include structured data and unstructured data; determining the embedding vector of each node, and clustering the nodes according to the similarity between the embedding vectors; updating the edge relationships in the initial knowledge graph according to the clustering results to obtain a target knowledge graph; and determining the access strategy of the target knowledge graph according to the node status in the target knowledge graph, by determining the graph structure corresponding to the initial knowledge graph, determining the embedding vector of each node in the graph structure, clustering according to the embedding vectors and updating the edges according to the clustering results, the purpose of mining possible associations between different data is achieved, thereby realizing the technical effect of comprehensive and efficient management of data, and further solving the technical problem that the possible associations between some data are omitted in related technologies and that the data cannot be managed comprehensively and efficiently.

[0089] According to the embodiment of the present application, there is also provided a Figure 4 The data management system shown is used to perform Figure 2The data management method shown in FIG. The self-evolving knowledge graph module 40 is used to generate a dynamically updateable knowledge graph based on metadata. The reinforcement learning strategy generation module 42 is used to generate a data access strategy based on the status (such as quality score, access frequency, etc.) of the knowledge graph generated by the self-evolving knowledge graph module. The digital twin verification module 44 is used to clone metadata from the self-evolving knowledge graph module 40 and verify the data access strategy sent by the reinforcement learning strategy generation module. The verification results are then fed back to the reinforcement learning strategy generation module 42.

[0090] The embodiment of the present application provides a data management device, Figure 5 It is a structural diagram of the device. Figure 5 It can be seen that the device includes: a first processing module 50, which is used to determine the initial graph structure corresponding to the initial knowledge graph, wherein the graph structure includes nodes and edges, the nodes correspond to data, the edges correspond to the association relationship between different data, and the data include structured data and unstructured data; a second processing module 52, which is used to determine the embedding vector of each node, and cluster the nodes according to the similarity between the embedding vectors; a third processing module 54, which is used to update the edge relationship in the initial knowledge graph according to the clustering results to obtain the target knowledge graph; a fourth processing module 56, which is used to determine the access strategy of the target knowledge graph according to the node status in the target knowledge graph.

[0091] In some embodiments of the present application, the step of the third processing module 56 updating the edge relationship in the initial knowledge graph based on the clustering results includes: determining the knowledge graph nodes in the knowledge graph belonging to the same cluster cluster based on the clustering results; establishing a target edge of a semantically similar type between two knowledge graph nodes belonging to the same cluster cluster, wherein the target edge is used to indicate that there is semantic similarity between the two connected knowledge graph nodes.

[0092] In some embodiments of the present application, the step of the fourth processing module 58 determining the access strategy of the target knowledge graph based on the node status in the target knowledge graph includes: determining the state space and the action space, wherein the state space includes the node status of each node in the target knowledge graph, and the action space includes the data processing strategy executed on the target indication graph; determining the reward function, wherein the reward function is used to compare the advantages and disadvantages between different node state sets, and the higher the reward function value, the better the node state set; determining the access strategy of the target knowledge graph based on the state space, the action space and the reward function.

[0093] In some embodiments of the present application, the node status includes a node data quality score, a node access frequency, and a node label type.

[0094] In some embodiments of the present application, the step in which the fourth processing module 58 determines the access strategy of the target knowledge graph based on the node status in the target knowledge graph includes: in a case where the node status includes the node access frequency, determining the target node whose access frequency is higher than the preset frequency threshold; in a case where the target node is marked as a sensitive node, determining the similar nodes of the target node based on the clustering result, wherein the similar nodes of the target node are nodes that are clustered into the same cluster as the target node; generating a data desensitizing access strategy for the target node and the similar nodes of the target node, wherein the data desensitizing strategy includes the type of access object that needs to be desensitized.

[0095] In some embodiments of the present application, after determining the access strategy of the target knowledge graph based on the node status in the target knowledge graph, the fourth processing module 58 is also used to: obtain user behavior simulation data, wherein the user behavior simulation data includes at least one of the following: historical query logs, custom test cases; copy the target knowledge graph in a virtual operating environment; determine the node status of the copied target knowledge graph after executing the access strategy in the virtual environment based on the user behavior simulation data and the copied target knowledge graph, and determine a first evaluation result of the access strategy based on the node status of the target knowledge graph; perform compliance detection on the access strategy to obtain a second evaluation result; perform conflict detection on the access strategy to obtain a third evaluation result; determine a comprehensive verification result of the access strategy based on the first evaluation result, the second evaluation result and the third evaluation result, and adjust the access strategy based on the comprehensive verification result.

[0096] In some embodiments of the present application, the data management device is also used to determine, when receiving new data, a subgraph structure corresponding to the newly added data based on the new data and the target knowledge graph, wherein the subgraph structure includes subgraph nodes and subgraph edges, the subgraph nodes corresponding to the newly added data, and data that has an association relationship between the data corresponding to the target knowledge graph and the newly added data; determine the embedding vectors of the subgraph nodes in the subgraph structure, and cluster the subgraph nodes based on the similarity between the embedding vectors of the subgraph nodes; update the subgraph structure based on the clustering results obtained by clustering the subgraph nodes, and update the target knowledge graph based on the updated subgraph structure.

[0097] It should be noted that the various modules in the above-mentioned data management device can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0098] According to an embodiment of the present application, a non-volatile storage medium is further provided. The non-volatile storage medium stores a program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the following data management method: determining an initial graph structure corresponding to an initial knowledge graph, wherein the graph structure includes nodes and edges, the nodes correspond to data, the edges correspond to associations between different data, and the data includes structured data and unstructured data; determining an embedding vector for each node, and clustering the nodes based on the similarity between the embedding vectors; updating the edge relationships in the initial knowledge graph based on the clustering results to obtain a target knowledge graph; and determining an access strategy for the target knowledge graph based on the node states in the target knowledge graph.

[0099] According to an embodiment of the present application, an electronic device is also provided, including: a memory and a processor, the processor being used to run a program stored in the memory, wherein the following data management method is executed when the program is running: determining an initial graph structure corresponding to an initial knowledge graph, wherein the graph structure includes nodes and edges, the nodes correspond to data, the edges correspond to associations between different data, and the data include structured data and unstructured data; determining the embedding vectors of each node, and clustering the nodes based on the similarity between the embedding vectors; updating the edge relationships in the initial knowledge graph based on the clustering results to obtain a target knowledge graph; determining the access strategy of the target knowledge graph based on the node status in the target knowledge graph.

[0100] According to an embodiment of the present application, a computer program product is also provided, including a computer program, which implements the following data management method when executed by a processor: determining an initial graph structure corresponding to an initial knowledge graph, wherein the graph structure includes nodes and edges, the nodes correspond to data, the edges correspond to the association relationship between different data, and the data includes structured data and unstructured data; determining the embedding vector of each node, and clustering the nodes based on the similarity between the embedding vectors; updating the edge relationship in the initial knowledge graph based on the clustering results to obtain a target knowledge graph; determining the access strategy of the target knowledge graph based on the node status in the target knowledge graph.

[0101] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0102] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0103] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0104] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0105] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0106] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data management method, characterized in that: include: Determine an initial graph structure corresponding to the initial knowledge graph, wherein the graph structure includes nodes and edges, the nodes correspond to data, the edges correspond to associations between different data, and the data include structured data and unstructured data; Determining an embedding vector of each of the nodes, and clustering the nodes based on similarities between the embedding vectors; Update the edge relationships in the initial knowledge graph according to the clustering results to obtain a target knowledge graph; An access strategy for the target knowledge graph is determined based on the node status in the target knowledge graph.

2. The data management method according to claim 1, wherein: Updating the edge relationships in the initial knowledge graph according to the clustering results includes: Determining knowledge graph nodes belonging to the same cluster in the knowledge graph according to the clustering result; A target edge of semantic similarity type is established between two knowledge graph nodes belonging to the same local cluster, wherein the target edge is used to indicate that there is semantic similarity between the two connected knowledge graph nodes.

3. The data management method according to claim 1, wherein: The method further comprises: Upon receiving newly added data, determining a subgraph structure corresponding to the newly added data based on the newly added data and the target knowledge graph, wherein the subgraph structure includes subgraph nodes and subgraph edges, the subgraph nodes corresponding to the newly added data, and data in the data corresponding to the target knowledge graph that has an association relationship with the newly added data; Determining embedding vectors of subgraph nodes in the subgraph structure, and clustering the subgraph nodes based on similarities between the embedding vectors of the subgraph nodes; The subgraph structure is updated according to the clustering result obtained by clustering the subgraph nodes, and the target knowledge graph is updated according to the updated subgraph structure.

4. The data management method according to claim 1, wherein: Determining the access strategy of the target knowledge graph based on the node status in the target knowledge graph includes: Determine a state space and an action space, wherein the state space includes the node state of each node in the target knowledge graph, and the action space includes the data processing strategy executed on the target instruction graph; Determining a reward function, wherein the reward function is used to compare the pros and cons of different node state sets, and a higher reward function value indicates a better node state set; An access strategy for the target knowledge graph is determined based on the state space, the action space, and the reward function.

5. The data management method according to claim 4, characterized in that: The node status includes node data quality score, node access frequency and node label type.

6. The data management method according to claim 1, wherein: Determining the access strategy of the target knowledge graph based on the node status in the target knowledge graph includes: In a case where the node status includes a node access frequency, determining a target node whose access frequency is higher than a preset frequency threshold; In the case where the target node is marked as a sensitive node, determining similar nodes of the target node according to the clustering result, wherein the similar nodes of the target node are nodes that are clustered into the same cluster as the target node; A data desensitization access policy is generated for the target node and similar nodes of the target node, wherein the data desensitization policy includes the access object type that needs to be desensitized.

7. The data management method according to claim 1, wherein: After determining the access strategy of the target knowledge graph according to the node status in the target knowledge graph, the method further includes: Acquire user behavior simulation data, wherein the user behavior simulation data includes at least one of the following: historical query logs, custom test cases; Copy the target knowledge graph in a virtual operating environment; Determining, in the virtual environment, a node state of the target knowledge graph copied after executing the access strategy based on the user behavior simulation data and the copied target knowledge graph, and determining a first evaluation result of the access strategy based on the node state of the target knowledge graph; Performing compliance testing on the access policy to obtain a second evaluation result; Performing conflict detection on the access policy to obtain a third evaluation result; A comprehensive verification result of the access policy is determined based on the first evaluation result, the second evaluation result, and the third evaluation result, and the access policy is adjusted based on the comprehensive verification result.

8. A data management device, characterized in that: include: A first processing module is configured to determine an initial graph structure corresponding to the initial knowledge graph, wherein the graph structure includes nodes and edges, the nodes correspond to data, the edges correspond to associations between different data, and the data include structured data and unstructured data; a second processing module, configured to determine an embedding vector of each of the nodes and cluster the nodes based on similarities between the embedding vectors; A third processing module is used to update the edge relationship in the initial knowledge graph according to the clustering result to obtain a target knowledge graph; The fourth processing module is used to determine the access strategy of the target knowledge graph based on the node status in the target knowledge graph.

9. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the data management method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is configured to run a program stored in the memory, wherein the program executes the data management method according to any one of claims 1 to 7 when running.

11. A computer program product, characterized in that The computer program comprises a computer program which, when executed by a processor, implements the data management method according to any one of claims 1 to 7.

Images