Data asset importance level marking method, device, medium and system

Abstract

The invention provides a data asset importance level marking method, device, medium and system, and the method comprises the steps: extracting data processing link information from a metadata management system, and constructing a data blood relationship map according to the data processing link information; solving an approximate minimum point coverage set for the data consanguinity map; and according to the approximate minimum point coverage set, carrying out importance level marking on the data assets, and displaying a marking result in a visual mode. Therefore, the problem that the protection range is too large or key nodes are omitted due to the fact that the minimum core asset set covering global dependence cannot be found within reasonable time in the prior art is solved.

Description

Technical Field

[0001] This application relates to the field of big data technology, and more specifically, to a method for marking the importance level of data assets, a device for marking the importance level of data assets, a computer-readable storage medium, and a system for marking the importance level of data assets. Background Technology

[0002] Data lineage graphs contain massive amounts of data, potentially reaching hundreds of millions of entities, and are of great importance for the maintenance, management, and protection of data assets. Assessing the importance of data assets in such large-scale graphs requires extremely high human resources.

[0003] Option 1 (Full Graph Protection): Provide the same level of security protection for all nodes in the data lineage graph. For example, deploy encryption and access control policies for all database tables.

[0004] Option 2 (Manual Rule Screening): Protect a subset of key assets manually based on predefined rules (such as data sensitivity tags and access frequency), ignoring global dependencies.

[0005] The drawbacks of existing technologies are: full-scale protection requires security resources for all data assets, resulting in redundant overhead (such as storage encryption and access control costs). Manual rule selection relies on subjective experience, cannot dynamically adapt to changes in lineage, and struggles to cover hidden but core derivative assets (such as intermediate tables). Existing methods cannot find the minimum set of core assets covering global dependencies within a reasonable timeframe, leading to excessively broad protection scope or omission of critical nodes. Summary of the Invention

[0006] The main objective of this application is to provide a method, device, computer-readable storage medium, and system for marking the importance level of data assets, so as to at least solve the problem that the prior art cannot find the minimum set of core assets that covers global dependencies within a reasonable time, resulting in an excessively large protection scope or omission of key nodes.

[0007] To achieve the above objectives, according to one aspect of this application, a method for marking the importance level of data assets is provided, comprising: extracting data processing link information from a metadata management system, and constructing a data lineage graph based on the data processing link information, wherein the data lineage graph is in the form of a directed graph, the nodes of the data lineage graph represent data assets, and the edges of the data lineage graph represent data flow directions; solving for an approximate minimum vertex cover set on the data lineage graph; marking the data assets with importance levels based on the approximate minimum vertex cover set, and displaying the marking results in a visual manner.

[0008] Optionally, solving for the approximate minimum vertex cover set of the data lineage graph includes: using an iterative algorithm to solve for the approximate minimum vertex cover set. The iterative algorithm initially has an empty protection set. In each subsequent iteration, the node with the highest current degree is selected and added to the protection set, and all edges associated with it are deleted from the data lineage graph until there are no more edges in the data lineage graph.

[0009] Optionally, in the process of solving the approximate minimum vertex cover set using an iterative algorithm, the method further includes: a selection step: each time, selecting the node currently associated with the most uncovered edges as the target node; a removal step: adding the target node to the initial protection set and removing all edges associated with the target node from the edge set; repeating the selection step and the removal step until the edge set is empty.

[0010] Optionally, in the process of extracting data processing link information from the metadata management system and constructing a data lineage graph based on the data processing link information, the method further includes: using a machine learning model to extract features of each link in the data lineage graph, wherein the features of the link include the frequency of data flow, the size of data, the sensitivity of data, and the business correlation between upstream and downstream data assets; using the machine learning model to calculate the weight of each link, and determining the score of each link based on the weight of each link; and filtering irrelevant or redundant links based on the scores of each link.

[0011] Optionally, in the process of solving the approximate minimum point cover set for the data lineage graph, the method further includes: constructing a target model, wherein the target model is a model that transforms the minimum point cover problem of the data lineage graph into a linear programming problem, wherein each node corresponds to a decision variable, the decision variable represents whether the node is selected into the cover set, the objective function is to minimize the total number of selected nodes, and the constraints ensure that each edge in the graph has at least one endpoint selected; and using a linear programming solver to calculate the optimal solution of the target model to obtain the decision variable value for each node.

[0012] Optionally, after marking the data assets with importance levels, the method further includes: using a sliding window to count the historical access frequency of the data assets to determine abnormal access patterns; and adjusting the importance level of the data assets according to the abnormal access patterns.

[0013] Optionally, adjusting the importance level of the data asset based on the abnormal access pattern includes: collecting all access records of the data asset and using statistical or machine learning algorithms to identify access patterns that deviate from normal behavior as abnormal access patterns; processing the abnormal access patterns using a neural network model to obtain a target importance level, and adjusting the importance level of the data asset to the target importance level.

[0014] According to another aspect of this application, a data asset importance level marking device is provided, comprising: a construction unit, configured to extract data processing link information from a metadata management system and construct a data lineage graph based on the data processing link information, wherein the data lineage graph is in the form of a directed graph, the nodes of the data lineage graph represent data assets, and the edges of the data lineage graph represent data flow directions; a first processing unit, configured to solve for an approximate minimum vertex cover set on the data lineage graph; and a second processing unit, configured to mark the data assets with importance levels based on the approximate minimum vertex cover set and display the marking results in a visual manner.

[0015] According to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform any of the methods described.

[0016] According to another aspect of this application, a data asset importance level marking system is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including methods for performing any one of the methods described.

[0017] Applying the technical solution of this application, firstly, data processing link information is extracted from the metadata management system. This step ensures that the lineage graph covers the flow paths and dependencies of all relevant data assets. Compared to screening methods that rely solely on manual rules or labels, this method can comprehensively capture the correlations between data assets, including those indirect dependencies or derived assets that may be overlooked. Representing this information in the form of a directed graph not only intuitively shows the flow of data but also facilitates subsequent algorithmic processing and analysis. Next, an approximate minimum vertex cover algorithm is used to find the set of core nodes required to cover all edges in the graph. This algorithm is based on a greedy strategy, selecting the node with the highest degree in the current graph in each iteration and adding it to the protection set, then removing all edges related to that node. Although this method does not guarantee absolute theoretical optimality, it can efficiently obtain a set close to the optimal solution in polynomial time, and its size is usually within about twice that of the theoretical optimal solution. This means that compared to full graph protection, this method greatly reduces the number of data assets that need to be protected, and reduces unnecessary costs such as storage encryption and access control. Finally, based on the solved approximate minimum point coverage set, the data assets in the graph are labeled with importance levels. Nodes within this set are considered "high importance" because they constitute the core of the solution, covering all data dependencies. Neighboring nodes directly connected to these high-importance nodes are labeled "medium importance," reflecting first-level derived dependencies. The remaining nodes are labeled "low importance." This hierarchical mechanism ensures that critical nodes are not overlooked, while also providing appropriate protection for non-core but still important data assets. It avoids the over- or under-protection issues that may arise under a single standard, thus solving the problem that existing technologies cannot find the minimum core asset set covering global dependencies within a reasonable timeframe, leading to excessive protection or the omission of critical nodes. Attached Figure Description

[0018] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 A flowchart illustrating a method for marking the importance level of data assets according to an embodiment of this application is shown.

[0020] Figure 2 A structural block diagram of a data asset importance level marking device provided according to an embodiment of this application is shown. Detailed Implementation

[0021] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0022] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] As described in the background section, existing technologies have the following drawbacks: full-map protection requires investment of security resources in all data assets, resulting in redundant overhead (such as storage encryption and access control costs). Manual rule selection relies on subjective experience, cannot dynamically adapt to changes in lineage, and is difficult to cover hidden but core derivative assets (such as intermediate tables). Existing methods cannot find the minimum set of core assets covering global dependencies within a reasonable time, leading to an overly broad protection scope or omission of key nodes. To address the problem that existing technologies cannot find the minimum set of core assets covering global dependencies within a reasonable time, resulting in an overly broad protection scope or omission of key nodes, embodiments of this application provide a method for marking the importance level of data assets, a device for marking the importance level of data assets, a computer-readable storage medium, and a system for marking the importance level of data assets.

[0025] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0026] This embodiment provides a method for marking the importance level of data assets. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0027] It should be noted that the information (including but not limited to) user device information and user personal information, and the data (including but not limited to data used for display and analysis) collected in this application are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of this data all comply with relevant laws, regulations, and standards, and necessary confidentiality measures have been taken. This process does not violate public order and good morals, and corresponding access points are provided for users to choose to authorize or refuse. For example, interfaces are set up between this system and relevant users or organizations, providing users with corresponding access points to choose to agree to or refuse automated decision-making results; if the user chooses to refuse, the process proceeds to the expert decision-making stage. If the user chooses to agree, the user can view the purpose of data use in real time through the authorization interface and has the right to withdraw authorization or delete data at any time. After authorization is withdrawn, the system will terminate the relevant data processing within 24 hours.

[0028] Figure 1 This is a flowchart of a method for marking the importance level of data assets according to an embodiment of this application. For example... Figure 1 As shown, the method includes the following steps:

[0029] Step S101: Extract data processing link information from the metadata management system, and construct a data lineage graph based on the data processing link information. The data lineage graph is in the form of a directed graph, and the nodes of the data lineage graph represent data assets, while the edges of the data lineage graph represent data flow directions.

[0030] A metadata management system is a software platform for organizing and managing data metadata. Metadata refers to data about data; it can describe information such as the data's source, format, storage location, business meaning, technical attributes, and relationships between data.

[0031] Data Processing Lineage Information describes a series of operations and transformations that data undergoes in an information system from the source system to the target system.

[0032] Step S102: Solve for the approximate minimum point cover set of the above pedigree data;

[0033] Step S103: Based on the above approximate minimum point coverage set, the data assets are marked with importance levels, and the marking results are displayed in a visual manner.

[0034] In the above steps, firstly, data processing link information is extracted from the metadata management system. This step ensures that the lineage graph covers the flow paths and dependencies of all relevant data assets. Compared to screening methods that rely solely on manual rules or labels, this method can comprehensively capture the correlations between data assets, including those indirect dependencies or derived assets that may be overlooked. Representing this information in the form of a directed graph not only intuitively shows the data flow but also facilitates subsequent algorithmic processing and analysis. Next, an approximate minimum vertex cover algorithm is used to find the core node set required to cover all edges in the graph. This algorithm is based on a greedy strategy, selecting the node with the highest degree in the current graph in each iteration and adding it to the protection set, then removing all edges related to that node. Although this method does not guarantee absolute theoretical optimality, it can efficiently obtain a set close to the optimal solution in polynomial time, and its size is usually within about twice that of the theoretical optimal solution. This means that compared to full graph protection, this method greatly reduces the number of data assets that need to be protected, and reduces unnecessary costs such as storage encryption and access control. Finally, based on the solved approximate minimum point coverage set, the data assets in the graph are labeled with importance levels. Nodes within this set are considered "high importance" because they constitute the core of the solution, covering all data dependencies. Neighboring nodes directly connected to these high-importance nodes are labeled "medium importance," reflecting first-level derived dependencies. The remaining nodes are labeled "low importance." This hierarchical mechanism ensures that critical nodes are not overlooked, while also providing appropriate protection for non-core but still important data assets. It avoids the over- or under-protection issues that may arise under a single standard, thus solving the problem that existing technologies cannot find the minimum core asset set covering global dependencies within a reasonable timeframe, leading to excessive protection or the omission of critical nodes.

[0035] Provide users with corresponding operation entry points, allowing them to choose to agree to or reject the automated decision results; if the user chooses to reject, the process will proceed to the expert decision-making process.

[0036] In one embodiment of this application, solving for the approximate minimum vertex cover set of the above-mentioned data lineage graph includes: using an iterative algorithm to solve for the approximate minimum vertex cover set. Initially, the protection set of the iterative algorithm is empty. In each subsequent iteration, the node with the largest current degree is selected and added to the protection set, and all edges associated with it are deleted from the above-mentioned data lineage graph until there are no more edges in the above-mentioned data lineage graph.

[0037] Iterative algorithms typically have low time complexity (often exponential), which can hinder computational bottlenecks. In each iteration, iterative algorithms independently select the node with the highest degree to add to the protection set, allowing them to easily adapt to real-time changes in the data lineage graph. When nodes or edges are added or deleted from the graph, the algorithm only needs to be rerun for the local region, rather than processing the entire graph from scratch, improving the algorithm's flexibility and responsiveness. Each step of the iterative algorithm can be executed independently, enabling parallel computing. In a distributed computing environment, the graph can be divided into multiple subgraphs, each executing the iterative algorithm simultaneously, and the results can be merged, further improving the efficiency of processing large-scale datasets.

[0038] In one embodiment of this application, during the process of solving the approximate minimum vertex cover set using an iterative algorithm, the above method further includes: a selection step: each time, selecting the node currently associated with the most uncovered edges as the target node; a removal step: adding the target node to the initial protection set and removing all edges associated with the target node from the edge set; repeating the selection step and the removal step until the edge set is empty.

[0039] Alternatively, a greedy algorithm can be used to find the approximate minimum point cover set for the pedigree graph of the data.

[0040] A greedy algorithm is an algorithm that makes the best or optimal choice at each step, aiming to find the globally optimal solution to the problem. In solving the approximate minimum vertex cover problem, a typical greedy strategy is to add the node with the highest degree among the remaining parts of the graph to the cover set in each iteration, and then remove all edges connected to that node. The key to this method is seeking a local optimum at each stage, although this does not guarantee a globally optimal solution. For the approximate minimum vertex cover problem, the advantage of greedy algorithms is their fast solution time and relatively low space complexity, typically completed in linear time, where |E| is the number of edges. However, a typical limitation is that it may not reach the theoretically optimal solution, although the quality of the solution is usually acceptable, and it is very practical in many real-world scenarios.

[0041] Iterative algorithms are a broader concept, referring to any algorithm that approximates the final solution through a series of steps. They can be non-greedy, meaning they don't necessarily pursue a local optimum at each step, but may explore more possibilities or gradually adjust the form of the solution according to a certain strategy. When solving for the approximate minimum point cover set, in addition to the greedy algorithms mentioned above, iterative algorithms can also include a variety of strategies:

[0042] Improve the greedy algorithm: Add extra logic to the basic greedy algorithm, such as considering some additional conditions when selecting nodes, or making local optimization adjustments after selection.

[0043] Randomized algorithms introduce randomness into the iterative process, seeking a better solution through multiple attempts. This may yield results different from those of greedy algorithms, and the quality of the results is affected by randomness.

[0044] Local search algorithm: The initial solution may be obtained through a greedy strategy, and then the solution is improved through a series of iterations, changing only a small part of the solution structure each time, in an attempt to find a better solution.

[0045] Distributed iterative algorithms: Considering that the scale of data lineage graphs may be very large, iterative algorithms can be executed in parallel in a distributed system.

[0046] Each iteration selects the node associated with the most uncovered edges, meaning that this node can simultaneously cover as many new edges as possible, thus maximizing coverage efficiency in each iteration, reducing the number of iterations, and speeding up the algorithm. Although the greedy algorithm may not always arrive at the theoretically optimal solution, by selecting the node with the most covered edges in each iteration, the algorithm's solution is guaranteed to be very close to the optimal solution for the minimum vertex cover in polynomial time. This is especially true when the graph structure resembles certain types, such as sparse graphs, where the greedy algorithm's solution may be very close to the optimal solution. During the iteration process, the algorithm's decision criterion is simple and intuitive: selecting the node currently associated with the most uncovered edges to add to the coverage set. This avoids complex optimization problems, making the algorithm easy to implement and understand. The efficiency and approximation of this method are particularly crucial when dealing with large-scale lineage graphs. Due to the complexity of lineage graphs, directly solving for the minimum vertex cover often involves enormous computational costs, while iteratively selecting the node with the most covered edges allows for the processing of large-scale graphs in a short time, achieving scalability of the algorithm.

[0047] In one embodiment of this application, during the process of extracting data processing link information from the metadata management system and constructing a data lineage graph based on the data processing link information, the method further includes: using a machine learning model to extract features of each link in the data lineage graph, wherein the features of the link include the frequency of data flow, the size of data volume, the sensitivity of data, and the business correlation between upstream and downstream data assets; using the machine learning model to calculate the weight of each link, and determining the score of each link based on the weight of each link; and filtering out irrelevant or redundant links based on the scores of each link.

[0048] By training machine learning models, historical data and pattern recognition capabilities can be used to distinguish which links are truly critical and which are irrelevant or redundant. This helps to construct data lineage maps more accurately, avoiding inaccurate assessments due to false positives or false negatives. Data processing links may contain a large amount of noisy data, such as temporary files, test data, and discarded processes. Machine learning models can identify and filter this noise, ensuring the map contains only valid and necessary information, thereby improving the efficiency and quality of subsequent analysis. Machine learning models can automate the processing and analysis of large amounts of data, reducing the need for manual intervention. This not only saves human resources but also avoids the subjectivity and errors introduced by human judgment. Data lineage maps are dynamic; models can continuously learn and adjust, automatically adapting to updates and changes in data processing flows, ensuring the map's real-time nature and effectiveness. After filtering out irrelevant or redundant links, more focus can be placed on the protection and management of critical data assets, rationally allocating limited security resources, and avoiding over-investment in non-critical links.

[0049] In one embodiment of this application, in the process of solving the approximate minimum point cover set of the above-mentioned data lineage graph, the method further includes: constructing a target model, wherein the target model is a model that transforms the minimum point cover problem of the above-mentioned data lineage graph into a linear programming problem, wherein each node corresponds to a decision variable, the decision variable represents whether the node is selected into the cover set, the objective function is to minimize the total number of selected nodes, and the constraint condition ensures that each edge in the graph has at least one endpoint selected; and using a linear programming solver to calculate the optimal solution of the above-mentioned target model to obtain the decision variable value of each node.

[0050] The linear programming relaxation method transforms the minimum point cover problem into a linear programming problem and then relaxes the integer constraints (i.e., converting integer variables into continuous variables), making it easier to find a solution close to the optimal one. This method can typically obtain a smaller or equal-sized set of point covers than the greedy algorithm, thus enabling more accurate identification of the importance and core nodes of data assets, reducing the scope of protection while maintaining complete coverage of dependencies.

[0051] In one embodiment of this application, after marking the data assets with importance levels, the method further includes: using a sliding window to count the historical access frequency of the data assets to determine abnormal access patterns; and adjusting the importance level of the data assets according to the abnormal access patterns.

[0052] The value and importance of data assets can change over time. For example, a field might be frequently accessed over a period of time because it's being used in new business processes or critical decisions. Sliding window statistics can capture these changes, allowing for timely adjustments to importance levels and ensuring security policies and resource allocation better align with actual needs. Abnormal access patterns are often associated with security incidents such as data breaches, internal misoperations, or external attack attempts. By detecting abnormal access frequency, potentially abused or attacked data assets can be quickly identified, enabling timely action such as strengthening access audits, adding access control policies, or implementing data encryption to mitigate risk. Focusing limited security resources and maintenance efforts on data assets with high access frequency or abnormal access patterns maximizes resource utilization efficiency. This avoids overprotecting data assets with low access frequency and no abnormalities, thus saving unnecessary costs and workload.

[0053] In one embodiment of this application, adjusting the importance level of the data asset according to the above-mentioned abnormal access pattern includes: collecting all access records of the data asset, and using statistical or machine learning algorithms to identify access patterns that deviate from normal behavior as the above-mentioned abnormal access patterns; processing the above-mentioned abnormal access patterns using a neural network model to obtain a target importance level, and adjusting the above-mentioned importance level of the data asset to the above-mentioned target importance level.

[0054] Machine learning and neural network models can automatically identify complex access patterns and distinguish between normal and abnormal behavior. Compared to traditional rule-based baseline methods, they can capture more multi-dimensional features, such as access time patterns, geographical location, and user behavior sequences, thus more accurately determining the authenticity of abnormal access. Statistical and machine learning algorithms can dynamically adjust the threshold for identifying anomalies based on historical data. This means the system can self-optimize as time and environment change, reducing false positives and false negatives and improving the accuracy of anomaly detection. Neural network models can convert abnormal access patterns into quantified importance levels. This step is crucial for data asset management because it not only reveals which data assets may be at risk but also provides an assessment of the severity of those risks, helping enterprises prioritize the most critical issues. Once an abnormal access pattern is identified, the system can automatically adjust the importance level of data assets according to pre-defined policies, triggering corresponding security measures, such as enhanced access control, initiating audit processes, or implementing data encryption, thereby responding promptly to potential security threats.

[0055] Randomization algorithm: Randomly select an edge and add its two endpoints to the set, repeating until the cover is complete.

[0056] Implementation of a distributed graph computing framework: Parallelizing greedy algorithms to handle ultra-large-scale graphs.

[0057] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0058] This application also provides a data asset importance level marking device. It should be noted that the data asset importance level marking device of this application embodiment can be used to execute the data asset importance level marking method provided in this application embodiment. This device is used to implement the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0059] The following describes the data asset importance level marking device provided in the embodiments of this application.

[0060] Figure 2 This is a structural block diagram of a data asset importance level marking device according to an embodiment of this application. For example... Figure 2 As shown, the device includes: a construction unit 21, used to extract data processing link information from the metadata management system and construct a data lineage graph based on the data processing link information, wherein the data lineage graph is in the form of a directed graph, the nodes of the data lineage graph represent data assets, and the edges of the data lineage graph represent data flow directions; a first processing unit 22, used to solve for an approximate minimum vertex cover set on the data lineage graph; and a second processing unit 23, used to mark the data assets with importance levels based on the approximate minimum vertex cover set and display the marking results in a visual manner.

[0061] First, data processing link information is extracted from the metadata management system. This step ensures that the lineage graph covers the flow paths and dependencies of all relevant data assets. Compared to screening methods that rely solely on manual rules or labels, this method can comprehensively capture the relationships between data assets, including those that might be overlooked as indirect dependencies or derived assets. Representing this information as a directed graph not only visually shows the data flow but also facilitates subsequent algorithmic processing and analysis. Next, an approximate minimum vertex cover algorithm is used to find the set of core nodes needed to cover all edges in the graph. This algorithm is based on a greedy strategy, selecting the node with the highest degree in the current graph in each iteration and adding it to the protection set, then removing all edges related to that node. While this method does not guarantee absolute theoretical optimality, it can efficiently obtain a set close to the optimal solution in polynomial time, typically within about twice the size of the theoretical optimal solution. This means that compared to full graph protection, this method significantly reduces the number of data assets requiring key protection, lowering unnecessary costs associated with storage encryption and access control. Finally, based on the solved approximate minimum vertex cover set, the data assets in the graph are marked with importance levels. Nodes within the set are considered "high importance" because they form the core of the solution, covering all data dependencies. Neighboring nodes directly connected to these high-importance nodes are marked as "medium importance," reflecting first-level derived dependencies. The remaining nodes are marked as "low importance." This hierarchical mechanism ensures that critical nodes are not overlooked, while also providing appropriate protection for non-core but still important data assets. It avoids the over- or under-protection issues that can arise from a single standard, thus addressing the problem that existing technologies cannot find the minimum set of core assets covering global dependencies within a reasonable timeframe, leading to overly broad protection or the omission of critical nodes.

[0062] In one embodiment of this application, the first processing unit includes: a first processing module for solving an approximate minimum point cover set using an iterative algorithm. Initially, the protection set is empty. In each iteration, the node with the largest current degree is selected and added to the protection set, and all edges associated with it are deleted from the data lineage graph until there are no more edges in the data lineage graph.

[0063] In one embodiment of this application, the first processing module includes: a first processing submodule for performing a selection step during the process of solving the approximate minimum point cover set using an iterative algorithm: each time, selecting the node currently associated with the most uncovered edges as the target node; a second processing submodule for a removal step: adding the target node to the initial protection set and removing all edges associated with the target node from the edge set; and a third processing submodule for repeatedly performing the selection step and the removal step until the edge set is empty.

[0064] In one embodiment of this application, the construction unit includes: a second processing module used to extract features of each link in the data lineage graph by using a machine learning model during the process of extracting data processing link information from the metadata management system and constructing a data lineage graph based on the data processing link information. The features of the links include the frequency of data flow, the size of data volume, the sensitivity of data, and the business correlation between upstream and downstream data assets; a third processing module used to calculate the weight of each link by using the machine learning model and determine the score of each link based on the weight of each link; and filtering out irrelevant or redundant links based on the scores of each link.

[0065] In one embodiment of this application, the first processing unit includes: a fourth processing module for constructing a target model during the process of solving the approximate minimum point cover set of the above-mentioned data lineage graph, wherein the target model is a model that transforms the minimum point cover problem of the above-mentioned data lineage graph into a linear programming problem, wherein each node corresponds to a decision variable, the decision variable represents whether the node is selected into the cover set, the objective function is to minimize the total number of selected nodes, and the constraint condition ensures that each edge in the graph has at least one endpoint selected; and a fifth processing module for calculating the optimal solution of the above-mentioned target model using a linear programming solver to obtain the decision variable value of each node.

[0066] In one embodiment of this application, the apparatus further includes: a third processing unit configured to determine an abnormal access pattern by using a sliding window to count the historical access frequency of the data assets after marking the data assets with an importance level; and a fourth processing unit configured to adjust the importance level of the data assets according to the abnormal access pattern.

[0067] In one embodiment of this application, the fourth processing unit includes: a sixth processing module for collecting all access records of the data asset and using statistical or machine learning algorithms to identify access patterns that deviate from normal behavior as the abnormal access patterns; and a seventh processing module for processing the abnormal access patterns using a neural network model to obtain a target importance level and adjusting the importance level of the data asset to the target importance level.

[0068] The aforementioned data asset importance level marking device includes a processor and a memory. The aforementioned building units, first processing unit, and second processing unit are all stored as program units in the memory. The processor executes these program units stored in the memory to achieve the corresponding functions. All of the aforementioned modules are located in the same processor; alternatively, the aforementioned modules may be located in different processors in any combination.

[0069] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and adjusting kernel parameters can address the problem that current technologies cannot find the minimum set of core assets covering global dependencies within a reasonable timeframe, leading to excessive protection or omissions of critical nodes.

[0070] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0071] This invention provides a computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform the data asset importance level marking method.

[0072] This invention provides a processor for running a program, wherein the program executes the method for marking the importance level of the data assets.

[0073] This invention provides a device including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs at least the following steps: extracting data processing link information from a metadata management system, and constructing a data lineage graph based on the data processing link information, wherein the data lineage graph is in the form of a directed graph, the nodes of the data lineage graph represent data assets, and the edges of the data lineage graph represent data flow directions; solving for an approximate minimum vertex cover set on the data lineage graph; and marking the data assets with importance levels based on the approximate minimum vertex cover set, and displaying the marking results in a visual manner. The device in this document can be a server, PC, PAD, mobile phone, etc.

[0074] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having at least the following method steps: extracting data processing link information from a metadata management system, and constructing a data lineage graph based on the data processing link information, wherein the data lineage graph is in the form of a directed graph, the nodes of the data lineage graph represent data assets, and the edges of the data lineage graph represent data flow directions; solving for an approximate minimum vertex cover set on the data lineage graph; marking the data assets with importance levels based on the approximate minimum vertex cover set, and displaying the marking results in a visual manner.

[0075] This application also provides a data asset importance level marking system, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include methods for performing any of the above-described methods.

[0076] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0077] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0078] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0079] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The function specified in one or more boxes.

[0080] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0081] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0082] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0083] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0084] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0085] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0086] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for marking the importance level of data assets, characterized in that, include: Data processing link information is extracted from the metadata management system, and a data lineage graph is constructed based on the data processing link information. The data lineage graph is in the form of a directed graph, where nodes represent data assets and edges represent data flow directions. Solve for the approximate minimum point cover set of the data pedigree map; Based on the approximate minimum point coverage set, the data assets are marked with importance levels, and the marking results are displayed in a visual manner.

2. The method according to claim 1, characterized in that, Solving for the approximate minimum point cover set of the data pedigree map includes: An iterative algorithm is used to solve for the approximate minimum vertex cover set. Initially, the protection set is empty. In each iteration, the node with the largest degree is selected and added to the protection set, and all edges associated with it are deleted from the data lineage graph until there are no more edges in the data lineage graph.

3. The method according to claim 2, characterized in that, In the process of solving the approximate minimum point cover set using an iterative algorithm, the method further includes: Selection steps: Each time, select the node currently associated with the most uncovered edges as the target node; Removal steps: Add the target node to the initial protection set, and remove all edges associated with the target node from the edge set; Repeat the selection and removal steps until the edge set is empty.

4. The method according to claim 1, characterized in that, In the process of extracting data processing link information from the metadata management system and constructing a data lineage map based on the data processing link information, the method further includes: A machine learning model is used to extract features of each link in the data lineage graph. The features of the link include the frequency of data flow, the size of data volume, the sensitivity of data, and the business correlation between upstream and downstream data assets. The weight of each link is calculated using the machine learning model, and the score of each link is determined based on the weight of each link. Irrelevant or redundant links are filtered based on the scores of each link.

5. The method according to claim 1, characterized in that, In the process of solving for the approximate minimum point cover set of the data lineage map, the method further includes: Construct a target model, which is a model that transforms the minimum point coverage problem of the data lineage graph into a linear programming problem. Each node corresponds to a decision variable, which represents whether the node is selected into the coverage set. The objective function is to minimize the total number of selected nodes. The constraints ensure that each edge in the graph has at least one endpoint selected. The optimal solution of the target model is calculated using a linear programming solver, and the decision variable values ​​for each node are obtained.

6. The method according to claim 1, characterized in that, After assigning importance levels to the data assets, the method further includes: An abnormal access pattern is determined by using a sliding window to count the historical access frequency of the data assets. Adjust the importance level of the data asset based on the abnormal access pattern.

7. The method according to claim 6, characterized in that, Adjusting the importance level of the data asset based on the abnormal access pattern includes: Collect all access records to the data assets and use statistical or machine learning algorithms to identify access patterns that deviate from normal behavior as abnormal access patterns; A neural network model is used to process the abnormal access pattern to obtain the target importance level, and the importance level of the data asset is adjusted to the target importance level.

8. A marking device for the importance level of data assets, characterized in that, include: A construction unit is used to extract data processing link information from the metadata management system and construct a data lineage graph based on the data processing link information. The data lineage graph is in the form of a directed graph, where the nodes of the data lineage graph represent data assets and the edges of the data lineage graph represent data flow directions. The first processing unit is used to solve for the approximate minimum point cover set of the data pedigree map; The second processing unit is used to mark the data assets according to the approximate minimum point coverage set, and to display the marking results in a visual manner.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the method according to any one of claims 1 to 7.

10. A system for marking the importance level of data assets, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising methods for performing any one of claims 1 to 7.

Images