Data capitalization method and device

Abstract

The invention discloses a data capitalization method and device, and the method comprises the steps: constructing a standardized node system which comprises a data production node, a data processing node and a data storage node, and configuring a basic attribute for each node; defining a node dependency relationship, and constructing a directed acyclic graph according to the dependency relationship among the nodes; sorting the directed acyclic graph by adopting a preset topological sorting algorithm to generate a node execution sequence; sequentially executing the nodes according to the node execution sequence to complete data asset creation, and reversely tracing full-link nodes by selecting final data resources to form a data asset consanguinity map. According to the method, the problems of flow disorder, repeated processing and uncontrollable quality can be fundamentally solved, and the data capitalization efficiency and the data asset quality are improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and apparatus for data assetization. Background Technology

[0002] Current data assetization methods rely on business personnel to manually define the mapping relationship between data resources and data assets, collect raw data through front-end machines, and then clean, transform, classify and organize the data with the help of scripts or manual labor to ultimately form usable data assets.

[0003] The existing process suffers from three core problems: process fragmentation: data collection, cleaning, transformation, and storage are often handled by different teams or independent systems, lacking a unified process modeling language and collaboration mechanism. This leads to frequent interruptions, idle periods, waiting periods, or conflicts in the data flow process, severely impacting the overall efficiency of data assetization.

[0004] Repeated processing: The same raw data is often processed repeatedly by multiple processing links when building different data assets, and the processing standards of each link are different, which wastes computing and storage resources and causes inconsistencies between the final data assets.

[0005] Difficulty in tracing quality: When quality problems occur in the generated data assets, it is difficult to quickly and accurately locate the specific processing stage where the problem is located and the relevant responsible persons, resulting in high repair costs and weak data quality control capabilities. Summary of the Invention

[0006] In view of this, there is an urgent need for a data assetization method that can achieve unified modeling, automatic scheduling, and full traceability in order to overcome the above-mentioned shortcomings.

[0007] A data assetization method includes:

[0008] Construct a standardized node system that includes data production nodes, data processing nodes, and data storage nodes, and configure basic attributes for each node;

[0009] Define node dependencies and construct a directed acyclic graph based on the dependencies between nodes;

[0010] The directed acyclic graph is sorted using a preset topological sorting algorithm to generate a node execution sequence;

[0011] The nodes are executed sequentially according to the node execution sequence to complete the creation of data assets. By selecting the final data resource, the entire chain of nodes is traced back to form a data asset lineage map.

[0012] Furthermore, the data production node is used for raw data collection and preliminary verification, the data processing node is used for data cleaning, transformation and fusion, and the data storage node is used for structured storage and access control of data assets.

[0013] Furthermore, the construction of the directed acyclic graph based on the dependencies between nodes includes:

[0014] Define node dependency rules, where the data production node is the starting node, the data processing node depends on the data production node or other data processing nodes, and the data storage node depends on the data production node or data processing node and serves as the end point of the link.

[0015] Automatically detect and alert on loop dependencies in the graph.

[0016] Furthermore, the step of sorting the directed acyclic graph using a preset topological sorting algorithm is to sort the directed acyclic graph using a greedy topological sorting algorithm based on keyness calculation, specifically including:

[0017] Calculate the criticality of each node, which is defined as the total weight of the longest path originating from that node;

[0018] Initialize a max-heap with key as the priority key, traverse all nodes, and insert nodes with an in-degree of 0 into the max-heap;

[0019] Iteratively extract the top node of the heap and add it to the sequence, update the in-degree of its subsequent nodes, and insert the node whose in-degree becomes 0 into the max heap;

[0020] When the max-heap is empty, output the topological sort sequence.

[0021] Furthermore, the calculation of the criticality of each node, defined as the total weight of the longest path originating from that node, also includes:

[0022] Find the path with the largest sum of weights among all paths from each node in a directed acyclic graph to any destination within a preset time complexity using reverse depth-first search or dynamic programming algorithms.

[0023] Furthermore, the iterative process of extracting the top node of the heap and adding it to the sequence, updating the in-degree of its subsequent nodes, and inserting nodes with an in-degree of 0 into the max-heap also includes:

[0024] While the max-heap is not empty, perform the following operation in a loop:

[0025] Extract the top node of the heap and append it to the end of the output sequence;

[0026] Traverse all outgoing edges of the top node of the heap, and decrement the in-degree of the outgoing edge node v by 1;

[0027] If the in-degree of an outgoing node becomes 0, then the outgoing node and its criticality value are inserted into the heap.

[0028] Furthermore, the step of sorting the directed acyclic graph using a preset topological sorting algorithm to generate a node execution sequence includes:

[0029] After constructing the directed acyclic graph, its topological characteristics are analyzed. If the directed acyclic graph has the characteristics of long chains and few branches, a topological sorting algorithm based on depth-first search is adopted; if the directed acyclic graph has the characteristics of short chains and many branches, a topological sorting algorithm based on breadth-first search is adopted.

[0030] Furthermore, the method also includes inserting a data verification node between the data processing node and the data storage node to perform data quality verification.

[0031] Furthermore, the method also includes dynamically adjusting the weights of nodes and edges based on the usage frequency and priority of data assets, dynamically adjusting the topology sorting scheme, and prioritizing the processing of high-frequency and high-priority data assets.

[0032] As another aspect of the present invention, an embodiment of the present invention provides a data assetization device, characterized in that it includes:

[0033] The node system construction module is used to build a standardized node system that includes data production nodes, data processing nodes, and data storage nodes, and to configure basic attributes for each node;

[0034] The node dependency graph construction module is used to define node dependencies and construct a directed acyclic graph based on the dependencies between nodes.

[0035] The sorting module is used to sort the directed acyclic graph using a preset topological sorting algorithm and generate a node execution sequence.

[0036] The data asset creation module is used to execute nodes sequentially according to the node execution sequence to complete the creation of data assets, and to trace back the entire link nodes by selecting the final data resource to form a data asset lineage map.

[0037] The beneficial effects of the above-described technical solutions provided in the embodiments of the present invention include at least the following:

[0038] This invention provides a data assetization method. Compared to existing technologies, this invention offers, for the first time, asset-level semantic modeling capabilities. Based on the characteristics of the data assetization process, it constructs a node system (standardized nodes) for the data assetization process. On this basis, the data asset generation process is abstracted as a directed acyclic graph (DAG), and automation, optimal order execution, and conflict detection are achieved through topological sorting, realizing a standardized, traceable, and schedulable data asset supply chain. Compared to current methods, this invention fundamentally solves the problems of chaotic processes, redundant processing, and uncontrollable quality, improving the efficiency and quality of data assetization.

[0039] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0040] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0041] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0042] Figure 1 This is a flowchart illustrating the data assetization method provided in this embodiment of the invention;

[0043] Figure 2 This refers to the data asset lineage map obtained based on the data assetization method provided in this embodiment of the invention.

[0044] Figure 3 This is a schematic diagram of the data assetization device provided in an embodiment of the present invention. Detailed Implementation

[0045] To facilitate understanding, some technical terms in the embodiments of this invention will now be introduced.

[0046] A Directed Acyclic Graph (DAG) consists of three parts:

[0047] (1) Graph: A data structure consisting of vertices and edges connecting them. It is like a network of relationships.

[0048] (2) Directed: The edges in the graph are directed, such as from A to B, which represents a one-way dependency or sequential relationship (A must come before B).

[0049] (3) No cycles: Cycles are not allowed in the graph. That is, it is impossible to start from a vertex, walk around the direction of the directed edge and return to the starting point. This ensures that the process has a "beginning and an end" and will not fall into an infinite loop.

[0050] The topological sorting algorithm based on greediness (Kahn's algorithm) is based on a directed acyclic graph (DAG). It constructs a linear sequence by repeatedly selecting nodes with an in-degree of 0 (nodes with no predecessor dependencies). The basic idea of ​​Kahn's algorithm is: find vertices with an in-degree of 0 and record them in a queue or stack; remove the found vertex with an in-degree of 0 and its corresponding edge originating from that vertex, add the removed vertex to a list, and decrement the in-degree of the endpoint of any edge originating from the removed vertex by 1; continue this process until the queue or stack is empty. The order of the vertices in the list is the result of the topological sort. If the number of elements in the list is less than the number of vertices, it indicates that the directed graph contains a cycle.

[0051] Data lineage mapping: Data lineage, also known as data pedigree, data origin, or data genealogy, refers to the natural relationships, much like human blood ties, that form between data throughout its entire lifecycle—from generation, processing, manipulation, fusion, and flow to its eventual demise. Simply put, it's the upstream and downstream source-to-destination relationship between data—where the data comes from and where it goes. Data lineage involves not only the physical flow of data but also its logical relationships and transformation processes. Lineage mapping utilizes visualization tools and technologies (such as D3.js, ECharts, Tableau, etc.) to display lineage relationships. It provides an interactive interface, allowing users to explore and analyze these relationships.

[0052] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0053] This embodiment uses the example of a manufacturing company converting real-time production data into two types of data assets to illustrate the implementation process of this method. Combined with... Figure 1 , Figure 2 As shown, a data assetization method includes:

[0054] Step S10: Construct a standardized node system including data production nodes, data processing nodes, and data storage nodes, and configure basic attributes for each node.

[0055] Three types of standardized nodes are defined as follows:

[0056] Node type Function Unified Interface Specification Basic attributes Data production nodes Raw data collection and preliminary verification (format, completeness) Output format: JSON / Parquet; Data transmission protocol: HTTPS / FTP Node name, associated data stream, responsible person, related information system (such as ERP, MES), and list of data resources (including field definitions and update frequency). Data processing nodes Data cleaning (duplicate removal, data completion), transformation (format standardization, field mapping), and fusion (multi-source data association). Input: Compatible with production / processing node outputs; Output: Standardized intermediate data format Node name, processing rules (such as deduplication algorithm, mapping table), processing time baseline, quality verification threshold Data storage nodes Structured storage of data assets, index building, and access control Input: Compatible with processing / production node outputs; Supported query interfaces: SQL / API Node name, storage path, asset category, storage period, access permission list

[0057] Data production node: Responsible for collecting raw data from sources (such as business systems and IoT devices) and performing preliminary checks on format and integrity. Its output follows a unified interface specification (such as JSON / Parquet format, transmitted via HTTPS / FTP protocol).

[0058] Data processing node: Responsible for processing input data, including cleaning (duplicate removal, data completion), transformation (format standardization, field mapping), and fusion (multi-source association). Its input is compatible with the output of production nodes or other processing nodes, and its output is a standardized intermediate data format.

[0059] Data storage nodes: Responsible for storing processed data in a structured format, building indexes, and implementing access control to form the final data assets. Their inputs are compatible with the outputs of processing or production nodes, and they provide standard query interfaces (such as SQL / API).

[0060] In this embodiment, basic attributes are configured for each node, including: node identifier, data stream to which it belongs, responsible person, associated information system, data resource list, and log address. Batch creation and import of nodes are supported, and node information entry is simplified through templated configuration.

[0061] In this embodiment, taking the assetization of enterprise production data as an example, the equipment operation data and quality inspection data in the production process are transformed into "production quality analysis assets" and "equipment operation and maintenance assets", so as to achieve a data assetization cycle of ≤5 working days and a quality problem tracing time of ≤1 hour.

[0062] Twelve nodes of three types were created in batches using templates. Four data production nodes correspond to the data acquisition tasks of the programmable logic controllers (PLCs) for four production lines. For example, "Production Node 1" is associated with the Manufacturing Execution System (MES) and is responsible for collecting equipment operation data from "Production Line A," with an update frequency set to 5 minutes per update. Six data processing nodes cover processing logic such as data cleaning (denoising and handling outliers), format conversion (converting time-series data into structured tables), and multi-source fusion (associating equipment operation data with batch quality inspection results from the quality inspection system). Each node is configured with specific processing rules, estimated processing time baselines, and quality verification thresholds. Two data storage nodes correspond to the target assets "Production Quality Analysis Asset" and "Equipment Operation and Maintenance Asset," respectively, and are configured with their respective storage paths, asset categories, storage periods, and access permission lists. Attributes such as "Production Node 1 - Production Line A" are associated with the MES system.

[0063] Step S20: Define node dependencies and construct a directed acyclic graph based on the dependencies between nodes.

[0064] Define node dependency rules to ensure that different types of nodes follow specific dependency principles. For example, starting with a data production node, data processing nodes depend on the data production node or other data processing nodes, and data storage nodes depend on the data production node or data processing node, typically serving as the endpoint of a link. That is, data processing nodes can have incoming edges (depending on upstream data production or data processing nodes) and outgoing edges (being depended on by downstream data processing or data storage nodes); data storage nodes typically only have incoming edges (depending on upstream nodes) and are the endpoint of a directed acyclic graph.

[0065] In some embodiments, many-to-many dependencies are allowed, such as two data processing nodes merging their outputs into a single data storage node. A directed acyclic graph (DAG) is not a simple chain, but a more complex network.

[0066] A directed acyclic graph (DAG) model is constructed based on node dependencies, denoted as [V, E, W], where V is the set of vertices in the graph (i.e., the set of all data production nodes, data processing nodes, and data storage nodes), and [V, E, W]. E is the set of edges, and the edges... Each edge represents a specific dependency relationship; W is the weight, which is the quantification part of the model, assigning each edge a node state transition cost (such as the estimated execution time and resource consumption for completing the previous node and starting the next node).

[0067] For example, in this embodiment, a dependency graph is constructed according to business rules, such as "Production node 1 → Processing node 1 (cleaning) → Processing node 3 (integrating quality inspection data) → Storage node 1 (production quality analysis asset)", and the edge weights are set to 0.5 hours (production → cleaning), 1 hour (cleaning → fusion), and 0.3 hours (fusion → storage) based on historical data.

[0068] Step S30: Sort the directed acyclic graph using a greedy topological sorting algorithm based on keyness calculation to generate a node execution sequence.

[0069] To improve the overall efficiency of the data assetization process (e.g., minimizing total time), a topological sorting algorithm (Kahn's algorithm) based on a greedy strategy is used to sort the weighted DAG. The specific steps are as follows:

[0070] Step S301: Calculate the criticality of each node, which is defined as the total weight of the longest path originating from that node.

[0071] For each node v in the graph, calculate its criticality LPT(v), defined as the total weight of the path with the largest sum of weights (including the node's own processing cost) among all paths from that node to any destination. This can be calculated in O(|E|+|V|) time complexity using a reverse depth-first search or dynamic programming algorithm. Here, V is the number of vertices in the DAG graph, E is the number of edges in the DAG graph, and O(|E|+|V|) indicates that the time complexity is proportional to the sum of the number of vertices and edges.

[0072] In some embodiments, the path with the largest weight is found by DFS or DP algorithm, and the time complexity of the algorithm does not exceed O(|E|+|V|). The time complexity of all DFS or DP algorithms in finding the best path on the graph is O(|E|+|V|).

[0073] Step S302: Initialize a max-heap with key as the priority key, traverse all nodes, and insert nodes with an in-degree of 0 into the max-heap.

[0074] Ready queue initialization: Build a max-heap (or priority queue) with key values ​​(LPT values) as priority keys. Traverse all nodes and insert nodes with an in-degree of 0 into the heap. The max-heap is a data structure used in Kahn's algorithm to temporarily store all nodes with a current in-degree of 0.

[0075] Step S303: Iteratively extract the top node of the heap and add it to the sequence, update the in-degree of its subsequent nodes, and insert the node with an in-degree of 0 into the max-heap. Specifically:

[0076] Iterative stripping and sequence generation: While the heap is not empty, perform the following operations in a loop:

[0077] a. Take out the top node u (i.e. the ready node with the highest criticality) and append it to the end of the output sequence.

[0078] b. Traverse all outgoing edges (u, v) of node u, and decrement the in-degree of outgoing node v by 1.

[0079] c. If the in-degree of the outgoing edge node v becomes 0, then insert the outgoing edge node v and its LPT(v) value into the heap.

[0080] In the data assetization process, at a certain moment, the in-degree of a processing node A is 0, indicating that node A can start without depending on other nodes. Therefore, node A can serve as the starting node for the current data assetization process. Simultaneously, the number of nodes that subsequent nodes depend on the processing results of node A will decrease by 1 (because node A has already started). The purpose of this process is to ensure that all nodes in the data assetization process can start sequentially according to the processing flow, avoiding data inconsistency and node waiting overhead.

[0081] Step S304: When the max-heap is empty, output the topological sort sequence.

[0082] Specifically, the algorithm terminates when the heap is empty, resulting in a "critical path first" topological sort. This sequence ensures that all nodes execute only after all their predecessors have completed, and it prioritizes nodes on the critical path, thus theoretically minimizing the overall completion time.

[0083] During the dependency graph construction and sorting process, the system automatically detects whether there are loops (circular dependencies) in the graph. If a loop is detected, it is determined that there is a logical error in the process definition, and the system automatically issues an alarm and prevents the process from being published, ensuring the rationality and executability of the data flow.

[0084] The aforementioned topology sorting algorithm can generate an ordered data asset chain, ensuring that each node executes after all its predecessor (dependent) nodes, and minimizing the total execution time. This clearly reveals the sequence and dependencies of data processing, and considering the differences in time or cost among different processing stages, it can generate a structured and efficient data asset supply chain.

[0085] In this embodiment, the system generates a sorted sequence using the Kahn algorithm: [Production node 1, Production node 2, Production node 3, Production node 4, Processing node 1, Processing node 2, Processing node 3, Processing node 4, Processing node 5, Processing node 6, Storage node 1, Storage node 2], and detects no circular dependencies.

[0086] Step S40: Execute nodes sequentially according to the node execution sequence to complete the creation of data assets, and trace back the entire link nodes by selecting the final data resource to form a data asset lineage map.

[0087] Specifically, data assets are defined as standardized data sets with business value that are output by data storage nodes after one or more data resources have been processed through the aforementioned node links.

[0088] When creating a data asset, the user specifies the associated final output data resource, and based on the directed acyclic graph established in step S20, automatically traces back (starting from the storage node and traversing backwards along the incoming edges) the complete processing chain that the data resource depends on, associating all relevant nodes on the chain. This traceability capability forms a "data asset lineage graph." Through this graph, users can intuitively view the complete processing path of any data asset from the original data source to the final product, each node it passes through, and its processing details, achieving end-to-end traceability of the data asset.

[0089] For example, execution nodes are scheduled according to a sorted sequence. Each node runs automatically based on its configured interface specifications and processing logic. In this example, after approximately 4.8 hours of automated processing, two target assets, "Production Quality Analysis Asset" and "Equipment Operation and Maintenance Asset," were successfully generated. During operation, it was discovered that a certain batch of data in the "Production Quality Analysis Asset" was missing. The operation and maintenance personnel traced back using the asset's "lineage map," which clearly showed that the asset depended on "Processing Node 3," and the data for "Processing Node 3" originated from "Production Node 2." Upon inspection, it was found that the PLC acquisition interface corresponding to "Production Node 2" had experienced a temporary failure. After the failure was repaired, the system only needed to re-execute the affected node link (starting from Production Node 2), and the asset data update and repair were completed in approximately 0.5 hours.

[0090] This invention provides, for the first time, asset-level semantic modeling capabilities. Based on the characteristics of the data assetization process, it constructs a node system (standardized nodes) for the data assetization process. On this basis, it abstracts the data asset generation process as a directed acyclic graph (DAG), and achieves automation, optimal order execution, and conflict detection through topological sorting, realizing a standardized, traceable, and schedulable data asset supply chain. Compared with existing technologies, this invention has the following beneficial effects:

[0091] Enhancing Management Standardization: By standardizing node definitions and clarifying topological dependencies, the data asset supply chain structure becomes transparent and the flow is clear, greatly enhancing the standardization and compliance of data asset management.

[0092] Improve conversion efficiency: The combination of automated and batch node configuration with topology sorting based on optimization algorithms enables orderly and efficient scheduling of processing nodes, effectively reducing process interruptions, idle time and manual intervention, significantly shortening the data assetization cycle and reducing the error rate.

[0093] Ensuring and improving data quality: By binding data quality indicators, security rules, and specific processing nodes, when quality or security issues arise in assets, the lineage graph can be used to quickly locate the problematic node and responsible party, enabling precise repair and thus systematically improving the overall quality and reliability of data assets.

[0094] In some embodiments, if higher requirements are required for data quality, a "data verification node" can be inserted between the "data processing node" and the "data storage node" to specifically perform complex business rule verification, consistency checks, etc., forming a four-layer node system of "production → processing → verification → storage", which further improves the quality confidence of the output assets.

[0095] In other embodiments, when faced with large-scale data assetization scenarios with complex and diverse topologies (such as enterprise-level data platforms), a fixed sorting algorithm may not maintain optimal performance in all cases. Therefore, a sorting strategy can be dynamically selected based on the structural characteristics of the graph. For example, for graphs with long chains and few branches, a topological sorting based on the depth-first search (DFS) approach can be used; for graphs with short chains and many branches, a topological sorting based on the breadth-first search (BFS) approach can be used to optimize algorithm performance.

[0096] Specifically, after constructing the DAG, the system automatically analyzes its topological characteristics, such as the total number of nodes (|V|), the total number of edges (|E|), the length of the longest path in the graph, and the average branch factor (i.e., the average out-degree of nodes). Strategy mapping and execution: When the DAG is detected to have "long chains and few branches" (such as linear or near-linear processes), the system tends to use a topology sorting algorithm based on depth-first search (DFS). This strategy can quickly find a path from the starting point to the ending point and advance its execution, which helps reduce the perceived latency of the overall process. When the DAG is detected to have "short chains and many branches" (such as extensive data aggregation or distribution), the system tends to use a topology sorting algorithm based on breadth-first search (BFS) (such as the Kahn algorithm). This strategy allows more nodes that can be executed in parallel to enter the ready state as early as possible, which helps improve the overall throughput and resource utilization of the system. Through this adaptive mechanism, the system can dynamically select a more efficient sorting strategy according to the specific processing chain pattern, thereby optimizing the overall execution performance in large-scale data processing scenarios.

[0097] In other embodiments, when the system needs to concurrently process multiple independent data assetization processes and there is competition for computing and storage resources, the static edge weights in the basic solution may not meet the differentiated business needs. Therefore, a dynamic weighting mechanism and intelligent scheduling are required. That is, when the system processes multiple data assetization processes simultaneously and resources are competing, dynamic weights can be assigned to nodes and edges. The weights can be dynamically calculated based on factors such as the usage frequency of data assets, business priority, and real-time system load. The topology sorting algorithm is adjusted based on the dynamic weights to achieve resource tilting and intelligent scheduling, prioritizing the generation of high-value, high-priority assets.

[0098] Specifically, a set of dynamic weighting factors is defined for each data assetization process (i.e., each DAG) and its internal nodes and edges. These factors may include business priority, asset usage frequency, Service Level Agreement (SLA) requirements, and real-time system load (current CPU, memory, and I / O usage). The intelligent scheduler periodically or based on events (such as new task arrivals or resource releases) recalculates the "equivalent execution cost" weight of nodes or edges based on preset optimization goals (such as minimum average completion time and highest on-time completion rate of high-priority tasks), incorporating the aforementioned factors. The scheduler then uses the updated dynamic weights to rerun or adjust the topology sorting algorithm, generating a new node execution sequence. This essentially achieves dynamic resource allocation, prioritizing the scheduling and execution of high-value, high-priority data asset processing links, thereby maximizing business efficiency in resource-constrained environments.

[0099] In other embodiments, when applied to scenarios involving classified or sensitive data, a "data anonymization rule" attribute can be added to the node configuration. During data processing or storage node execution, the anonymization rule is automatically invoked to process sensitive fields. Simultaneously, the operation log recording in the lineage graph is strengthened, detailing the data access, processing operations, and operator information for each node to meet stringent data security audit requirements.

[0100] In some embodiments, the present invention also discloses a data assetization apparatus, such as Figure 3 As shown, it includes:

[0101] The node system construction module 10 is used to build a standardized node system that includes data production nodes, data processing nodes, and data storage nodes, and to configure basic attributes for each node.

[0102] The node dependency graph construction module 20 is used to define node dependencies and construct a directed acyclic graph based on the dependencies between nodes.

[0103] The sorting module 30 is used to sort the directed acyclic graph using a preset topological sorting algorithm to generate a node execution sequence.

[0104] The data asset creation module 40 is used to execute nodes sequentially according to the node execution sequence to complete the creation of data assets, and to trace back the entire link nodes by selecting the final data resource to form a data asset lineage map.

[0105] This invention provides, for the first time, asset-level semantic modeling capabilities. Based on the characteristics of the data assetization process, it constructs a node system (standardized nodes) for the data assetization process. On this basis, the data asset generation process is abstracted as a directed acyclic graph (DAG), and automation, optimal order execution, and conflict detection are achieved through topological sorting, realizing a standardized, traceable, and schedulable data asset supply chain. Compared to current methods, this invention fundamentally solves the problems of chaotic processes, redundant processing, and uncontrollable quality, improving the efficiency and quality of data assetization.

[0106] As another aspect of the present invention, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the relevant steps of the above-described data assetization method.

[0107] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the relevant steps of the aforementioned data assetization method.

[0108] The principles by which the above-described apparatus, client, medium, related equipment and system in this embodiment of the invention solve the problem are similar to those of the aforementioned method. Therefore, their implementation can refer to the implementation of the aforementioned method, and repeated details will not be repeated.

[0109] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0110] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0111] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0112] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0113] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of equivalents of this invention, this invention is also intended to include these modifications and variations.

Claims

1. A method for data assetization, characterized in that, include: Construct a standardized node system that includes data production nodes, data processing nodes, and data storage nodes, and configure basic attributes for each node; Define node dependencies and construct a directed acyclic graph based on the dependencies between nodes; The directed acyclic graph is sorted using a preset topological sorting algorithm to generate a node execution sequence; The nodes are executed sequentially according to the node execution sequence to complete the creation of data assets. By selecting the final data resource, the entire chain of nodes is traced back to form a data asset lineage map.

2. The data assetization method as described in claim 1, characterized in that: The data production node is used for raw data collection and preliminary verification, the data processing node is used for data cleaning, transformation and fusion, and the data storage node is used for structured storage and access control of data assets.

3. The data assetization method as described in claim 1, characterized in that, The construction of the directed acyclic graph based on the dependencies between nodes includes: Define node dependency rules, where the data production node is the starting node, the data processing node depends on the data production node or other data processing nodes, and the data storage node depends on the data production node or data processing node and serves as the end point of the link. Automatically detect and alert on loop dependencies in the graph.

4. The data assetization method as described in claim 1, characterized in that, The process of sorting the directed acyclic graph using a preset topological sorting algorithm is specifically described as using a greedy topological sorting algorithm based on keyity calculation. Calculate the criticality of each node, which is defined as the total weight of the longest path originating from that node; Initialize a max-heap with key as the priority key, traverse all nodes, and insert nodes with an in-degree of 0 into the max-heap; Iteratively extract the top node of the heap and add it to the sequence, update the in-degree of its subsequent nodes, and insert the node whose in-degree becomes 0 into the max heap; When the max-heap is empty, output the topological sort sequence.

5. The data assetization method as described in claim 4, characterized in that, The calculation of the criticality of each node, defined as the total weight of the longest path originating from that node, also includes: Find the path with the largest sum of weights among all paths from each node in a directed acyclic graph to any destination within a preset time complexity using reverse depth-first search or dynamic programming algorithms.

6. The data assetization method as described in claim 4, characterized in that, The iterative process of extracting the top node of the heap and adding it to the sequence, updating the in-degree of its subsequent nodes, and inserting nodes with an in-degree of 0 into the max-heap also includes: While the max-heap is not empty, perform the following operation in a loop: Extract the top node of the heap and append it to the end of the output sequence; Traverse all outgoing edges of the top node of the heap, and decrement the in-degree of the outgoing edge node v by 1; If the in-degree of an outgoing node becomes 0, then the outgoing node and its criticality value are inserted into the heap.

7. The data assetization method as described in claim 1, characterized in that, The step of sorting the directed acyclic graph using a preset topological sorting algorithm to generate a node execution sequence includes: After constructing the directed acyclic graph, its topological characteristics are analyzed. If the directed acyclic graph has the characteristics of long chains and few branches, a topological sorting algorithm based on depth-first search is adopted; if the directed acyclic graph has the characteristics of short chains and many branches, a topological sorting algorithm based on breadth-first search is adopted.

8. The method according to claim 1, characterized in that, The method also includes inserting a data verification node between the data processing node and the data storage node to perform data quality verification.

9. The method according to claim 1, characterized in that, The method also includes dynamically adjusting the weights of nodes and edges based on the usage frequency and priority of data assets, dynamically adjusting the topology sorting scheme, and prioritizing the processing of high-frequency and high-priority data assets.

10. A data assetization device, characterized in that, include: The node system construction module is used to build a standardized node system that includes data production nodes, data processing nodes, and data storage nodes, and to configure basic attributes for each node; The node dependency graph construction module is used to define node dependencies and construct a directed acyclic graph based on the dependencies between nodes. The sorting module is used to sort the directed acyclic graph using a preset topological sorting algorithm and generate a node execution sequence. The data asset creation module is used to execute nodes sequentially according to the node execution sequence to complete the creation of data assets, and to trace back the entire link nodes by selecting the final data resource to form a data asset lineage map.

Images