A heterogeneous system global data asset automatic inventory and intelligent value quantification evaluation method

Abstract

This invention discloses a method for automatic inventory and intelligent value quantification assessment of data assets across heterogeneous systems, belonging to the field of data asset inventory and assessment technology. Addressing issues such as complex access to heterogeneous data sources, difficulty in automatic asset discovery, and missing lineage relationships, this method constructs a unified adaptation layer for heterogeneous data sources, shielding underlying access differences; it builds an automatic discovery engine based on metadata scanning, collecting metadata such as table structure and file patterns and generating feature description vectors; it constructs a data lineage analysis and link tracing module to analyze data processing tasks, establish data flow graphs, and support link tracing and change impact analysis. Furthermore, it integrates automatic API interface detection and dynamic adaptation, unified directory management across the entire system, and a unified identification model for structured / semi-structured / unstructured data, enabling proactive inventory, correlation analysis, classification labeling, and standardized management of cross-platform data assets. This invention enhances the ability to manage data assets with precision and provides comprehensive support for data governance.

Description

Technical Field

[0001] This invention belongs to the field of data asset inventory and evaluation technology, specifically involving a method for automatic inventory and intelligent value quantification of data assets across heterogeneous systems. Background Technology

[0002] During enterprise digital transformation, data assets are scattered across various heterogeneous storage environments such as relational databases, NoSQL databases, distributed file systems, and external API interfaces. How to achieve unified inventory and automated identification of data assets across heterogeneous platforms is a core challenge in the field of data governance.

[0003] Common heterogeneous data management solutions typically design independent data acquisition interfaces for different types of data sources, achieving one-to-one access to various data sources through customized development. This approach exposes serious systemic flaws when facing complex heterogeneous environments: due to the lack of a unified standardized access framework and proactive asset discovery mechanism, different types of data sources need to repeatedly develop independent interfaces, resulting in high system integration complexity, high maintenance costs, and the inability to automatically traverse and identify data assets distributed across the entire domain. Discovered data assets are difficult to form a unified and standardized catalog, and the overall data asset panorama remains in a black box state. This problem has become a key bottleneck restricting the improvement of enterprises' refined data asset management capabilities and urgently needs to be solved through technical means. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method for automatic inventory and intelligent value quantification of heterogeneous system full-domain data assets, which can effectively solve the problems mentioned in the background technology.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: In a first aspect, the present invention provides a method for automatic inventory and intelligent value quantification of data assets across heterogeneous systems, the method comprising the following steps: A unified adaptation layer architecture for heterogeneous data sources is constructed, which shields the differences in access to underlying data sources by defining standardized data access interfaces; An automatic data asset discovery engine is built based on metadata scanning. The automatic data asset discovery engine automatically traverses and queries data storage nodes in heterogeneous environments through the standardized data access interface, collects metadata including table structure, file mode and data dictionary, and automatically identifies data assets based on the collected metadata. A data lineage analysis and link tracing module is constructed. This module parses data processing tasks, extracts data input and output relationships from the data processing tasks, and constructs a data flow graph to characterize the relationships and dependency links between data assets.

[0006] Furthermore, the construction of a unified adaptation layer architecture for heterogeneous data sources includes: When the data source type identification unit identifies the target data source as an unknown type not found in the feature fingerprint database, it marks the connection configuration information of the target data source as a type to be defined. The data asset auto-discovery engine still performs metadata scanning on the target data source marked as undefined type through the standardized data access interface, and performs correlation analysis between the collected metadata and the metadata of known type data assets to infer the position of the target data source marked as undefined type in the data flow map.

[0007] Furthermore, the data asset automatic discovery engine built based on metadata scanning specifically includes: The data asset auto-discovery engine extracts features from semantic, structural, and quality dimensions based on the collected metadata, and generates feature description vectors for data assets. The extraction of semantic dimension features involves using a pre-trained Chinese language representation model to convert data asset names and field annotations into semantic vectors, calculating the cosine similarity between the semantic vectors and each topic tag vector in a preset topic tag vector library, and determining the topic tag with the highest cosine similarity as the semantic tag of the data asset.

[0008] Furthermore, the data asset automatic discovery engine stores the generated data asset feature description vectors into a data asset feature library. The data asset feature library uses an inverted index structure to store each data asset feature description vector to support feature similarity-based retrieval and clustering analysis.

[0009] Furthermore, the data lineage analysis and link tracing module includes: The data flow graph construction unit includes parsing, extraction, transformation, and loading jobs, structured query language query scripts, and data synchronization tasks, to extract the data flow mapping from the source to the target. In the data flow graph, data nodes represent data asset instances, data edges represent data processing task instances, and the direction of the data edges indicates the direction of data flow.

[0010] Furthermore, the data lineage analysis and link tracing module also includes: The impact analysis unit receives a data change request containing the identifier of the data asset to be changed, and determines all downstream data nodes of the data asset to be changed based on the data flow map. The impact analysis unit calculates the impact level score for each downstream data node based on the number of data edge hops between the downstream data node and the data asset to be changed, the proportion of the number of fields involved in the change to the total number of fields in the downstream data node, and the time interval between the change trigger time and the most recent consumption time of the downstream data, and generates a comprehensive impact analysis report.

[0011] Furthermore, the method also includes: The API interface auto-detection and access adapter function's interface detection unit automatically detects and extracts the service address, interface path, and request method of external API interfaces; The code generation unit dynamically generates data acquisition driver code adapted to the external API interface based on the input parameter definition and response data structure identified by the parameter parsing unit, using a template-driven code generation strategy. The generated data acquisition driver code is then registered to the protocol conversion unit of the heterogeneous data source unified adaptation layer architecture, so that the external API interface becomes a standard data source type that can be accessed by the data asset auto-discovery engine and the data lineage analysis and link tracing module through the standardized data access interface.

[0012] Furthermore, the data lineage analysis and link tracing module also includes: According to the data flow map, the link tracing unit recursively traverses downstream data nodes along the data edge direction or recursively traverses upstream data nodes in the opposite direction along the data edge from the starting point of the data asset tracing, and collects the identification information of the data nodes and the data processing task instance information of the data nodes. Based on the data lineage analysis and link tracing results, the data lineage analysis and link tracing module automatically generates and labels behavioral tags that characterize the flow characteristics of data assets.

[0013] Furthermore, the method also includes: A unified identification model for structured, semi-structured, and unstructured data is designed. When the data asset auto-discovery engine reads a file whose format cannot be identified by its file extension and header identifier, the data format identification unit in the unified identification model performs a deep analysis strategy: sampling the file content, statistically analyzing the frequency of delimiters and the depth of nesting levels in the file content, and comprehensively calculating the degree of structure to determine whether the file content is structured, semi-structured, or unstructured data.

[0014] Furthermore, when the data format recognition unit executes the deep analysis strategy, the comprehensive calculation of the structuredness score specifically involves: Preset weighting coefficients are assigned to the frequency of the separator and the depth of the nesting level, respectively. A quantitative score is obtained by weighted calculation, and the data format type is determined based on the comparison result of the quantitative score and the preset threshold.

[0015] In summary, this application includes at least one of the following beneficial technical effects: 1. The method for automatic inventory and intelligent value quantification of heterogeneous system data assets provided by this invention firstly constructs a unified adaptation layer architecture for heterogeneous data sources, defines standardized data acquisition interfaces and a unified protocol conversion mechanism, and realizes standard access to multiple types of data sources such as relational databases, NoSQL databases, distributed file systems, and external API interfaces; then, it constructs an automatic data asset discovery engine based on metadata scanning, automatically traverses data storage nodes in the heterogeneous environment through a metadata collector, and automatically identifies data assets and generates feature descriptions based on metadata such as table structure, file mode, and data dictionary; 2. This invention constructs a data lineage analysis and link tracing module, builds a data lineage graph model, automatically analyzes the complete flow path of data from source to application, and identifies data relationships and dependency links; furthermore, it realizes the automatic detection and access adapter function of API interfaces, and completes the automatic detection, parameter identification, response format parsing and dynamic adaptation access code generation of external API interfaces; in addition, it establishes a unified catalog management mechanism for data assets across the entire domain, supports the classification, labeling, lineage association and version management of data assets, and forms a unified data asset list across the entire domain; finally, it designs a unified identification model for structured, semi-structured and unstructured data, which supports the automatic identification and standardized conversion of multi-format data such as relational tables, JSON documents, XML documents and files.

[0016] 3. This invention achieves standardized access to multiple heterogeneous data sources through a unified adaptation layer architecture, effectively solving the problems of high system integration complexity and maintenance costs caused by the need to develop independent interfaces for different types of data sources in existing technologies. This invention also achieves proactive automatic discovery of data assets through a metadata scanning-based data asset discovery engine, effectively solving the problem of lacking proactive discovery mechanisms and being unable to automatically identify data assets distributed across the entire domain in existing technologies. Furthermore, this invention enables cross-heterogeneous environment data asset relationship analysis through data lineage analysis and link tracing modules, effectively solving the problem of not being able to grasp the full picture of data in existing technologies. Finally, this invention achieves automated access to external API interfaces through automatic API interface detection and access adapters, effectively solving the problem of lacking automatic detection capabilities for external API interfaces in existing technologies.

[0017] 4. This invention achieves standardized management of data assets through a unified catalog management mechanism for all data assets, effectively solving the problem of the inability to create a unified and standardized catalog for discovered assets. This invention also achieves unified identification and registration of multiple data types through a unified identification model for structured, semi-structured, and unstructured data, effectively addressing the problem that existing technologies often focus on single data types and lack the ability to uniformly identify multiple data types. Overall, this invention significantly improves the refined management capabilities of enterprise data assets, providing comprehensive technical support for data governance. Attached Figure Description

[0018] Figure 1 A schematic diagram of the overall scheme for automatic inventory and intelligent value quantification assessment of data assets across heterogeneous systems.

[0019] Figure 2 A schematic diagram illustrating the core principles of a unified adaptation layer architecture for heterogeneous data sources.

[0020] Figure 3 This is a flowchart illustrating the logic of an automatic data asset discovery engine based on metadata scanning.

[0021] Figure 4 This is a schematic diagram of the data lineage analysis and link tracing module.

[0022] Figure 5 This is a diagram illustrating the automatic detection and connection of the API interface to the adapter.

[0023] Figure 6 A schematic diagram of a unified catalog management mechanism for all data assets.

[0024] Figure 7 A schematic diagram of a unified identification model for structured, semi-structured, and unstructured data. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the following description is provided in conjunction with the appendix. Figure 1 To be continued Figure 7 Specific embodiments are provided to further illustrate the present invention in detail.

[0026] This invention provides a method for automatic inventory and intelligent value quantification assessment of data assets across heterogeneous systems. The method includes six core steps and modules, which cooperate with each other to achieve unified access to data sources across heterogeneous platforms, automatic discovery and identification of data assets across the entire system, and standardized management of data assets, as detailed below.

[0027] The first step, S1, involves constructing a unified adaptation layer architecture for heterogeneous data sources. This unified adaptation layer is the core infrastructure layer for achieving unified access across multiple data source types. It is responsible for standardized access to various data sources, including relational databases, NoSQL databases, distributed file systems, and external application programming interfaces. This architecture consists of three core functional units: a data source type identification unit, a protocol conversion unit, and a connection management unit. The implementation follows these steps.

[0028] In step S101, the data source type identification unit extracts the connection configuration information of the target data source from the preset data source configuration file. The connection configuration information includes, but is not limited to, the connection address, port number, database name, username, and authentication credentials.

[0029] The data source type identification unit uses feature identifiers extracted from the connection configuration information to determine the specific type of the target data source. To this end, the data source type identification unit has a built-in feature fingerprint database covering multiple data source types. This database pre-stores driver class name features, connection string format features, and default port features for various mainstream relational databases, as well as connection protocol features for NoSQL databases and access protocol features for distributed file systems. When a new data source connection request is received, the data source type identification unit performs pattern matching between the extracted connection parameters and various features in the feature fingerprint database. If a match is successful, the type identifier of the target data source is determined.

[0030] After determining the type identifier, the data source type identification unit searches for and loads the corresponding data acquisition driver in the driver registry based on the type identifier. When the identification result is MySQL type, the data source type identification unit loads the MySQL-specific JDBC driver; when the identification result is MongoDB type, the data source type identification unit loads the MongoDB-specific client driver.

[0031] For unknown data sources that are not matched in the feature fingerprint database, the data source type identification unit marks the connection configuration information as "type to be defined" and records the connection protocol features for subsequent manual confirmation or type inference through machine learning models.

[0032] The training process of the machine learning model is as follows: The training dataset consists of a large number of known connection protocol features and their corresponding correct type labels; feature vectors are extracted from the historical connection configuration information with labeled type labels. Each feature vector contains the keyword distribution dimension of the protocol header, port number, and driver class name fields in the connection string, and manually labeled data source types are used as supervision labels; the machine learning model adopts a multilayer perceptron structure, with the number of input layer nodes equal to the dimension of the feature vectors and the number of output layer nodes equal to the total number of categories of all known data source types; the cross-entropy loss function is used to measure the difference between the type probability distribution predicted by the machine learning model and the true type label; the optimization goal of the machine learning model training is to minimize the cross-entropy loss function through gradient descent, so that the machine learning model can output the corresponding data source type according to the input connection protocol features. The model parameters are iteratively updated through the backpropagation algorithm until the prediction accuracy of the machine learning model on the validation set reaches a preset threshold.

[0033] Through the training process described above, the machine learning model can be used to perform type inference on unknown data sources labeled as "type to be defined".

[0034] In step S102, the protocol conversion unit maintains an abstract set of data access interface definitions, which explicitly define standardized data operation method signatures such as read, write, query, and metadata. Simultaneously, the protocol conversion unit configures corresponding protocol adapter plugins for each supported data source type.

[0035] The protocol adapter plugin translates standardized interface method calls into native protocol calls for the corresponding data source type, and then reconstitutes the execution results into a standardized data representation format. For relational databases, the plugin translates standardized interface calls into Structured Query Language (SCL) statements for execution; for NoSQL databases, it translates them into NoSQL-specific query languages ​​or application programming interfaces (APIs); for distributed file systems, it translates them into file system read / write operations; and for external application programming interfaces (APIs), it translates them into the construction and parsing of Hypertext Transfer Protocol (HTTP) requests and responses.

[0036] The protocol conversion mechanism ensures that the upper-layer processing logic is decoupled from the lower-layer data source type. When a new data source type needs to be added, only the corresponding protocol adapter plugin needs to be developed and registered to the protocol conversion unit to be integrated into the overall technical framework without modifying any upper-layer processing logic.

[0037] In step S103, the connection management unit maintains a connection pool manager, which maintains an independent connection pool instance for each configured data source type. During the system initialization phase, the connection pool manager pre-establishes a certain number of initial connections and stores these pre-established connections in a connection queue to await invocation.

[0038] When an upper-layer application needs to access a data source, the connection management unit first checks whether there are any idle connections in the connection pool corresponding to the data source. If an idle connection exists, it is immediately allocated to the requester; if no idle connection exists and the connection pool has not yet reached its maximum connection limit, a new connection is created and allocated to the requester; if the connection pool has reached its maximum connection limit, the requester enters a waiting queue until a connection is released.

[0039] The connection management unit is also configured with a scheduled connection health check task. This task periodically selects a number of connections from each connection pool and sends lightweight probe requests to verify the validity of the connections. Once a failed connection is detected, the connection management unit automatically removes it from the connection pool and marks it as unavailable, while simultaneously triggering connection reconstruction logic.

[0040] The connection management unit is also responsible for the unified release management of connections. After the upper-layer business completes the data access operation, the connection is returned to the connection pool instead of being closed directly. The connection reuse mechanism reduces the performance overhead caused by frequently establishing and closing connections.

[0041] Step S1, through the collaborative work of three core functional units—the data source type identification unit, the protocol conversion unit, and the connection management unit—completely realizes the construction of a unified adaptation layer architecture for heterogeneous data sources. This unified adaptation layer architecture shields the subsequent steps of automatic data asset discovery, lineage analysis, and unified directory management from the access differences of different underlying data sources, providing a standardized data access interface to ensure the feasibility and universality of the overall technical solution in heterogeneous environments.

[0042] Regarding step S2, an automatic data asset discovery engine is built based on metadata scanning. This engine is the core functional module for proactive and automatic data asset discovery. The automatic data asset discovery engine automatically traverses data storage nodes in heterogeneous environments through a metadata collector, automatically identifying data assets and generating feature descriptions based on metadata such as table structure, file schema, and data dictionary. The automatic data asset discovery engine is implemented according to the following steps.

[0043] In step S201, after the metadata collection task scheduler starts, it triggers metadata scanning jobs according to the preset scanning strategy and time plan. The metadata collection task scheduler is the central control unit of the data asset auto-discovery engine, and it maintains the task queue and execution plan configuration.

[0044] The task queue stores data source scanning tasks to be executed. Each task record contains the target data source identification information, scan range definition, priority setting, and creation time.

[0045] The execution plan configuration defines the timing rules for triggering scan jobs. These rules include immediate execution, single-time scheduled execution, and periodic recurring execution. When the timing conditions configured in the execution plan are met, the metadata collection task scheduler retrieves the corresponding scan task from the task queue and sorts them according to their priority and creation time, generating a scan job execution sequence. After the scan job execution sequence is generated, the metadata collection task scheduler distributes the jobs to a preset number of worker threads for concurrent execution, enabling parallel scanning of multiple data sources.

[0046] In step S202, the data asset auto-discovery engine connects to the target data source through the heterogeneous data source unified adaptation layer, and then sends a metadata query request to the target data source to read the metadata catalog information of the target data source.

[0047] For relational databases, the metadata catalog information of the target data source is stored in system tables or system views, such as the `information_schema` database in MySQL and the `data dictionary` view in Oracle. The data asset auto-discovery engine retrieves the definition information of all database instances, tablespaces, schemas, tables, views, stored procedures, and functions in the relational database by querying system tables or system views.

[0048] For NoSQL databases, the data asset auto-discovery engine obtains a list of database instances, collections, and indexes through a client application programming interface.

[0049] For distributed file systems, the data asset auto-discovery engine recursively scans the directory structure under a specified path through the file system's directory traversal application programming interface to obtain file path, file size, creation time, modification time, and file owner information.

[0050] In step S203, the data asset auto-discovery engine performs targeted metadata collection operations based on the type of the target data source.

[0051] For relational database type data assets, metadata collection operations include the following: Collect table structure definition information, including table name, schema, storage engine, and table comments; Collect field type definition information, which includes field name, data type, length, precision, whether it can be nullable, default value, and field comments; Collect index definition information, which includes index name, index type, index method, included fields, and index status; Collect constraint definition information, which includes constraint name, constraint type, primary key constraint, unique constraint, check constraint, and foreign key constraint; Collect view definition information, which includes view name, mode, view definition text, and view status; Collect stored procedure and function definition information, including object name, object type, parameter list, code body, and mode.

[0052] For NoSQL database type data assets, metadata collection operations include the following: Collect collection definition information, which includes collection name, collection type, database, sharding strategy, and compression configuration; Collect document schema information, which includes document structure templates, field definitions, and nested hierarchical structures. Collect key-value mapping information, which includes key name, key type, value type, and time-to-live configuration. Collect index definition information, which includes index name, index fields, index type, index status, and sparsity. Collect cluster topology information, including shard node information, replica set configuration, and arbitrator node information.

[0053] For data assets of the distributed file system type, metadata collection operations include the following: Collect file directory structure information, which includes directory hierarchy, parent directory identifier, and subdirectory list; Collect file type definition information, which includes file extensions, multipurpose Internet Mail extensions, and content encoding formats. Collect file size distribution information, which includes file size range statistics and storage space usage statistics; Collect file attribute information, including file creation time, file modification time, file access time, file owner, and permission flags.

[0054] In step S204, the data asset automatic discovery engine extracts data asset features based on the collected metadata, generates data asset feature description vectors, and stores these vectors in the data asset feature library. The data asset feature extraction employs a multi-dimensional feature vector construction method, which characterizes data assets from three perspectives: semantic, structural, and quality dimensions.

[0055] Semantic dimension features include natural language semantic analysis results for data asset names, keyword extraction results for field annotations, and topic classification tags for data content. The specific implementation of natural language semantic analysis in the semantic dimension features is as follows: The data asset automatic discovery engine uses a pre-trained Chinese language representation model to convert the text content in the data asset name and field annotations into fixed-length semantic vectors. It then calculates the cosine similarity between the semantic vector and each topic tag vector in a pre-defined topic tag vector library, and uses the topic tag with the highest cosine similarity as the semantic tag for the data asset. The cosine similarity is calculated as follows:

[0056] in, The semantic vector obtained after transforming the text content representing the data asset name or field annotation by a pre-trained Chinese language representation model. This represents the topic tag vector corresponding to a specific topic tag in the preset topic tag vector library. This represents the dot product of the semantic vector and the topic tag vector. The magnitude of the semantic vector is represented. This represents the magnitude of the topic tag vector.

[0057] The pre-built topic tag vector library is constructed as follows: collect a set of common topic terms in the field of enterprise data governance. The topic term set includes, but is not limited to, topics such as customer information, product data, transaction records, financial data, and log data. Input each topic term into a pre-trained Chinese language representation model to obtain the corresponding topic tag vector.

[0058] The training process of the pre-trained Chinese language representation model is as follows: The training dataset consists of a large-scale general Chinese corpus, with sources including online text, technical documents, and professional literature in the field of data governance; the training objective is a masked language modeling task, i.e., randomly masking some words in the input text, and training the pre-trained Chinese language representation model to predict the masked words based on the context; the cross-entropy loss function is used to measure the difference between the word probability distribution predicted by the pre-trained Chinese language representation model and the actual masked words; the parameters of the pre-trained Chinese language representation model are iteratively updated through the backpropagation algorithm until the cross-entropy loss function value converges to a preset threshold. After the pre-trained Chinese language representation model is trained, its parameters are fixed, and it can be directly used to convert the input text into semantic vectors without additional fine-tuning training for the data asset inventory task.

[0059] Structural dimension features include the type identifier of data assets, the number of fields, the depth of nesting levels, and the standardization of data formats.

[0060] The quality dimension features include data completeness rate, null value ratio, duplicate record ratio, and data timeliness score. The calculation methods for each indicator in the quality dimension features are as follows: The data integrity rate is calculated as follows: for non-null constraint fields in relational database tables, the ratio of the actual number of non-null records in the non-null constraint fields to the total number of records; for documents in NoSQL database collections, the ratio of the number of fields actually existing in the document to the total number of fields defined in the document schema.

[0061] The null value ratio is calculated as follows: the null value ratio equals 1 minus the data integrity rate.

[0062] The calculation method for the duplicate record ratio is as follows: the data asset auto-discovery engine performs a hash operation on the data content and counts the proportion of records with the same hash value to the total number of records.

[0063] The data timeliness score is calculated as follows: the data asset auto-discovery engine reads the latest update time of the data, calculates the time difference between the latest update time and the current system time, and maps the time difference to a preset interval as the timeliness score value.

[0064] The data asset auto-discovery engine vectorizes and concatenates the features from the three dimensions mentioned above to generate feature description vectors for each data asset. The data asset feature library uses an inverted index structure to store the feature description vectors of each data asset, supporting fast retrieval and clustering analysis based on feature similarity.

[0065] Step S2 achieves proactive and automatic discovery and feature description of data assets through the sequential execution of four stages: metadata collection task scheduling, metadata catalog information reading, targeted metadata collection, and multi-dimensional feature extraction. The automatic data asset discovery engine provides a basic data asset list and feature information for subsequent steps such as data lineage analysis, unified catalog management, and value quantification assessment.

[0066] In step S3 of this embodiment, a data lineage analysis and link tracing module is constructed. This module is the core functional module for visualizing and analyzing the relationships between data assets. The data lineage analysis and link tracing module constructs a data lineage graph model, automatically analyzes the complete flow path of data from its source to its application, and identifies data relationships and dependent links. The data lineage analysis and link tracing module consists of three core functional units: a data flow graph construction unit, a link tracing unit, and an impact analysis unit, and is implemented according to the following steps.

[0067] Step S301: The data flow graph construction unit obtains the task definition of the data processing task, parses the data processing task, extracts the data input relationship and data output relationship defined in the data processing task, constructs the association between data nodes and data edges based on the data input relationship and data output relationship, and generates a data flow graph.

[0068] Data processing tasks originate from sources including extraction, transformation, and loading jobs; structured query language query scripts; data synchronization tasks; and data processing pipelines. The data flow graph construction unit configures different parsing rules for different data processing task types: For the extraction, transformation, and loading job, the data flow graph construction unit reads the source table definition and target table definition in the extraction, transformation, and loading job configuration file and establishes a data flow mapping from the source table to the target table.

[0069] For Structured Query Language (SCL) query scripts, the data flow graph construction unit extracts the FROM and INSERT INTO clauses from the SCL query statements through lexical and syntactic analysis, and establishes a data flow mapping from the query table to the written table.

[0070] For data synchronization tasks, the data flow graph construction unit reads the source and target definitions in the data synchronization configuration and establishes a data flow mapping from the source to the target.

[0071] The three parsing rules described above correspond to different types of data processing tasks, collectively covering the main task forms in data flow scenarios. The data flow graph construction unit converts the parsed data flow relationships into a graph structure representation. In the graph structure of the data flow graph, data nodes correspond to data asset instances, data edges correspond to data processing task instances, and the direction of the data edges indicates the direction of data flow. To support the storage and querying of large-scale graph data, the data flow graph uses a graph database for persistent storage. The node labels, edge types, and attribute fields in the graph database are organized according to a preset graph pattern.

[0072] In step S302, the link tracing unit receives the data asset selected by the user as the starting point for tracing, performs link tracing along the data flow map, and generates a lineage link report. The link tracing unit provides two tracing modes: forward tracing mode and reverse tracing mode.

[0073] In the forward tracing mode, the link tracing unit starts from the data asset selected by the user, recursively traverses all reachable downstream data nodes along the direction of the data edge in the data flow graph, collects the identification information of the downstream data nodes, the system information of the system to which the downstream data nodes belong, and the data usage information of the downstream data nodes, and finally generates a downstream lineage link report.

[0074] In the reverse tracing mode, the link tracing unit starts from the data asset selected by the user, recursively traverses all reachable upstream data nodes in the opposite direction of the data edge in the data flow graph, collects the identification information of the upstream data nodes, the data source information of the upstream data nodes, and the original definition information of the upstream data nodes, and finally generates an upstream lineage link report.

[0075] During the tracing process, the link tracing unit simultaneously tracks the lineage relationships at the data level and the dependencies at the data processing task level. Upon traversing each data edge, the link tracing unit records the identifier information, task type, and execution order of the corresponding data processing task instance.

[0076] Step S303: The impact analysis unit receives a data change request submitted by the user. The data change request includes the identifier of the data asset to be changed and a description of the change content.

[0077] The impact analysis unit obtains all downstream data nodes corresponding to the data nodes of the data assets to be changed from the data flow graph based on the identifier of the data assets to be changed, forming a set of downstream nodes.

[0078] The impact analysis unit performs an impact assessment calculation for each downstream data node in the downstream node set. The impact assessment calculation considers three factors: First, the number of data edge hops between the downstream data node and the data node of the data asset to be changed; the smaller the number of data edge hops, the closer the relationship. Second, the proportion of the number of fields involved in the change to the total number of fields in the downstream data node. Third, the time interval between the change trigger time and the last consumption time of the downstream data.

[0079] The impact analysis unit calculates the impact level score based on the three factors mentioned above. The calculation method is as follows: take the reciprocal of the number of data jumps, directly take the field percentage, and take the reciprocal of the time interval. The three transformed values ​​are then weighted and summed to obtain the impact level score. The weight coefficients for each of the three factors are preset by the administrator; all three weight coefficients are positive real numbers and their sum is 1. Based on the numerical range of the impact level score, the impact analysis unit classifies the impact level into three levels: high impact, medium impact, and low impact. A high impact level indicates that the change will have a decisive impact on downstream data; a medium impact level indicates that the change will have a partial impact on downstream data; and a low impact level indicates that the impact of the change on downstream data is negligible.

[0080] The impact analysis unit summarizes the impact assessment results of all downstream data nodes in the downstream node set and generates a comprehensive impact analysis report. The comprehensive impact analysis report includes statistical information on the scope of impact, a list of high-impact assets, and recommended handling measures.

[0081] In summary, step S3, through the sequential execution of three stages—data flow graph construction, link tracing, and impact analysis—achieves visualization of data asset relationships and quantitative assessment of the impact of data changes. The data lineage analysis and link tracing modules provide support for the display of data asset relationships in the subsequent unified catalog management steps.

[0082] For step S4, the API interface auto-detection and access adapter function is implemented. The API interface auto-detection and access adapter is the core functional module for achieving automated access to external API interfaces. It performs automatic detection of external API interfaces, parameter identification, response format parsing, and dynamic adaptation access code generation. The API interface auto-detection and access adapter consists of three core functional units: an interface detection unit, a parameter parsing unit, and a code generation unit, and is implemented according to the following steps.

[0083] Step S401: The interface detection unit automatically discovers available API interfaces through the interface registry center or interface specification document, and extracts the service address, interface path, request method, and authentication method of the API interface. The interface detection unit supports the following three interface discovery modes: The first mode is designed for system environments that support service registration mechanisms. The interface detection unit connects to an interface registry center, which may include service registration and discovery platforms such as Eureka, Nacos, and Consul. The interface detection unit queries the list of registered service instances and extracts the access address of the API interface definition document from the metadata of the service instances.

[0084] The second mode is designed for system environments that do not support service registration mechanisms. The interface detection unit discovers interfaces by parsing a pre-configured set of interface specification documents, including OpenAPI specification documents, Swagger documents, and WSDL documents.

[0085] The third mode is designed for system environments where standard documentation is unavailable. The interface detection unit discovers interfaces through network probing, specifically by sending probe requests to known service addresses and identifying the existence and basic information of API interfaces based on response characteristics.

[0086] After discovering an API, the API discovery unit stores the extracted API metadata in the API manifest database. The API manifest database records the service identifier, API path, request method type, authentication requirements, and documentation link for each API.

[0087] In step S402, the parameter parsing unit automatically identifies the input parameter definitions, parameter types, parameter constraints, and response data structures of the API interface by sending test requests or parsing interface response samples. The parameter parsing unit uses a combination of active probing and passive parsing strategies for parameter identification.

[0088] The specific implementation of the active probing strategy is as follows: the parameter parsing unit sends a probing request containing common parameter types to the target API interface, collects the response behavior characteristics of the target API interface, including changes in response status codes and changes in response content, and infers the existence and valid value range of parameters based on the response behavior characteristics.

[0089] The specific implementation of the passive parsing strategy is as follows: The parameter parsing unit first obtains the specification document of the target API interface. If the specification document is unavailable, it collects the historical call records of the target API interface. Through structured analysis of the specification document or historical call records, it extracts the parameter name list, parameter position definition, parameter type constraint, and required field identifier.

[0090] The parameter parsing unit internally maintains a parameter recognition rule base, which defines parameter extraction rules for various standardized document formats and recognition patterns for common interface parameter types. The parameter recognition rule base supports dynamic expansion and updates of the recognition rules.

[0091] In step S403, the code generation unit dynamically generates data acquisition driver code adapted to the API interface based on the parameter parsing results, and registers the data acquisition driver code to the heterogeneous data source unified adaptation layer. The code generation unit adopts a template-driven code generation strategy, which defines data acquisition code templates for different programming languages, including Java, Python, and Go.

[0092] The code generation unit fills in the placeholders in the data acquisition code template based on the interface metadata and parameter information provided by the parameter parsing unit, generating complete interface adapter code. The interface adapter code includes the following core components: The HTTP client initialization code is responsible for establishing a connection with the target API interface and configuring authentication information. The request build code is responsible for assembling request parameters and setting request headers according to the parameter definition. The response parsing code is responsible for parsing the data returned by the API interface according to the response data structure definition. Error handling code is responsible for capturing and handling exceptions during API call processes.

[0093] After the generated interface adapter code passes compilation and verification, the code generation unit registers the interface adapter code with the protocol conversion unit of the heterogeneous data source unified adaptation layer, enabling the interface adapter code to be called by the upper-layer business as a standard data source type.

[0094] Step S4 automates the process of discovering and connecting external API interfaces by sequentially executing three stages: interface detection, parameter parsing, and code generation. Automatic API interface detection and adapter integration enable external API interfaces to be connected to the unified adaptation layer of heterogeneous data sources as standard data source types, providing support for subsequent unified data access.

[0095] Next, for step S5, a unified data asset catalog management mechanism will be established across the entire domain. This mechanism is the core functional module for achieving standardized management of data assets. It supports the classification, tagging, lineage association, and version management of data assets, forming a unified data asset list across the entire domain. The unified data asset catalog management mechanism consists of four core functional units: asset registration, classification management, tagging management, and version management. It will be implemented according to the following steps.

[0096] In step S501, the asset registration unit receives the asset information submitted by the data asset auto-discovery engine, verifies the completeness and format of the asset information, generates asset registration records according to the standardized asset registration template after the verification is passed, and stores the asset registration records in the asset catalog database.

[0097] The asset registration template defines a standard format for asset registration information, including the following registration fields: Asset identifier, a globally unique identifier automatically generated by the asset registration unit; Asset name, a human-readable name for user identification and retrieval; Asset type, identifying the classification type of the data asset; The system to which the data asset belongs is the business system or technology platform that identifies it. Storage location, which identifies the physical or logical storage address of the data asset; Data format identifies the types of data formats contained in the data asset; Data scale refers to the size of the data assets. Creation time records the time when the data asset is first recognized by the data asset auto-discovery engine. Update time records the most recent change to the asset registration record; The asset manager is responsible for recording the contact information of the person in charge of managing the data assets. Asset description: Provides a detailed text description of the data asset.

[0098] Step S502: The classification management unit establishes a multi-dimensional classification system based on the type, source, and purpose of data assets, supporting the classification retrieval and statistics of data assets. The classification management unit internally maintains a configurable classification dimension model, which defines multiple independent classification dimensions, each containing several classification nodes. The default configuration of the classification dimension model includes the following classification dimensions: The technology type dimension includes relational database table category nodes, NoSQL collection category nodes, file resource category nodes, and API interface category nodes; In terms of business areas, it includes customer management category nodes, product management category nodes, order management category nodes, and financial management category nodes; The data domain dimension includes basic data domain classification nodes, main data domain classification nodes, transaction data domain classification nodes, and analytical data domain classification nodes. The data sensitivity dimension includes public data classification nodes, internal data classification nodes, sensitive data classification nodes, and confidential data classification nodes.

[0099] The category management unit supports dynamic configuration of category dimensions and category nodes, allowing administrators to add new category dimensions, modify the names of existing category dimensions, delete unused category dimensions, add new category nodes under category dimensions, modify the names of category nodes, or delete category nodes according to actual management needs.

[0100] When data assets are registered in the asset catalog database, the classification management unit supports selecting and associating classification nodes under multiple classification dimensions for the data assets, thereby achieving multi-dimensional classification labeling of data assets. The classification management unit also provides asset statistics functions based on classification dimensions, which support generating asset quantity distribution reports and asset growth rate trend reports under each classification dimension.

[0101] In step S503, the tag management unit adds custom tags to the data assets and supports adding, deleting, modifying, and querying tags, enabling flexible labeling and grouping of data assets. The tag management unit organizes tag information in key-value pairs; each tag consists of two parts: a tag key and a tag value. The tag key defines the semantic category of the tag; the tag value defines the specific value within that semantic category.

[0102] For example, the tag key can be set to "Business Department", and the corresponding tag value can be set to "Sales Department"; the tag key can be set to "Project Affiliation", and the corresponding tag value can be set to "Digital Transformation Project"; the tag key can be set to "Quality Level", and the corresponding tag value can be set to "Grade A".

[0103] The tag management unit supports adding multiple tags to a single data asset, and also supports batch adding the same tags to data assets that meet specific tag conditions. The tag management unit provides full-text search capabilities for tags, allowing users to query assets using tag keys, tag values, or combinations of tag keys and tag values. The tag management unit also supports tag inheritance; when there are hierarchical or lineage relationships between data assets, lower-level or downstream data assets automatically inherit specified tags from higher-level or upstream data assets, achieving batch tag propagation.

[0104] Step S504: The version management unit records the structural change history of data assets, supporting data asset version tracing and comparison. The version management unit maintains a version history linked list for each data asset, recording version snapshots of all changes made to the data asset since its initial registration. Each version snapshot contains the following information: Version number, an automatically incrementing version identifier assigned by the version management unit; Change time: Records the specific time when the version change occurred; The change type indicates whether the change is a structural change or an attribute change. Change details: Describe the specific changes. For structural changes, record the differences in the structural definitions before and after the changes. Change the operator and record the identity of the user who performed the change operation.

[0105] The version management unit provides a version rollback function, allowing users to view the structure definition and attribute values ​​of data assets at any historical version point. The version management unit also provides a version comparison function; when a user selects two different versions for comparison, the unit automatically analyzes the differences in changes between the two versions and generates a version comparison report, which includes field-level change details.

[0106] Step S5, through the sequential execution of four stages—asset registration, classification management, tagging management, and version management—achieves standardized registration, multi-dimensional classification, flexible labeling, and version traceability of data assets. This unified data asset catalog management mechanism across the entire domain provides a structured data asset list and management support for subsequent quantitative assessment of data asset value.

[0107] Finally, in step S6, a unified identification model for structured, semi-structured, and unstructured data is designed. This model is the core functional module for achieving unified identification and standardized conversion of multiple types of data assets. The unified identification model supports automatic identification and standardized conversion of multi-format data such as relational tables, JSON documents, XML documents, and files. The model consists of three core functional units: a data format identification unit, a pattern extraction unit, and a standardized conversion unit, and is implemented according to the following steps.

[0108] In step S601, the data format recognition unit automatically determines the data format type based on the header identifier, file extension, or structural characteristics of the data content. The data format recognition unit has a built-in multi-level data format recognition strategy.

[0109] The first-layer identification strategy performs rapid classification based on file extensions. The data format identification unit maintains a mapping table between common file extensions and data format types, mapping common extensions to structured data formats, semi-structured data formats, or unstructured data formats, respectively.

[0110] The second-layer recognition strategy performs precise classification based on file header identifiers. The data format recognition unit reads the header byte sequence of the file to be recognized and matches it with the header feature codes of known data formats. Different format types have their own feature codes; for example, JSON, XML, and PDF formats all have identifiable header markers.

[0111] The third-layer identification strategy performs in-depth analysis based on the structural features of the data content. When the file extension and header identifier cannot determine the format type, the data format identification unit samples and analyzes the file content, statistically scores the frequency of delimiters, the depth of nesting levels, and the degree of structure, and comprehensively judges the data format type based on these indicators.

[0112] After completing the format recognition, the data format recognition unit writes the recognition result into the metadata storage to provide input for subsequent pattern extraction.

[0113] In step S602, the pattern extraction unit automatically extracts data pattern information from the data content. The pattern extraction unit adopts different pattern extraction strategies according to the data type.

[0114] For structured data, the pattern extraction unit parses the table header information to extract the field names and data types of each field. If the table header is unavailable, the number of fields and the value types of each field are inferred by scanning the data row content. The pattern extraction unit also needs to identify the format parameters between fields.

[0115] For semi-structured data, the pattern extraction unit parses the root node type, recursively traverses the node structure at each level, and extracts the path expression, field type, and nesting level of each field. The pattern extraction unit also needs to identify constraints such as field optionality, repetition, and default values.

[0116] For unstructured data, the pattern extraction unit extracts feature information through content analysis techniques, including language type, topic classification, keywords, and entity recognition results for text-based data, and format, resolution, and content tags for image-based data. Pattern extraction for unstructured data supports the configuration of custom feature extraction rules, allowing users to define domain-specific feature extraction logic according to business needs.

[0117] Step S603: The standardization conversion unit converts various types of data into a unified standard data representation format, facilitating subsequent unified processing and analysis. The standardization conversion unit defines two standard representation formats: the structured data standard format and the feature data standard format.

[0118] The structured data standard format is used to represent data assets with a defined schema. It uses a unified table structure model to describe the data schema and organizes the data content in a row and column matrix format.

[0119] The feature data standard format is used to represent data assets that lack a clear pattern. It transforms the raw data content into a multi-dimensional feature vector, with each dimension of the feature vector corresponding to various extracted feature indicators.

[0120] When performing the conversion operation, the standardization conversion unit first selects the corresponding conversion strategy based on the recognition results of the data format recognition unit. Then, it calls the pattern extraction unit to obtain the original pattern definition of the data. Finally, it maps the original data content to the standard representation format according to the selected conversion strategy. The standardization conversion unit also provides a conversion quality assessment function, comparing the degree of data information retention before and after conversion, calculating the conversion fidelity score, and appending the score result as metadata to the conversion result.

[0121] Step S6, through the sequential execution of three stages—data format recognition, pattern extraction, and standardization conversion—completes the entire process from format recognition to standardized representation of various data assets. The unified recognition model for structured, semi-structured, and unstructured data provides a unified representation for all types of data assets, facilitating unified processing and analysis.

[0122] To facilitate understanding, we will use a real-world application scenario from a specific enterprise as an example. This enterprise has deployed a complex technical architecture that includes multiple heterogeneous data sources such as a MySQL relational database cluster, a MongoDB NoSQL database cluster, an HDFS distributed file system, and external third-party data service API interfaces. It uses the method provided in this invention to achieve unified management and inventory of its entire data asset portfolio.

[0123] In the first step, enterprise administrators add connection configuration information for each data source through the system configuration interface, including the address, port, authentication credentials, etc. of each data source. After the system starts, the data source type identification unit automatically parses the connection configuration information and identifies the type of each data source. The protocol conversion unit loads the corresponding protocol adapter plugin for each data source type based on the identification results, converting the native access protocols of various data sources into a unified standardized data access protocol. The connection management unit establishes connection pools for each data source and maintains connection reuse and health status monitoring.

[0124] In the second step, the administrator configures the metadata scanning task and sets the execution plan. The metadata collection task scheduler triggers the scanning job according to the execution plan, with multiple worker threads concurrently connecting to each target data source. Metadata is collected through methods such as system table queries, application programming interface calls, and file system traversal. After the metadata collection is completed, the feature extraction module performs multi-dimensional analysis on the collected metadata, generates feature description vectors for each data asset, and stores them in the data asset feature library.

[0125] In the third step, the data lineage analysis module's data flow graph construction unit parses the configuration file of the data processing task, extracts the input data source and output data target of the data processing task, constructs the association relationship between data nodes and data edges, and forms a data flow graph. The link tracing unit allows administrators to query the upstream source and downstream destination of any data asset.

[0126] In the fourth step, the API interface auto-detection and access adapter's interface detection unit connects to the service registry center, queries the list of registered third-party service instances, obtains the interface specification document, and parses the document content to extract interface metadata. The parameter parsing unit verifies the actual behavior of the interface, and the code generation unit generates the adapter code for the API interface based on the verified interface metadata and registers it to the protocol conversion unit of the heterogeneous data source unified adaptation layer.

[0127] In the fifth step, all data assets identified by the data asset auto-discovery engine are automatically registered to the unified global data asset directory through the asset registration unit. Administrators set category tags for various data assets through the category management unit, add custom tags for important data assets through the tag management unit, and the version management unit continuously tracks structural changes of each data asset and records version snapshots.

[0128] In the sixth step, the data format recognition unit determines the format type of the data sample to be identified based on the file extension or the structural features of the data content. The pattern extraction unit executes the corresponding pattern extraction strategy for different format types. The standardization conversion unit converts each type of data to a standard representation format according to its pattern information, thereby realizing the unified recognition and standardization conversion of multiple types of data.

[0129] After executing the above six steps in sequence, the enterprise completed the entire process from unified access to heterogeneous data sources to full-domain data asset inventory, lineage analysis, automated API interface access, unified directory management, and unified identification of multiple types of data, realizing automated inventory and standardized management of full-domain data assets.

[0130] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, the embodiments should be regarded as exemplary and non-limiting in all respects.

[0131] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for automatic inventory and intelligent value quantification of data assets across heterogeneous systems, characterized in that, include: A unified adaptation layer architecture for heterogeneous data sources is constructed, which shields the differences in access to underlying data sources by defining standardized data access interfaces; An automatic data asset discovery engine is built based on metadata scanning. The automatic data asset discovery engine automatically traverses and queries data storage nodes in heterogeneous environments through the standardized data access interface, collects metadata including table structure, file mode and data dictionary, and automatically identifies data assets based on the collected metadata. A data lineage analysis and link tracing module is constructed. This module parses data processing tasks, extracts data input and output relationships from the data processing tasks, and constructs a data flow graph to characterize the relationships and dependency links between data assets.

2. The method according to claim 1, characterized in that, The construction of a unified adaptation layer architecture for heterogeneous data sources includes: When the data source type identification unit identifies the target data source as an unknown type not found in the feature fingerprint database, it marks the connection configuration information of the target data source as a type to be defined. The data asset auto-discovery engine still performs metadata scanning on the target data source marked as undefined type through the standardized data access interface, and performs correlation analysis between the collected metadata and the metadata of known type data assets to infer the position of the target data source marked as undefined type in the data flow map.

3. The method according to claim 1, characterized in that, The data asset automatic discovery engine built based on metadata scanning specifically includes: The data asset auto-discovery engine extracts features from semantic, structural, and quality dimensions based on the collected metadata, and generates feature description vectors for data assets. The extraction of semantic dimension features involves using a pre-trained Chinese language representation model to convert data asset names and field annotations into semantic vectors, calculating the cosine similarity between the semantic vectors and each topic tag vector in a preset topic tag vector library, and determining the topic tag with the highest cosine similarity as the semantic tag of the data asset.

4. The method according to claim 3, characterized in that, The automatic data asset discovery engine stores the generated data asset feature description vectors into a data asset feature library. The data asset feature library uses an inverted index structure to store each data asset feature description vector to support feature similarity-based retrieval and clustering analysis.

5. The method according to claim 1, characterized in that, The data lineage analysis and link tracing module includes: The data flow graph construction unit includes parsing, extraction, transformation, and loading jobs, structured query language query scripts, and data synchronization tasks, to extract the data flow mapping from the source to the target. In the data flow graph, data nodes represent data asset instances, data edges represent data processing task instances, and the direction of the data edges indicates the direction of data flow.

6. The method according to claim 5, characterized in that, The data lineage analysis and link tracing module also includes: The impact analysis unit receives a data change request containing the identifier of the data asset to be changed, and determines all downstream data nodes of the data asset to be changed based on the data flow map. The impact analysis unit calculates the impact level score for each downstream data node based on the number of data edge hops between the downstream data node and the data asset to be changed, the proportion of the number of fields involved in the change to the total number of fields in the downstream data node, and the time interval between the change trigger time and the most recent consumption time of the downstream data, and generates a comprehensive impact analysis report.

7. The method according to claim 1, characterized in that, The method further includes: The API interface auto-detection and access adapter function's interface detection unit automatically detects and extracts the service address, interface path, and request method of external API interfaces; The code generation unit dynamically generates data acquisition driver code adapted to the external API interface based on the input parameter definition and response data structure identified by the parameter parsing unit, using a template-driven code generation strategy. The generated data acquisition driver code is then registered to the protocol conversion unit of the heterogeneous data source unified adaptation layer architecture, so that the external API interface becomes a standard data source type that can be accessed by the data asset auto-discovery engine and the data lineage analysis and link tracing module through the standardized data access interface.

8. The method according to claim 1, characterized in that, The data lineage analysis and link tracing module also includes: According to the data flow map, the link tracing unit recursively traverses downstream data nodes along the data edge direction or recursively traverses upstream data nodes in the opposite direction along the data edge from the starting point of the data asset tracing, and collects the identification information of the data nodes and the data processing task instance information of the data nodes. Based on the data lineage analysis and link tracing results, the data lineage analysis and link tracing module automatically generates and labels behavioral tags that characterize the flow characteristics of data assets.

9. The method according to claim 1, characterized in that, The method further includes: A unified identification model for structured, semi-structured, and unstructured data is designed. When the data asset auto-discovery engine reads a file whose format cannot be identified by its file extension and header identifier, the data format identification unit in the unified identification model performs a deep analysis strategy: sampling the file content, statistically analyzing the frequency of delimiters and the depth of nesting levels in the file content, and comprehensively calculating the degree of structure to determine whether the file content is structured, semi-structured, or unstructured data.

10. The method according to claim 9, characterized in that, When the data format recognition unit executes the deep analysis strategy, the comprehensive calculation of the structuredness score is specifically as follows: Preset weighting coefficients are assigned to the frequency of the separator and the depth of the nesting level, respectively. A quantitative score is obtained by weighted calculation, and the data format type is determined based on the comparison result of the quantitative score and the preset threshold.

Images