A data asset model management method, device, equipment and medium facing the whole process

Abstract

This application relates to a method, apparatus, device, and medium for managing data asset models across the entire process, belonging to the field of data processing technology. The method includes: sequentially extracting metadata from multiple data sources of a target business system, constructing a data asset catalog, and accessing data assets based on metadata association and catalog binding to generate a full-process data asset flowchart showcasing the entire data asset access path; performing missing metadata prediction and completion using a dual algorithm based on data lineage logic reasoning and knowledge graph rule judgment; conducting a correctable, multi-stage, multi-round recursive quality assessment on the data asset flowchart after missing metadata prediction and completion; and simultaneously executing a three-layer parallel joint retrieval mechanism based on the quality assessment results to complete the joint search of multiple data assets and display high-quality retrieval results. This method enables efficient and accurate management of the data asset model of the target business system.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus, equipment and medium for managing data asset models across the entire process. Background Technology

[0002] With the deep integration and development of information technology, multi-source, heterogeneous, and highly dynamic data resources have become key assets supporting the efficient operation of various complex business scenarios. In specific fields, such as command and control scenarios with extremely high requirements for real-time performance and accuracy, the rapid flow of data, high-quality governance, and intelligent application level directly determine the execution efficiency and final results of core businesses. Therefore, building a data platform for specific fields to achieve full lifecycle management of data assets from "collection and aggregation, governance and integration to analysis and application" has become an important technical path to improve overall business responsiveness.

[0003] Against this backdrop, data asset management, as the cornerstone of a data platform, centers on building a dedicated data asset model adapted to the characteristics of the domain, enabling accurate description and closed-loop management of data across the entire domain. Currently, in the general commercial field, technical solutions such as data lakes and big data platforms have established relatively mature basic capabilities in data storage and batch processing. However, when these existing technologies are directly applied to business domains with special requirements (such as data platform scenarios requiring high real-time performance and high reliability), significant limitations are exposed, mainly in the following aspects:

[0004] Insufficient real-time response and processing capabilities: While existing data lake technologies can store massive amounts of raw data, data retrieval requires complex extraction, cleaning, and standardization processes, introducing significant latency and making it difficult to meet the instantaneous decision-making needs of business scenarios. Furthermore, mainstream big data platform technologies focus on batch data processing models, and their support for real-time data stream processing and analysis is still inadequate. Existing technologies generally lack dynamic priority scheduling mechanisms for data tasks, which can easily lead to delays in the flow of critical business information during resource contention, affecting overall timeliness.

[0005] High barriers to entry and low efficiency in data application: The raw data stored in data lakes is diverse in form and semantically ambiguous, making it extremely difficult for business analysts who are not professional data engineers to understand, discover, and directly use the data, severely restricting the efficiency of business application of data assets. The technology stack of big data platforms usually relies on professional programming for data processing, which is complex to operate and has high maintenance costs, hindering the rapid empowerment and popularization of data capabilities.

[0006] Weak proactive governance and quality assurance: Without rigorous governance strategies, data lakes can easily degenerate into unmanageable and unusable "data swamps," significantly increasing the difficulty of data screening and value extraction. While existing big data platforms possess powerful storage and computing capabilities, they have shortcomings in core capabilities such as data quality measurement, lineage tracing, and proactive governance. Of particular note is that general data quality assessment dimensions (such as completeness and consistency) fail to cover the core requirements of specific business domains for data accuracy, timeliness, and reliability. If poor-quality data is not effectively identified and intercepted, it may directly impact the business layer, triggering decision-making risks.

[0007] In summary, general data management technologies are insufficient to directly meet the stringent requirements of domain-specific data platforms for end-to-end asset model management. Therefore, there is an urgent need to research and construct a new data asset management methodology and technology system that deeply integrates domain knowledge, possesses real-time responsiveness, low barriers to entry, and strong governance capabilities, in order to overcome existing bottlenecks and fully unleash the core value of data assets. Summary of the Invention

[0008] Therefore, it is necessary to provide a data asset model management method, device, equipment, and medium for the entire process to address the above-mentioned technical problems.

[0009] A data asset model management method for the entire process, the method comprising:

[0010] Step 1, Data Asset Metadata Collection: By sequentially extracting metadata from multiple data sources of the target business system, constructing a data asset catalog, and accessing data assets based on metadata association and catalog binding, a full-process diagram of data assets is generated to show the entire process path of data asset access.

[0011] Step 2: Dynamically simulate and predict the entire data asset flowchart: For missing metadata in the preceding, intermediate, or subsequent nodes of the data asset flowchart, two algorithms are used to predict and fill in the missing metadata. The algorithms include logical reasoning based on existing data lineage clues and rule judgment based on knowledge accumulated in the knowledge graph from historical information.

[0012] Step 3: Modifiable Multi-Stage, Multi-Round Recursive Quality Assessment: Responding to the full lifecycle regulatory requirements of data assets in the target business system, based on a phased quality assessment logic, phased assessments are conducted on the convergence, governance, data analysis, and data application stages within the full-process path displayed in the data asset full-process diagram after missing metadata prediction and supplementation. Data quality test cases are generated for each stage based on data semantics to conduct data quality assessments. During the assessment process, test cases are run and modified in multiple rounds to dynamically adjust the constraints between test cases until the quality assessment results are output. Data quality is quantitatively calculated based on the quality assessment results, and quality level labels are assigned, generating a full-process data asset quality analysis report.

[0013] Step 4, Joint Search Based on Quality Assessment: In response to the data asset search needs of the target business system, based on the full process diagram of the data assets after missing metadata prediction and completion, the joint search of multiple data assets is carried out simultaneously by global search based on vertical distributed search engine, word segmentation search based on semantic understanding, and weighted search based on relevance. The search results are optimized and filtered according to the quality level tags generated by the quality assessment of the entire process path, and high-quality search results are given priority to users.

[0014] In one embodiment, metadata extraction includes:

[0015] By configuring data source connection information and metadata extraction tasks for the target business system, metadata extraction and updating from multiple data sources can be performed automatically. Among them, the multiple data sources that support metadata extraction include relational databases, non-relational databases, message queues, and file storage systems.

[0016] The extraction process includes:

[0017] Identify and parse metadata from multiple data sources, and automatically obtain the structure and format of the metadata;

[0018] The acquired metadata is stored in a central metadata repository, and a visual metadata management page is configured to provide query, update, and delete operations for the metadata.

[0019] In one embodiment, the data asset catalog construction includes:

[0020] A first-level directory is planned for data assets, and a data asset catalog is constructed based on the first-level directory and its sub-directories. The first-level directory includes a front-end library, a source library, an integration library, and an application library. The sub-directories of the front-end library are the database table classification directories before the data assets are introduced and aggregated. The sub-directories of the source library are the data source systems and business data classification directories after the data assets are introduced and aggregated. The sub-directories of the integration library are the business data classification directories after the data assets are governed and integrated. The sub-directories of the application library are the application data classification directories after the data assets are analyzed and applied.

[0021] In one embodiment, data asset access based on metadata association and directory binding includes:

[0022] Data asset ingestion and aggregation includes identifying data assets stored in multiple data sources in the target business system and extracting and aggregating digital assets.

[0023] The governance and integration of data assets includes quality control of extracted and aggregated data assets, and loading the quality-controlled data assets into the storage system of the data asset management platform;

[0024] The analysis and application of data assets includes associating metadata and binding data asset catalogs for each data asset within the storage system, configuring data service interfaces, and calling data assets for analysis and application based on the data service interfaces.

[0025] In one embodiment, a full-process diagram for generating data assets is constructed to illustrate the entire process path for data asset access, including:

[0026] Static analysis technology based on SQL parsing is used to parse the DML statements involved in the data asset access process into an abstract syntax tree; among which, SQL parsing includes lexical analysis, syntax analysis and semantic analysis.

[0027] Traverse the abstract syntax tree to extract the metadata information it contains, including tables and fields;

[0028] The extracted metadata information is analyzed to obtain the relationships and flow paths between data assets. By using the data asset catalogs bound to the data assets as nodes and the relationships and flow paths between the data assets as lines, a full flow diagram of data assets is generated to show the entire flow path of data asset access through the front-end library, the original library, the integration library and finally the application library.

[0029] In one embodiment, data quality test cases are generated at each stage based on data semantics to perform data quality assessment. During the assessment process, the constraints between test cases are dynamically adjusted through multiple rounds of recursive execution and modification until the assessment ends and the quality assessment results are output, including:

[0030] Data quality test cases for each stage are generated based on data semantics, and the constraints between test cases are initialized. The test cases include test cases focusing on integrity data quality assessment in the induction and convergence stage, test cases focusing on consistency data quality assessment in the governance and integration stage, test cases focusing on accuracy data quality assessment in the data analysis stage, and test cases focusing on usability data quality assessment in the data application stage. The constraints include dependency or mutual exclusion.

[0031] Using a task scheduler, test cases for each stage are run recursively in multiple rounds. If the constraints between test cases are mutually exclusive, the content and constraints of the test cases are adjusted in the next round of testing.

[0032] After multiple rounds of testing to achieve the preset test case coverage requirements, the data quality assessment ends and outputs the quality assessment results for each stage.

[0033] In one embodiment, based on the data asset end-to-end flowchart after missing metadata prediction, a joint search of multiple data assets is simultaneously performed using a vertically distributed search engine-based global retrieval, semantic understanding-based word segmentation retrieval, and relevance-based weighted retrieval, including:

[0034] Perform global retrieval based on a vertically distributed search engine. The vertically distributed search engine uses the Elasticsearch search engine, which has distributed, master-slave sharding and replication mechanisms, to achieve global retrieval coverage of multiple data assets in the entire data asset process diagram after missing metadata prediction and completion.

[0035] Perform semantic understanding-based word segmentation retrieval, process long and difficult sentences in the retrieval request using grammatical word segmentation algorithm, identify uncommon words in the retrieval request using dictionary word segmentation algorithm to achieve accurate word segmentation, and embed the word segmentation results of the retrieval request into the query component of the Elasticsearch search engine for multi-data asset retrieval;

[0036] Perform relevance-based weighted retrieval. During the retrieval request parsing process, assign high relevance weights to the key fields obtained from the parsing, and embed the weighted retrieval request into the query component of the Elasticsearch search engine for multi-data asset retrieval.

[0037] Finally, by combining global search results, word segmentation search results, and weighted search results, a joint search of multiple data assets can be achieved.

[0038] A data asset model management device for the entire process, the device comprising:

[0039] The data asset metadata acquisition module is used to extract metadata from multiple data sources of the target business system, build a data asset catalog, and access data assets based on metadata association and catalog binding, thereby generating a full-process diagram of data assets to show the entire process path of data asset access.

[0040] The Dynamic Simulation and Prediction Data Asset Flowchart module is used to predict and fill in missing metadata in the preceding, intermediate, or subsequent nodes of the data asset flowchart using two algorithms. The algorithms include logical reasoning based on existing data lineage clues and rule judgment based on knowledge accumulated in the knowledge graph from historical information.

[0041] The modifiable, multi-stage, multi-round recursive quality assessment module is designed to respond to the full lifecycle regulatory requirements of data assets in target business systems. Based on a phased quality assessment logic, it performs phased assessments of the convergence, governance, data analysis, and data application stages within the data asset full-process diagram after missing metadata prediction and supplementation. It generates data quality test cases for each stage based on data semantics to conduct data quality assessments. During the assessment process, it dynamically adjusts the constraints between test cases through multiple rounds of recursive execution and correction until the quality assessment results are output. Based on the quality assessment results, it performs quantitative calculations of data quality, assigns quality level labels, and generates a full-process data asset quality analysis report.

[0042] The joint retrieval module based on quality assessment is used to respond to the data asset search needs of the target business system. Based on the data asset full-process diagram after missing metadata prediction and completion, it simultaneously performs global retrieval based on vertical distributed search engine, word segmentation retrieval based on semantic understanding, and weighted retrieval based on relevance to achieve joint search of multiple data assets. Based on the quality level tags generated by the full-process path quality assessment, the retrieval results are optimized and filtered to prioritize the display of high-quality retrieval results to users.

[0043] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:

[0044] Step 1, Data Asset Metadata Collection: By sequentially extracting metadata from multiple data sources of the target business system, constructing a data asset catalog, and accessing data assets based on metadata association and catalog binding, a full-process diagram of data assets is generated to show the entire process path of data asset access.

[0045] Step 2: Dynamically simulate and predict the entire data asset flowchart: For missing metadata in the preceding, intermediate, or subsequent nodes of the data asset flowchart, two algorithms are used to predict and fill in the missing metadata. The algorithms include logical reasoning based on existing data lineage clues and rule judgment based on knowledge accumulated in the knowledge graph from historical information.

[0046] Step 3: Modifiable Multi-Stage, Multi-Round Recursive Quality Assessment: Responding to the full lifecycle regulatory requirements of data assets in the target business system, based on a phased quality assessment logic, phased assessments are conducted on the convergence, governance, data analysis, and data application stages within the full-process path displayed in the data asset full-process diagram after missing metadata prediction and supplementation. Data quality test cases are generated for each stage based on data semantics to conduct data quality assessments. During the assessment process, test cases are run and modified in multiple rounds to dynamically adjust the constraints between test cases until the quality assessment results are output. Data quality is quantitatively calculated based on the quality assessment results, and quality level labels are assigned, generating a full-process data asset quality analysis report.

[0047] Step 4, Joint Search Based on Quality Assessment: In response to the data asset search needs of the target business system, based on the full process diagram of the data assets after missing metadata prediction and completion, the joint search of multiple data assets is carried out simultaneously by global search based on vertical distributed search engine, word segmentation search based on semantic understanding, and weighted search based on relevance. The search results are optimized and filtered according to the quality level tags generated by the quality assessment of the entire process path, and high-quality search results are given priority to users.

[0048] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0049] Step 1, Data Asset Metadata Collection: By sequentially extracting metadata from multiple data sources of the target business system, constructing a data asset catalog, and accessing data assets based on metadata association and catalog binding, a full-process diagram of data assets is generated to show the entire process path of data asset access.

[0050] Step 2: Dynamically simulate and predict the entire data asset flowchart: For missing metadata in the preceding, intermediate, or subsequent nodes of the data asset flowchart, two algorithms are used to predict and fill in the missing metadata. The algorithms include logical reasoning based on existing data lineage clues and rule judgment based on knowledge accumulated in the knowledge graph from historical information.

[0051] Step 3: Modifiable Multi-Stage, Multi-Round Recursive Quality Assessment: Responding to the full lifecycle regulatory requirements of data assets in the target business system, based on a phased quality assessment logic, phased assessments are conducted on the convergence, governance, data analysis, and data application stages within the full-process path displayed in the data asset full-process diagram after missing metadata prediction and supplementation. Data quality test cases are generated for each stage based on data semantics to conduct data quality assessments. During the assessment process, test cases are run and modified in multiple rounds to dynamically adjust the constraints between test cases until the quality assessment results are output. Data quality is quantitatively calculated based on the quality assessment results, and quality level labels are assigned, generating a full-process data asset quality analysis report.

[0052] Step 4, Joint Search Based on Quality Assessment: In response to the data asset search needs of the target business system, based on the full process diagram of the data assets after missing metadata prediction and completion, the joint search of multiple data assets is carried out simultaneously by global search based on vertical distributed search engine, word segmentation search based on semantic understanding, and weighted search based on relevance. The search results are optimized and filtered according to the quality level tags generated by the quality assessment of the entire process path, and high-quality search results are given priority to users.

[0053] The aforementioned data asset model management method, apparatus, equipment, and media for the entire process firstly extract metadata from multiple data sources in the target business system, construct a data asset catalog, and access data assets sequentially. This generates a complete data asset process diagram, forming a closed-loop data asset management path from "introduction and aggregation" to "governance and integration" and then to "analysis and application." This provides a unified view foundation for data asset model management, facilitating an understanding of the lineage and flow of data assets at each stage and improving the responsiveness of data asset business applications. Then, through the collaborative algorithm of data lineage logical reasoning and knowledge graph rule judgment, accurate prediction and completion of missing metadata are achieved, ensuring the accuracy of metadata prediction and completion. The system prioritizes the integrity, consistency, and business adaptability of the data system. Secondly, by conducting phased, multi-round recursive quality assessments throughout the entire data asset access process, it can not only identify data quality issues but also locate and track them, providing detailed rectification suggestions for data governance to improve data quality and enhance the value of data assets. Finally, based on the quality assessment, a three-layer parallel joint retrieval mechanism integrates the advantages of global scanning, semantic understanding, and relevance weighting technologies. This enables not only rapid location of current data assets but also accurate discovery of high-value, high-quality data, significantly improving the management efficiency and ease of use of data assets and providing strong support for their full utilization. Attached Figure Description

[0054] Figure 1This is a flowchart illustrating a data asset model management method for the entire process in one embodiment;

[0055] Figure 2 This is a schematic diagram of an abstract syntax tree in one embodiment;

[0056] Figure 3 This is a visualization and standardized representation of the data asset workflow in one embodiment;

[0057] Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0059] In one embodiment, such as Figure 1 As shown, a data asset model management method for the entire process is provided, including the following steps:

[0060] Step 1: Data Asset Metadata Collection: By sequentially extracting metadata from multiple data sources of the target business system, constructing a data asset catalog, and accessing data assets based on metadata association and catalog binding, a full-process diagram of data assets is generated to show the entire process path of data asset access.

[0061] Step 2: Dynamically simulate and predict the entire data asset flowchart: For missing metadata in the preceding, intermediate, or subsequent nodes of the data asset flowchart, two algorithms are used to predict and fill in the missing metadata. The algorithms include logical reasoning based on existing data lineage clues and rule judgment based on knowledge accumulated in the knowledge graph from historical information.

[0062] Among these, logical reasoning based on existing data lineage clues refers to deducing the attributes or content of missing metadata by analyzing the lineage relationships of metadata already extracted from the target business system. Rule-based judgment based on knowledge accumulated in a knowledge graph from historical information refers to accumulating historically extracted metadata from the target business system into a knowledge graph, and then judging and supplementing missing metadata by matching rules in the knowledge graph.

[0063] Step 3: Modifiable Multi-Stage, Multi-Round Recursive Quality Assessment: Responding to the full lifecycle regulatory requirements of data assets in the target business system, based on a phased quality assessment logic, phased assessments are conducted on the convergence, governance, data analysis, and data application phases within the full-process path displayed in the data asset full-process diagram after missing metadata prediction and supplementation. Data quality test cases are generated for each phase based on data semantics to conduct data quality assessments. During the assessment process, test cases are run and modified in multiple rounds to dynamically adjust the constraints between test cases until the quality assessment results are output. Data quality is quantitatively calculated based on the quality assessment results, and quality level labels are assigned, generating a full-process data asset quality analysis report.

[0064] Step 4, Joint Search Based on Quality Assessment: In response to the data asset search needs of the target business system, based on the full process diagram of the data assets after missing metadata prediction and completion, the joint search of multiple data assets is carried out simultaneously by global search based on vertical distributed search engine, word segmentation search based on semantic understanding, and weighted search based on relevance. The search results are optimized and filtered according to the quality level tags generated by the quality assessment of the entire process path, and high-quality search results are given priority to users.

[0065] The aforementioned end-to-end data asset model management method firstly extracts metadata from multiple data sources in the target business system, constructs a data asset catalog, and integrates data assets. This generates a complete data asset workflow diagram, forming a closed-loop data asset management path from "introduction and aggregation" to "governance and integration" and then to "analysis and application." This provides a unified view foundation for data asset model management, facilitating an understanding of the lineage and flow of data assets at each stage and improving the responsiveness of data asset business applications. Then, through the collaborative use of dual algorithms—data lineage logical reasoning and knowledge graph rule judgment—accurate prediction and completion of missing metadata are achieved, ensuring the metadata system's integrity. First, it ensures the integrity, consistency, and business adaptability of data assets. Second, by conducting phased, multi-round recursive quality assessments throughout the entire data asset access process, it can not only identify data quality issues but also locate and track these issues, providing detailed rectification suggestions for data governance to improve data quality and enhance the value of data assets. Finally, based on the quality assessment, a three-layer parallel joint retrieval mechanism is used, integrating the advantages of global scanning, semantic understanding, and relevance weighting technologies. This enables not only the rapid location of current data assets but also the accurate discovery of high-value, high-quality data, significantly improving the management efficiency and ease of use of data assets and providing strong support for their full utilization.

[0066] In one embodiment, metadata extraction includes:

[0067] By configuring data source connection information and metadata extraction tasks for the target business system, the system automatically extracts and updates metadata from multiple data sources. These supported data sources include relational databases, non-relational databases, message queues, and file storage systems. Specifically, the extraction process includes: first, identifying and parsing metadata from multiple data sources to automatically obtain the metadata's structure and format; second, storing the acquired metadata in a central metadata repository and providing query, update, and delete operations through a configured visual metadata management page to meet the needs of different business scenarios.

[0068] It should be understood that metadata is data that describes data assets and plays a vital role in the organization, management, and utilization of those assets. Metadata extraction technology can automatically extract metadata from data sources through simple configuration, greatly reducing the complexity and time cost of manual operations and ensuring the accuracy and real-time nature of metadata extraction.

[0069] In one embodiment, the data asset catalog construction includes: planning a first-level catalog for data assets, and constructing a data asset catalog based on the first-level catalog and its subordinate catalogs; wherein, the first-level catalog includes a pre-positioning library, a raw library, an integration library, and an application library; the subordinate catalogs of the pre-positioning library are the catalogs of database tables before data assets are introduced and aggregated; the subordinate catalogs of the raw library are the catalogs of data source systems and business data after data assets are introduced and aggregated; the subordinate catalogs of the integration library are the catalogs of business data after data asset governance and integration; and the subordinate catalogs of the application library are the catalogs of application data such as thematic libraries, interfaces, and models after data asset analysis and application.

[0070] It should be understood that a data asset catalog is built upon automated metadata extraction and user-defined classification rules. Cataloging data assets enables orderly organization and convenient management. Each data asset catalog can access multiple metadata-based data assets. For example, each table and its data in a relational database is a data asset, and each file and its information in a file storage system is a data asset.

[0071] In one embodiment, data asset access based on metadata association and directory binding includes:

[0072] (1) Data asset ingestion and aggregation, including identifying data assets stored in multiple data sources in the target business system and extracting and aggregating digital assets.

[0073] (2) Governance and integration of data assets, including quality control of the extracted and aggregated data assets, including cleaning, transformation and normalization, and loading the quality-controlled data assets into the storage system of the data asset management platform, such as a distributed file system or database.

[0074] (3) Data asset analysis and application, including metadata association and data asset catalog binding for each data asset in the storage system, and configuration of data service interfaces, and calling data assets for analysis and application based on the data service interfaces. Among them, metadata association, such as associating data name, data source, structure, format, purpose, etc., facilitates user understanding and use. Metadata association and data asset catalog binding enable users to easily discover and access the required data assets.

[0075] Specifically, the main technologies and commonly used tools adopted in each stage of data asset access are shown in Table 1:

[0076] Table 1. Key Technologies and Commonly Used Tools at Each Stage of Data Asset Access

[0077]

[0078] In one embodiment, a full-process diagram for generating data assets is constructed to illustrate the entire process path for data asset access, including:

[0079] First, static analysis technology based on SQL parsing is employed to parse the DML (Data Manipulation Language) statements involved in the data asset access process into an abstract syntax tree. SQL parsing includes lexical analysis, syntax analysis, and semantic analysis. DML statements refer to SQL statements used to manipulate data in the database, including data queries, data insertions, data updates, and data deletions. The abstract syntax tree is an abstract representation of SQL statements; it displays the complete SQL syntax structure through a tree structure. By traversing the abstract syntax tree, syntactic elements in the SQL statements can be analyzed and extracted. An example of an abstract syntax tree for an SQL statement is shown below. Figure 2 As shown. Figure 2Shows the syntactic structure analysis of the SQL query statement "SELECT id, age FROM t_user WHERE name='Zhang San';". This statement is decomposed into lexical units of different categories through lexical analysis: Keywords include SELECT, FROM, WHERE; Identifiers include id, age, t_user, name; Operator is the equal sign (=); Literal is the string Zhang San; Delimiters include commas and semicolons. In the syntactic analysis stage, these units are organized into an Abstract Syntax Tree (AST) rooted at root, whose structure includes a SELECT clause (SELECT Clause) and its query items (ITEMS) id and age, a FROM clause (FROM Clause) and its source table (TABLES) t_user, and a WHERE clause (WHERE Clause) and its filtering condition (CONDITIONS) "name = 'Zhang San'". This tree structure clearly presents the syntactic hierarchy of the SQL statement and provides a basis for subsequent semantic analysis and query processing.

[0080] Secondly, traverse the abstract syntax tree and extract the metadata information contained therein, including elements such as tables and fields.

[0081] Finally, perform lineage analysis on the extracted metadata information to obtain the relationships and transfer paths between data assets. By taking the data asset catalogs bound to the data assets as nodes and the relationships and transfer paths between the data assets as connections, a full-flow chart of data assets is constructed and generated to show the full-flow path of data assets accessing and flowing through the pre-library, original library, integration library, and application library. The visualization effect and standardized expression of the full-flow chart of data assets are as Figure 3As shown, the standardized representation uses a standardized JSON array structure. This array contains multiple node objects, each of which fully defines a data node through a set of standardized fields: `id` is the unique identifier of the node; `nodeName` and `assetsName` represent the node name and asset name, respectively; `nodeType` and `platformType` distinguish the node type and platform type; `contextPath` specifies the context path; `assetsId` is the data asset ID; and `preId` and `nextId` clarify the flow order between nodes through their successor relationship, forming a complete link (such as the coherent path from front-end library 1 to application library table 1 in the example). This structure, in a machine-readable standardized form, clearly expresses the topological relationships and metadata information of data assets across multiple processing stages (such as front-end library, original library, integration library, and application library), providing a structured foundation for data lineage tracing and end-to-end visualization.

[0082] It should be understood that a full-process diagram of data assets can help users intuitively understand the relationships and flow paths between data assets, quickly obtain the required data assets and their lineage, help users take stock of the data assets of the target business system, understand the entire process of data induction, aggregation, governance, integration, analysis and application, and comprehensively improve the value of data assets.

[0083] In one embodiment, data quality test cases are generated at each stage based on data semantics to perform data quality assessment. During the assessment process, the constraints between test cases are dynamically adjusted through multiple rounds of recursive execution and modification until the assessment ends and the quality assessment results are output, including:

[0084] Data quality test cases for each stage are generated based on data semantics, and the constraints between test cases are initialized. The test cases include test cases focusing on integrity data quality assessment in the induction and convergence stage, test cases focusing on consistency data quality assessment in the governance and integration stage, test cases focusing on accuracy data quality assessment in the data analysis stage, and test cases focusing on usability data quality assessment in the data application stage. The constraints include dependency or mutual exclusion.

[0085] Using a task scheduler, test cases for each stage are run recursively in multiple rounds. If the constraints between test cases are mutually exclusive, the content and constraints of the test cases are adjusted in the next round of testing.

[0086] After multiple rounds of testing to achieve the preset test case coverage requirements, the data quality assessment ends and outputs the quality assessment results for each stage.

[0087] In one embodiment, based on the data asset end-to-end flowchart after missing metadata prediction, a joint search of multiple data assets is simultaneously performed using a vertically distributed search engine-based global retrieval, semantic understanding-based word segmentation retrieval, and relevance-based weighted retrieval, including:

[0088] (1) Execute global retrieval based on a vertically distributed search engine. The vertically distributed search engine uses Elasticsearch (ES) with distributed, master-slave sharding and replication mechanisms to achieve global retrieval coverage of multiple data assets in the entire data asset process diagram after missing metadata prediction and completion. It should be understood that vertically distributed search engines, represented by ES, have the advantages of full-text search, data analysis and distributed storage in one. The distributed feature enables it to have millisecond-level response speed while ensuring retrieval accuracy when facing the storage and retrieval of massive amounts of data. The master-slave sharding and replication mechanism also makes it perform very well in terms of scalability and maintainability when facing a large amount of data. Because of the advantages of vertically distributed search engines in the professional search field, it is the best choice for analyzing and retrieving data assets. ES also supports hot updates of stop words, which means that the stop word library is dynamically updated in ES. The word library can be adjusted in real time without restarting the service. Traditional stop word libraries usually require modification of the configuration file and restarting the service to take effect, while hot updates automatically load the updated word library through external interfaces or scheduled tasks to ensure the timeliness and accuracy of search results. Elasticsearch is developed based on Java, and its underlying storage and retrieval are implemented based on the Lucene engine.

[0089] (2) Perform semantic understanding-based word segmentation retrieval. Long and complex sentences in the retrieval request are segmented using a grammatical word segmentation algorithm. Uncommon words in the retrieval request are identified using a dictionary-based word segmentation algorithm to achieve accurate word segmentation. The word segmentation results of the retrieval request are then embedded into the Elasticsearch search engine's query component for multi-data asset retrieval. It should be understood that semantic understanding-based word segmentation retrieval can accurately parse retrieval requests, support the transformation of retrieval requests, and improve the retrieval accuracy of the required data assets.

[0090] (3) Perform relevance-based weighted retrieval. During the retrieval request parsing process, assign high relevance weights to the key fields obtained from the parsing, and embed the weighted retrieval request into the query component of the Elasticsearch search engine for multi-data asset retrieval. It should be understood that relevance-based weighted retrieval avoids the situation where the traditional equal-weighted retrieval is prone to excessively biased retrieval results, and improves the recall rate of data asset retrieval. Among them, recall (R) represents the proportion of the text returned by the retrieval to all relevant text, and precision (P) represents the proportion of relevant text in the text returned by the retrieval. The concepts of P and R can be more intuitively understood through Table 2, so the calculation formula of P is: The formula for calculating R is: To resolve the case where P and R are mutually exclusive, their harmonic mean is defined as F, where... When P and R are made to have equal weights, At this point, there is a formula. .

[0091] Table 2. Concepts related to precision and recall

[0092]

[0093] Finally, by combining the global search results, word segmentation search results, and weighted search results, a joint search of multiple data assets is completed.

[0094] In one embodiment, a data asset model management device for the entire process is provided, comprising:

[0095] The data asset metadata acquisition module is used to extract metadata from multiple data sources of the target business system, build a data asset catalog, and access data assets based on metadata association and catalog binding, thereby generating a full-process diagram of data assets to show the entire process path of data asset access.

[0096] The Dynamic Simulation and Prediction Data Asset Flowchart module is used to predict and fill in missing metadata in the preceding, intermediate, or subsequent nodes of the data asset flowchart using two algorithms. The algorithms include logical reasoning based on existing data lineage clues and rule judgment based on knowledge accumulated in the knowledge graph from historical information.

[0097] The modifiable, multi-stage, multi-round recursive quality assessment module is designed to respond to the full lifecycle regulatory requirements of data assets in target business systems. Based on a phased quality assessment logic, it performs phased assessments of the convergence, governance, data analysis, and data application stages within the data asset full-process diagram after missing metadata prediction and supplementation. It generates data quality test cases for each stage based on data semantics to conduct data quality assessments. During the assessment process, it dynamically adjusts the constraints between test cases through multiple rounds of recursive execution and correction until the quality assessment results are output. Based on the quality assessment results, it performs quantitative calculations of data quality, assigns quality level labels, and generates a full-process data asset quality analysis report.

[0098] The joint retrieval module based on quality assessment is used to respond to the data asset search needs of the target business system. Based on the data asset full-process diagram after missing metadata prediction and completion, it simultaneously performs global retrieval based on vertical distributed search engine, word segmentation retrieval based on semantic understanding, and weighted retrieval based on relevance to achieve joint search of multiple data assets. Based on the quality level tags generated by the full-process path quality assessment, the retrieval results are optimized and filtered to prioritize the display of high-quality retrieval results to users.

[0099] Specific limitations regarding the end-to-end data asset model management device can be found in the limitations of the end-to-end data asset model management method described above, and will not be repeated here. Each module in the aforementioned end-to-end data asset model management device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0100] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 4As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a data asset model management method oriented towards the entire process. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0101] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0102] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to perform the following steps:

[0103] Step 1, Data Asset Metadata Collection: By sequentially extracting metadata from multiple data sources of the target business system, constructing a data asset catalog, and accessing data assets based on metadata association and catalog binding, a full-process diagram of data assets is generated to show the entire process path of data asset access.

[0104] Step 2: Dynamically simulate and predict the entire data asset flowchart: For missing metadata in the preceding, intermediate, or subsequent nodes of the data asset flowchart, two algorithms are used to predict and fill in the missing metadata. The algorithms include logical reasoning based on existing data lineage clues and rule judgment based on knowledge accumulated in the knowledge graph from historical information.

[0105] Step 3: Modifiable Multi-Stage, Multi-Round Recursive Quality Assessment: Responding to the full lifecycle regulatory requirements of data assets in the target business system, based on a phased quality assessment logic, phased assessments are conducted on the convergence, governance, data analysis, and data application stages within the full-process path displayed in the data asset full-process diagram after missing metadata prediction and supplementation. Data quality test cases are generated for each stage based on data semantics to conduct data quality assessments. During the assessment process, test cases are run and modified in multiple rounds to dynamically adjust the constraints between test cases until the quality assessment results are output. Data quality is quantitatively calculated based on the quality assessment results, and quality level labels are assigned, generating a full-process data asset quality analysis report.

[0106] Step 4, Joint Search Based on Quality Assessment: In response to the data asset search needs of the target business system, based on the full process diagram of the data assets after missing metadata prediction and completion, the joint search of multiple data assets is carried out simultaneously by global search based on vertical distributed search engine, word segmentation search based on semantic understanding, and weighted search based on relevance. The search results are optimized and filtered according to the quality level tags generated by the quality assessment of the entire process path, and high-quality search results are given priority to users.

[0107] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0108] Step 1, Data Asset Metadata Collection: By sequentially extracting metadata from multiple data sources of the target business system, constructing a data asset catalog, and accessing data assets based on metadata association and catalog binding, a full-process diagram of data assets is generated to show the entire process path of data asset access.

[0109] Step 2: Dynamically simulate and predict the entire data asset flowchart: For missing metadata in the preceding, intermediate, or subsequent nodes of the data asset flowchart, two algorithms are used to predict and fill in the missing metadata. The algorithms include logical reasoning based on existing data lineage clues and rule judgment based on knowledge accumulated in the knowledge graph from historical information.

[0110] Step 3: Modifiable Multi-Stage, Multi-Round Recursive Quality Assessment: Responding to the full lifecycle regulatory requirements of data assets in the target business system, based on a phased quality assessment logic, phased assessments are conducted on the convergence, governance, data analysis, and data application stages within the full-process path displayed in the data asset full-process diagram after missing metadata prediction and supplementation. Data quality test cases are generated for each stage based on data semantics to conduct data quality assessments. During the assessment process, test cases are run and modified in multiple rounds to dynamically adjust the constraints between test cases until the quality assessment results are output. Data quality is quantitatively calculated based on the quality assessment results, and quality level labels are assigned, generating a full-process data asset quality analysis report.

[0111] Step 4, Joint Search Based on Quality Assessment: In response to the data asset search needs of the target business system, based on the full process diagram of the data assets after missing metadata prediction and completion, the joint search of multiple data assets is carried out simultaneously by global search based on vertical distributed search engine, word segmentation search based on semantic understanding, and weighted search based on relevance. The search results are optimized and filtered according to the quality level tags generated by the quality assessment of the entire process path, and high-quality search results are given priority to users.

[0112] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0113] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0114] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application.

Claims

1. A data asset model management method for the entire process, characterized in that, The method includes: Step 1, Data Asset Metadata Collection: By sequentially extracting metadata from multiple data sources of the target business system, constructing a data asset catalog, and accessing data assets based on metadata association and catalog binding, a full-process diagram of data assets is generated to show the entire process path of data asset access. Step 2: Dynamically simulate and predict the entire data asset flowchart: For missing metadata in the preceding, intermediate, or subsequent nodes of the data asset flowchart, two algorithms are used to predict and fill in the missing metadata. The algorithms include logical reasoning based on existing data lineage clues and rule judgment based on knowledge accumulated in the knowledge graph from historical information. Step 3: Modifiable Multi-Stage, Multi-Round Recursive Quality Assessment: Responding to the full lifecycle regulatory requirements of data assets in the target business system, based on a phased quality assessment logic, phased assessments are conducted on the convergence, governance, data analysis, and data application stages within the full-process path displayed in the data asset full-process diagram after missing metadata prediction and supplementation. Data quality test cases are generated for each stage based on data semantics to conduct data quality assessments. During the assessment process, test cases are run and modified in multiple rounds to dynamically adjust the constraints between test cases until the quality assessment results are output. Data quality is quantitatively calculated based on the quality assessment results, and quality level labels are assigned, generating a full-process data asset quality analysis report. Step 4, Joint Search Based on Quality Assessment: In response to the data asset search needs of the target business system, based on the full process diagram of the data assets after missing metadata prediction and completion, the joint search of multiple data assets is carried out simultaneously by global search based on vertical distributed search engine, word segmentation search based on semantic understanding, and weighted search based on relevance. The search results are optimized and filtered according to the quality level tags generated by the quality assessment of the entire process path, and high-quality search results are given priority to users.

2. The data asset model management method for the entire process according to claim 1, characterized in that, The metadata extraction includes: By configuring data source connection information and metadata extraction tasks for the target business system, metadata extraction and updating from multiple data sources can be performed automatically. Among them, the multiple data sources that support metadata extraction include relational databases, non-relational databases, message queues, and file storage systems. The extraction process includes: Identify and parse metadata from multiple data sources, and automatically obtain the structure and format of the metadata; The acquired metadata is stored in a central metadata repository, and a visual metadata management page is configured to provide query, update, and delete operations for the metadata.

3. The data asset model management method for the entire process according to claim 2, characterized in that, The construction of the data asset catalog includes: A first-level directory is planned for the data assets, and a data asset directory is constructed based on the first-level directory and its sub-directories. The first-level directory includes a pre-positioning library, a raw library, an integration library, and an application library. The sub-directories of the pre-positioning library are the database table classification directories before the data assets are introduced and aggregated. The sub-directories of the raw library are the data source systems and business data classification directories after the data assets are introduced and aggregated. The sub-directories of the integration library are the business data classification directories after the data assets are governed and integrated. The sub-directories of the application library are the application data classification directories after the data assets are analyzed and applied.

4. The data asset model management method for the entire process according to claim 3, characterized in that, Data asset access based on metadata association and directory binding includes: Data asset ingestion and aggregation includes identifying data assets stored in multiple data sources in the target business system and extracting and aggregating digital assets. The governance and integration of data assets includes quality control of extracted and aggregated data assets, and loading the quality-controlled data assets into the storage system of the data asset management platform; The analysis and application of data assets includes associating metadata and binding data asset catalogs for each data asset within the storage system, configuring data service interfaces, and calling data assets for analysis and application based on the data service interfaces.

5. The data asset model management method for the entire process according to claim 4, characterized in that, Construct a full-process diagram for generating data assets to illustrate the entire process of data asset access, including: Static analysis technology based on SQL parsing is used to parse the DML statements involved in the data asset access process into an abstract syntax tree; wherein, the SQL parsing includes lexical analysis, syntax analysis and semantic analysis; Traverse the abstract syntax tree to extract the metadata information contained therein, including tables and fields; The extracted metadata information is analyzed to obtain the relationships and flow paths between data assets. By using the data asset catalogs bound to the data assets as nodes and the relationships and flow paths between the data assets as lines, a full flow diagram of data assets is generated to show the entire flow path of data asset access through the front-end library, the original library, the integration library and finally the application library.

6. The data asset model management method for the entire process according to claim 1, characterized in that, Data quality assessment is performed by generating data quality test cases for each stage based on data semantics. During the assessment process, test cases are dynamically adjusted through multiple rounds of recursive execution and modification until the assessment concludes and the quality assessment results are output, including: Data quality test cases for each stage are generated based on data semantics, and the constraints between the test cases are initialized. The test cases include test cases focusing on integrity data quality assessment in the induction and convergence stage, test cases focusing on consistency data quality assessment in the governance and integration stage, test cases focusing on accuracy data quality assessment in the data analysis stage, and test cases focusing on usability data quality assessment in the data application stage. The constraints include dependencies or mutual exclusions. Using a task scheduler, test cases for each stage are run recursively in multiple rounds. If the constraints between test cases are mutually exclusive, the content and constraints of the test cases are adjusted in the next round of testing. After multiple rounds of testing to achieve the preset test case coverage requirements, the data quality assessment ends and outputs the quality assessment results for each stage.

7. The data asset model management method for the entire process according to claim 1, characterized in that, Based on the complete data asset workflow diagram after missing metadata prediction, simultaneous execution of global retrieval based on a vertically distributed search engine, word segmentation retrieval based on semantic understanding, and relevance-based weighted retrieval enables joint search of multiple data assets, including: Perform a global search based on a vertically distributed search engine, which uses the Elasticsearch search engine with distributed, master-slave sharding and replication mechanisms to achieve global search coverage of multiple data assets in the entire process diagram of data assets after missing metadata prediction and completion. Perform semantic understanding-based word segmentation retrieval, process long and difficult sentences in the retrieval request using grammatical word segmentation algorithm, identify uncommon words in the retrieval request using dictionary word segmentation algorithm to achieve accurate word segmentation, and embed the word segmentation results of the retrieval request into the query component of the Elasticsearch search engine for multi-data asset retrieval; Perform relevance-based weighted retrieval. During the retrieval request parsing process, assign high relevance weights to the key fields obtained from the parsing, and embed the weighted retrieval request into the query component of the Elasticsearch search engine for multi-data asset retrieval. Finally, by combining global search results, word segmentation search results, and weighted search results, a joint search of multiple data assets can be achieved.

8. A data asset model management device for the entire process, characterized in that, The device includes: The data asset metadata acquisition module is used to extract metadata from multiple data sources of the target business system, build a data asset catalog, and access data assets based on metadata association and catalog binding, thereby generating a full-process diagram of data assets to show the entire process path of data asset access. The Dynamic Simulation and Prediction Data Asset Flowchart module is used to predict and fill in missing metadata in the preceding, intermediate, or subsequent nodes of the data asset flowchart using two algorithms. The algorithms include logical reasoning based on existing data lineage clues and rule judgment based on knowledge accumulated in the knowledge graph from historical information. The modifiable, multi-stage, multi-round recursive quality assessment module is designed to respond to the full lifecycle regulatory requirements of data assets in target business systems. Based on a phased quality assessment logic, it performs phased assessments of the convergence, governance, data analysis, and data application stages within the data asset full-process diagram after missing metadata prediction and supplementation. It generates data quality test cases for each stage based on data semantics to conduct data quality assessments. During the assessment process, it dynamically adjusts the constraints between test cases through multiple rounds of recursive execution and correction until the quality assessment results are output. Based on the quality assessment results, it performs quantitative calculations of data quality, assigns quality level labels, and generates a full-process data asset quality analysis report. The joint retrieval module based on quality assessment is used to respond to the data asset search needs of the target business system. Based on the data asset full-process diagram after missing metadata prediction and completion, it simultaneously performs global retrieval based on vertical distributed search engine, word segmentation retrieval based on semantic understanding, and weighted retrieval based on relevance to achieve joint search of multiple data assets. Based on the quality level tags generated by the full-process path quality assessment, the retrieval results are optimized and filtered to prioritize the display of high-quality retrieval results to users.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images