Full-process-oriented data asset model management method and device, equipment and medium

Abstract

The invention relates to a full-process-oriented data asset model management method and device, equipment and a medium, and belongs to the technical field of data processing. The method comprises the following steps: sequentially performing metadata extraction, data asset directory construction and data asset access based on metadata association and directory binding on multiple data sources of a target business system, and constructing and generating a data asset full-flow chart to display a full-flow path of data asset access; predicting and supplementing missing metadata based on the double algorithms of data consanguinity logical reasoning and knowledge graph rule judgment; performing correctable multi-stage multi-round recursive quality evaluation on the data asset full-flow chart after missing metadata prediction and completion; and synchronously executing a three-layer parallel joint retrieval mechanism based on a quality evaluation result to complete joint search of multiple data assets and high-quality retrieval result display. By adopting the method, efficient and accurate management of the data asset model of the target business system can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a full-process-oriented data asset model management method, device, equipment and medium. BACKGROUND

[0002] With the deep integration development of information technology, multi-source, heterogeneous and high-dynamic data resources have become the key assets to support the efficient operation of various complex business scenarios. In specific fields, such as command and control scenarios with extremely high requirements for real-time and accuracy, the rapid flow, high-quality management and intelligent application level of data directly determine the execution efficiency and final effectiveness of core business. Therefore, building a data middle platform oriented to specific fields to realize the full life cycle management of data assets from "collection, management, fusion to analysis and application" has become an important technical path to improve the overall business response capability.

[0003] Under this background, data asset management as the cornerstone of data middle platform, its core is to build a special data asset model that adapts to the characteristics of the field, to realize the accurate description and closed-loop control of global data. At present, in the general commercial field, the technical solutions represented by data lake and big data platform have formed mature basic capabilities in data storage and batch processing. However, when these existing technologies are directly applied to business fields with special requirements (such as high real-time and high-reliable data middle platform scenarios), significant limitations are exposed, mainly in the following aspects: Insufficient real-time response and processing capability: although the existing data lake technology can store massive raw data, the data needs to go through complex extraction, cleaning and standardization processes before being called, introducing high latency and making it difficult to meet the instantaneous decision-making needs of business scenarios. At the same time, mainstream big data platform technology focuses on batch data processing mode, and the support for real-time data stream processing and analysis is not perfect. The existing technology generally lacks a dynamic priority scheduling mechanism for data tasks, which may lead to key business information flow lagging behind in resource contention, affecting overall timeliness.

[0004] High data application threshold and low efficiency: the raw data stored in the data lake has diverse forms and unclear semantics, making it extremely difficult for non-professional data engineers to understand, discover and directly use the data, which seriously hinders the efficiency of business application of data assets. The technology stack of big data platform usually relies on professional programming for data processing, which is complex to operate and high in maintenance cost, hindering the rapid empowerment and popularization of data capabilities.

[0005] Active governance and weak quality assurance: Data lakes are prone to degenerate into "data swamps" that are difficult to manage and use without strict governance strategies, making data filtering and value extraction more difficult. Existing big data platforms have strong storage and computing capabilities, but they lack core capabilities such as data quality measurement, bloodline tracking, and active governance. In particular, the general data quality evaluation dimensions (such as completeness and consistency) do not cover the core requirements of data accuracy, timeliness, and credibility in specific business domains. Poor data that is not effectively identified and intercepted may directly affect the business layer and cause decision-making risks.

[0006] In summary, general data management techniques cannot directly meet the stringent requirements of specific domain data middle platforms for full-process asset model management. Therefore, it is necessary to research and build a new data asset management method and technical system that deeply integrates domain knowledge, has real-time response, low usage threshold, and strong governance capabilities, to break through existing bottlenecks and fully release the core value of data assets. SUMMARY

[0007] Therefore, it is necessary to provide a full-process-oriented data asset model management method, device, equipment and medium.

[0008] A full-process-oriented data asset model management method, the method comprising: Step 1, data asset metadata acquisition: by sequentially performing metadata extraction, data asset directory construction, and data asset access based on metadata association and directory binding on multiple data sources of a target business system, a data asset full-process diagram is constructed and generated to show the full-process path of data asset access; Step 2, dynamically simulating and predicting the data asset full-process diagram: for missing metadata in the pre-node, intermediate node, or post-node in the data asset process diagram, two algorithms are used for prediction and completion; the algorithms include logical reasoning based on existing data bloodline clues and rule judgment based on knowledge in the knowledge graph that is deposited from historical information; Step 3, multi-stage and multi-round recursive quality evaluation that can be modified: in response to the data asset full-life cycle monitoring needs of the target business system, based on the quality evaluation logic of different stages, the full-process path shown by the data asset full-process diagram after the missing metadata is predicted and completed is evaluated in stages, including the introduction, connection, convergence stage, governance integration stage, data analysis stage, and data application stage; and according to the data semantics, data quality test cases for each stage are generated for data quality evaluation. In the evaluation process, the constraint relationships between the test cases are dynamically adjusted through multiple rounds of recursive running and modification of the test cases until the evaluation ends and the quality evaluation result is output. According to the quality evaluation result, data quality quantification calculation is performed, and a quality level label is assigned, generating a data asset full-process quality analysis report; Step 4: Joint retrieval based on quality evaluation: In response to the data asset search demand of the target business system, based on the data asset whole process diagram after the missing metadata is filled in, the joint search of multiple data assets is implemented by synchronously performing global retrieval based on vertical distributed search engine, segmented retrieval based on semantic understanding, and weighted retrieval based on relevance, and the retrieval results are optimized and screened according to the quality level label generated by the whole process path quality evaluation, and the high-quality retrieval results are preferentially displayed to the user.

[0009] In one of the embodiments, the metadata extraction includes: By configuring the data source connection information and the metadata extraction task of the target business system, the metadata extraction and update of multiple data sources are automatically performed; wherein the multiple data sources supporting metadata extraction include relational databases, non-relational databases, message queues and file storage systems; The extraction process includes: The metadata in the multiple data sources is identified and parsed to automatically obtain the structure and format of the metadata; The obtained metadata is stored in the central metadata warehouse, and the metadata query, update and deletion operations are provided through the configuration of the visual metadata management page.

[0010] In one of the embodiments, the data asset directory construction includes: The data assets are planned in the first-level directory, and the data asset directory is constructed based on the first-level directory and the sub-directories thereof; wherein the first-level directory includes a pre-library, an original library, an integrated library and an application library; the sub-directories of the pre-library are the library table classification directories before the data asset connection and convergence; the sub-directories of the original library are the data source system and business data classification directories after the data asset connection and convergence; the sub-directories of the integrated library are the business data classification directories after the governance and fusion of the data assets; and the sub-directories of the application library are the application data classification directories after the analysis and application of the data assets.

[0011] In one of the embodiments, the data asset access based on metadata correlation and directory binding includes: Data asset connection and convergence, including identifying and extracting the data assets stored in the multiple data sources of the target business system; Governance and fusion of data assets, including quality control of the extracted and converged data assets, and loading the quality controlled data assets into the storage system of the data asset management platform; Analysis and application of data assets, including metadata correlation and data asset directory binding of each data asset in the storage system, and configuring a data service interface to call the data assets for analysis and application based on the data service interface.

[0012] In one embodiment, a full-process diagram for generating data assets is constructed to illustrate the entire process path for data asset access, including: Static analysis technology based on SQL parsing is used to parse the DML statements involved in the data asset access process into an abstract syntax tree; among which, SQL parsing includes lexical analysis, syntax analysis and semantic analysis. Traverse the abstract syntax tree to extract the metadata information it contains, including tables and fields; The extracted metadata information is analyzed to obtain the relationships and flow paths between data assets. By using the data asset catalogs bound to the data assets as nodes and the relationships and flow paths between the data assets as lines, a full flow diagram of data assets is generated to show the entire flow path of data asset access through the front-end library, the original library, the integration library and finally the application library.

[0013] In one embodiment, data quality test cases are generated at each stage based on data semantics to perform data quality assessment. During the assessment process, the constraints between test cases are dynamically adjusted through multiple rounds of recursive execution and modification until the assessment ends and the quality assessment results are output, including: Data quality test cases for each stage are generated based on data semantics, and the constraints between test cases are initialized. The test cases include test cases focusing on integrity data quality assessment in the induction and convergence stage, test cases focusing on consistency data quality assessment in the governance and integration stage, test cases focusing on accuracy data quality assessment in the data analysis stage, and test cases focusing on usability data quality assessment in the data application stage. The constraints include dependency or mutual exclusion. Using a task scheduler, test cases for each stage are run recursively in multiple rounds. If the constraints between test cases are mutually exclusive, the content and constraints of the test cases are adjusted in the next round of testing. After multiple rounds of testing to achieve the preset test case coverage requirements, the data quality assessment ends and outputs the quality assessment results for each stage.

[0014] In one embodiment, based on the data asset end-to-end flowchart after missing metadata prediction, a joint search of multiple data assets is simultaneously performed using a vertically distributed search engine-based global retrieval, semantic understanding-based word segmentation retrieval, and relevance-based weighted retrieval, including: Perform global retrieval based on a vertically distributed search engine. The vertically distributed search engine uses the Elasticsearch search engine, which has distributed, master-slave sharding and replication mechanisms, to achieve global retrieval coverage of multiple data assets in the entire data asset process diagram after missing metadata prediction and completion. Perform semantic understanding-based word segmentation retrieval, process long and difficult sentences in the retrieval request using grammatical word segmentation algorithm, identify uncommon words in the retrieval request using dictionary word segmentation algorithm to achieve accurate word segmentation, and embed the word segmentation results of the retrieval request into the query component of the Elasticsearch search engine for multi-data asset retrieval; Perform relevance-based weighted retrieval. During the retrieval request parsing process, assign high relevance weights to the key fields obtained from the parsing, and embed the weighted retrieval request into the query component of the Elasticsearch search engine for multi-data asset retrieval. Finally, by combining global search results, word segmentation search results, and weighted search results, a joint search of multiple data assets can be achieved.

[0015] A data asset model management device for the entire process, the device comprising: The data asset metadata acquisition module is used to extract metadata from multiple data sources of the target business system, build a data asset catalog, and access data assets based on metadata association and catalog binding, thereby generating a full-process diagram of data assets to show the entire process path of data asset access. The Dynamic Simulation and Prediction Data Asset Flowchart module is used to predict and fill in missing metadata in the preceding, intermediate, or subsequent nodes of the data asset flowchart using two algorithms. The algorithms include logical reasoning based on existing data lineage clues and rule judgment based on knowledge accumulated in the knowledge graph from historical information. The modifiable, multi-stage, multi-round recursive quality assessment module is designed to respond to the full lifecycle regulatory requirements of data assets in target business systems. Based on a phased quality assessment logic, it performs phased assessments of the convergence, governance, data analysis, and data application stages within the data asset full-process diagram after missing metadata prediction and supplementation. It generates data quality test cases for each stage based on data semantics to conduct data quality assessments. During the assessment process, it dynamically adjusts the constraints between test cases through multiple rounds of recursive execution and correction until the quality assessment results are output. Based on the quality assessment results, it performs quantitative calculations of data quality, assigns quality level labels, and generates a full-process data asset quality analysis report. The joint retrieval module based on quality assessment is used to respond to the data asset search needs of the target business system. Based on the data asset full-process diagram after missing metadata prediction and completion, it simultaneously performs global retrieval based on vertical distributed search engine, word segmentation retrieval based on semantic understanding, and weighted retrieval based on relevance to achieve joint search of multiple data assets. Based on the quality level tags generated by the full-process path quality assessment, the retrieval results are optimized and filtered to prioritize the display of high-quality retrieval results to users.

[0016] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps: Step 1, Data Asset Metadata Collection: By sequentially extracting metadata from multiple data sources of the target business system, constructing a data asset catalog, and accessing data assets based on metadata association and catalog binding, a full-process diagram of data assets is generated to show the entire process path of data asset access. Step 2: Dynamically simulate and predict the entire data asset flowchart: For missing metadata in the preceding, intermediate, or subsequent nodes of the data asset flowchart, two algorithms are used to predict and fill in the missing metadata. The algorithms include logical reasoning based on existing data lineage clues and rule judgment based on knowledge accumulated in the knowledge graph from historical information. Step 3: Modifiable Multi-Stage, Multi-Round Recursive Quality Assessment: Responding to the full lifecycle regulatory requirements of data assets in the target business system, based on a phased quality assessment logic, phased assessments are conducted on the convergence, governance, data analysis, and data application stages within the full-process path displayed in the data asset full-process diagram after missing metadata prediction and supplementation. Data quality test cases are generated for each stage based on data semantics to conduct data quality assessments. During the assessment process, test cases are run and modified in multiple rounds to dynamically adjust the constraints between test cases until the quality assessment results are output. Data quality is quantitatively calculated based on the quality assessment results, and quality level labels are assigned, generating a full-process data asset quality analysis report. Step 4, Joint Search Based on Quality Assessment: In response to the data asset search needs of the target business system, based on the full process diagram of the data assets after missing metadata prediction and completion, the joint search of multiple data assets is carried out simultaneously by global search based on vertical distributed search engine, word segmentation search based on semantic understanding, and weighted search based on relevance. The search results are optimized and filtered according to the quality level tags generated by the quality assessment of the entire process path, and high-quality search results are given priority to users.

[0017] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Step 1, Data Asset Metadata Collection: By sequentially extracting metadata from multiple data sources of the target business system, constructing a data asset catalog, and accessing data assets based on metadata association and catalog binding, a full-process diagram of data assets is generated to show the entire process path of data asset access. Step 2: Dynamically simulate and predict the entire data asset flowchart: For missing metadata in the preceding, intermediate, or subsequent nodes of the data asset flowchart, two algorithms are used to predict and fill in the missing metadata. The algorithms include logical reasoning based on existing data lineage clues and rule judgment based on knowledge accumulated in the knowledge graph from historical information. Step 3: Modifiable Multi-Stage, Multi-Round Recursive Quality Assessment: Responding to the full lifecycle regulatory requirements of data assets in the target business system, based on a phased quality assessment logic, phased assessments are conducted on the convergence, governance, data analysis, and data application stages within the full-process path displayed in the data asset full-process diagram after missing metadata prediction and supplementation. Data quality test cases are generated for each stage based on data semantics to conduct data quality assessments. During the assessment process, test cases are run and modified in multiple rounds to dynamically adjust the constraints between test cases until the quality assessment results are output. Data quality is quantitatively calculated based on the quality assessment results, and quality level labels are assigned, generating a full-process data asset quality analysis report. Step 4, Joint Search Based on Quality Assessment: In response to the data asset search needs of the target business system, based on the full process diagram of the data assets after missing metadata prediction and completion, the joint search of multiple data assets is carried out simultaneously by global search based on vertical distributed search engine, word segmentation search based on semantic understanding, and weighted search based on relevance. The search results are optimized and filtered according to the quality level tags generated by the quality assessment of the entire process path, and high-quality search results are given priority to users.

[0018] The aforementioned data asset model management method, apparatus, equipment, and media for the entire process firstly extract metadata from multiple data sources in the target business system, construct a data asset catalog, and access data assets sequentially. This generates a complete data asset process diagram, forming a closed-loop data asset management path from "introduction and aggregation" to "governance and integration" and then to "analysis and application." This provides a unified view foundation for data asset model management, facilitating an understanding of the lineage and flow of data assets at each stage and improving the responsiveness of data asset business applications. Then, through the collaborative algorithm of data lineage logical reasoning and knowledge graph rule judgment, accurate prediction and completion of missing metadata are achieved, ensuring the accuracy of metadata prediction and completion. The system prioritizes the integrity, consistency, and business adaptability of the data system. Secondly, by conducting phased, multi-round recursive quality assessments throughout the entire data asset access process, it can not only identify data quality issues but also locate and track them, providing detailed rectification suggestions for data governance to improve data quality and enhance the value of data assets. Finally, based on the quality assessment, a three-layer parallel joint retrieval mechanism integrates the advantages of global scanning, semantic understanding, and relevance weighting technologies. This enables not only rapid location of current data assets but also accurate discovery of high-value, high-quality data, significantly improving the management efficiency and ease of use of data assets and providing strong support for their full utilization. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating a data asset model management method for the entire process in one embodiment; Figure 2 This is a schematic diagram of an abstract syntax tree in one embodiment; Figure 3 This is a visualization and standardized representation of the data asset workflow in one embodiment; Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0021] In one embodiment, such as Figure 1 As shown, a data asset model management method for the entire process is provided, including the following steps: Step 1: Data Asset Metadata Collection: By sequentially extracting metadata from multiple data sources of the target business system, constructing a data asset catalog, and accessing data assets based on metadata association and catalog binding, a full-process diagram of data assets is generated to show the entire process path of data asset access.

[0022] Step 2: Dynamically simulate and predict the entire data asset flowchart: For missing metadata in the preceding, intermediate, or subsequent nodes of the data asset flowchart, two algorithms are used to predict and fill in the missing metadata. The algorithms include logical reasoning based on existing data lineage clues and rule judgment based on knowledge accumulated in the knowledge graph from historical information.

[0023] Among these, logical reasoning based on existing data lineage clues refers to deducing the attributes or content of missing metadata by analyzing the lineage relationships of metadata already extracted from the target business system. Rule-based judgment based on knowledge accumulated in a knowledge graph from historical information refers to accumulating historically extracted metadata from the target business system into a knowledge graph, and then judging and supplementing missing metadata by matching rules in the knowledge graph.

[0024] Step 3: Modifiable Multi-Stage, Multi-Round Recursive Quality Assessment: Responding to the full lifecycle regulatory requirements of data assets in the target business system, based on a phased quality assessment logic, phased assessments are conducted on the convergence, governance, data analysis, and data application phases within the full-process path displayed in the data asset full-process diagram after missing metadata prediction and supplementation. Data quality test cases are generated for each phase based on data semantics to conduct data quality assessments. During the assessment process, test cases are run and modified in multiple rounds to dynamically adjust the constraints between test cases until the quality assessment results are output. Data quality is quantitatively calculated based on the quality assessment results, and quality level labels are assigned, generating a full-process data asset quality analysis report.

[0025] Step 4, Joint Search Based on Quality Assessment: In response to the data asset search needs of the target business system, based on the full process diagram of the data assets after missing metadata prediction and completion, the joint search of multiple data assets is carried out simultaneously by global search based on vertical distributed search engine, word segmentation search based on semantic understanding, and weighted search based on relevance. The search results are optimized and filtered according to the quality level tags generated by the quality assessment of the entire process path, and high-quality search results are given priority to users.

[0026] The aforementioned end-to-end data asset model management method firstly extracts metadata from multiple data sources in the target business system, constructs a data asset catalog, and integrates data assets. This generates a complete data asset workflow diagram, forming a closed-loop data asset management path from "introduction and aggregation" to "governance and integration" and then to "analysis and application." This provides a unified view foundation for data asset model management, facilitating an understanding of the lineage and flow of data assets at each stage and improving the responsiveness of data asset business applications. Then, through the collaborative use of dual algorithms—data lineage logical reasoning and knowledge graph rule judgment—accurate prediction and completion of missing metadata are achieved, ensuring the metadata system's integrity. First, it ensures the integrity, consistency, and business adaptability of data assets. Second, by conducting phased, multi-round recursive quality assessments throughout the entire data asset access process, it can not only identify data quality issues but also locate and track these issues, providing detailed rectification suggestions for data governance to improve data quality and enhance the value of data assets. Finally, based on the quality assessment, a three-layer parallel joint retrieval mechanism is used, integrating the advantages of global scanning, semantic understanding, and relevance weighting technologies. This enables not only the rapid location of current data assets but also the accurate discovery of high-value, high-quality data, significantly improving the management efficiency and ease of use of data assets and providing strong support for their full utilization.

[0027] In one embodiment, metadata extraction includes: By configuring data source connection information and metadata extraction tasks for the target business system, the system automatically extracts and updates metadata from multiple data sources. These supported data sources include relational databases, non-relational databases, message queues, and file storage systems. Specifically, the extraction process includes: first, identifying and parsing metadata from multiple data sources to automatically obtain the metadata's structure and format; second, storing the acquired metadata in a central metadata repository and providing query, update, and delete operations through a configured visual metadata management page to meet the needs of different business scenarios.

[0028] It should be understood that metadata is data that describes data assets and plays a vital role in the organization, management, and utilization of those assets. Metadata extraction technology can automatically extract metadata from data sources through simple configuration, greatly reducing the complexity and time cost of manual operations and ensuring the accuracy and real-time nature of metadata extraction.

[0029] In one embodiment, the data asset catalog construction includes: planning a first-level catalog for data assets, and constructing a data asset catalog based on the first-level catalog and its subordinate catalogs; wherein, the first-level catalog includes a pre-positioning library, a raw library, an integration library, and an application library; the subordinate catalogs of the pre-positioning library are the catalogs of database tables before data assets are introduced and aggregated; the subordinate catalogs of the raw library are the catalogs of data source systems and business data after data assets are introduced and aggregated; the subordinate catalogs of the integration library are the catalogs of business data after data asset governance and integration; and the subordinate catalogs of the application library are the catalogs of application data such as thematic libraries, interfaces, and models after data asset analysis and application.

[0030] It should be understood that a data asset catalog is built upon automated metadata extraction and user-defined classification rules. Cataloging data assets enables orderly organization and convenient management. Each data asset catalog can access multiple metadata-based data assets. For example, each table and its data in a relational database is a data asset, and each file and its information in a file storage system is a data asset.

[0031] In one embodiment, data asset access based on metadata association and directory binding includes: (1) Data asset ingestion and aggregation, including identifying data assets stored in multiple data sources in the target business system and extracting and aggregating digital assets.

[0032] (2) Governance and integration of data assets, including quality control of the extracted and aggregated data assets, including cleaning, transformation and normalization, and loading the quality-controlled data assets into the storage system of the data asset management platform, such as a distributed file system or database.

[0033] (3) Data asset analysis and application, including metadata association and data asset catalog binding for each data asset in the storage system, and configuration of data service interfaces, and calling data assets for analysis and application based on the data service interfaces. Among them, metadata association, such as associating data name, data source, structure, format, purpose, etc., facilitates user understanding and use. Metadata association and data asset catalog binding enable users to easily discover and access the required data assets.

[0034] Specifically, the main technologies and commonly used tools adopted in each stage of data asset access are shown in Table 1: Table 1. Key Technologies and Commonly Used Tools at Each Stage of Data Asset Access

[0035] In one embodiment, a full-process diagram for generating data assets is constructed to illustrate the entire process path for data asset access, including: First, adopt static analysis technology based on SQL parsing to parse DML statements (data manipulation language statements) involved in the data asset access process into an abstract syntax tree. Among them, SQL parsing includes technologies such as lexical analysis, syntax analysis, and semantic analysis. DML statements refer to SQL statements used to operate data in a database, including data query, data insertion, data update, and data deletion. The abstract syntax tree is an abstract representation of an SQL statement, which shows the complete SQL syntax structure through a tree structure. By traversing the abstract syntax tree, syntax elements in the SQL statement can be analyzed and extracted. The abstract syntax tree of the SQL statement is as Figure 2 shown. Figure 2 Figure 2 shows the syntax structure analysis of the SQL query statement "SELECT id, age FROM t_user WHERE name='Zhang San';". This statement is decomposed into different types of lexical units through lexical analysis: Keywords include SELECT, FROM, WHERE; Identifiers include id, age, t_user, name; Operator is the equal sign (=); Literal is the string Zhang San; Delimiters include commas and semicolons. In the syntax analysis stage, these units are organized into an abstract syntax tree (Abstract Syntax Tree, AST) with root as the root, and its structure includes a SELECT clause (SELECT Clause) and its query items (ITEMS) id and age, a FROM clause (FROMClause) and its source table (TABLES) t_user, and a WHERE clause (WHERE Clause) and its filtering condition (CONDITIONS) "name = 'Zhang San'". This tree structure clearly presents the syntax hierarchy of the SQL statement and provides a basis for subsequent semantic analysis and query processing.

[0036] Second, traverse the abstract syntax tree and extract the metadata information it contains, including elements such as tables and fields.

[0037] Finally, perform lineage analysis on the extracted metadata information to obtain the relationships and transfer paths between data assets. By using the data asset catalogs bound to data assets as nodes and the relationships and transfer paths between data assets as connections, a full-flow chart of data assets is constructed and generated to show the full-flow path of data asset access flowing through the pre-library, original library, integration library, and application library. The visualization effect and standardized expression of the full-flow chart of data assets are as Figure 3As shown, the standardized representation uses a standardized JSON array structure. This array contains multiple node objects, each of which fully defines a data node through a set of standardized fields: `id` is the unique identifier of the node; `nodeName` and `assetsName` represent the node name and asset name, respectively; `nodeType` and `platformType` distinguish the node type and platform type; `contextPath` specifies the context path; `assetsId` is the data asset ID; and `preId` and `nextId` clarify the flow order between nodes through their successor relationship, forming a complete link (such as the coherent path from front-end library 1 to application library table 1 in the example). This structure, in a machine-readable standardized form, clearly expresses the topological relationships and metadata information of data assets across multiple processing stages (such as front-end library, original library, integration library, and application library), providing a structured foundation for data lineage tracing and end-to-end visualization.

[0038] It should be understood that a full-process diagram of data assets can help users intuitively understand the relationships and flow paths between data assets, quickly obtain the required data assets and their lineage, help users take stock of the data assets of the target business system, understand the entire process of data induction, aggregation, governance, integration, analysis and application, and comprehensively improve the value of data assets.

[0039] In one embodiment, data quality test cases are generated at each stage based on data semantics to perform data quality assessment. During the assessment process, the constraints between test cases are dynamically adjusted through multiple rounds of recursive execution and modification until the assessment ends and the quality assessment results are output, including: Data quality test cases for each stage are generated based on data semantics, and the constraints between test cases are initialized. The test cases include test cases focusing on integrity data quality assessment in the induction and convergence stage, test cases focusing on consistency data quality assessment in the governance and integration stage, test cases focusing on accuracy data quality assessment in the data analysis stage, and test cases focusing on usability data quality assessment in the data application stage. The constraints include dependency or mutual exclusion. Using a task scheduler, test cases for each stage are run recursively in multiple rounds. If the constraints between test cases are mutually exclusive, the content and constraints of the test cases are adjusted in the next round of testing. After multiple rounds of testing to achieve the preset test case coverage requirements, the data quality assessment ends and outputs the quality assessment results for each stage.

[0040] In one embodiment, based on the data asset end-to-end flowchart after missing metadata prediction, a joint search of multiple data assets is simultaneously performed using a vertically distributed search engine-based global retrieval, semantic understanding-based word segmentation retrieval, and relevance-based weighted retrieval, including: (1) Execute global retrieval based on a vertically distributed search engine. The vertically distributed search engine uses Elasticsearch (ES) with distributed, master-slave sharding and replication mechanisms to achieve global retrieval coverage of multiple data assets in the entire data asset process diagram after missing metadata prediction and completion. It should be understood that vertically distributed search engines, represented by ES, have the advantages of full-text search, data analysis and distributed storage in one. The distributed feature enables it to have millisecond-level response speed while ensuring retrieval accuracy when facing the storage and retrieval of massive amounts of data. The master-slave sharding and replication mechanism also makes it perform very well in terms of scalability and maintainability when facing a large amount of data. Because of the advantages of vertically distributed search engines in the professional search field, it is the best choice for analyzing and retrieving data assets. ES also supports hot updates of stop words, which means that the stop word library is dynamically updated in ES. The word library can be adjusted in real time without restarting the service. Traditional stop word libraries usually require modification of the configuration file and restarting the service to take effect, while hot updates automatically load the updated word library through external interfaces or scheduled tasks to ensure the timeliness and accuracy of search results. Elasticsearch is developed based on Java, and its underlying storage and retrieval are implemented based on the Lucene engine.

[0041] (2) Perform semantic understanding-based word segmentation retrieval. Long and complex sentences in the retrieval request are segmented using a grammatical word segmentation algorithm. Uncommon words in the retrieval request are identified using a dictionary-based word segmentation algorithm to achieve accurate word segmentation. The word segmentation results of the retrieval request are then embedded into the Elasticsearch search engine's query component for multi-data asset retrieval. It should be understood that semantic understanding-based word segmentation retrieval can accurately parse retrieval requests, support the transformation of retrieval requests, and improve the retrieval accuracy of the required data assets.

[0042] (3) Perform relevance-based weighted retrieval. During the retrieval request parsing process, assign high relevance weights to the key fields obtained from the parsing, and embed the weighted retrieval request into the query component of the Elasticsearch search engine for multi-data asset retrieval. It should be understood that relevance-based weighted retrieval avoids the situation where the traditional equal-weighted retrieval is prone to excessively biased retrieval results, and improves the recall rate of data asset retrieval. Among them, recall (R) represents the proportion of the text returned by the retrieval to all relevant text, and precision (P) represents the proportion of relevant text in the text returned by the retrieval. The concepts of P and R can be more intuitively understood through Table 2, so the calculation formula of P is: The formula for calculating R is: To resolve the case where P and R are mutually exclusive, their harmonic mean is defined as F, where... When P and R are made to have equal weights, At this point, there is a formula. .

[0043] Table 2. Concepts related to precision and recall

[0044] Finally, by combining the global search results, word segmentation search results, and weighted search results, a joint search of multiple data assets is completed.

[0045] In one embodiment, a data asset model management device for the entire process is provided, comprising: The data asset metadata acquisition module is used to extract metadata from multiple data sources of the target business system, build a data asset catalog, and access data assets based on metadata association and catalog binding, thereby generating a full-process diagram of data assets to show the entire process path of data asset access. The Dynamic Simulation and Prediction Data Asset Flowchart module is used to predict and fill in missing metadata in the preceding, intermediate, or subsequent nodes of the data asset flowchart using two algorithms. The algorithms include logical reasoning based on existing data lineage clues and rule judgment based on knowledge accumulated in the knowledge graph from historical information. The modifiable, multi-stage, multi-round recursive quality assessment module is designed to respond to the full lifecycle regulatory requirements of data assets in target business systems. Based on a phased quality assessment logic, it performs phased assessments of the convergence, governance, data analysis, and data application stages within the data asset full-process diagram after missing metadata prediction and supplementation. It generates data quality test cases for each stage based on data semantics to conduct data quality assessments. During the assessment process, it dynamically adjusts the constraints between test cases through multiple rounds of recursive execution and correction until the quality assessment results are output. Based on the quality assessment results, it performs quantitative calculations of data quality, assigns quality level labels, and generates a full-process data asset quality analysis report. The joint retrieval module based on quality assessment is used to respond to the data asset search needs of the target business system. Based on the data asset full-process diagram after missing metadata prediction and completion, it simultaneously performs global retrieval based on vertical distributed search engine, word segmentation retrieval based on semantic understanding, and weighted retrieval based on relevance to achieve joint search of multiple data assets. Based on the quality level tags generated by the full-process path quality assessment, the retrieval results are optimized and filtered to prioritize the display of high-quality retrieval results to users.

[0046] Specific limitations regarding the end-to-end data asset model management device can be found in the limitations of the end-to-end data asset model management method described above, and will not be repeated here. Each module in the aforementioned end-to-end data asset model management device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0047] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a data asset model management method oriented towards the entire process. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0048] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0049] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to perform the following steps: Step 1, Data Asset Metadata Collection: By sequentially extracting metadata from multiple data sources of the target business system, constructing a data asset catalog, and accessing data assets based on metadata association and catalog binding, a full-process diagram of data assets is generated to show the entire process path of data asset access. Step 2: Dynamically simulate and predict the entire data asset flowchart: For missing metadata in the preceding, intermediate, or subsequent nodes of the data asset flowchart, two algorithms are used to predict and fill in the missing metadata. The algorithms include logical reasoning based on existing data lineage clues and rule judgment based on knowledge accumulated in the knowledge graph from historical information. Step 3: Modifiable Multi-Stage, Multi-Round Recursive Quality Assessment: Responding to the full lifecycle regulatory requirements of data assets in the target business system, based on a phased quality assessment logic, phased assessments are conducted on the convergence, governance, data analysis, and data application stages within the full-process path displayed in the data asset full-process diagram after missing metadata prediction and supplementation. Data quality test cases are generated for each stage based on data semantics to conduct data quality assessments. During the assessment process, test cases are run and modified in multiple rounds to dynamically adjust the constraints between test cases until the quality assessment results are output. Data quality is quantitatively calculated based on the quality assessment results, and quality level labels are assigned, generating a full-process data asset quality analysis report. Step 4, Joint Search Based on Quality Assessment: In response to the data asset search needs of the target business system, based on the full process diagram of the data assets after missing metadata prediction and completion, the joint search of multiple data assets is carried out simultaneously by global search based on vertical distributed search engine, word segmentation search based on semantic understanding, and weighted search based on relevance. The search results are optimized and filtered according to the quality level tags generated by the quality assessment of the entire process path, and high-quality search results are given priority to users.

[0050] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Step 1, Data Asset Metadata Collection: By sequentially extracting metadata from multiple data sources of the target business system, constructing a data asset catalog, and accessing data assets based on metadata association and catalog binding, a full-process diagram of data assets is generated to show the entire process path of data asset access. Step 2: Dynamically simulate and predict the entire data asset flowchart: For missing metadata in the preceding, intermediate, or subsequent nodes of the data asset flowchart, two algorithms are used to predict and fill in the missing metadata. The algorithms include logical reasoning based on existing data lineage clues and rule judgment based on knowledge accumulated in the knowledge graph from historical information. Step 3: Modifiable Multi-Stage, Multi-Round Recursive Quality Assessment: Responding to the full lifecycle regulatory requirements of data assets in the target business system, based on a phased quality assessment logic, phased assessments are conducted on the convergence, governance, data analysis, and data application stages within the full-process path displayed in the data asset full-process diagram after missing metadata prediction and supplementation. Data quality test cases are generated for each stage based on data semantics to conduct data quality assessments. During the assessment process, test cases are run and modified in multiple rounds to dynamically adjust the constraints between test cases until the quality assessment results are output. Data quality is quantitatively calculated based on the quality assessment results, and quality level labels are assigned, generating a full-process data asset quality analysis report. Step 4, Joint Search Based on Quality Assessment: In response to the data asset search needs of the target business system, based on the full process diagram of the data assets after missing metadata prediction and completion, the joint search of multiple data assets is carried out simultaneously by global search based on vertical distributed search engine, word segmentation search based on semantic understanding, and weighted search based on relevance. The search results are optimized and filtered according to the quality level tags generated by the quality assessment of the entire process path, and high-quality search results are given priority to users.

[0051] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0052] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0053] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application.

Claims

1. A data asset model management method for the entire process, characterized in that, The method includes: Step 1, Data Asset Metadata Collection: By sequentially extracting metadata from multiple data sources of the target business system, constructing a data asset catalog, and accessing data assets based on metadata association and catalog binding, a full-process diagram of data assets is generated to show the entire process path of data asset access. Step 2: Dynamically simulate and predict the entire data asset flowchart: For missing metadata in the preceding, intermediate, or subsequent nodes of the data asset flowchart, two algorithms are used to predict and fill in the missing metadata. The algorithms include logical reasoning based on existing data lineage clues and rule judgment based on knowledge accumulated in the knowledge graph from historical information. Step 3: Modifiable Multi-Stage, Multi-Round Recursive Quality Assessment: Responding to the full lifecycle regulatory requirements of data assets in the target business system, based on a phased quality assessment logic, phased assessments are conducted on the convergence, governance, data analysis, and data application stages within the full-process path displayed in the data asset full-process diagram after missing metadata prediction and supplementation. Data quality test cases are generated for each stage based on data semantics to conduct data quality assessments. During the assessment process, test cases are run and modified in multiple rounds to dynamically adjust the constraints between test cases until the quality assessment results are output. Data quality is quantitatively calculated based on the quality assessment results, and quality level labels are assigned, generating a full-process data asset quality analysis report. Step 4, Joint Search Based on Quality Assessment: In response to the data asset search needs of the target business system, based on the full process diagram of the data assets after missing metadata prediction and completion, the joint search of multiple data assets is carried out simultaneously by global search based on vertical distributed search engine, word segmentation search based on semantic understanding, and weighted search based on relevance. The search results are optimized and filtered according to the quality level tags generated by the quality assessment of the entire process path, and high-quality search results are given priority to users.

2. The data asset model management method for the entire process according to claim 1, characterized in that, The metadata extraction includes: By configuring data source connection information and metadata extraction tasks for the target business system, metadata extraction and updating from multiple data sources can be performed automatically. Among them, the multiple data sources that support metadata extraction include relational databases, non-relational databases, message queues, and file storage systems. The extraction process includes: Identify and parse metadata from multiple data sources, and automatically obtain the structure and format of the metadata; The acquired metadata is stored in a central metadata repository, and a visual metadata management page is configured to provide query, update, and delete operations for the metadata.

3. The data asset model management method for the entire process according to claim 2, characterized in that, The construction of the data asset catalog includes: A first-level directory is planned for the data assets, and a data asset directory is constructed based on the first-level directory and its sub-directories. The first-level directory includes a pre-positioning library, a raw library, an integration library, and an application library. The sub-directories of the pre-positioning library are the database table classification directories before the data assets are introduced and aggregated. The sub-directories of the raw library are the data source systems and business data classification directories after the data assets are introduced and aggregated. The sub-directories of the integration library are the business data classification directories after the data assets are governed and integrated. The sub-directories of the application library are the application data classification directories after the data assets are analyzed and applied.

4. The data asset model management method for the entire process according to claim 3, characterized in that, Data asset access based on metadata association and directory binding includes: Data asset ingestion and aggregation includes identifying data assets stored in multiple data sources in the target business system and extracting and aggregating digital assets. The governance and integration of data assets includes quality control of extracted and aggregated data assets, and loading the quality-controlled data assets into the storage system of the data asset management platform; The analysis and application of data assets includes associating metadata and binding data asset catalogs for each data asset within the storage system, configuring data service interfaces, and calling data assets for analysis and application based on the data service interfaces.

5. The data asset model management method for the entire process according to claim 4, characterized in that, Construct a full-process diagram for generating data assets to illustrate the entire process of data asset access, including: Static analysis technology based on SQL parsing is used to parse the DML statements involved in the data asset access process into an abstract syntax tree; wherein, the SQL parsing includes lexical analysis, syntax analysis and semantic analysis; Traverse the abstract syntax tree to extract the metadata information contained therein, including tables and fields; The extracted metadata information is analyzed to obtain the relationships and flow paths between data assets. By using the data asset catalogs bound to the data assets as nodes and the relationships and flow paths between the data assets as lines, a full flow diagram of data assets is generated to show the entire flow path of data asset access through the front-end library, the original library, the integration library and finally the application library.

6. The data asset model management method for the entire process according to claim 1, characterized in that, Data quality assessment is performed by generating data quality test cases for each stage based on data semantics. During the assessment process, test cases are dynamically adjusted through multiple rounds of recursive execution and modification until the assessment concludes and the quality assessment results are output, including: Data quality test cases for each stage are generated based on data semantics, and the constraints between the test cases are initialized. The test cases include test cases focusing on integrity data quality assessment in the induction and convergence stage, test cases focusing on consistency data quality assessment in the governance and integration stage, test cases focusing on accuracy data quality assessment in the data analysis stage, and test cases focusing on usability data quality assessment in the data application stage. The constraints include dependencies or mutual exclusions. Using a task scheduler, test cases for each stage are run recursively in multiple rounds. If the constraints between test cases are mutually exclusive, the content and constraints of the test cases are adjusted in the next round of testing. After multiple rounds of testing to achieve the preset test case coverage requirements, the data quality assessment ends and outputs the quality assessment results for each stage.

7. The data asset model management method for the entire process according to claim 1, characterized in that, Based on the complete data asset workflow diagram after missing metadata prediction, simultaneous execution of global retrieval based on a vertically distributed search engine, word segmentation retrieval based on semantic understanding, and relevance-based weighted retrieval enables joint search of multiple data assets, including: Perform a global search based on a vertically distributed search engine, which uses the Elasticsearch search engine with distributed, master-slave sharding and replication mechanisms to achieve global search coverage of multiple data assets in the entire process diagram of data assets after missing metadata prediction and completion. Perform semantic understanding-based word segmentation retrieval, process long and difficult sentences in the retrieval request using grammatical word segmentation algorithm, identify uncommon words in the retrieval request using dictionary word segmentation algorithm to achieve accurate word segmentation, and embed the word segmentation results of the retrieval request into the query component of the Elasticsearch search engine for multi-data asset retrieval; Perform relevance-based weighted retrieval. During the retrieval request parsing process, assign high relevance weights to the key fields obtained from the parsing, and embed the weighted retrieval request into the query component of the Elasticsearch search engine for multi-data asset retrieval. Finally, by combining global search results, word segmentation search results, and weighted search results, a joint search of multiple data assets can be achieved.

8. A data asset model management device for the entire process, characterized in that, The device includes: The data asset metadata acquisition module is used to extract metadata from multiple data sources of the target business system, build a data asset catalog, and access data assets based on metadata association and catalog binding, thereby generating a full-process diagram of data assets to show the entire process path of data asset access. The Dynamic Simulation and Prediction Data Asset Flowchart module is used to predict and fill in missing metadata in the preceding, intermediate, or subsequent nodes of the data asset flowchart using two algorithms. The algorithms include logical reasoning based on existing data lineage clues and rule judgment based on knowledge accumulated in the knowledge graph from historical information. The modifiable, multi-stage, multi-round recursive quality assessment module is designed to respond to the full lifecycle regulatory requirements of data assets in target business systems. Based on a phased quality assessment logic, it performs phased assessments of the convergence, governance, data analysis, and data application stages within the data asset full-process diagram after missing metadata prediction and supplementation. It generates data quality test cases for each stage based on data semantics to conduct data quality assessments. During the assessment process, it dynamically adjusts the constraints between test cases through multiple rounds of recursive execution and correction until the quality assessment results are output. Based on the quality assessment results, it performs quantitative calculations of data quality, assigns quality level labels, and generates a full-process data asset quality analysis report. The joint retrieval module based on quality assessment is used to respond to the data asset search needs of the target business system. Based on the data asset full-process diagram after missing metadata prediction and completion, it simultaneously performs global retrieval based on vertical distributed search engine, word segmentation retrieval based on semantic understanding, and weighted retrieval based on relevance to achieve joint search of multiple data assets. Based on the quality level tags generated by the full-process path quality assessment, the retrieval results are optimized and filtered to prioritize the display of high-quality retrieval results to users.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images