Efficient management methods and systems for data catalogs in data spaces

Abstract

This invention relates to the field of data management technology, and specifically discloses an efficient method and system for managing data catalogs in a data space. The method includes: incrementally updating metadata information, data product registration information, and cross-node data catalog information related to the data catalog to a distributed data storage system; constructing a full-text index and a vector index for the metadata in the distributed data storage system, and matching the vector generated based on the request content with the full-text index or vector index to obtain retrieval response results; marking semantically equivalent datasets based on the semantic vectors of all datasets, constructing a dataset node graph based on a list of all semantic entities, and outputting a list of semantically equivalent fields based on the semantically equivalent datasets and the dataset node graph; determining associated recommendation results based on the list of semantically equivalent fields, and simultaneously returning the retrieval response results and associated recommendation results; to support the management and efficient utilization of data catalogs in a distributed, heterogeneous, cross-domain data space.

Description

Technical Field

[0001] This invention relates to the field of data management technology, and in particular to an efficient method and system for managing data catalogs in a data space. Background Technology

[0002] In today's era of rapid development in cutting-edge technologies such as big data, cloud computing, artificial intelligence, and the industrial internet, the value of data is increasingly prominent, making it a crucial strategic resource. To fully unlock the potential of data and achieve data sharing, collaboration, and value release among different organizations, the innovative concept of "DataSpace" has emerged in recent years and gradually gained widespread attention. DataSpace emphasizes distributed data ownership, collaborative governance, semantic interoperability, and data sovereignty protection, demonstrating significant application value in numerous key scenarios such as government collaboration, supply chain data interoperability, and medical and scientific research data sharing. Within the overall architecture of DataSpace, the DataCatalog system occupies a fundamental and crucial position. Like a precise "data resource map," it undertakes important tasks such as metadata extraction, semantic modeling, quality assessment, and classification of accessed data. Simultaneously, it provides data consumers with convenient entry points for searching, browsing, applying for, and analyzing data, helping users efficiently explore and utilize data resources. Through the DataCatalog system, different organizations can gain a clearer understanding of their own and other organizations' data assets, thereby providing strong support for data-driven decision-making and promoting cross-organizational data collaboration and innovative development. As data space applications continue to expand and deepen, the demand for efficient management of data catalog systems is becoming increasingly urgent. A scientific and efficient data catalog management method and system can not only improve the availability and usability of data, but also further promote the healthy development of the data space ecosystem, injecting strong momentum into the digital transformation of various industries.

[0003] However, existing data catalog systems have numerous limitations. Regarding semantic description, inconsistencies in data field naming across multiple organizations and systems, coupled with severe metadata gaps, make it difficult for data catalogs to accurately understand and match data from different sources. The lack of a unified semantic description capability significantly hinders data interoperability and integration. In terms of information updates, as data sources undergo continuous dynamic changes, critical data such as field structure, data volume, and interface information are prone to becoming invalid. Existing systems lack effective incremental update mechanisms, resulting in untimely or inaccurate catalog information updates, failing to reflect the latest data status and impacting effective data utilization. At the security and sovereignty control level, data catalogs in data space scenarios must not only describe data but also be tightly bound to data access permissions, data sovereignty claims, and authorization agreements. Existing systems generally have weak capabilities in this area, failing to meet the stringent requirements of data space for data security and sovereignty protection. Furthermore, most existing data catalog platforms adopt a centralized, single-tenant architecture. This architecture struggles to support collaborative data governance across multiple organizations and lacks the ability to access heterogeneous systems, limiting the widespread sharing and collaboration of data within the data space.

[0004] In summary, there is an urgent need for a new data catalog management method and system to meet the requirements of efficient construction and sustainable operation of data space, realize the automated, intelligent, standardized description, classification and governance of data resources, and possess semantic connectivity, sovereignty control, cross-domain indexing and an orchestratable publishing mechanism.

[0005] Therefore, this invention proposes an efficient management method and system for data catalogs in a data space. Summary of the Invention

[0006] This invention provides an efficient management method and system for data catalogs in a data space, which is used to unify, describe, catalog, index, and retrieve multi-source heterogeneous data resources in cross-organizational and cross-platform data collaboration scenarios, thereby solving at least one of the above-mentioned defects.

[0007] This invention provides an efficient method for managing data directories in a data space, comprising:

[0008] Incremental updates of metadata information related to data catalogs, data product registration information, and cross-node data catalog information obtained through multiple channels are performed and centrally stored in a distributed data storage system.

[0009] Construct full-text indexes and vector indexes for metadata in a distributed data storage system. Based on the retrieval pattern corresponding to the request type of the user's retrieval request, match the vector generated based on the request content with the full-text index or vector index to obtain the retrieval response results.

[0010] The large language model is called to extract the list of semantic entities and semantic vectors for each dataset. Based on the semantic vectors of all datasets, semantically equivalent datasets are marked. A dataset node graph is constructed based on the list of all semantic entities. Based on the semantically equivalent datasets and the dataset node graph, a list of semantically equivalent fields is output.

[0011] The search results are used to determine the associated recommendations based on the list of semantically equivalent fields, and the search results and associated recommendations are returned synchronously.

[0012] Preferably, the distributed data storage system includes a distributed storage subsystem compatible with object storage protocols, a distributed relational data subsystem, and a vector indexing subsystem. The distributed storage subsystem compatible with object storage protocols is used to store unstructured data and serves as the underlying storage support for the distributed relational data subsystem and the vector indexing subsystem.

[0013] Preferably, the metadata information related to the data catalog, data product registration information, and cross-node data catalog information obtained through multiple channels is incrementally updated, and centrally stored in a distributed data storage system, including:

[0014] The system receives metadata information from manual registration through the directory upload interface of the data connector, and at the same time acts as a backend reverse proxy of the data connector to directly access the application system of the data provider to extract metadata and generate data product registration information according to the data product registration template.

[0015] When the metadata update service is deployed near a regional or global functional node, cross-node data catalog information is retrieved through the query service of the functional node.

[0016] Metadata information, data product registration information, and cross-node data directory information are encoded into metadata operation logs and written into the distributed data storage system to complete incremental metadata updates.

[0017] Preferably, a full-text index and a vector index are constructed for metadata in the distributed data storage system. Based on the retrieval pattern corresponding to the user's retrieval request type, the vector generated based on the request content is matched with the full-text index or the vector index to obtain the retrieval response results, including:

[0018] Full-text indexes and vector indexes are built based on metadata in a distributed data storage system;

[0019] If the user's search request is a keyword or semi-structured keyword request, the first search response result is generated by accurately matching the keywords in the user's search request during the full-text index matching process. At the same time, the user's search request is converted into a basic semantic vector through the large language model, and the semantically related dataset of the basic semantic vector is matched in the vector index to generate a second search response result.

[0020] If the user's search request is a semantic request, the vector index matching process prioritizes accurate matching based on the high-precision semantic vector converted from the user's search request to generate a second search response result. At the same time, the core keywords in the user's search request are extracted, and the dataset containing the core keywords is matched in the full-text index to generate a first search response result.

[0021] The search results include the first search result and the second search result generated simultaneously.

[0022] Preferably, a large language model is invoked to extract the semantic entity list and semantic vectors for each dataset. Semantically equivalent datasets are then identified based on the semantic vectors of all datasets. A dataset node graph is constructed based on the list of all semantic entities. Finally, a list of semantically equivalent fields is output based on the semantically equivalent datasets and the dataset node graph, including:

[0023] The large language model is invoked to extract the list of semantic entities and semantic vectors for each dataset;

[0024] Calculate the cosine similarity of the semantic vectors of any two datasets, and mark any two datasets with a similarity greater than a preset similarity threshold as semantically equivalent datasets;

[0025] Analyze the field names and field comments of the structured dataset and extract semantic entities to supplement the semantic entity list;

[0026] Construct a node graph for the dataset with the dataset as the vertex and the semantic entity lists having an intersection.

[0027] Verify the semantic consistency of fields in semantically equivalent datasets and datasets whose correlation in the node graph of the datasets is greater than a preset correlation threshold, and output a list of semantically equivalent fields.

[0028] Preferably, the associated recommendation results of the search response are determined based on a list of semantically equivalent fields, and the search response results and associated recommendation results are returned synchronously, including:

[0029] Provides list categorization browsing and graph node navigation based on dataset node graphs;

[0030] The first and second search response results in the search results are merged based on the result fusion algorithm to obtain the final search result;

[0031] Based on the correlation between the node corresponding to the final search result and the remaining nodes in the dataset node graph, and the list of semantically equivalent fields, related recommendation results are generated.

[0032] The final search results and related recommendation results are returned synchronously.

[0033] Preferably, the specific execution method of the result fusion algorithm includes:

[0034] Calculate the reciprocal ranking of each dataset in the first search response result and the reciprocal ranking of each dataset in the second search response result;

[0035] Determine the weights of the first and second search response results;

[0036] The fusion score of the corresponding dataset is obtained by weighting the two inverse rankings based on the weights of the first and second search response results.

[0037] The top-k datasets are selected as the final retrieval results based on their fusion scores, ranked from highest to lowest.

[0038] Preferably, determining the weights of the first search response result and the second search response result includes:

[0039] Identify the current application scenario of the data space and define multiple search requirement indicators for the current application scenario, including: accuracy indicators, comprehensiveness indicators, and timeliness indicators.

[0040] Based on the historical search logs within the latest historical period in the current application scenario, the actual values ​​of the accuracy, comprehensiveness, and timeliness indicators of each historical search result, as well as user satisfaction, are determined.

[0041] Using the actual values ​​of accuracy, comprehensiveness, and timeliness indicators of each historical search result as independent variables and user satisfaction as the dependent variable, a multiple linear regression model is used to determine the linear relationship between multiple search demand indicators and user satisfaction in the current application scenario.

[0042] Based on the linear relationship between multiple search demand indicators and user satisfaction in the current application scenario, the optimal weight combination of multiple search demand indicators in the current application scenario is determined.

[0043] Based on the optimal weight combination of multiple search demand indicators in the current application scenario, an indicator weight matrix is ​​constructed. Based on the correlation coefficients between the multiple search demand indicators and the full-text index and vector index respectively, an indicator-index correlation matrix is ​​constructed. Based on the constraint factors of the full-text index and vector index in the current application scenario, a scenario constraint matrix is ​​constructed.

[0044] The result of multiplying the indicator weight matrix, the indicator-index association matrix, and the scenario constraint matrix in sequence is used as the optimized index weight matrix. Based on the dual index weights, a normalization operation is performed on all elements in the optimized index weight matrix to obtain the final index weight matrix. The weights of the first search response result and the second search response result are determined based on the elements in the final index weight matrix.

[0045] Preferably, based on historical search logs within the latest historical period in the current application scenario, the actual values ​​of accuracy, comprehensiveness, and timeliness indicators, as well as user satisfaction, are determined for each historical search result, including:

[0046] Based on historical search logs within the latest historical period in the current application scenario, the following parameters are determined: the ratio of the number of results containing core keywords or semantic entities to the total number of results in each historical search is used as the search result matching degree; the ratio of the number of results with a click matching degree not less than a preset matching degree threshold to the total number of clicks in each historical search is used as the precision click-through rate; the number of business dimensions covered by the results returned by a single search request in each historical search is used as the number of associated dimensions; the ratio of the number of requests using associated results to the total number of requests in each historical search is used as the association usage rate; the proportion of data in each historical search result whose update time does not exceed the preset period of the current application scenario is used as the freshness; and the ratio of the number of times results with a click freshness not less than a preset freshness threshold to the total number of clicks in each historical search is used as the real-time click-through rate.

[0047] The actual values ​​of the accuracy indicator are determined based on the matching degree and precise click-through rate of each historical search result; the actual values ​​of the comprehensiveness indicator are determined based on the number of associated dimensions and the usage rate of associated dimensions of each historical search result; and the actual values ​​of the timeliness indicator are determined based on the freshness and real-time click-through rate of each historical search result.

[0048] User satisfaction with each historical search result is calculated based on the click-through rate, collection rate, and usage rate of the search results in the historical search logs within the latest historical period under the current application scenario.

[0049] This invention provides an efficient management system for data catalogs in a data space, comprising:

[0050] The metadata collection and update module is used to incrementally update metadata information related to data catalogs, data product registration information, and cross-node data catalog information obtained through multiple channels, and to centrally store them in a distributed data storage system.

[0051] The index building and retrieval module is used to build full-text indexes and vector indexes for metadata in the distributed data storage system. Based on the retrieval mode corresponding to the request type of the user's retrieval request, it matches the vector generated based on the request content with the full-text index or vector index to obtain the retrieval response results.

[0052] The consistency analysis and fusion module is used to call the large language model to extract the semantic entity list and semantic vector of each dataset, mark the semantically equivalent datasets based on the semantic vectors of all datasets, construct the dataset node graph based on the list of all semantic entities, and output the semantically equivalent field list based on the semantically equivalent datasets and the dataset node graph.

[0053] The related recommendation and output module is used to determine the related recommendation results of the search response results based on the list of semantically equivalent fields, and synchronously return the search response results and related recommendation results.

[0054] The beneficial effects of this invention compared to existing technologies are as follows: It incrementally updates metadata information related to data catalogs, data product registration information, and cross-node data catalog information obtained through multiple channels, and centrally stores this information in a distributed data storage system, ensuring data real-time performance and integrity, and facilitating unified data management and retrieval. It constructs full-text and vector indexes for metadata in the distributed data storage system. Based on the retrieval mode corresponding to the user's retrieval request type, it matches the vector generated based on the request content with the corresponding index to obtain retrieval response results, improving retrieval efficiency and accuracy, and quickly meeting diverse user retrieval needs. It calls a large language model to extract the semantic entity list and semantic vectors of each dataset, thereby marking semantically equivalent datasets, constructing a dataset node graph, and outputting a list of semantically equivalent fields, uncovering semantic relationships between data, and deepening the understanding of the data. Based on the list of semantically equivalent fields, it determines the associated recommendation results of the retrieval response results and returns them synchronously, providing users with more comprehensive and valuable information, further improving the user experience, and helping users to explore and utilize the data catalog in the data space more deeply.

[0055] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in this application.

[0056] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0057] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0058] Figure 1 This is a flowchart illustrating an efficient data directory management method in a data space according to an embodiment of the present invention.

[0059] Figure 2 This is a flowchart of the metadata consistency analysis in an embodiment of the present invention;

[0060] Figure 3 This is a semantic network graph of the dataset in an embodiment of the present invention;

[0061] Figure 4 This is a flowchart of the intelligent semantic retrieval and recommendation process in an embodiment of the present invention;

[0062] Figure 5 This is a system description diagram in an embodiment of the present invention. Detailed Implementation

[0063] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0064] like Figure 1 As shown, this invention provides an efficient method for managing data directories in a data space, including:

[0065] Incremental updates of metadata information related to data catalogs, data product registration information, and cross-node data catalog information obtained through multiple channels are performed and centrally stored in a distributed data storage system.

[0066] Construct full-text indexes and vector indexes for metadata in a distributed data storage system. Based on the retrieval pattern corresponding to the request type of the user's retrieval request, match the vector generated based on the request content with the full-text index or vector index to obtain the retrieval response results.

[0067] The large language model is called to extract the list of semantic entities and semantic vectors for each dataset. Based on the semantic vectors of all datasets, semantically equivalent datasets are marked. A dataset node graph is constructed based on the list of all semantic entities. Based on the semantically equivalent datasets and the dataset node graph, a list of semantically equivalent fields is output.

[0068] The search results are used to determine the associated recommendations based on the list of semantically equivalent fields, and the search results and associated recommendations are returned synchronously.

[0069] In this embodiment, the metadata information related to the data catalog covers the industry, source, product interaction method, data type, format, and structured data field definitions of the data. Data product registration information is generated based on the data product registration template, combined with metadata extracted from the data provider's application system, and involves information related to data product transactions. Cross-node data catalog information refers to the data catalog information between different regional or global functional nodes within the data space architecture.

[0070] In this embodiment, a user retrieval request is a data query request sent by the user to the data catalog management system. For example, if a user wants to obtain sales data for a specific time and location, they will submit such a request to the system.

[0071] In this embodiment, the request content is the specific description of the user's search request, such as "querying the sales volume of clothing in Beijing in the first quarter of 2024".

[0072] In this embodiment, the vector generated based on the request content is transformed into vector form using a large language model. For example, "Query the sales volume of clothing in Beijing in the first quarter of 2024" is transformed into a vector containing semantic features using a large language model, where the numerical values ​​in the vector represent feature values ​​of different semantic dimensions.

[0073] In this embodiment, the retrieval response result is the result returned to the user by the system after matching the full-text index and the vector index according to the user's retrieval request.

[0074] In this embodiment, the large language model is a model with powerful language understanding and generation capabilities. In this embodiment, it is used to transform user search requests into semantic vectors, extract semantic entity lists and semantic vectors from datasets, etc. Models such as ChatGPT can understand natural language and perform semantic analysis and transformation.

[0075] In this embodiment, a dataset is a collection of data. In the data catalog management system, each dataset has specific metadata descriptions. For example, an employee information dataset contains data such as employee name, age, and position.

[0076] In this embodiment, the semantic entity list is composed of meaningful entities extracted from the dataset name, description, and structured data fields using a large language model. For example, semantic entities such as "employee," "salary," and "bonus" are extracted from the "employee salary table (employee ID, salary, bonus)" to form a list. The semantic vector is a vector obtained by transforming the dataset name and description through the large language model. It is used to represent the semantic features of the dataset, and the vector value reflects the semantic features in different dimensions.

[0077] In this embodiment, calling the large language model to extract the semantic entity list and semantic vector of each dataset means using the large language model to analyze the name and description information of each dataset, outputting the semantic entity list, and converting the dataset name and description into semantic vectors.

[0078] In this embodiment, the data directory, along with corresponding descriptive information and tags, constitutes a complete description of the corresponding data, which is the metadata. For example... Figure 5As shown, the data catalog management system comprises the following five modules: data storage service; metadata update service; metadata indexing and query service; data catalog browsing service; and metadata consistency analysis and fusion service. Among them:

[0079] Data storage service: This service stores metadata, index data, and metadata update logs in a distributed system. Through an abstract storage interface, it can flexibly adapt to distributed key-value storage systems and relational database systems, providing flexible scalability and high-performance read / write support. Currently, the chosen systems are the S3-compatible distributed key-value storage system and the distributed relational database system Apache Doris, along with Milvus for vector indexing. The S3-compatible distributed key-value storage system can not only directly store unstructured data, such as metadata update logs and indexes, but also serve as the underlying file system for Apache Doris and Milvus.

[0080] Metadata update service: In accordance with the latest national data infrastructure / data space standards, data is first accessed by the data provider's business system / or data sharing service system through a data connector to the data space. For example... Figure 5 Connection 2 in the data connector allows the metadata service to accept user-registered metadata information for data resources / products via the data directory upload interface. This includes the industry, data source, product interaction method, data type, format (structured data: field definitions), and other transaction-related information. The metadata service encodes this information into metadata operation logs and writes them to the underlying data storage system through the storage service.

[0081] In addition to the push-based streaming update schemes mentioned above, the metadata update service can also act as a backend reverse proxy for the data connector, such as... Figure 5 Connection 3 in the diagram directly accesses the data provider's application system, such as accessing the backend relational database system via JDBC or accessing the data provider's Hive metadata service via the Hive Metastore service. In this way, metadata can be directly extracted from the service provider's business system, and data product registration information can be generated based on the data product registration template. Figure 5 The connection line 4 is written into the data connector, which greatly reduces the workload during data product registration.

[0082] In the national data infrastructure / data space standards, the data connector's data directory upload interface requires it to upload its own data directory information to the regional functional node, which in turn uploads it to the global functional node. See also... Figure 5Connections 10 and 11: When the metadata update service is deployed near the above two nodes, it can retrieve data catalog information through the query service of the above functional nodes and then submit it to the data storage service for storage.

[0083] Metadata indexing and query services: There are two types of metadata indexes. One is full-text indexing, which is supported by the built-in full-text indexing function of Apache Doris. The second is vector indexing based on the large language model (currently supporting deepseek and chatgpt), which is supported by Milvus.

[0084] The search service supports: a) simple keywords, such as "weather cloud map of Beijing area". b) semi-structured keyword search, such as "region: Beijing; data type: weather cloud map". c) intelligent semantic search and recommendation; users submit analysis or modeling requirements, and the search service returns relevant datasets based on the semantics of the user's needs.

[0085] Data Catalog Browsing Service: Although descriptive information from various data sources in the data space is stored in tabular form, real-world data is actually interconnected in a network structure. For example, personal information from the public security department can be linked to medical visit information, consumer information from the commercial sector, and traffic information from the transportation department. Traffic information can then be linked to vehicle information, as well as various parts, manufacturers, and other related information. Based on this understanding, the data catalog descriptive information obtained from data connectors, regional functional nodes, and global functional nodes is used to perform semantic analysis on a large-scale model, assigning different semantic entity labels, such as: category, region, vehicle, medical, etc. Different datasets can establish semantic relationships with each other based on their semantic labels.

[0086] For structured data, a subsequent metadata consistency analysis service is used to identify fields with the same semantics but different names, or fields with the same name but different semantics. Then, associations are established based on the fields with the same semantics in the dataset.

[0087] Through the two levels of semantic analysis described above, the dataset in the data space constitutes a graph with multiple connections. Based on the above processing results, the data catalog browsing service provides list-based categorized browsing and graph-based node navigation starting from a specific node, such as... Figure 3 As shown in the semantic network graph of the dataset, nodes of the same color represent semantically equivalent datasets, while other nodes represent semantically related data. Users can navigate from one node to other related nodes on this graph based on their associations.

[0088] like Figure 2 As shown, the steps for metadata consistency analysis and fusion services are as follows:

[0089] S101: Using a large language model, extract semantic information from dataset names and descriptions, outputting:

[0090] Semantic entity list (entity_list);

[0091] Dataset name + semantic vector corresponding to the dataset description (dataset_semantic_vector);

[0092] S102: Calculate the cosine similarity of all dataset_semantic_vectors. Datasets with a similarity greater than a specified threshold (e.g., 0.95) are considered equivalent datasets, meaning they describe the same information but come from different sources. Examples include weather cloud imagery data from different provinces, traffic camera data, and road mapping maps published by different organizations.

[0093] S103: For structured datasets, analyze their field annotations and field names to extract their semantic entities. For example, in an order table (order number, product number, shipping method, delivery date), the output semantic entities are: order and product. Add these semantic entities to their corresponding entity list (entity_list) extracted from the dataset name and description.

[0094] S104: Find the union of the semantic entity lists of each dataset and finally construct a graph with the datasets as vertices. If the semantic entity lists corresponding to two vertices have an intersection, each identical element is converted into an edge connecting the two vertices.

[0095] S105: For each node pair with a correlation degree greater than the specified threshold, add it to the semantically related vertex pair list related_pair_list.

[0096] S106: For semantically equivalent structured datasets, if the field names are the same, they can be considered to represent the same information (if there are field descriptions, the semantics of the field descriptions can be compared to determine if they are completely consistent; if not, sampling can be used to compare field values). For fields with similar names, such as "username, user_name" and "user_id, userId", their field comments or field values ​​can be compared to determine if they are semantically the same.

[0097] For related datasets A and B, based on their associations on the node graph, the field semantics of their neighboring associated nodes are compared, and the comparison method is similar to the method in the steps above.

[0098] In summary, the efficient data catalog management method in this data space achieves the following technical effects:

[0099] Consistency: Through consistency analysis and fusion of metadata, a unified semantic model view is provided to users. At the dataset level, datasets with different names are given the same semantic name. For example, vehicle capture information from traffic management departments may be named differently in different provinces, sometimes as "vehicle capture" and sometimes as "road capture." Through semantic analysis of the data description, it is unified as "traffic management vehicle capture." At the field level, different fields have different names. For example, different ID card information may be named "user_id" on some datasets and "userID" on others. After consistency analysis, it is unified as "user_id_code." After unifying field naming and semantic equivalence analysis, different datasets can be linked. For example, hotel registration information and public security personnel information can be linked through ID card identifiers, thus providing a more complete dataset view for subsequent data analysis, leading to more analytical needs and uncovering more data value.

[0100] Real-time performance: When the data source changes, the metadata update service can detect the changes immediately and automatically or semi-automatically update the data catalog information on the data connector and the relevant metadata items on the metadata storage system. This avoids the problem of users querying outdated information, leading to discrepancies between the data and the catalog information, which could result in the failure of data transaction contracts.

[0101] Flexibility: The service-based architecture allows the system to be flexibly deployed alongside different components in different data spaces as supplementary public support services. Standardized data infrastructure interfaces enable the data catalog management system to adapt to various heterogeneous data sources. A flexible user interface supports everything from simple keyword searches to complex semantic natural language queries, as well as intuitive data graph-based browsing and navigation. This improves the effectiveness and convenience of users accessing the data catalog.

[0102] To achieve efficient storage of various types of data and provide underlying support for other components in a distributed data storage system, this paper proposes a distributed data storage system comprised of a distributed storage subsystem compatible with object storage protocols, a distributed relational data subsystem, and a vector indexing subsystem. The distributed storage subsystem compatible with object storage protocols plays a crucial role, capable of storing unstructured data such as metadata update logs and indexes. It also provides underlying storage support for the distributed relational data subsystem and the vector indexing subsystem. These subsystems rely on it for stable operation. In practical applications, a distributed key-value storage system compatible with S3 falls under this category of distributed storage subsystems compatible with object storage protocols, providing basic storage support for distributed relational data systems like Apache Doris and Milvus for vector indexing.

[0103] To comprehensively and timely update and centrally store metadata related to data catalogs, product registration, and cross-node information obtained from multiple channels, this paper proposes incremental updates of metadata related to data catalogs, data product registration information, and cross-node data catalog information obtained through multiple channels, and centralized storage in a distributed data storage system, including:

[0104] Receiving manually registered metadata information through the data connector's directory upload interface means that data providers or users can manually input data-related metadata information, such as industry, data source, product interaction method, data type, format (and field definitions for structured data), and other transaction-related information, into the system through the data connector's dedicated directory upload interface. This provides basic information for subsequent data management and processing. Simultaneously, acting as a backend reverse proxy for the data connector, it directly accesses the data provider's application system to extract metadata and generates data product registration information based on the data product registration template. This means that after obtaining the metadata extracted from the data provider's application system, it organizes and populates this metadata according to a pre-defined data product registration template to generate data product registration information containing data product transaction-related information, such as data product specifications, usage restrictions, and pricing.

[0105] When the metadata update service is deployed near a regional or global functional node, it retrieves cross-node data catalog information through the query service of the functional node. This indicates that in the data space architecture, when the metadata update service is in a specific location, it can obtain cross-node data catalog information shared between different nodes by leveraging the query service provided by the regional or global functional nodes.

[0106] Metadata information, data product registration information, and cross-node data directory information are encoded into metadata operation logs and written to the distributed data storage system to complete incremental metadata updates. This involves converting the collected information into metadata operation log format according to specific encoding rules and then storing it in the distributed data storage system.

[0107] To efficiently and accurately obtain retrieval results using full-text indexes and vector indexes based on different user retrieval request types, this paper proposes constructing full-text indexes and vector indexes for metadata in a distributed data storage system. Based on the retrieval pattern corresponding to the user's retrieval request type, the vector generated based on the request content is matched with the full-text index or vector index to obtain the retrieval results, including:

[0108] Full-text indexes and vector indexes are built based on metadata in a distributed data storage system;

[0109] If the user's search request is a keyword or semi-structured keyword request, the first search response result is generated by accurately matching the keywords in the user's search request during the full-text index matching process. At the same time, the user's search request is converted into a basic semantic vector through the large language model, and the semantically related dataset of the basic semantic vector is matched in the vector index to generate a second search response result.

[0110] If the user's search request is a semantic request, the vector index matching process prioritizes accurate matching based on the high-precision semantic vector converted from the user's search request to generate a second search response result. At the same time, the core keywords in the user's search request are extracted, and the dataset containing the core keywords is matched in the full-text index to generate a first search response result.

[0111] The search results include the first search result and the second search result generated simultaneously.

[0112] In this embodiment, the full-text index utilizes the built-in functionality of Apache Doris to perform full-text indexing of the text content in metadata, facilitating the rapid location of data containing specific keywords. The vector index uses Milvus to transform metadata into vector form, enabling semantic-based retrieval.

[0113] In this embodiment, keyword or semi-structured keyword requests are a type of user search request. A keyword request refers to a user directly entering simple keywords for searching, such as "sales data." A semi-structured keyword request involves a user entering keywords according to a specific structured format, such as "Region: Beijing; Data Type: Sales Data."

[0114] In this embodiment, during the full-text index matching process, priority is given to generating the first search response result based on the precise matching of keywords in the user's search request. For example, if the user enters the keyword "electronic product sales", the full-text index will search for data records that precisely contain these keywords in the metadata of the distributed data storage system, and present the data that meets the requirements as the first search response result to the user.

[0115] In this embodiment, a large language model converts the user's search request into a basic semantic vector, and matches semantically related datasets of the basic semantic vector in the vector index to generate a second search response result. For example, if a user requests "analyze mobile phone sales trends," the large language model converts this request into a basic semantic vector. The vector index, based on semantic similarity, finds datasets in the dataset that are semantically related to this request, such as historical mobile phone sales data and market share data, as the second search response result.

[0116] In this embodiment, semantic request is another type of user retrieval request. This type of request focuses more on expressing complex semantic needs, proposing analysis, modeling, and other requirements in natural language, such as "predicting changes in the market share of new energy vehicles in the next year."

[0117] In this embodiment, during the vector index matching process, priority is given to accurate matching based on the high-precision semantic vector converted from the user's retrieval request to generate a second retrieval response result. When a user makes a semantic request, the large language model converts it into a high-precision semantic vector, and the vector index uses this high-precision vector to accurately match semantically highly relevant datasets in the dataset. For example, for a request to "predict changes in the market share of new energy vehicles in the next year," the vector index can find relevant datasets such as new energy vehicle production, sales, and market competition trends as the second retrieval response result.

[0118] High-precision semantic vector generation involves a three-stage process: building a domain vocabulary for the data space, fine-tuning the model, and optimizing the vectors.

[0119] A dynamically updated "data space domain thesaurus" is constructed, which is divided into a three-level structure: core layer, extension layer, and association layer. The core layer contains basic entities of the data space, such as cross-node directories, data sovereignty declarations, authorization agreement identifiers, and metadata operation logs. Each entity is labeled with an industry benchmark weight, such as a data sovereignty declaration weight of 0.9 in the government affairs scenario and a patient privacy identifier weight of 0.95 in the medical scenario. The extension layer contains derived concepts of entities, such as data sovereignty declarations derived from cross-border data sovereignty and hierarchical sovereignty control, with a weight of 0.6-0.8 times that of the core layer entities. The association layer contains business terms associated with entities, such as "authorization agreement identifier" associated with "data usage period" and "access permission granularity", with a weight of 0.3-0.5 times that of the core layer entities. The thesaurus is automatically updated every 7 days based on the frequency of entities added to the data space metadata, and extension layer / association layer terms that appear less than a threshold (e.g., less than 5 times per week) are removed.

[0120] Incremental fine-tuning of the large language model is performed based on the aforementioned domain lexicon, employing a "domain sample generation-contrastive learning training" strategy: First, over 100,000 semantic request entries are extracted from historical retrieval logs in the data space, with each request labeled with 3-5 core entities from the domain lexicon as tags; then, a "domain entity masking task" is constructed, randomly masking domain entities in the input text, allowing the model to predict the masked entities. Simultaneously, contrastive learning is introduced, using texts containing domain entities and synonymous texts without domain entities as positive and negative samples to optimize the model's semantic capture ability of domain entities; during fine-tuning, a dynamic learning rate decay strategy is employed, with an initial learning rate of 5e... -5 The model decays by 10% every 2000 steps to ensure that it retains its general semantic understanding capabilities while enhancing its semantic sensitivity in the data space domain.

[0121] Attention-weighted optimization is applied to the initial semantic vector output by the fine-tuned model. The weight of each domain entity in the vocabulary in the user's search request is calculated, and the weight is mapped to the adjustment coefficient of the vector dimension through the attention mechanism. The feature dimensions of the corresponding domain entities in the initial semantic vector are weighted and enhanced. For example, the value of the dimension corresponding to cross-border data sovereignty is multiplied by 1.2-1.5 times. Finally, a high-precision semantic vector is generated. According to the test, the retrieval recall rate of this vector in the data space scenario is improved compared with the general semantic vector, and the false match rate is reduced.

[0122] In this embodiment, core keywords are extracted from the user's search request, and datasets containing these core keywords are matched in the full-text index to generate the first search response result. For example, for a request to "predict changes in the market share of new energy vehicles in the next year," the system extracts core keywords such as "new energy vehicles" and "market share," and searches the full-text index for datasets containing these core keywords, such as new energy vehicle market research reports and related policy documents, as the first search response result.

[0123] like Figure 2 As shown, to delve deeper into the semantic relationships between datasets and output a list of semantically equivalent fields to provide a foundation for data association analysis, this paper proposes to call a large language model to extract the semantic entity list and semantic vectors of each dataset, label semantically equivalent datasets based on the semantic vectors of all datasets, construct a dataset node graph based on the list of all semantic entities, and output a list of semantically equivalent fields based on the semantically equivalent datasets and the dataset node graph, including:

[0124] The large language model is invoked to extract the list of semantic entities and semantic vectors for each dataset;

[0125] Calculate the cosine similarity of the semantic vectors of any two datasets, and mark any two datasets with a similarity greater than a preset similarity threshold as semantically equivalent datasets;

[0126] Analyzing the field names and annotations of a structured dataset and extracting semantic entities to supplement the semantic entity list is an operation for in-depth mining of the dataset's semantic information. For a structured dataset, such as an employee information table, field names might include "Employee ID," "Name," and "Department," and field annotations might further explain these names. Extracting meaningful semantic entities such as "Employee" and "Department" from these field names and annotations and adding them to the semantic entity list previously extracted from the dataset's names and descriptions using a large language model enriches the semantic entity list and more comprehensively reflects the dataset's semantics.

[0127] Constructing a dataset node graph, using datasets as vertices and the existence of intersections in their semantic entity lists, is a graphical structure that visually represents the semantic relationships between datasets. Each dataset is considered a vertex. If two datasets have the same elements in their semantic entity lists (i.e., they intersect), connections are established between the two vertices using these shared elements. For example, dataset A has "products" and "sales" in its semantic entity list, while dataset B has "sales" and "market." Because they both have the intersection of "sales," connections are established between the corresponding vertices of datasets A and B. A graph can be constructed from numerous datasets in this way.

[0128] Verifying the semantic consistency of fields in semantically equivalent datasets and datasets with a correlation greater than a preset correlation threshold in their node graphs, and outputting a list of semantically equivalent fields, aims to identify semantically identical fields across different datasets. For semantically equivalent datasets and datasets with high correlation (greater than the preset correlation threshold), their fields are analyzed in detail. For example, for two semantically equivalent employee information datasets, one with the field "Employee ID" and the other with the field "Employee Number," their semantic consistency is verified by comparing field comments, field values, etc. The semantically consistent fields are then compiled into a list for output, i.e., the list of semantically equivalent fields.

[0129] In this embodiment, the cosine similarity of the semantic vectors of any two datasets is calculated in order to measure the degree of similarity between the two datasets at the semantic level.

[0130] In this embodiment, the preset similarity threshold is a pre-defined standard value used to determine whether two datasets are semantically equivalent. For example, the preset similarity threshold is set to 0.95.

[0131] In this embodiment, the preset correlation threshold is a pre-set numerical standard, for example, 0.8.

[0132] like Figure 4 As shown, in order to provide related recommendations for search response results based on semantic analysis results and improve the comprehensiveness and usability of the search service, a method is proposed to determine related recommendations for search response results based on a list of semantically equivalent fields, and to simultaneously return the search response results and related recommendation results, including:

[0133] This approach, based on a dataset node graph, provides users with a convenient way to browse data, offering both list-based categorization and graph node navigation. List-based categorization involves classifying datasets according to their characteristics or relationships within the node graph and presenting them to the user in a list format. For example, datasets can be categorized by industry, such as medical, financial, and educational. Users can quickly browse datasets within a category by clicking on different categories. Graph node navigation allows users to start from a specific node (i.e., a particular dataset) and navigate along the lines connecting nodes in the node graph (representing semantic relationships) to other related dataset nodes. For instance, starting from the "Medical Patient Information Dataset" node, a user can navigate to related nodes such as the "Medical Diagnosis Results Dataset" through the connections between nodes.

[0134] The first and second search response results in the search results are merged based on the result fusion algorithm to obtain the final search result;

[0135] Generating related recommendation results based on the correlation between the nodes corresponding to the final search results and the remaining nodes in the dataset node graph, as well as the list of semantically equivalent fields, is the process of providing relevant data recommendations to users. When the system provides the final search results, the datasets corresponding to these results are represented as corresponding nodes in the dataset node graph. The correlation between these nodes and the remaining nodes in the graph is calculated; the higher the correlation, the stronger the semantic connection between the dataset corresponding to the remaining node and the dataset of the final search results. Simultaneously, the list of semantically equivalent fields is used to analyze the semantically equivalent fields in these related datasets, thereby determining which datasets to recommend to the user. For example, if the final search result is a dataset related to "electronic product sales data," by analyzing the correlation of the node graph and the list of semantically equivalent fields, it is found that the dataset "electronic product production data" has a high correlation with it and contains semantically equivalent fields; therefore, the dataset "electronic product production data" is provided to the user as a related recommendation result.

[0136] The final search results and related recommendation results are returned synchronously.

[0137] like Figure 4 As shown, in order to scientifically and rationally integrate the results generated by different retrieval methods and obtain higher-quality final retrieval results, a specific execution method for the result fusion algorithm is proposed, including:

[0138] Calculate the reciprocal ranking of each dataset in the first search response result and the reciprocal ranking of each dataset in the second search response result. Assuming there are 10 datasets in the first search response result, sorted from high to low according to relevance and other rules, if a dataset is ranked 3rd, its reciprocal ranking is 1 / 3. Similarly, in the second search response result, if the dataset is ranked 5th, its reciprocal ranking is 1 / 5.

[0139] Determine the weights of the first and second search response results; for example, after a series of analyses, the weight of the first search response result is determined to be 0.6, and the weight of the second search response result is determined to be 0.4.

[0140] The fusion score of the corresponding dataset is obtained by weighting the two inverse rankings based on the weights of the first and second search response results.

[0141] The top-k datasets are selected as the final search results based on their fusion scores, from highest to lowest. This involves filtering from numerous datasets to find the ones that best meet the user's needs. For example, if k is set to 5, then all datasets are sorted by fusion score from highest to lowest, and the top 5 datasets are selected as the final search results presented to the user.

[0142] In this embodiment, taking intelligent semantic retrieval by the user as an example, for instance, query Q1: "During the epidemic, analyze the speed and scope of the epidemic's spread, and track possible close contacts." The specific processing flow is as follows: Figure 4 As shown in the intelligent semantic retrieval.

[0143] S401: Process query Q1 using a full-text search system and return the top-k results R1;

[0144] S402: Using a large language model, convert query Q1 into a semantic vector, query from the vector index database, and return the top-k query results R2;

[0145] S403: Use the query result fusion algorithm "reciprocalrankfusion" to merge the query results R1 and R2 into the final top-k results R3.

[0146] S404: Using R3, based on the semantic association graph of the dataset, and according to the association degree between nodes in the R3 set and other nodes, as well as the list of semantically equivalent fields, another result R4 is returned. Finally, result R3 is returned as the query result, and R4 is returned as the recommended association.

[0147] After the above processing, the result set R3 of Q1 is: hospital visit information, mobile phone trajectory movement information, and shopping mall / supermarket shopping information; the recommended dataset R4 is: highway vehicle capture information, police personnel information, and vehicle registration information.

[0148] To accurately determine the weights of the first and second search response results based on the current application scenario of the data space and optimize the fusion effect of search results, this paper proposes a method for determining the weights of the first and second search response results, including:

[0149] Determining the current application scenario of the data space means identifying the specific scenario within which the current data retrieval operation takes place, such as a government data collaboration scenario, a medical data sharing scenario, or a supply chain data interoperability scenario. This can be done automatically by identifying the user's organization type or by analyzing the scenario entities in the search keywords. Furthermore, it involves defining multiple retrieval requirement indicators for the current application scenario, including: accuracy indicators, comprehensiveness indicators, and timeliness indicators.

[0150] Historical search logs based on the latest historical period within the current application scenario refer to search logs generated within a specific time period in a defined current application scenario, such as government data collaboration. As previously set, this historical period is generally 30 days. These logs record the specific content of the search request, the feedback on search results, including whether it was clicked, saved, or used, as well as data attributes such as update time, correlation dimensions, and matching degree. This allows for the determination of the actual values ​​of accuracy, comprehensiveness, and timeliness indicators, as well as user satisfaction, for each historical search result.

[0151] Using the actual values ​​of accuracy, comprehensiveness, and timeliness indicators from each historical search result as independent variables, and user satisfaction as the dependent variable, a multiple linear regression model is used to determine the linear relationship between various search demand indicators and user satisfaction in the current application scenario. Specifically, the actual values ​​of accuracy, comprehensiveness, and timeliness indicators are extracted and calculated from historical search logs and used as input variables. User satisfaction is calculated by weighting the "search result usage rate plus collection rate" in the logs, i.e., satisfaction equals 0.6 multiplied by usage rate plus 0.4 multiplied by collection rate, and is used as the output variable. A multiple linear regression model is then used, where satisfaction equals β1 multiplied by accuracy score, plus β2 multiplied by comprehensiveness score, plus β3 multiplied by timeliness score, and finally, the error term ε is added (where β1, β2, and β3 are standardized regression coefficients, and accuracy, comprehensiveness, and timeliness scores are normalized values ​​for each indicator), to find the linear relationship between these search demand indicators and user satisfaction.

[0152] Based on the linear relationship between various search demand indicators and user satisfaction in the current application scenario, the optimal weight combination for these indicators—that is, the weight combination that maximizes user satisfaction—is determined. After obtaining the aforementioned linear relationship, the standardized regression coefficients β1, β2, and β3 output by the model represent the weights of accuracy, comprehensiveness, and timeliness, respectively, ensuring that the sum of the accuracy weight, comprehensiveness weight, and timeliness weight equals 1. Under this condition, the weight combination that maximizes user satisfaction is the optimal weight combination for the various search demand indicators in the current application scenario.

[0153] Based on the optimal weight combination of multiple search demand indicators in the current application scenario, an indicator weight matrix is ​​constructed. For example, in the government data collaboration scenario, if the optimal weight combination is an accuracy weight of 0.4, a comprehensiveness weight of 0.3, and a timeliness weight of 0.3, then the constructed indicator weight matrix W is a 1-row, 3-column matrix, written as W equal to [0.4, 0.3, 0.3]. Here, the rows represent the scenario, and the columns correspond to the accuracy, comprehensiveness, and timeliness indicators, respectively.

[0154] Based on the correlation coefficients between various retrieval demand indicators and full-text indexes and vector indexes, an indicator-index correlation matrix is ​​constructed. The element A[i][j] in the matrix represents the correlation strength between the i-th type of indicator and the j-th type of index. This strength value is between 0 and 1, determined according to the characteristics of full-text indexes and vector indexes. For example, full-text indexes perform better in terms of accuracy, meaning strong keyword matching ability, but are relatively weaker in terms of comprehensiveness and timeliness; vector indexes, on the other hand, perform well in terms of comprehensiveness, meaning strong semantic association ability, but are relatively weaker in terms of accuracy. The resulting indicator-index correlation matrix A might look like this: the first row indicates that the correlation strength between accuracy and the full-text index is 0.8, and the correlation strength with the vector index is 0.2; the second row indicates that the correlation strength between comprehensiveness and the full-text index is 0.3, and the correlation strength with the vector index is 0.7; the third row indicates that the correlation strength between timeliness and both the full-text index and the vector index is 0.5.

[0155] Based on the constraint factors of full-text indexing and vector indexing under the current application scenario, a scenario constraint matrix is ​​constructed. The diagonal elements of this matrix are the basic constraint coefficients of the scenario on the indexes, aiming to ensure that the index weights are non-negative and reasonable. Off-diagonal elements are 0, indicating that there is no direct constraint relationship between the indexes. This setting is based on the compliance requirements of the scenario. For example, in the scenario of government data collaboration, to strengthen the constraints of full-text indexing and weaken the constraints of vector indexing, the constructed scenario constraint matrix Cgovernment is equal to... .

[0156] The result of multiplying the indicator weight matrix, the indicator-index association matrix, and the scenario constraint matrix sequentially is used as the optimized index weight matrix. Then, the indicator weight matrix W is multiplied by the indicator-index association matrix A to obtain the initial index weight matrix S. This matrix is ​​1 row and 2 columns, where rows represent scenarios and columns represent indexes. The principle is to distribute the scenario weights to the indicators according to the strength of the association between the indicators and the indexes, across both full-text indexes and vector indexes.

[0157] Based on the double-index weights and normalizing all elements in the optimized index weight matrix, the final index weight matrix is obtained. To ensure that the sum of the final double-index weights is a value that conforms to the actual business situation, according to the verification of the retrieval log, this sum value is set to a specific value. Based on this specific value, all elements in the optimized index weight matrix are normalized to obtain the final index weight matrix. For example, assume that the specific value of the double-index weight sum is 2.2, and the elements in the optimized index weight matrix are S' government[0][0] equal to 0.672 and S' government[0][1] equal to 0.396, and their sum is 1.068. Then the elements in the final index weight matrix are calculated as follows: The first corresponding element in the final index weight matrix is equal to 0.672 divided by 1.068 and then multiplied by 2.2, approximately equal to 1.38, and approximately taken as 1.4; the second corresponding element is equal to 0.396 divided by 1.068 and then multiplied by 2.2, approximately equal to 0.82, and approximately taken as 0.8. In this way, the final index weight matrix is obtained.

[0158] And based on the elements in the final index weight matrix, the weights of the first retrieval response result and the second retrieval response result are determined. The elements in the final index weight matrix directly correspond to the weights of the first retrieval response result (full-text index) and the second retrieval response result (vector index). For example, in the final index weight matrix calculated above, the first element 1.4 is the weight of the first retrieval response result, and the second element 0.8 is the weight of the second retrieval response result.

[0159] In this embodiment, the accuracy index, comprehensiveness index, and timeliness index are important measurement criteria for evaluating retrieval requirements and retrieval results. The accuracy index is to see the degree of exact match between the retrieval result and the user's needs, which can be reflected by the retrieval result match degree and the user's accurate click-through rate. The comprehensiveness index mainly refers to the coverage range of associated data. In the medical scenario, it is reflected in the number of business dimensions covered by the results returned by a single retrieval request, such as dimensions like medical treatment, medication, and trajectory. It can also be measured by the usage rate of associated results. The timeliness index focuses on the freshness of data updates and the real-time result click-through rate.

[0160] In this embodiment, the double-index weight sum refers to the sum of the weight of the first retrieval response result corresponding to the full-text index and the weight of the second retrieval response result corresponding to the vector index.

[0161] In order to accurately extract the actual values of the accuracy, comprehensiveness, and timeliness indexes of each historical retrieval result and the user satisfaction from the historical retrieval log, and provide an effective basis for weight determination, it is proposed to determine the actual values of the accuracy index, comprehensiveness index, and timeliness index of each historical retrieval result and the user satisfaction based on the historical retrieval log in the latest historical period under the current application scenario, including:

[0162] Based on historical search logs within the latest historical period in the current application scenario, the ratio of the number of results containing core keywords or semantic entities in each historical search to the total number of results is determined as the search result matching degree. This means that in the current specific data space application scenario (such as government affairs, medical care, or industrial chain scenarios), based on the search logs of the most recent historical period (usually 30 days), for each historical search, the number of results containing core keywords or semantic entities is counted, and then this number is divided by the total number of search results. The resulting ratio is the search result matching degree, used to measure the degree to which the search results match the user's core needs. For example, in a government affairs data search, there are 100 total results, of which 80 results contain the core keyword "tax data". Then the search result matching degree is 80 ÷ 100 = 0.8.

[0163] The exact click-through rate (CTR) is determined by the ratio of the number of results with a match score of at least a preset threshold to the total number of clicks in each historical search. Specifically, for each historical search, a match score threshold (e.g., 90%) is set, and the number of clicks on results with a match score at or above this threshold is counted. This number is then divided by the total number of clicks to obtain the exact CTR, reflecting the user's focus on exact matches. For example, if a search had 50 clicks, and 30 of them were for results with a match score of at least 90%, the exact CTR would be 30 ÷ 50 = 0.6.

[0164] The number of business dimensions covered by the results returned by a single search request in each historical search result is determined as the number of associated dimensions. In each historical search result, the number of business dimensions covered by the results returned by a single search request is the number of associated dimensions, used to evaluate the comprehensiveness of the search results. Taking a medical scenario as an example, if a search request returns results involving three business dimensions—patient visit information, medication information, and disease trajectory information—then the number of associated dimensions is 3.

[0165] The association utilization rate is determined by the ratio of requests using related results to the total number of requests in each historical search. For each historical search, the number of requests using related results is counted, and then divided by the total number of search requests. The resulting ratio is the association utilization rate, which reflects the degree to which users utilize related results. Assuming there are a total of 100 search requests over a period of time, and 60 of them used related results, the association utilization rate is 60 ÷ 100 = 0.6.

[0166] The freshness of search results is determined by the percentage of data whose update time does not exceed a preset period for the current application scenario within each historical search result. For each historical search result, a time period is set based on the current application scenario (12 hours for the industrial chain scenario, 7 days for the government scenario, and 3 days for the medical scenario). The amount of data whose update time is within this period is counted, and this percentage is divided by the total amount of data. This percentage is used to measure the timeliness of the search results. For example, in a medical scenario search with 200 results, 150 of them were updated within 3 days, so the freshness is 150 ÷ ​​200 = 0.75.

[0167] The real-time click-through rate (CTR) is determined by the ratio of the number of clicks on results with a freshness level of at least a preset freshness threshold to the total number of clicks in each historical search. For each historical search, a freshness threshold (e.g., 80%) is set, and the number of clicks on results with a freshness level of 80% or higher is counted. This count is then divided by the total number of clicks to obtain the real-time CTR, which is used to further measure user attention to fresh data. For example, if there are 80 total clicks, and 50 of them are for results with a freshness level of at least 80%, the real-time CTR would be 50 ÷ 80 = 0.625.

[0168] The actual value of the accuracy metric is determined based on the search result matching degree and precise click-through rate (CTR) of each historical search result. This is then combined using weighted calculations or other statistical methods to obtain the actual value of the accuracy metric. For example, the search result matching degree can be weighted at 0.6, and the precise click-through rate at 0.4. Similarly, the actual value of the comprehensiveness metric is determined based on the number of associated dimensions and the associated usage rate of each historical search result. This can also be combined using weighted calculations to obtain the actual value of the comprehensiveness metric. For example, the number of associated dimensions can be weighted at 0.5, and the associated usage rate at 0.5. Finally, the actual value of the timeliness metric is determined based on the freshness and real-time click-through rate of each historical search result. These two data points are then integrated using similar weighted calculations to obtain the actual value of the timeliness metric. For example, the freshness weight can be 0.6, and the real-time click-through rate weight can be 0.4.

[0169] User satisfaction with each historical search result is calculated based on the click-through rate, collection rate, and usage rate of the search results in the historical search logs within the latest historical period under the current application scenario.

[0170] like Figure 5 As shown, this invention proposes an implementation method for an efficient data catalog management system in a data space, comprising:

[0171] The metadata collection and update module is used to incrementally update metadata information related to data catalogs, data product registration information, and cross-node data catalog information obtained through multiple channels, and to centrally store them in a distributed data storage system.

[0172] The index building and retrieval module is used to build full-text indexes and vector indexes for metadata in the distributed data storage system. Based on the retrieval mode corresponding to the request type of the user's retrieval request, it matches the vector generated based on the request content with the full-text index or vector index to obtain the retrieval response results.

[0173] The consistency analysis and fusion module is used to call the large language model to extract the semantic entity list and semantic vector of each dataset, mark the semantically equivalent datasets based on the semantic vectors of all datasets, construct the dataset node graph based on the list of all semantic entities, and output the semantically equivalent field list based on the semantically equivalent datasets and the dataset node graph.

[0174] The related recommendation and output module is used to determine the related recommendation results of the search response results based on the list of semantically equivalent fields, and synchronously return the search response results and related recommendation results.

[0175] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of this invention and its equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for efficient management of data catalogs in data spaces, characterized in that, The method comprises the following steps: Incremental updating metadata information related to data directory obtained by using a multi-channel mode, data product registration information, and cross-node data directory information, and centrally storing in a distributed data storage system; Building full-text index and vector index of metadata in the distributed data storage system, matching the vector generated based on the request content with the full-text index or the vector index based on the retrieval mode corresponding to the request type of the user retrieval request, and obtaining a retrieval response result; Calling a large language model to extract a semantic entity list and a semantic vector of each data set, marking semantic equivalent data sets based on the semantic vectors of all data sets, constructing a data set node graph based on all semantic entity lists, and outputting a semantic equivalent field list based on the semantic equivalent data sets and the data set node graph, comprising: Calling a large language model to extract a semantic entity list and a semantic vector of each data set; Calculating the cosine similarity of the semantic vectors of any two data sets, and marking any two data sets with a similarity greater than a preset similarity threshold as semantic equivalent data sets; Analyzing the field name and field annotation of the structured data set and supplementing the semantic entities to the semantic entity list; Constructing a data set node graph with the data set as the vertex and the intersection of the semantic entity list as the condition; Verifying the semantic consistency of the data sets with an association degree greater than a preset association degree threshold in the semantic equivalent data sets and the data set node graph, and outputting a semantic equivalent field list; Determining the associated recommendation result of the retrieval response result based on the semantic equivalent field list, and synchronously returning the retrieval response result and the associated recommendation result, comprising: Providing list classification browsing and graph node navigation based on the data set node graph; Fusing the first retrieval response result and the second retrieval response result in the retrieval response result based on a result fusion algorithm to obtain a final retrieval result; Generating an associated recommendation result based on the association degree of the node corresponding to the final retrieval result and the remaining nodes in the data set node graph and the semantic equivalent field list; Synchronously returning the final retrieval result and the associated recommendation result.

2. The method for efficient management of data catalog in data space according to claim 1, characterized in that, The distributed data storage system comprises a distributed storage subsystem compatible with an object storage protocol, a distributed relational data subsystem, and a vector index subsystem. The distributed storage subsystem compatible with the object storage protocol is used to store unstructured data and serves as the underlying storage support for the distributed relational data subsystem and the vector index subsystem.

3. The method for efficient management of data catalog in data space according to claim 1, characterized in that, Incremental updating metadata information related to data directory obtained by using a multi-channel mode, data product registration information, and cross-node data directory information, and centrally storing in a distributed data storage system, comprising: Receiving manually registered metadata information through a directory upload interface of a data connector, and directly accessing a data provider application system to extract metadata as a backend reverse proxy of the data connector, and generating data product registration information according to a data product registration template; When the metadata updating service is deployed near a regional function node or a global function node, retrieving cross-node data directory information through a query service of the function node; Encode metadata information, data product registration information and cross-node data directory information into a metadata operation log and write into a distributed data storage system to complete metadata incremental update.

4. The method for efficient management of data catalog in data space according to claim 1, characterized in that, Construct full-text index and vector index of metadata in the distributed data storage system, and based on the retrieval mode corresponding to the request type of the user retrieval request, match the vector generated based on the request content with the full-text index or the vector index to obtain a retrieval response result, including: Constructing full-text index and vector index based on metadata in the distributed data storage system; If the request type of the user retrieval request is a keyword or semi-structured keyword request, preferentially perform accurate matching based on the keyword in the user retrieval request in the full-text index matching process to generate a first retrieval response result, and at the same time, convert the user retrieval request into a basic semantic vector through a large language model and match the semantic related data set of the basic semantic vector in the vector index to generate a second retrieval response result; If the request type of the user retrieval request is a semantic demand request, preferentially perform accurate matching based on the high-precision semantic vector converted from the user retrieval request in the vector index matching process to generate a second retrieval response result, and at the same time, extract the core keyword in the user retrieval request and match the data set containing the core keyword in the full-text index to generate a first retrieval response result; The retrieval response result includes the first retrieval response result and the second retrieval response result generated synchronously.

5. The method for efficient management of data catalog in data space according to claim 1, characterized in that, The specific execution method of the result fusion algorithm includes: Calculate the reciprocal of the ranking of each data set in the first retrieval response result and the reciprocal of the ranking of each data set in the second retrieval response result, respectively; Determine the weight of the first retrieval response result and the second retrieval response result; Based on the weight of the first retrieval response result and the second retrieval response result, perform weighted operation on the two reciprocals to obtain the fusion score of the corresponding data set; Select the top-k data sets as the final retrieval result in the order from high to low of the fusion score.

6. The method for efficient management of data catalog in data space according to claim 5, characterized in that, Determining the weight of the first retrieval response result and the second retrieval response result includes: Determine the current application scenario of the data space and define multiple retrieval demand indicators of the current application scenario, wherein the multiple retrieval demand indicators include: accuracy indicator, comprehensiveness indicator, timeliness indicator; Based on the historical retrieval log in the latest historical period under the current application scenario, determine the actual value of the accuracy indicator, the comprehensiveness indicator, and the timeliness indicator of each historical retrieval result and the user satisfaction; Take the actual value of the accuracy indicator, the comprehensiveness indicator, and the timeliness indicator of each historical retrieval result as the independent variable, and take the user satisfaction as the dependent variable, and use a multiple linear regression model to determine the linear relationship between the multiple retrieval demand indicators and the user satisfaction under the current application scenario; Based on the linear relationship between the multiple retrieval demand indicators and the user satisfaction under the current application scenario, determine the best weight combination of the multiple retrieval demand indicators under the current application scenario; Based on the best weight combination of the multi-class retrieval demand indicators in the current application scenario, an indicator weight matrix is constructed, and based on the correlation coefficients between the multi-class retrieval demand indicators and the full-text index and vector index, an indicator-index correlation matrix is constructed, and based on the constraint factors of the full-text index and vector index in the current application scenario, a scene constraint matrix is constructed; The result obtained by multiplying the indicator weight matrix, the indicator-index correlation matrix and the scene constraint matrix in turn is taken as the optimized index weight matrix, the elements in the optimized index weight matrix are normalized based on the double index weight, and the final index weight matrix is obtained, and the weights of the first retrieval response result and the second retrieval response result are determined based on the elements in the final index weight matrix.

7. The method for efficient management of data catalog in data space according to claim 6, characterized in that, Based on the historical retrieval logs in the latest historical period under the current application scenario, the actual values of the accuracy indicator, the comprehensiveness indicator, the timeliness indicator and the user satisfaction of each historical retrieval result are determined, including: Based on the historical retrieval logs in the latest historical period under the current application scenario, the ratio of the number of results containing core keywords or semantic entities in each historical retrieval result to the total number of results is determined as the retrieval result matching degree, the ratio of the number of results with a click matching degree not less than a preset matching degree threshold to the total number of clicks in each historical retrieval result is determined as the accurate click rate, the number of business dimensions covered by the returned results of each single retrieval request in each historical retrieval result is determined as the associated dimension number, the ratio of the number of requests using associated results to the total number of requests in each historical retrieval result is determined as the associated usage rate, the proportion of data with an update time not exceeding a preset period of time of the current application scenario in each historical retrieval result is determined as the freshness, and the ratio of the number of results with a click freshness not less than a preset freshness threshold to the total number of results in each historical retrieval result is determined as the real-time click rate; Based on the retrieval result matching degree and the accurate click rate of each historical retrieval result, the actual value of the accuracy indicator is determined, based on the associated dimension number and the associated usage rate of each historical retrieval result, the actual value of the comprehensiveness indicator is determined, and based on the freshness and the real-time click rate of each historical retrieval result, the actual value of the timeliness indicator is determined; Based on the click rate, the collection rate and the usage rate of each historical retrieval result in the historical retrieval logs in the latest historical period under the current application scenario, the user satisfaction of each historical retrieval result is calculated.

8. A system for efficient management of data catalogs in data spaces, characterized in that, It includes: A metadata acquisition and update module for performing metadata incremental update on metadata information related to data directory obtained by using a multi-channel method, data product registration information and cross-node data directory information, and centrally storing in a distributed data storage system; An index construction and retrieval module for constructing full-text index and vector index of metadata in a distributed data storage system, matching the vector generated based on the request content with the full-text index or the vector index based on the retrieval mode corresponding to the request type of the user retrieval request, and obtaining a retrieval response result; The consistency analysis and fusion module is configured to call a large language model to extract a semantic entity list and a semantic vector of each data set, mark semantic equivalent data sets based on the semantic vectors of all data sets, construct a data set node graph based on all semantic entity lists, and output a semantic equivalent field list based on the semantic equivalent data sets and the data set node graph, including: The consistency analysis and fusion module is configured to call a large language model to extract a semantic entity list and a semantic vector of each data set, mark semantic equivalent data sets based on the semantic vectors of all data sets, construct a data set node graph based on all semantic entity lists, and output a semantic equivalent field list based on the semantic equivalent data sets and the data set node graph, including: The consistency analysis and fusion module is configured to call a large language model to extract a semantic entity list and a semantic vector of each data set, mark semantic equivalent data sets based on the semantic vectors of all data sets, construct a data set node graph based on all semantic entity lists, and output a semantic equivalent field list based on the semantic equivalent data sets and the data set node graph, including: The consistency analysis and fusion module is configured to call a large language model to extract a semantic entity list and a semantic vector of each data set, mark semantic equivalent data sets based on the semantic vectors of all data sets, construct a data set node graph based on all semantic entity lists, and output a semantic equivalent field list based on the semantic equivalent data sets and the data set node graph, including: The consistency analysis and fusion module is configured to call a large language model to extract a semantic entity list and a semantic vector of each data set, mark semantic equivalent data sets based on the semantic vectors of all data sets, construct a data set node graph based on all semantic entity lists, and output a semantic equivalent field list based on the semantic equivalent data sets and the data set node graph, including: The association recommendation and output module is configured to determine an association recommendation result of a search response result based on the semantic equivalent field list, and synchronously return the search response result and the association recommendation result, including: The association recommendation and output module is configured to determine an association recommendation result of a search response result based on the semantic equivalent field list, and synchronously return the search response result and the association recommendation result, including: The association recommendation and output module is configured to determine an association recommendation result of a search response result based on the semantic equivalent field list, and synchronously return the search response result and the association recommendation result, including: The association recommendation and output module is configured to determine an association recommendation result of a search response result based on the semantic equivalent field list, and synchronously return the search response result and the association recommendation result, including: ​ ​

Images