Metadata processing method and system

Abstract

The application provides a metadata processing method and system, the method comprising: obtaining a search semantic vector and a search intention vector of a metadata search request; obtaining a target classification dimension corresponding to the metadata search request according to the search semantic vector and the search intention vector; calculating a semantic similarity between the search semantic vector and data content features of each metadata in a metadata database, a classification matching degree between the target classification dimension and classification labels of each metadata, and an association closeness between context data corresponding to the search intention vector and each metadata; and performing a matching search in the metadata database according to the semantic similarity, the classification matching degree, the association closeness, and access activity and timeliness parameters of each metadata, to obtain response information corresponding to the metadata search request. The application improves the search efficiency and the search accuracy through multi-dimensional intelligent search.

Description

Technical Field

[0001] This invention relates to the field of data management technology, and in particular to a metadata processing method and system. Background Technology

[0002] With the development of big data technology, data lakes have become the core carrier for centralized storage of massive amounts of heterogeneous data from multiple sources. Metadata, as the navigation label of a data lake, directly determines the efficiency of data asset value mining through its management level. Therefore, in order to manage and utilize data assets in a data lake, metadata is typically collected and retrieved through a metadata management system.

[0003] Existing metadata retrieval methods mainly rely on user-input keywords for retrieval. However, this method is primarily based on literal character matching logic. When the search terms are incorrect or differ from the description in the metadata, repeated trial and error are required, making it difficult to quickly and accurately retrieve relevant data. Summary of the Invention

[0004] This invention provides a metadata processing method and system to address the shortcomings of existing technologies that rely on literal characters for metadata retrieval, resulting in low retrieval efficiency and poor retrieval accuracy. This method improves both retrieval efficiency and retrieval accuracy.

[0005] This invention provides a metadata processing method, comprising: In response to a metadata retrieval request, obtain the retrieval semantic vector and retrieval intent vector of the metadata retrieval request; Based on the retrieval semantic vector and the retrieval intent vector, the target classification dimension corresponding to the metadata retrieval request is obtained; Calculate the semantic similarity between the retrieval semantic vector and the data content features of each metadata in the metadata database, the classification matching degree between the target classification dimension and the classification labels of each metadata, and the correlation between the context data corresponding to the retrieval intent vector and each metadata. Based on the semantic similarity, classification matching degree, association tightness, and access activity and timeliness parameters of each metadata, a matching search is performed in the metadata database to obtain the response information corresponding to the metadata retrieval request.

[0006] According to a metadata processing method provided by the present invention, for each piece of metadata in the metadata database, the step of obtaining the classification tag of the metadata includes: Obtain the classification model for the target classification period; the target classification period is the classification period to which the metadata belongs; The multi-dimensional information of the metadata is input into the feature extraction layer of the classification model to obtain a multi-dimensional feature vector; the multi-dimensional information includes the descriptive text of the metadata, the association weight between the metadata and each business, and the user behavior information associated with the metadata; The multi-dimensional feature vector is input into the attention mechanism layer of the classification model to obtain the fused feature vector; The fused feature vector is input into the classification layer of the classification model to obtain the classification label of the metadata.

[0007] According to a metadata processing method provided by the present invention, obtaining the classification model for the target classification period includes: The statistics include the proportion of new business data in the data lake and the rate of change of user search intent in the previous classification period of the target classification period, as well as the classification accuracy of the classification model in the previous classification period. When the percentage reaches the percentage threshold, the rate of change reaches the rate of change threshold, or the classification accuracy is lower than the accuracy threshold, the training dataset of the previous classification period is updated based on the multi-dimensional information and classification labels of the metadata corresponding to the new business data to obtain the training dataset of the target classification period. Based on the training dataset of the target classification period, the classification model of the previous classification period is iteratively trained to obtain the classification model of the target classification period.

[0008] According to a metadata processing method provided by the present invention, the step of inputting the multi-dimensional information of the metadata into the feature extraction layer of the classification model to obtain a multi-dimensional feature vector includes: The descriptive text is input into the convolutional network layer of the feature extraction layer to obtain the data content feature vector of the metadata. The association weights are input into the feature embedding layer of the feature extraction layer to obtain the business feature vector of the metadata; The user behavior information is input into the temporal feature processing layer of the feature extraction layer to obtain the user behavior feature vector of the metadata. The multi-dimensional feature vector is determined based on the data content feature vector, the business feature vector, and the user behavior feature vector.

[0009] According to a metadata processing method provided by the present invention, the steps for obtaining multi-dimensional information of the metadata include: When new target source data is detected in the data lake through any of the multiple detection modes, the metadata is generated according to the data type of the target source data, and the descriptive text is extracted from the metadata; the target source data is the source data to which the metadata belongs; In the first knowledge graph, the data entities of the metadata and the business entities of each business are determined, and the association weights are determined based on the topological connection relationships between the data entities and each business entity. In the first knowledge graph, obtain the target user entity associated with the data entity, and obtain the user behavior information based on the behavior operation record formed by the user object corresponding to the target user entity for the metadata operation; The various detection modes include multiple modes such as a detection mode that sniffs network transmission characteristics, a detection mode that sends heartbeat probe packets to the data source port, and a detection mode that actively obtains data source access information through the database management protocol.

[0010] According to a metadata processing method provided by the present invention, generating the metadata based on the data type of the target source data includes: When the data type is structured data, the content information of the changed fields of the target source data is obtained, and the semantic information and business tags of the content information of the changed fields are parsed using a pre-trained language model. Based on the parsing results, the metadata is obtained. When the data type is semi-structured data, extract the hierarchical structure information of the target source data, and obtain the metadata based on the hierarchical structure information; When the data type is unstructured data, the target source data is parsed according to the parsing algorithm corresponding to the modality type to which the target source data belongs, and the metadata is obtained based on the parsing result; The modal types include text, images, audio, or video.

[0011] According to a metadata processing method provided by the present invention, obtaining the retrieval semantic vector and retrieval intent vector of the metadata retrieval request includes: The metadata retrieval request is segmented to obtain a sequence of retrieval keywords; The search keyword sequence is input into a pre-trained language model to generate a search semantic vector; Based on the retrieval semantic vector, the associated paths are obtained in the second knowledge graph; the associated paths include user entities, behavioral entities, and data entities associated with the retrieval semantic vector; The intent is inferred based on the associated path to obtain the retrieval intent vector.

[0012] According to a metadata processing method provided by the present invention, the method further includes: Obtain real-time operation logs and real-time data fingerprints of each source data in the data lake; If any source data has a data change operation in its real-time operation log, or if the real-time data fingerprint of any source data is inconsistent with the original data fingerprint in the metadata of any source data, then the source data is determined to be changed data. Based on the business importance level of the changed data, a target update strategy is determined from multiple preset update strategies; According to the target update strategy, the metadata of the changed data is updated in the metadata database; The multiple preset update strategies include a second-level update strategy, a minute-level update strategy, and an hour-level update strategy; the second-level update strategy is used to atomically update the metadata of the changed data through a distributed lock mechanism; the minute-level update strategy is used to update the metadata of the changed data through a batch asynchronous processing mechanism.

[0013] According to a metadata processing method provided by the present invention, the method further includes: Obtain the update record of the metadata of the changed data; the update record includes at least one of the following: change time, change type, change content, operator information, and real-time data fingerprint of the changed data; The update record is written into the blockchain network to obtain the change log chain; In response to a query request for the change log chain, the change log chain is retrieved according to the query parameters in the query request, and the target change record is returned based on the retrieval results; The query parameters include at least one of the following: time range, change type, operator, or data identifier.

[0014] According to a metadata processing method provided by the present invention, the step of performing a matching search in the metadata database based on the semantic similarity, the classification matching degree, the association tightness, and the access activity and timeliness parameters of each of the metadata, to obtain the response information corresponding to the metadata search request, includes: The semantic similarity, classification matching degree, association tightness, access activity and timeliness parameters are weighted and fused to obtain the comprehensive score of each metadata. The response information is generated based on the target metadata with the highest comprehensive score and the associated information of the target metadata; the associated information includes the source data of the target metadata, as well as metadata and lineage maps similar to the target metadata.

[0015] The present invention also provides a metadata processing system, comprising: The first acquisition unit is used to acquire the retrieval semantic vector and retrieval intent vector of the metadata retrieval request in response to the metadata retrieval request. The second acquisition unit is used to acquire the target classification dimension corresponding to the metadata retrieval request based on the retrieval semantic vector and the retrieval intent vector. The calculation unit is used to calculate the semantic similarity between the retrieval semantic vector and the data content features of each metadata in the metadata database, the classification matching degree between the target classification dimension and the classification labels of each metadata, and the correlation between the context data corresponding to the retrieval intent vector and each metadata. The retrieval unit is used to perform matching retrieval in the metadata database based on the semantic similarity, the classification matching degree, the association tightness, and the access activity and timeliness parameters of each of the metadata, and obtain the response information corresponding to the metadata retrieval request.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the metadata processing method described above.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the metadata processing method as described above.

[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the metadata processing method described above.

[0019] The metadata processing method and system provided by this invention obtains retrieval semantic vectors and retrieval intent vectors in response to metadata retrieval requests. This not only understands the literal meaning of the user's words but also provides insights into the user's business intent. By further calculating semantic similarity, classification matching degree, and correlation tightness, and combining data access activity and timeliness parameters for multi-dimensional matching retrieval, this method changes the inefficient traditional model that relies solely on keyword matching. It can effectively achieve rapid and accurate capture of retrieval information, effectively solving the problem of retrieval failures caused by spelling errors or terminology differences. At the same time, it can prioritize the display of data with high popularity, high timeliness, and strong business relevance, thereby significantly improving the recall rate, precision rate, and efficiency of users in obtaining high-value data in metadata retrieval. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the metadata processing method provided by the present invention.

[0022] Figure 2 This is a schematic diagram of the metadata processing system provided by the present invention.

[0023] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0025] In the wave of digital transformation, data has become a core strategic asset for enterprises and organizations, and data lakes, as a key architecture for carrying massive amounts of multi-source heterogeneous data, are experiencing explosive growth in development and application. Data lakes break away from the traditional data warehouse's sole reliance on structured data, being compatible with unstructured data such as text, images, audio, and video, as well as semi-structured data such as JSON and Extensible Markup Language (XML). Simultaneously, they integrate structured business data generated by Enterprise Resource Planning (ERP) systems and Customer Relationship Management (CRM) systems, forming a data aggregation center covering all business scenarios. Sectors such as finance, healthcare, manufacturing, and the internet all regard data lakes as a core infrastructure for enabling data-driven decision-making.

[0026] Metadata, as the nerve center of a data lake, is defined as data describing its attributes, sources, structure, relationships, usage permissions, and business implications. It is a prerequisite for the orderly management and efficient utilization of a data lake. In the data lake's operational system, metadata plays multiple key roles: First, it serves as a catalog index for data assets, helping users quickly locate the data they need; second, it acts as a data quality monitoring probe, ensuring data reliability by recording indicators such as data integrity and accuracy; third, it is a data lineage tracer, clearly presenting the entire process of data from generation and processing to application, providing support for data traceability and troubleshooting; and fourth, it acts as a data security barrier, achieving hierarchical data control by labeling data sensitivity levels and access permissions. Therefore, the maturity of metadata management directly determines the efficiency and value output capability of the data lake.

[0027] However, as data lakes continue to expand in scale and data complexity increases, existing metadata management technologies have gradually revealed many insurmountable flaws, becoming the main bottleneck restricting the release of data lake value. These flaws are specifically manifested as follows: Existing metadata retrieval functions exhibit low accuracy and efficiency. Current metadata retrieval methods primarily rely on exact keyword matching, lacking semantic understanding and association analysis capabilities. On the one hand, users must accurately input complete keywords from the metadata to obtain search results. When the search terms are incorrect or differ from the description in the metadata, repeated trial and error are necessary, making it difficult to quickly and accurately retrieve relevant data, severely impacting work efficiency.

[0028] To address this issue, this application provides a metadata processing method.

[0029] It should be noted that this method is a metadata processing method applicable to various data lake scenarios, including but not limited to data lake scenarios in finance, healthcare, manufacturing, and the internet. This embodiment does not specifically limit these scenarios. The execution entity of this method is a metadata processing system, which can be a server, network attached storage, personal computer, etc., and this invention does not specifically limit it.

[0030] Figure 1 This is a flowchart illustrating the metadata processing method provided by the present invention. Figure 1 As shown, the method includes: Step 110: In response to the metadata retrieval request, obtain the retrieval semantic vector and retrieval intent vector of the metadata retrieval request.

[0031] Optionally, after obtaining the metadata retrieval request, a retrieval engine can be invoked to analyze the metadata retrieval request to obtain the retrieval semantic vector and retrieval intent vector of the metadata retrieval request. In this process, the corresponding retrieval semantic vector and retrieval intent vector can be obtained by mapping the metadata retrieval request with a pre-established association between the metadata retrieval request and the retrieval semantic vector and retrieval intent vector; or, an existing network model with the ability to parse retrieval semantic vector and retrieval intent vector, such as an existing Large Language Model (LLM) or Natural Language Processing (NLP) model, can be invoked to encode and parse the metadata retrieval request to obtain the corresponding retrieval semantic vector and retrieval intent vector, etc. This embodiment does not specifically limit this.

[0032] The search engine here can be a distributed search and analysis (such as the Elasticsearch engine) underlying storage and retrieval framework, deployed in a distributed cluster, supporting horizontal scaling, with a maximum index capacity of 1 billion in a single cluster.

[0033] The metadata retrieval request here refers to a retrieval command entered to find specific metadata. This can be user input on the front-end interface, or information generated under certain conditions by other controllers or devices within the storage system, such as a timed retrieval request. User input can include information input through command-line interfaces, graphical interfaces, touch input, drop-down selection, voice input, gesture input, visual input, brain-computer interface input, etc., and this embodiment does not specifically limit this.

[0034] The retrieval semantic vector here refers to the numerical representation of the textual information in the metadata retrieval request mapped to a high-dimensional vector space. It is used to characterize the semantic information of the retrieval request at the word and phrase levels, and can capture the inherent relationships between synonyms and near-synonyms. The retrieval intent vector refers to the numerical vector representing the intent behind initiating the metadata retrieval request, such as the business purpose, task type, or desired data format.

[0035] Step 120: Obtain the target classification dimension corresponding to the metadata retrieval request based on the retrieval semantic vector and the retrieval intent vector.

[0036] Optionally, after obtaining the retrieval semantic vector and the retrieval intent vector, the distance between the retrieval semantic vector and the retrieval intent vector and the feature vectors of each preset classification dimension can be calculated, and the dimension with the closest distance can be determined as the target classification dimension, etc. This embodiment does not specifically limit this.

[0037] The target classification dimension here refers to one or more classification categories, levels, or tag sets that match the current metadata retrieval request in the classification system of the metadata processing system.

[0038] Step 130: Calculate the semantic similarity between the retrieval semantic vector and the data content features of each metadata in the metadata database, the classification matching degree between the target classification dimension and the classification labels of each metadata, and the correlation between the context data corresponding to the retrieval intent vector and each metadata.

[0039] Optionally, the semantic similarity between the retrieval semantic vector and the data content features of each metadata in the metadata database, the classification matching degree between the target classification dimension and the classification labels of each metadata, and the correlation between the context data corresponding to the retrieval intent vector and each metadata can be further calculated.

[0040] The metadata database here refers to a storage system that stores collected, cleaned, and processed metadata. The data content features of each metadata element refer to the vector representation generated after feature extraction from the descriptive text of the metadata.

[0041] Semantic similarity can be calculated by measuring the cosine similarity, Euclidean distance, or Manhattan distance between the retrieval semantic vector and the data content features, in order to measure whether the retrieval terms and metadata match at the semantic level.

[0042] Category tags refer to category identifiers assigned to metadata, including but not limited to multi-level category tags such as industry general, enterprise business, department specific, data type, and sensitivity level.

[0043] The category matching degree here can be calculated by determining whether the category label of the metadata is included in or belongs to the target category dimension. For example, if it belongs, the matching degree is 1, and if it does not belong, it is 0; or a value between 0 and 1 can be calculated based on the similarity between the category label and the target category dimension.

[0044] Contextual data refers to auxiliary information entities that are directly or indirectly related to the core object pointed to by the retrieval intent vector in business logic or data flow relationships. For example, when the retrieval intent vector points to a specific dataset, the contextual data implied by the intent may include the upstream data source of the dataset (such as the source system that generated the data, the original file), the downstream application reports (such as the analysis charts and business reports generated based on the data), the relevant business process nodes, and the departmental entities responsible for the data.

[0045] The correlation tightness here can be calculated by measuring the graph distance, path hops, or connection weight between the nodes corresponding to each metadata and the nodes corresponding to the context data of the retrieval intent vector in the knowledge graph or association network.

[0046] Step 140: Based on the semantic similarity, classification matching degree, association tightness, and access activity and timeliness parameters of each metadata, perform matching retrieval in the metadata database to obtain the response information corresponding to the metadata retrieval request.

[0047] Optionally, after obtaining semantic similarity, classification matching degree, and association tightness, multi-dimensional data retrieval can be performed by combining semantic similarity, classification matching degree, association tightness, and access activity and timeliness parameters of each metadata to obtain the corresponding response information. During the matching retrieval process, a weighted sum of semantic similarity, classification matching degree, association tightness, access activity, and timeliness parameters can be used to obtain a comprehensive score for each metadata, and a preset number of metadata can be selected in descending order of the comprehensive score; alternatively, a multi-round sorting strategy can be adopted, first filtering based on classification matching degree, then coarse sorting based on semantic similarity and association tightness, and finally fine sorting based on access activity and timeliness parameters, etc. This embodiment does not specifically limit this approach.

[0048] The access activity here includes statistical metrics on the frequency of metadata access, browsing, downloading, referencing, and bookmarking within a preset historical time period.

[0049] The timeliness parameter here refers to an indicator that characterizes the freshness of metadata or its corresponding source data in the time dimension, i.e., data freshness. Specifically, it can be a time decay factor calculated based on the time difference between the creation time and last modification time of the data file corresponding to the metadata and the current system time, or a comprehensive measure based on the update frequency of the data source and the last update time, used to reflect whether the data is the latest business status, so as to avoid retrieving outdated historical data.

[0050] The response information here refers to the final search results returned to the requesting party. This may include a list of matched metadata, metadata detail cards, a data lineage graph, and a list of recommended similar metadata. The data lineage graph uses a visual graph component, supporting node dragging, hierarchical expansion, and highlighting of related paths, allowing users to intuitively view the upstream and downstream relationships of the data. The search results can be visually displayed on the front-end page, which supports various data operations, including but not limited to viewing metadata details, data previews, and obtaining download links.

[0051] It should be noted that, after verification, the method provided in this embodiment can achieve an average retrieval response time of ≤300 milliseconds, and the accuracy of the top 5 search results is over 95%, thereby significantly improving the user's search experience.

[0052] The metadata processing method provided in this embodiment obtains retrieval semantic vectors and retrieval intent vectors in response to metadata retrieval requests. This not only understands the literal meaning of the user's words but also provides insights into the user's business intent. By further calculating semantic similarity, classification matching degree, and correlation tightness, and combining data access activity and timeliness parameters for multi-dimensional matching retrieval, it changes the inefficient mode of relying solely on keyword matching in the traditional way. This effectively enables the rapid and accurate capture of retrieval information, effectively solving the problem of retrieval failures caused by spelling errors or terminology differences. At the same time, it can prioritize the display of data with high popularity, high timeliness, and strong business relevance, thereby significantly improving the recall rate, precision rate, and efficiency of users in obtaining high-value data in metadata retrieval.

[0053] In some embodiments, for each piece of metadata, the dynamic classification steps are as follows: Step 210: Obtain the classification model for the target classification period; the target classification period is the classification period to which the metadata belongs.

[0054] Step 220: Input the multi-dimensional information of the metadata into the feature extraction layer of the classification model to obtain a multi-dimensional feature vector.

[0055] Step 230: Input the multi-dimensional feature vector into the attention mechanism layer of the classification model to obtain the fused feature vector.

[0056] Step 240: Input the fused feature vector into the attention mechanism layer of the classification model to obtain the classification label of the metadata.

[0057] It's important to note that existing metadata classification systems are static and simplistic. Specifically, current technologies often use pre-defined static tags to classify metadata, such as categorizing it solely by data format (structured / unstructured) or business system (financial / human resources). This classification method fails to reflect the dynamic characteristics and complex relationships of data: on the one hand, as business scenarios expand, the application scenarios of data may change, and static tags are difficult to update synchronously, leading to a disconnect between classification results and actual needs; on the other hand, the relationships between data, such as the relationship between user behavior data and order data, or the relationship between supply chain data and production data, cannot be reflected through a single classification dimension. Users find it difficult to integrate data from a business logic perspective, severely impacting data processing efficiency.

[0058] To address this, this application dynamically acquires a classification model and then uses this dynamically acquired learning model to obtain multi-level classification labels for metadata, thereby achieving both accuracy and adaptability in metadata classification. The specific implementation steps are as follows: When classifying each piece of metadata, the classification model for the target classification period is first dynamically obtained. During this process, the classification period to which the metadata belongs is determined based on its classification requirement timestamp, thus obtaining the target classification period, and the classification model corresponding to that target classification period is loaded. It should be noted that the classification model for the target classification period can be the classification model of the previous classification period, or it can be a model inherited from the previous classification period and fine-tuned or incrementally updated based on the incremental data of the current period. This ensures that the model can continue historical knowledge and adapt to new business changes, etc. This embodiment does not specifically limit this.

[0059] After obtaining the classification model for the target classification period, multi-dimensional information of the metadata can be extracted and input into the feature extraction layer of the classification model for parallel processing to output the feature vectors corresponding to each dimension, i.e., multi-dimensional feature vectors. In this process, the feature extraction layer may contain multiple independent sub-neural network modules, each used to process input data of different modalities, etc. This embodiment does not specifically limit this.

[0060] The multi-dimensional information here includes the descriptive text of the metadata, the association weights between the metadata and various business entities, and user behavior information associated with the metadata. The descriptive text includes natural language descriptions such as the metadata's name, annotations, and summary; the association weights include the connection strength values ​​between the metadata and various business entities (such as departments, processes, and metrics) in the knowledge graph; and the user behavior information includes the user's historical operation sequence on the metadata within a specific historical period, such as search, access, and usage records for the past three months. This embodiment does not specifically limit this.

[0061] The feature extraction layer here refers to the set of neural network layers in the classification model used to extract high-level abstract features from the original input data. The multi-dimensional feature vector refers to the set of vectors output by the feature extraction layer that respectively represent the metadata's features in three dimensions: content, business scenario, and user behavior.

[0062] Optionally, after obtaining the multi-dimensional feature vectors, they can be input into the attention mechanism layer of the classification model. The attention mechanism can then be used to weight and fuse the feature vectors of each dimension to obtain a fused feature vector. In this process, a self-attention mechanism can be used to calculate the dependencies within each feature vector; or a multi-head attention mechanism can be used to capture the feature interaction information in different subspaces and then weighted and summed and concatenated the feature vectors according to the learned attention weights. This embodiment does not specifically limit the specific implementation of these methods.

[0063] The attention mechanism layer here refers to the existing network layer in the classification model used to automatically learn the importance weights of different features to the classification result. The fused feature vector refers to a unified feature representation that, after weighted fusion, comprehensively reflects metadata content attributes, business scenario attributes, and user behavior preferences.

[0064] Optionally, after obtaining the fused feature vector, it can be input into the classification layer of the classification model for calculation to obtain the classification label of the metadata. In this process, a fully connected layer can be used to map the fused feature vector to the label space, and the probability distribution of each classification label can be output through the Softmax activation function, selecting the label with the highest probability as the final result; or, a classifier such as a support vector machine can be used for discrimination, etc. This embodiment does not specifically limit this.

[0065] Category tags refer to the category identifiers to which metadata belongs. They support multi-level classification systems, including general industry classifications, enterprise business classifications, departmental special classifications, data type classifications, and sensitivity level classifications. Each level of tags can be adjusted independently.

[0066] The classification tag acquisition method provided in this embodiment constructs a deep learning classification model that includes a feature extraction layer, an attention mechanism layer, and a classification layer. It integrates multi-dimensional information such as metadata description text, business association weights, and user behavior information, solving the problem that traditional static classification methods have a single dimension and cannot reflect complex relationships. In particular, the introduction of the attention mechanism layer can dynamically adjust the weights of features of different dimensions, making the classification results more accurately match the actual business scenarios and user habits, thus providing a precise data foundation for efficient and accurate retrieval.

[0067] In some embodiments, step 210 specifically includes: Step 211: Calculate the proportion of new business data in the data lake and the rate of change of user search intent in the previous classification period of the target classification period, as well as the classification accuracy of the classification model in the previous classification period.

[0068] Optionally, upon entering the target classification period, statistical analysis can be performed on the proportion of new business data in the data lake during the previous classification period, the rate of change in user search intent, and the classification accuracy of the classification model from the previous classification period. During this process, backend monitoring components can be used to aggregate data access logs and user search logs in real time, calculating statistical values ​​for various indicators (i.e., proportion, rate of change, classification accuracy), etc. This embodiment does not specifically limit this.

[0069] The percentage of newly added business data here refers to the proportion of data newly added to the data lake in the previous period to the total data volume. The rate of change in user search intent refers to the frequency of significant changes in user search terms or search behavior patterns, such as the growth rate of search frequency for a certain type of emerging business terminology. Classification accuracy refers to the proportion of correct predictions made by the classification model on the actual business validation set or user feedback samples in the previous period.

[0070] Step 212: If the percentage reaches the percentage threshold, the rate of change reaches the rate of change threshold, or the classification accuracy is lower than the accuracy threshold, the training dataset of the previous classification period is updated according to the multi-dimensional information and classification labels of the metadata corresponding to the new business data, so as to obtain the training dataset of the target classification period.

[0071] Step 213: Based on the training dataset of the target classification period, iteratively train the classification model of the previous classification period to obtain the classification model of the target classification period.

[0072] Optionally, after obtaining the statistical indicators, they can be compared with preset thresholds. During this process, the thresholds can be set according to actual needs, such as a percentage threshold of 15%, a change rate threshold of 20%, and an accuracy threshold of 92%. If any triggering condition is met—that is, the data changes, the requirements change, or the model becomes inaccurate—the incremental training process is initiated. The specific implementation steps are as follows: This could involve collecting multi-dimensional information about the metadata of newly added business data and its correct classification labels, which have been manually verified or automatically labeled, and adding it to the historical training dataset (i.e., the training dataset of the previous classification period), or replacing some samples in the historical training dataset, thereby constructing a training dataset for the target classification period that is adapted to the current business characteristics. This embodiment does not specifically limit this.

[0073] After preparing the training dataset, the model training task can be started. During this process, a fine-tuning strategy can be adopted, loading the model parameters from the previous classification cycle as initial weights, and performing several rounds of iterative training using the training dataset of the target classification cycle. For example, if the incremental training takes no more than 30 minutes, training can be stopped to update the model parameters and adapt them to the new data distribution, so as to finally obtain the classification model for the target classification cycle. This embodiment does not make specific limitations on this.

[0074] It should be noted that, after verification, the classification accuracy of the classification model trained based on this method can be consistently maintained above 92%, effectively ensuring that the classification results keep pace with changes in business development and user needs.

[0075] The method provided in this embodiment, by establishing a closed-loop incremental learning mechanism of monitoring-triggering-updating, can respond in real time to changes in business scenarios and adjustments in user needs. It does not require manual redefinition of classification rules, supports multi-level classification to meet the needs of complex business scenarios, realizes the adaptive evolution of the classification model, and ensures that the metadata classification results can be dynamically adjusted with changes in business development and user needs. This avoids the rigidity problem of static classification, improves the adaptability of classification results to business development and user needs, and thus ensures the correctness of metadata management.

[0076] In some embodiments, step 220 specifically includes: Step 221: Input the description text into the convolutional network layer of the feature extraction layer to obtain the data content feature vector of the metadata.

[0077] Optionally, when processing descriptive text, local semantic features can be extracted through convolutional network layers. In this process, the convolutional network layer can be an existing convolutional neural network (CNN), which uses multiple convolutional kernels to perform a sliding window scan on the word vector sequence of the descriptive text, extracts keyword combination features, and outputs a fixed-dimensional (e.g., 256-dimensional) data content feature vector through pooling operations.

[0078] Step 222: Input the association weights into the feature embedding layer of the feature extraction layer to obtain the business feature vector of the metadata.

[0079] Optionally, when processing association weights, they can be mapped to dense vectors through a feature embedding layer to generate business scenario features. In this process, the association weights corresponding to the metadata are first one-hot encoded to obtain a high-dimensional sparse vector; then, the high-dimensional sparse vector is input to the feature embedding layer for dimensionality reduction, outputting a low-dimensional dense (e.g., 128-dimensional) business feature vector. In this way, discrete business entity labels can be transformed into continuous vector representations containing graph association information, thereby capturing the implicit relationships between business entities.

[0080] Step 223: Input the user behavior information into the temporal feature processing layer of the feature extraction layer to obtain the user behavior feature vector of the metadata.

[0081] Optionally, when processing user behavior information, its time-series patterns can be mined through a temporal feature processing layer. In this process, the temporal feature processing layer can employ existing recurrent neural networks (RNNs), preferably using Long Short-Term Memory (LSTM) units to improve the temporal feature extraction capability. Specifically, user behavior information, such as the user's operation records generated chronologically over the past three months, can be used as the input sequence, and LSTM units can be used to predict a fixed-dimensional (e.g., 64-dimensional) user behavior feature vector.

[0082] Step 224: Determine the multi-dimensional feature vector based on the data content feature vector, the business feature vector, and the user behavior feature vector.

[0083] Optionally, after obtaining the above three vectors, they can be concatenated or stacked to form a multi-dimensional feature vector.

[0084] The method provided in this embodiment adopts an adapted neural network structure for data of different modalities, which maximizes the preservation of effective information in each dimension and ensures the accuracy of metadata classification.

[0085] In some embodiments, the steps for collecting metadata and its multidimensional information include: When new target source data is detected in the data lake through any of the multiple detection modes, the metadata is generated according to the data type of the target source data, and the descriptive text is extracted from the metadata; the target source data is the source data to which the metadata belongs; In the first knowledge graph, the data entities of the metadata and the business entities of each business are determined, and the association weights are determined based on the topological connection relationships between the data entities and each business entity. In the first knowledge graph, obtain the target user entity associated with the data entity, and obtain the user behavior information based on the behavior operation record formed by the user object corresponding to the target user entity for the metadata operation; The various detection modes include multiple modes such as a detection mode that sniffs network transmission characteristics (referred to as traffic sniffing mode), a detection mode that sends heartbeat probe packets to the data source port (referred to as heartbeat probe mode), and a detection mode that actively obtains data source access information through the database management protocol (referred to as active query mode).

[0086] It's important to note that existing metadata collection processes are passive and limited. Current mainstream metadata collection methods primarily rely on manual configuration or fixed script development: for structured data, field mapping relationships and parsing rules must be manually defined; for unstructured data, only basic attributes such as file name, size, and creation time can often be collected, failing to extract in-depth content features and business semantics. This approach not only consumes significant manpower but also exhibits extremely poor adaptability to new data sources. For instance, when a data lake integrates real-time streaming data from IoT devices or interactive data from social media, parsing scripts must be rewritten, resulting in severe delays in metadata collection. Furthermore, manual collection is prone to rule definition biases, leading to missing or incorrect metadata information, making it difficult to meet the demands of high-precision data management.

[0087] To address this, this embodiment employs a three-tier architecture of distributed perception, hierarchical parsing, and unified fusion to achieve full and efficient collection of metadata from multi-source heterogeneous data. The specific implementation steps are as follows: First, to adapt to large-scale network environments, lightweight multi-source data sensing nodes can be deployed at the edge layer of the data lake. These sensing nodes are deployed using Kubernetes containers, supporting horizontal scaling and auto-scaling. A single sensing node can simultaneously monitor more than 1000 data source ports, ensuring stability under high concurrency.

[0088] Furthermore, when sensing new source data, sensing nodes can be used to perform sensing through various detection modes, including traffic sniffing, heartbeat detection, and proactive inquiry.

[0089] The traffic sniffing mode is used to capture data transmission characteristics in the network in real time and analyze potential data interaction behaviors. The heartbeat detection mode is used to send probe packets to potential data sources at preset time intervals (such as every 5 seconds) and confirm the service's liveness through the response status. The proactive query mode is used to automatically obtain data source information by connecting to common data source management interfaces such as MySQL, Hadoop, and MongoDB through preset protocols.

[0090] Furthermore, when new target source data is detected in the data lake under any detection mode, the data type of the new target source data (such as structured, semi-structured or unstructured) will be automatically identified, and the data will be routed to the corresponding parsing module. The parsing algorithm in the corresponding parsing module will then be used to parse the target source data to generate the corresponding metadata.

[0091] For example, in some embodiments, generating the metadata based on the data type of the target source data includes: When the data type is structured data, the content information of the changed fields of the target source data is obtained, and the semantic information and business tags of the content information of the changed fields are parsed using a pre-trained language model. Based on the parsing results, the metadata is obtained. When the data type is semi-structured data, extract the hierarchical structure information of the target source data, and obtain the metadata based on the hierarchical structure information; When the data type is unstructured data, the target source data is parsed according to the parsing algorithm corresponding to the modality type to which the target source data belongs, and the metadata is obtained based on the parsing result; The modal types include text, images, audio, or video.

[0092] Optionally, during the metadata parsing process, a hybrid algorithm framework combining pre-trained models and domain-adaptive algorithms can be used to employ differentiated parsing strategies for source data of different data types, in order to accurately generate the corresponding metadata. The specific implementation steps are as follows: For structured data, such as source data in comma-separated values ​​(CSV), JavaScript object notation (JSON), and columnar storage file formats (such as Parquet), Flink Change Data Capture can be used. Capture (CDC) the content information of changed fields in the target source data, and parse the semantic information and business tags of the changed field content information based on a pre-trained language model. Simultaneously, a standardized application programming interface (API) gateway is constructed. This gateway supports multiple interface protocols, including but not limited to RESTful, Java Database Connectivity (JDBC), and Open Database Connectivity (ODBC) protocols. It can interface with various source data, such as Enterprise Resource Planning (ERP), Customer Relationship Management (CRM), IoT platforms, social media, and applications, actively pulling attribute information such as business ownership, responsible person, and sensitivity level from each source data. Then, the parsed semantic information and business tags, along with the attribute information such as business ownership, responsible person, and sensitivity level pulled by the standardized API gateway, and the content information such as the field name and data type of the changed fields, are associated, integrated, and standardized to construct the corresponding metadata. The pre-trained language model here refers to an existing large language model, such as the pre-trained BERT model; this embodiment does not specifically limit it.

[0093] For semi-structured data, such as Extensible Markup Language (XML) and Hypertext Markup Language (HTML), hierarchical structure information can be extracted using a schema inference algorithm, and standardized metadata description information can be generated based on this hierarchical structure information. Then, this standardized metadata description information can be associated, integrated, and standardized with attribute information such as business affiliation, responsible person, and sensitivity level retrieved by a standardized application programming interface gateway, as well as content information such as field names and data types of changed fields, to construct the corresponding metadata.

[0094] For unstructured data, such as text, images, audio, and video, a multimodal parsing strategy is employed. For example, for text data, keywords and topic distributions can be extracted using existing Latent Dirichlet Allocation (LDA) topic models; for image data, existing residual models, such as the ResNet50 model, can be used to identify scene and object features; for audio or video data, content tagging is achieved through Mel-Frequency Cepstral Coefficients (MFCC) feature extraction and existing audio models based on Visual Geometry Group (VGG) architectures (such as the VGGish model). Then, the information parsed using the corresponding parsing algorithm for each modality type is associated, integrated, and standardized with attribute information such as business affiliation, responsible person, and sensitivity level retrieved from the standardized application programming interface gateway, as well as content information such as field names and data types of changed fields, to construct the corresponding metadata.

[0095] After the metadata generation is completed, in order to ensure the quality and timeliness of the data, the following cleaning, transmission, and storage steps can be performed on the metadata: The collected metadata undergoes integrity, accuracy, and consistency checks. Alarms are triggered for metadata that does not meet quality standards, and a data cleaning process is initiated, including deduplication, error correction, and format standardization. The cleaned metadata is then pushed to the metadata database in real-time via a distributed message middleware (such as Kafka) to ensure metadata quality. Kafka supports high-throughput and low-latency data transmission, ensuring that the end-to-end latency of the entire collection process is controlled within 10 seconds.

[0096] At the storage level, the metadata database adopts a hybrid storage architecture, such as a hybrid storage architecture that combines an object-relational database management system (such as PostgreSQL) with a distributed search and analysis engine (such as Elasticsearch). Specifically, structured attributes are stored in PostgreSQL to leverage its relational characteristics to ensure transaction consistency, while unstructured features are stored in Elasticsearch to take advantage of its powerful full-text search and vector search capabilities.

[0097] In addition, the metadata database can adopt a master-slave backup architecture, with the master node responsible for writing data and the slave node responsible for reading data. When the master node fails, it automatically switches to the slave node to ensure high availability.

[0098] Verification has shown that the end-to-end latency of the entire data collection process can be controlled within 10 seconds, effectively improving the efficiency of metadata collection.

[0099] The method provided in this embodiment breaks through the limitations of traditional manual configuration by intelligent metadata collection, achieving fully automated coverage of multi-source heterogeneous data sources. The distributed perception architecture can flexibly adapt to new data sources, and the pre-trained model's parsing capabilities cover various data formats, enabling deep extraction of business semantics from structured data as well as mining content features from unstructured data. By proactively integrating supplementary information from business systems through standardized interfaces, it significantly reduces manual intervention, avoids human error, and ensures the integrity and reliability of metadata information from the source, laying a solid foundation for data lake management.

[0100] Furthermore, after collecting the metadata through the above metadata collection process, multi-dimensional information can be obtained from the metadata. The specific acquisition steps include: Extract descriptive text from the generated metadata. For example, for a database table (structured data), extract its table name, field comments, and semantic tags generated by a pre-trained model as descriptive text; for a document (unstructured data), extract its file name, summary, and topic keywords generated by an LDA model as descriptive text.

[0101] In addition, in order to accurately quantify the correlation between data and business scenarios, this embodiment also introduces a pre-built first knowledge graph (also known as a multi-dimensional metadata knowledge graph). This graph adopts a multi-dimensional architecture that supports automatic updates, manual editing, and external fusion and expansion. Internally, it consists of three core node types: data entities (covering datasets, fields, files, etc.), business entities (covering departments, processes, metrics, etc.), and user entities (covering roles, permissions, preferences, etc.). Various complex topological relationships, including attribution, association, usage, and dependency, are defined between these nodes. The graph's construction employs a dual-track strategy of automatic extraction and manual completion. On one hand, it utilizes existing entity recognition models, such as the BERT-CRF model combining Transformer bidirectional encoder representation and conditional random field, for entity recognition, and combines this with existing relationship extraction models, such as the Span-based Pre-training of BERT (SpanBERT), to extract relationships between entities. This automatically extracts basic graph information from collected metadata and business documents. On the other hand, business experts are introduced to manually verify and supplement key entity relationships, ensuring the accuracy and completeness of the graph and providing a solid semantic foundation for accurately calculating association weights based on topological connections.

[0102] In practical applications, the data entities of the metadata can be mapped in the first knowledge graph. Then, the topological connections between the data entity and the business entities of each business are traversed in the first knowledge graph to calculate the association weight. The association weight here can be determined by calculating at least one of the following: the shortest path length between the data entity and the business entity, the number of connecting edges, and the membership degree. For example, if the association requires multiple hops, the weight value is reduced by a decay coefficient. This association weight objectively reflects the importance of the metadata in different business scenarios.

[0103] Furthermore, it is also possible to search for target user entities that have interacted with the data entity in the first knowledge graph, that is, user entities that have accessed, modified or referenced the metadata.

[0104] Subsequently, from the log audit center or operation log database, retrieve the user action records of these target user entities within a preset time window (e.g., the last 3 months) regarding the source data to which the metadata belongs. These records include, but are not limited to: search keyword history, click to view actions, data download actions, and collection or citation actions. These chronologically ordered sequences of actions are defined as user behavior information. This information will later be used to mine the temporal correlation between user intent and data value.

[0105] The method provided in this embodiment achieves automated and comprehensive collection of multi-dimensional metadata information through multi-mode proactive perception and knowledge graph association analysis, solving the problems of lagging manual configuration and incomplete information.

[0106] In some embodiments, step 110 specifically includes: The metadata retrieval request is segmented to obtain a sequence of retrieval keywords; The search keyword sequence is input into a pre-trained language model to generate a search semantic vector; Based on the retrieval semantic vector, the associated paths are obtained in the second knowledge graph; the associated paths include user entities, behavioral entities, and data entities associated with the retrieval semantic vector; The intent is inferred based on the associated path to obtain the retrieval intent vector.

[0107] Optionally, upon receiving a metadata retrieval request, a word segmenter can be used to segment the request into a sequence of search keywords. For example, this process can employ an open-source Chinese word segmentation tool (such as IK Segmenter) combined with an industry-customized dictionary to segment the natural language request into a sequence of search keywords. This industry-customized dictionary is configured according to industry characteristics, supports regular updates, and new terms can be added after administrator approval.

[0108] After obtaining the search keyword sequence, it can be input into an existing pre-trained language model, such as a pre-trained BERT model. During this process, the pre-trained language model can be used to convert the discrete search keyword sequence into a continuous, dense vector containing contextual information, i.e., a search semantic vector, such as a 512-dimensional semantic vector. This embodiment does not specifically limit the dimension of the search semantic vector.

[0109] Simultaneously, entity linking and intent inference can be performed in the second knowledge graph using the retrieval semantic vector. Specifically, the association paths formed by user entities, behavioral entities, and data entities associated with the retrieval semantic vector are first determined in the second knowledge graph. Then, the retrieval intent is clarified based on the association paths, thereby obtaining the retrieval intent vector. The retrieval intent vector can be obtained by associating it with a pre-established mapping relationship between the association paths and the retrieval intent vector.

[0110] The construction method of the second knowledge graph here is similar to that of the first knowledge graph. For details, please refer to the construction method of the first knowledge graph. The second knowledge graph is composed of three core node types: user entities, behavioral entities (download, retrieval, collection, access, etc.), and data entities. Various complex topological relationship types, including ownership, association, use, and dependency, are defined between the nodes.

[0111] The method provided in this embodiment combines the deep semantic extraction capabilities of a pre-trained model with the topological association paths between users, behaviors, and data entities in a knowledge graph. It can accurately infer the true business intent behind a user's search behavior based on multi-dimensional contextual logic, thereby generating a search intent vector with clear directionality. This effectively solves the problem that traditional keyword retrieval is unable to capture deep business logic, improves the efficiency of cross-departmental data collaboration, enables users in different business processes to quickly locate data resources that meet their needs, breaks down the barriers of data usage scenarios, and significantly improves the accuracy and matching efficiency of metadata retrieval.

[0112] In some embodiments, this application may also update the metadata in real time, and the specific update steps include: Obtain real-time operation logs and real-time data fingerprints of each source data in the data lake; If any source data has a data change operation in its real-time operation log, or if the real-time data fingerprint of any source data is inconsistent with the original data fingerprint in the metadata of any source data, then the source data is determined to be changed data. Based on the business importance level of the changed data, a target update strategy is determined from multiple preset update strategies; According to the target update strategy, the metadata of the changed data is updated in the metadata database; The multiple preset update strategies include a second-level update strategy, a minute-level update strategy, and an hour-level update strategy; the second-level update strategy is used to atomically update the metadata of the changed data through a distributed lock mechanism; the minute-level update strategy is used to update the metadata of the changed data through a batch asynchronous processing mechanism.

[0113] It should be noted that existing metadata update mechanisms suffer from lag and incompleteness. Data in a data lake is constantly and dynamically changing, including data additions, content modifications, and file deletions. However, current metadata updates mostly employ a timed synchronization strategy, with synchronization cycles typically lasting several hours or even a day. This batch update method results in a significant time lag between metadata and source data. When users perform data analysis based on lagging metadata, they are prone to drawing incorrect conclusions. Furthermore, existing metadata management tools only record data modification operations but neglect to track deletion operations. This leads to a large amount of metadata in the metadata database that indicates the source data has been deleted, consuming storage resources and interfering with normal user searches.

[0114] To address this, this application establishes a real-time update mechanism of dual monitoring, three-level updates, and full traceability to ensure the consistency between metadata and source data. The specific implementation steps are as follows: To detect subtle changes in the massive amounts of data within the data lake, the system employs a dual monitoring mechanism. Firstly, log collection agents (such as Flink) deployed on the data lake's storage nodes collect real-time operation logs of each source data within the data lake. These logs include, but are not limited to, records of creation, modification, deletion, renaming, and permission changes. Each log entry includes an operation timestamp, Internet Protocol (IP) address, and account information. Secondly, the system periodically (e.g., every 10 seconds) calculates the data fingerprint of each source data. This fingerprint is generated using a 256-bit secure hash algorithm (SHA-256), and the periodically calculated fingerprints are updated in the metadata of each source data. Furthermore, every second, the real-time data fingerprint of active source data is recalculated and compared with the original data fingerprint in its metadata. For inactive source data, the real-time data fingerprint is recalculated every 5 minutes and compared with the original data fingerprint in its metadata.

[0115] If any source data is found to have data change operations in its real-time operation log, or if the real-time data fingerprint of the source data is inconsistent with the original data fingerprint in its metadata, the source data is determined to be changed data.

[0116] When data changes are detected, different update strategies are triggered based on the business importance level of the changed data: If the business importance level of the changed data is Level 1, that is, the changed data is core business data, such as transaction data, real-time monitoring data, etc., then the metadata of the changed data is updated using a second-level update strategy. Specifically, the metadata of the changed data is atomically updated through a distributed lock mechanism to ensure that the update delay between the metadata and the source data is less than or equal to 1 second, so as to avoid decision delays. If the business importance level of the changed data is level two, that is, the changed data is general business data, such as report data, then the metadata of the changed data is updated using a minute-level update strategy. Specifically, the metadata of the changed data is updated through a batch asynchronous processing mechanism to ensure that the update delay between the metadata and the source data is less than or equal to 5 minutes, thus balancing system load and timeliness. If the business importance level of the changed data is level three, that is, the changed data is cold data, such as historical archive data that has been inactive for a long time, then the metadata of the changed data will be updated using an hourly update strategy to reduce resource consumption.

[0117] The metadata update content includes, but is not limited to, basic data attributes such as size, modification time, and storage path, as well as content features such as keywords and themes; in addition, it also includes relationships such as lineage and dependency relationships, which are not specifically limited in this embodiment.

[0118] The method provided in this embodiment ensures that no data changes are missed during real-time metadata updates through dual monitoring, and the hierarchical update strategy balances timeliness and system performance, effectively guaranteeing dynamic synchronization between metadata and source data, thereby improving data retrieval performance.

[0119] In some embodiments, during the metadata update process, the update records of the metadata can also be stored. Specific implementation steps include: Obtain the update record of the metadata of the changed data; the update record includes at least one of the following: change time, change type, change content, operator information, and real-time data fingerprint of the changed data; The update record is written into the blockchain network to obtain the change log chain; In response to a query request for the change log chain, the change log chain is retrieved according to the query parameters in the query request, and the target change record is returned based on the retrieval results; The query parameters include at least one of the following: time range, change type, operator, or data identifier.

[0120] Optionally, each time a metadata update is completed, a structured update record will be automatically generated for the updated metadata. This record includes: change time, change type, change content, operator information (related user information or system service information), and real-time data fingerprint.

[0121] To ensure that the metadata change history is immutable and traceable, blockchain technology (such as a consortium blockchain architecture) is used to package the above update records into blocks, which are then synchronized to each node through a consensus mechanism to form a chain-like change log chain. This chain records the entire lifecycle of metadata changes. Each update record includes information such as change time, change type, change content, operator, and real-time data fingerprint. This change log chain supports multi-dimensional (multi-condition combination) queries by time range, change type, operator, data identifier, etc. The traceability response time is less than or equal to 100 milliseconds, which can effectively meet compliance audit requirements.

[0122] The method provided in this embodiment records the entire lifecycle of metadata through an immutable change log chain. This not only avoids decision-making biases caused by information lag, but also meets the stringent compliance and auditing requirements of industries such as finance and healthcare. It provides traceable and auditable security for data management and reduces data governance risks.

[0123] In some embodiments, step 140 specifically includes: The semantic similarity, classification matching degree, association tightness, access activity and timeliness parameters are weighted and fused to obtain the comprehensive score of each metadata. The response information is generated based on the target metadata with the highest comprehensive score and the associated information of the target metadata; the associated information includes the source data of the target metadata, as well as metadata and lineage maps similar to the target metadata.

[0124] It should be noted that the existing search results only present a single metadata entry and cannot display information such as its related data lineage and similar data. Users need to perform multiple searches to integrate the required data, resulting in metadata retrieval taking an average of 25%-40% of the total data analysis time, which seriously affects work efficiency.

[0125] In response, this application proposes to simplify the data search process through multi-dimensional retrieval and intelligent sorting, and to provide visualization and related recommendations to help users quickly grasp the data lineage, thereby reducing the time cost of data exploration. In this way, by improving the efficiency of metadata usage, the process of data lake value mining can be accelerated, allowing data assets to be transformed into business decision-making basis more quickly, and promoting the implementation of data-driven business development model.

[0126] Specifically, after calculating the indicators for each dimension, a weighted score can be applied. In this process, existing weight learning models, such as the eXtreme Gradient Boosting (XGBoost) model, can be used to learn the weights of each dimension indicator. For example, the weights for semantic similarity, classification matching, association tightness, access activity, and timeliness parameters are 0.4, 0.2, 0.1, 0.2, and 0.1, respectively. Then, the weights are used to weight and sum the semantic similarity, classification matching, association tightness, access activity, and timeliness parameters corresponding to each metadata element to obtain a comprehensive score for each metadata element. This embodiment does not specifically limit the specific implementation of this method.

[0127] Finally, the results are extracted based on the scores. During this process, the target metadata with the highest overall score can be selected, and its related information can be aggregated, including the source data of the target metadata, recommended data similar to the target metadata, and lineage graphs, etc., and then packaged into response information for display on the front-end page.

[0128] The method provided in this embodiment simplifies the data search process through multi-dimensional retrieval and intelligent sorting functions, and helps users quickly grasp the data lineage through visualization and related recommendations, ensuring high accuracy and richness of search results, reducing the time cost of data exploration, and greatly improving the user experience.

[0129] The metadata processing system provided by the present invention is described below. The metadata processing system described below can be referred to in correspondence with the metadata processing method described above.

[0130] Figure 2 This is a schematic diagram of the metadata processing system provided by the present invention; as shown below. Figure 2 As shown, the system includes: The first acquisition unit is used to acquire the retrieval semantic vector and retrieval intent vector of the metadata retrieval request in response to the metadata retrieval request. The second acquisition unit is used to acquire the target classification dimension corresponding to the metadata retrieval request based on the retrieval semantic vector and the retrieval intent vector. The calculation unit is used to calculate the semantic similarity between the retrieval semantic vector and the data content features of each metadata in the metadata database, the classification matching degree between the target classification dimension and the classification labels of each metadata, and the correlation between the context data corresponding to the retrieval intent vector and each metadata. The retrieval unit is used to perform matching retrieval in the metadata database based on the semantic similarity, the classification matching degree, the association tightness, and the access activity and timeliness parameters of each of the metadata, and obtain the response information corresponding to the metadata retrieval request.

[0131] The system provided in this embodiment obtains retrieval semantic vectors and retrieval intent vectors in response to metadata retrieval requests. This not only understands the user's literal meaning but also provides insights into the user's business intent. By further calculating semantic similarity, classification matching degree, and correlation tightness, and combining data access activity and timeliness parameters for multi-dimensional matching retrieval, it changes the inefficient mode of relying solely on keyword matching. This effectively enables the rapid and accurate capture of retrieval information, effectively solving the problem of retrieval failures caused by spelling errors or terminology differences. At the same time, it can prioritize the display of high-popularity, high-timeliness, and highly business-related data, thereby significantly improving the recall rate, precision rate, and efficiency of users in obtaining high-value data.

[0132] The system provided by this invention is used to execute the above-described method embodiments. For specific processes and details, please refer to the above embodiments, which will not be repeated here.

[0133] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3As shown, the electronic device may include a processor 310, a communications interface 320, a memory 330, and a communication bus 340, wherein the processor 310, communications interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 can invoke logical instructions in the memory 330 to execute a metadata processing method. This method includes: in response to a metadata retrieval request, obtaining a retrieval semantic vector and a retrieval intent vector of the metadata retrieval request; obtaining a target classification dimension corresponding to the metadata retrieval request based on the retrieval semantic vector and the retrieval intent vector; calculating the semantic similarity between the retrieval semantic vector and the data content features of each metadata in the metadata database, the classification matching degree between the target classification dimension and the classification labels of each metadata, and the correlation tightness between the context data corresponding to the retrieval intent vector and each metadata; and performing a matching retrieval in the metadata database based on the semantic similarity, the classification matching degree, the correlation tightness, and the access activity and timeliness parameters of each metadata to obtain response information corresponding to the metadata retrieval request.

[0134] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0135] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the metadata processing method provided by the above methods. The method includes: in response to a metadata retrieval request, obtaining a retrieval semantic vector and a retrieval intent vector of the metadata retrieval request; obtaining a target classification dimension corresponding to the metadata retrieval request based on the retrieval semantic vector and the retrieval intent vector; calculating the semantic similarity between the retrieval semantic vector and the data content features of each metadata in the metadata database, the classification matching degree between the target classification dimension and the classification tags of each metadata, and the correlation tightness between the context data corresponding to the retrieval intent vector and each metadata; and performing a matching retrieval in the metadata database based on the semantic similarity, the classification matching degree, the correlation tightness, and the access activity and timeliness parameters of each metadata to obtain response information corresponding to the metadata retrieval request.

[0136] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the metadata processing method provided by the methods described above. The method includes: in response to a metadata retrieval request, obtaining a retrieval semantic vector and a retrieval intent vector of the metadata retrieval request; obtaining a target classification dimension corresponding to the metadata retrieval request based on the retrieval semantic vector and the retrieval intent vector; calculating the semantic similarity between the retrieval semantic vector and the data content features of each metadata in the metadata database, the classification matching degree between the target classification dimension and the classification labels of each metadata, and the correlation tightness between the context data corresponding to the retrieval intent vector and each metadata; and performing a matching retrieval in the metadata database based on the semantic similarity, the classification matching degree, the correlation tightness, and the access activity and timeliness parameters of each metadata to obtain response information corresponding to the metadata retrieval request.

[0137] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0138] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A metadata processing method, characterized in that, include: In response to a metadata retrieval request, obtain the retrieval semantic vector and retrieval intent vector of the metadata retrieval request; Based on the retrieval semantic vector and the retrieval intent vector, the target classification dimension corresponding to the metadata retrieval request is obtained; Calculate the semantic similarity between the retrieval semantic vector and the data content features of each metadata in the metadata database, the classification matching degree between the target classification dimension and the classification labels of each metadata, and the correlation between the context data corresponding to the retrieval intent vector and each metadata. Based on the semantic similarity, classification matching degree, association tightness, and access activity and timeliness parameters of each metadata, a matching search is performed in the metadata database to obtain the response information corresponding to the metadata retrieval request.

2. The metadata processing method according to claim 1, characterized in that, For each piece of metadata in the metadata database, the steps for obtaining the category label of the metadata include: Obtain the classification model for the target classification period; the target classification period is the classification period to which the metadata belongs; The multi-dimensional information of the metadata is input into the feature extraction layer of the classification model to obtain a multi-dimensional feature vector; the multi-dimensional information includes the descriptive text of the metadata, the association weight between the metadata and each business, and the user behavior information associated with the metadata; The multi-dimensional feature vector is input into the attention mechanism layer of the classification model to obtain the fused feature vector; The fused feature vector is input into the classification layer of the classification model to obtain the classification label of the metadata.

3. The metadata processing method according to claim 2, characterized in that, The classification model for obtaining the target classification period includes: The statistics include the proportion of new business data in the data lake and the rate of change of user search intent in the previous classification period of the target classification period, as well as the classification accuracy of the classification model in the previous classification period. When the percentage reaches the percentage threshold, the rate of change reaches the rate of change threshold, or the classification accuracy is lower than the accuracy threshold, the training dataset of the previous classification period is updated based on the multi-dimensional information and classification labels of the metadata corresponding to the new business data to obtain the training dataset of the target classification period. Based on the training dataset of the target classification period, the classification model of the previous classification period is iteratively trained to obtain the classification model of the target classification period.

4. The metadata processing method according to claim 2, characterized in that, The step of inputting the multi-dimensional information of the metadata into the feature extraction layer of the classification model to obtain a multi-dimensional feature vector includes: The descriptive text is input into the convolutional network layer of the feature extraction layer to obtain the data content feature vector of the metadata. The association weights are input into the feature embedding layer of the feature extraction layer to obtain the business feature vector of the metadata; The user behavior information is input into the temporal feature processing layer of the feature extraction layer to obtain the user behavior feature vector of the metadata. The multi-dimensional feature vector is determined based on the data content feature vector, the business feature vector, and the user behavior feature vector.

5. The metadata processing method according to claim 2, characterized in that, The steps for obtaining multi-dimensional information of the metadata include: When new target source data is detected in the data lake through any of the multiple detection modes, the metadata is generated according to the data type of the target source data, and the descriptive text is extracted from the metadata; the target source data is the source data to which the metadata belongs; In the first knowledge graph, the data entities of the metadata and the business entities of each business are determined, and the association weights are determined based on the topological connection relationships between the data entities and each business entity. In the first knowledge graph, obtain the target user entity associated with the data entity, and obtain the user behavior information based on the behavior operation record formed by the user object corresponding to the target user entity for the metadata operation; The various detection modes include multiple modes such as a detection mode that sniffs network transmission characteristics, a detection mode that sends heartbeat probe packets to the data source port, and a detection mode that actively obtains data source access information through the database management protocol.

6. The metadata processing method according to claim 5, characterized in that, The step of generating the metadata based on the data type of the target source data includes: When the data type is structured data, the content information of the changed fields of the target source data is obtained, and the semantic information and business tags of the content information of the changed fields are parsed using a pre-trained language model. Based on the parsing results, the metadata is obtained. When the data type is semi-structured data, extract the hierarchical structure information of the target source data, and obtain the metadata based on the hierarchical structure information; When the data type is unstructured data, the target source data is parsed according to the parsing algorithm corresponding to the modality type to which the target source data belongs, and the metadata is obtained based on the parsing result; The modal types include text, images, audio, or video.

7. The metadata processing method according to any one of claims 1-6, characterized in that, The step of obtaining the retrieval semantic vector and retrieval intent vector of the metadata retrieval request includes: The metadata retrieval request is segmented to obtain a sequence of retrieval keywords; The search keyword sequence is input into a pre-trained language model to generate a search semantic vector; Based on the retrieval semantic vector, the associated paths are obtained in the second knowledge graph; the associated paths include user entities, behavioral entities, and data entities associated with the retrieval semantic vector; The intent is inferred based on the associated path to obtain the retrieval intent vector.

8. The metadata processing method according to any one of claims 1-6, characterized in that, The method further includes: Obtain real-time operation logs and real-time data fingerprints of each source data in the data lake; If any source data has a data change operation in its real-time operation log, or if the real-time data fingerprint of any source data is inconsistent with the original data fingerprint in the metadata of any source data, then the source data is determined to be changed data. Based on the business importance level of the changed data, a target update strategy is determined from multiple preset update strategies; According to the target update strategy, the metadata of the changed data is updated in the metadata database; The multiple preset update strategies include a second-level update strategy, a minute-level update strategy, and an hour-level update strategy; the second-level update strategy is used to atomically update the metadata of the changed data through a distributed lock mechanism; the minute-level update strategy is used to update the metadata of the changed data through a batch asynchronous processing mechanism.

9. The metadata processing method according to claim 8, characterized in that, The method further includes: Obtain the update record of the metadata of the changed data; the update record includes at least one of the following: change time, change type, change content, operator information, and real-time data fingerprint of the changed data; The update record is written into the blockchain network to obtain the change log chain; In response to a query request for the change log chain, the change log chain is retrieved according to the query parameters in the query request, and the target change record is returned based on the retrieval results; The query parameters include at least one of the following: time range, change type, operator, or data identifier.

10. The metadata processing method according to any one of claims 1-6, characterized in that, The step involves performing a matching search in the metadata database based on the semantic similarity, classification matching degree, association tightness, and access activity and timeliness parameters of each metadata element, to obtain the response information corresponding to the metadata retrieval request, including: The semantic similarity, classification matching degree, association tightness, access activity and timeliness parameters are weighted and fused to obtain the comprehensive score of each metadata. The response information is generated based on the target metadata with the highest comprehensive score and the associated information of the target metadata; the associated information includes the source data of the target metadata, as well as metadata and lineage maps similar to the target metadata.

11. A metadata processing system, characterized in that, include: The first acquisition unit is used to acquire the retrieval semantic vector and retrieval intent vector of the metadata retrieval request in response to the metadata retrieval request. The second acquisition unit is used to acquire the target classification dimension corresponding to the metadata retrieval request based on the retrieval semantic vector and the retrieval intent vector. The calculation unit is used to calculate the semantic similarity between the retrieval semantic vector and the data content features of each metadata in the metadata database, the classification matching degree between the target classification dimension and the classification labels of each metadata, and the correlation between the context data corresponding to the retrieval intent vector and each metadata. The retrieval unit is used to perform matching retrieval in the metadata database based on the semantic similarity, the classification matching degree, the association tightness, and the access activity and timeliness parameters of each of the metadata, and obtain the response information corresponding to the metadata retrieval request.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the metadata processing method as described in any one of claims 1 to 10.

Images