Metadata management methods, devices, equipment, and storage media based on lineage analysis

Abstract

This application discloses a metadata management method, apparatus, device, and storage medium based on lineage analysis, belonging to the field of artificial intelligence technology. This application obtains the metadata description corresponding to the data to be processed, performs lineage analysis based on the metadata description to obtain the lineage information of the data to be processed, divides the data according to the lineage information to obtain a lineage dataset, inputs the lineage dataset into a pre-trained data classification model to obtain data classification results, wherein the data classification results record the mapping relationship between each data to be processed, constructs a data map of the data to be processed based on the mapping relationship and the metadata description, and manages the data to be processed based on the data map. Furthermore, this application also relates to blockchain technology, and the data to be processed can be stored in the blockchain. This application combines the lineage information and metadata description of the data to classify and manage the data to be processed, improving the reliability of data management and thus enhancing the value of data utilization.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, specifically relating to a metadata management method, apparatus, device, and storage medium based on lineage analysis. Background Technology

[0002] Data management is the process of effectively collecting, storing, processing, and applying data using computer hardware and software technologies. However, in actual data management, the storage, processing, and application of much data do not fully comply with enterprise management methods. Some small and medium-sized enterprises even lack complete data management methods, which leads to difficulties in data acquisition, use, and analysis, thereby affecting the overall operation, management, and business efficiency of the enterprise.

[0003] Current data asset management platforms on the market only offer data display functionality; however, they lack comprehensive solutions for deeper issues such as data governance, data quality monitoring, and data standards management. Taking company data table metadata management as an example, current metadata management suffers from the following problems: due to the lack of in-depth analysis of data tables, data monitoring only focuses on monitoring the data on the tables themselves, while metadata management information is not effectively displayed and processed. When anomalies occur in the metadata corresponding to a data table, developers cannot promptly understand the cause of the anomaly and therefore cannot handle data anomalies in a timely manner. Summary of the Invention

[0004] The purpose of this application is to propose a metadata management method, device, computer equipment, and storage medium based on lineage analysis, in order to solve the technical problem that existing data asset management solutions only monitor data on data tables, and when the metadata corresponding to the data tables is abnormal, developers cannot understand the cause of the abnormality and handle the data abnormality in a timely manner.

[0005] To address the aforementioned technical problems, this application provides a metadata management method based on lineage analysis, employing the following technical solution:

[0006] A metadata management method based on lineage analysis includes:

[0007] The data to be processed is obtained from the preset data table, and the metadata description corresponding to each data to be processed is obtained to obtain the first metadata description;

[0008] Based on the first metadata description, a lineage analysis is performed on the data to be processed to obtain the lineage information of the data to be processed.

[0009] Based on the lineage information of the data to be processed, the data is divided into lineage datasets.

[0010] The kinship dataset is input into a pre-trained data classification model to obtain the data classification results of the data to be processed. The data classification results record the mapping relationship between each data to be processed.

[0011] Construct a data map of the data to be processed based on the mapping relationship between each data to be processed in the data classification results and the metadata description corresponding to each data to be processed.

[0012] Data management is performed based on the data map.

[0013] Furthermore, based on the first metadata description, a lineage analysis is performed on the data to be processed to obtain the lineage information of the data to be processed, specifically including:

[0014] A script file that converts the data to be processed, carrying the first metadata description, into SQL code;

[0015] Extract the regularized SQL statements from the script file containing the SQL code, and then convert the SQL statements into an abstract syntax tree;

[0016] Traverse the abstract syntax tree to obtain the logical relationships between all tree nodes in the abstract syntax tree;

[0017] The lineage information of the data to be processed is obtained based on the logical relationships of all tree nodes.

[0018] Furthermore, tree nodes include root nodes and leaf nodes. Traversing the abstract syntax tree (AST) retrieves the logical relationships between all tree nodes, specifically including:

[0019] Traverse the abstract syntax tree downwards from the root node until the leaf node at the bottom of the abstract syntax tree;

[0020] Extract the logical relationships between all adjacent tree nodes to obtain the logical relationships between all tree nodes in the abstract syntax tree.

[0021] Furthermore, the data classification model includes an encoding layer and a decoding layer. The kinship dataset is input into the pre-trained data classification model to obtain the data classification results for the data to be processed, specifically including:

[0022] Feature extraction and feature vector transformation are performed on the data to be processed in the kinship dataset to obtain data feature vectors;

[0023] The data feature vector is encoded by the encoding layer of the data classification model to obtain the data encoding vector;

[0024] Spatial mapping is performed on the data encoding vector to obtain the spatial mapping result of the data to be processed;

[0025] The spatial mapping result of the data to be processed is decoded by the decoding layer of the data classification model to obtain the data classification result of the data to be processed.

[0026] Furthermore, before inputting the kinship dataset into a pre-trained data classification model to obtain the data classification results for the data to be processed, the following steps are also included:

[0027] Sample data is obtained from a preset database, along with the metadata description corresponding to each sample data, to obtain a second metadata description;

[0028] Based on the second metadata description, a bloodline analysis was performed on the sample data to obtain the bloodline information of the sample data;

[0029] The sample data is divided according to the bloodline information of the sample data to obtain the sample bloodline dataset;

[0030] Import the sample bloodline dataset into a pre-trained transformer model, which includes an encoding layer and a decoding layer.

[0031] Feature extraction and feature vector transformation are performed on the sample data in the sample kinship dataset to obtain sample feature vectors;

[0032] The sample feature vector is encoded by the encoding layer of the transformer pre-trained model to obtain the sample encoding vector;

[0033] Spatial mapping is performed on the sample encoding vector to obtain the spatial mapping result of the sample data;

[0034] The spatial mapping result of the sample data is decoded by the decoding layer of the transformer pre-trained model to obtain the data classification result of the sample data;

[0035] The transformer pre-trained model is iteratively updated based on the data classification results of the sample data to obtain a well-trained data classification model.

[0036] Furthermore, the transformer pre-trained model is iteratively updated based on the data classification results of the sample data to obtain a trained data classification model, specifically including:

[0037] Obtain the loss function of the pre-trained transformer model;

[0038] The classification error is obtained by calculating the relative error between the data classification result and the preset standard classification result based on the loss function.

[0039] The classification error is passed into the transformer pre-trained model and compared with a preset error threshold.

[0040] If the classification error is greater than the preset error threshold, the transformer pre-trained model is iteratively updated until the model fits, and a well-trained data classification model is obtained.

[0041] Furthermore, a data map of the data to be processed is constructed based on the mapping relationships between the data to be processed in the data classification results and the metadata descriptions corresponding to each data to be processed. Specifically, this includes:

[0042] Draw an initial data map based on the mapping relationship between the various data to be processed in the data classification results;

[0043] Add metadata descriptions for each piece of data to be processed to the initial data map to form a data map of the data to be processed.

[0044] To address the aforementioned technical problems, this application also provides a metadata management device based on lineage analysis, employing the following technical solution:

[0045] A metadata management device based on lineage analysis, comprising:

[0046] The data acquisition module is used to acquire data to be processed from a preset data table, and to acquire the metadata description corresponding to each piece of data to be processed, thereby obtaining the first metadata description.

[0047] The first analysis module is used to perform lineage analysis on the data to be processed based on the first metadata description to obtain the lineage information of the data to be processed.

[0048] The first partitioning module is used to partition the data to be processed based on the bloodline information of the data to be processed, so as to obtain the bloodline dataset.

[0049] The data classification module is used to input the kinship dataset into a pre-trained data classification model to obtain the data classification results of the data to be processed. The data classification results record the mapping relationship between each data to be processed.

[0050] The data aggregation module is used to construct a data map of the data to be processed based on the mapping relationship between each data to be processed in the data classification results and the metadata description corresponding to each data to be processed.

[0051] The data management module is used to manage the data to be processed based on the data map.

[0052] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution:

[0053] A computer device includes a memory and a processor, the memory storing computer-readable instructions, the processor executing the computer-readable instructions to implement the steps of the metadata management method based on lineage analysis as described in any of the preceding claims.

[0054] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below:

[0055] A computer-readable storage medium storing computer-readable instructions, which, when executed by a processor, implement the steps of the metadata management method based on lineage analysis as described in any one of the preceding descriptions.

[0056] Compared with the prior art, the embodiments of this application have the following main advantages:

[0057] This application discloses a metadata management method, apparatus, device, and storage medium based on lineage analysis, belonging to the field of artificial intelligence technology. This application obtains the metadata description corresponding to the data to be processed, performs lineage analysis based on the metadata description to obtain the lineage information of the data to be processed, divides the data according to the lineage information to obtain a lineage dataset, inputs the lineage dataset into a pre-trained data classification model to obtain data classification results, wherein the data classification results record the mapping relationship between each data to be processed, constructs a data map of the data to be processed based on the mapping relationship and the metadata description, and manages the data to be processed based on the data map. This application combines the lineage information and metadata description of the data to perform a simple initial classification of the data to be processed, and then uses a pre-trained data classification model to perform data classification management in one step, improving the reliability of data management and thus enhancing the value of data utilization. Attached Figure Description

[0058] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 An exemplary system architecture diagram is shown, in which this application can be applied;

[0060] Figure 2 A flowchart illustrating an embodiment of the kinship analysis-based metadata management method according to this application is shown;

[0061] Figure 3A schematic diagram of the structure of one embodiment of a metadata management device based on lineage analysis according to this application is shown;

[0062] Figure 4 A schematic diagram of the structure of one embodiment of a computer device according to this application is shown. Detailed Implementation

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0064] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0065] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0066] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0067] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0068] Terminal devices 101, 102, and 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers, etc.

[0069] Server 105 can be a server that provides various services, such as a backend server that supports the pages displayed on terminal devices 101, 102, and 103. The server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0070] It should be noted that the metadata management method based on lineage analysis provided in this application embodiment is generally executed by the server, and correspondingly, the metadata management device based on lineage analysis is generally set in the server.

[0071] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0072] Continue to refer to Figure 2 The diagram illustrates a flowchart of an embodiment of the metadata management method based on kinship analysis according to this application. Embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0073] Foundational artificial intelligence technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly include computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning. The aforementioned metadata management method based on lineage analysis includes the following steps:

[0074] S201, obtain the data to be processed from the preset data table, and obtain the metadata description corresponding to each data to be processed, to obtain the first metadata description.

[0075] Metadata, also known as intermediary or relay data, is data about data, primarily describing data properties. It supports functions such as indicating storage location, historical data, resource lookup, and file records. Metadata describes information resources or data objects, and its purpose is to: identify resources; evaluate resources; track changes in resources during use; achieve simple and efficient management of large amounts of networked data; and enable effective discovery, retrieval, integrated organization, and management of information resources. Metadata can be considered a type of electronic catalog. To achieve the purpose of cataloging, it is necessary to describe and collect the content or characteristics of the data, thereby assisting in data retrieval.

[0076] Specifically, a preset data table stores data to be processed and corresponding metadata descriptions. The server retrieves multiple data items to be processed from the preset data table, and simultaneously retrieves the metadata descriptions corresponding to each data item to obtain the first metadata description. For example, the server retrieves the data item to be processed from the data table as "XX critical illness insurance coverage is 500,000", where "500,000" represents the data itself, and "XX critical illness insurance coverage" represents the corresponding metadata description.

[0077] S202, perform lineage analysis on the data to be processed based on the first metadata description to obtain the lineage information of the data to be processed.

[0078] One approach to expressing the relationship between data is by drawing on kinship in human society. This is called data kinship. Data kinship includes some unique characteristics: data attribution (the organization or individual to which the data belongs), data source diversity (the same data comes from more than one source), data traceability (the entire process of data from its creation to its extinction), and data hierarchy (the classification, induction, and summarization of data form data hierarchy). Analyzing data kinship is called kinship analysis.

[0079] Specifically, the server transforms the data to be processed, which carries metadata descriptions, into SQL statements, and then transforms the SQL statements into an abstract syntax tree. The leaf nodes of the abstract syntax tree represent the lineage relationships between the data to be processed. The server traverses the abstract syntax tree to obtain the logical relationships between all tree nodes in the abstract syntax tree.

[0080] S203, the data to be processed is divided according to the bloodline information of the data to be processed to obtain the bloodline dataset.

[0081] Specifically, the server partitions the data to be processed based on its lineage information to obtain a lineage dataset. In a specific embodiment of this application, data to be processed that are related by blood are grouped into the same dataset. After all data to be processed has been partitioned, multiple lineage datasets are obtained.

[0082] S204. Input the kinship dataset into the pre-trained data classification model to obtain the data classification results of the data to be processed. The data classification results record the mapping relationship between each data to be processed.

[0083] The data classification model is trained on a Transformer pre-trained model, which is an encoder-decoder structure where both the encoder and decoder consist of attention modules and preceding neural networks. It was the first model built using pure attention, resulting in faster computation and better performance on translation tasks. Initially proposed as a sequence-to-sequence model for machine translation, Transformer-based pre-trained models (PTMs) have shown to be optimal across various tasks. Therefore, Transformer has become the preferred architecture in NLP, especially PTMs. Beyond language applications, Transformer is also used in computer vision, audio processing, and even chemistry and life sciences.

[0084] Specifically, the server pre-trains a data classification model based on a transformer pre-trained model. When classifying data, the kinship dataset is input into the pre-trained data classification model, and the data classification results of the data to be processed can be obtained directly. The data classification results record the mapping relationship between each data to be processed.

[0085] It should be noted that when using a data classification model to process a kinship dataset, after extracting and encoding features from the data to be processed in the kinship dataset, mapping the encoded features to the same feature space can reveal the relationship between the data to be processed in different kinship datasets. Based on the relationship between the data to be processed in different kinship datasets, the data to be processed can be further classified.

[0086] S205, construct a data map of the data to be processed based on the mapping relationship between each data to be processed in the data classification results and the metadata description corresponding to each data to be processed.

[0087] Among them, the data map is an enterprise data catalog management module provided on the basis of metadata. It covers functions such as global data retrieval, metadata detail viewing, data preview, data lineage and data category management. The existence of the data map can make it easier to find, understand and use data.

[0088] Specifically, the server constructs an initial data map based on the mapping relationships between the various data to be processed in the data classification results, and adds metadata descriptions corresponding to each data to be processed to the initial data map, forming a data map of the data to be processed. The data map helps developers to more intuitively understand the data information of each data to be processed.

[0089] S206, manage the data to be processed based on the data map.

[0090] Specifically, the server manages the data to be processed based on the data map. For example, when it is necessary to change the data to be processed, the server receives the data change instruction, parses the data change instruction, obtains the data change field and metadata description change field, and adaptively modifies the data to be processed and the metadata description in the data map according to the data change field and metadata description change field.

[0091] In this embodiment, the metadata management method based on lineage analysis runs on electronic devices (e.g., Figure 1 The server shown can receive data modification commands via wired or wireless connection. It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultra-wideband) connections, and other currently known or future wireless connection methods.

[0092] In the above embodiments, this application performs a simple initial classification of the data to be processed by combining the data lineage information and metadata description, and then performs data classification management of the data to be processed by a trained data classification model, thereby improving the reliability of data management and enhancing the value of data utilization.

[0093] Furthermore, based on the first metadata description, a lineage analysis is performed on the data to be processed to obtain the lineage information of the data to be processed, specifically including:

[0094] A script file that converts the data to be processed, carrying the first metadata description, into SQL code;

[0095] Extract the regularized SQL statements from the script file containing the SQL code, and then convert the SQL statements into an abstract syntax tree;

[0096] Traverse the abstract syntax tree to obtain the logical relationships between all tree nodes in the abstract syntax tree;

[0097] The lineage information of the data to be processed is obtained based on the logical relationships of all tree nodes.

[0098] Specifically, the server transforms the data to be processed, which carries the first metadata description, into SQL code, combines the SQL code to form an SQL script file, extracts regularized SQL statements from the SQL script file, transforms the SQL statements into an abstract syntax tree, traverses the abstract syntax tree, obtains the logical relationships of all tree nodes in the abstract syntax tree, and obtains the lineage information of the data to be processed based on the logical relationships of all tree nodes.

[0099] In another specific embodiment of this application, the data to be processed can also be pre-stored in a preset database in the form of SQL code. In this case, the servers obtain the data to be processed in the form of SQL code, and complete the conversion of the SQL statement into an abstract syntax tree. The abstract syntax tree is traversed to obtain the logical relationship of all tree nodes in the abstract syntax tree, and the lineage information of the data to be processed is obtained based on the logical relationship of all tree nodes.

[0100] Furthermore, tree nodes include root nodes and leaf nodes. Traversing the abstract syntax tree (AST) retrieves the logical relationships between all tree nodes, specifically including:

[0101] Traverse the abstract syntax tree downwards from the root node until the leaf node at the bottom of the abstract syntax tree;

[0102] Extract the logical relationships between all adjacent tree nodes to obtain the logical relationships between all tree nodes in the abstract syntax tree.

[0103] Specifically, a tree node includes a root node and multiple leaf nodes. The server traverses the abstract syntax tree downwards from the root node until the leaf node at the bottom level of the abstract syntax tree, extracting the logical relationships between all adjacent tree nodes to obtain the logical relationships of all tree nodes in the abstract syntax tree.

[0104] It should be noted that after obtaining the lineage information of the data to be processed, the lineage information also needs to be verified. During the lineage information verification, the server traverses the entire abstract syntax tree backward from the bottom leaf node to obtain the reverse logical relationship. The lineage information is verified by comparing the logical relationship and the reverse logical relationship of all tree nodes in the abstract syntax tree. When the lineage information is verified, the verified lineage information is output.

[0105] Furthermore, the data classification model includes an encoding layer and a decoding layer. The kinship dataset is input into the pre-trained data classification model to obtain the data classification results for the data to be processed, specifically including:

[0106] Feature extraction and feature vector transformation are performed on the data to be processed in the kinship dataset to obtain data feature vectors;

[0107] The data feature vector is encoded by the encoding layer of the data classification model to obtain the data encoding vector;

[0108] Spatial mapping is performed on the data encoding vector to obtain the spatial mapping result of the data to be processed;

[0109] The spatial mapping result of the data to be processed is decoded by the decoding layer of the data classification model to obtain the data classification result of the data to be processed.

[0110] Specifically, the data classification model includes an encoding layer and a decoding layer. After the server inputs the kinship dataset into the pre-trained data classification model, it first extracts features and transforms feature vectors from the data to be processed in the kinship dataset to obtain data feature vectors. Then, the encoding layer of the data classification model encodes the data feature vectors to obtain data encoding vectors. The data encoding vectors are then spatially mapped to the same feature space, obtaining the spatial mapping result of the data to be processed. Finally, the decoding layer of the data classification model decodes the spatial mapping result of the data to be processed to obtain the data classification result of the data to be processed.

[0111] Furthermore, before inputting the kinship dataset into a pre-trained data classification model to obtain the data classification results for the data to be processed, the following steps are also included:

[0112] Sample data is obtained from a preset database, along with the metadata description corresponding to each sample data, to obtain a second metadata description;

[0113] Based on the second metadata description, a bloodline analysis was performed on the sample data to obtain the bloodline information of the sample data;

[0114] The sample data is divided according to the bloodline information of the sample data to obtain the sample bloodline dataset;

[0115] Import the sample bloodline dataset into a pre-trained transformer model, which includes an encoding layer and a decoding layer.

[0116] Feature extraction and feature vector transformation are performed on the sample data in the sample kinship dataset to obtain sample feature vectors;

[0117] The sample feature vector is encoded by the encoding layer of the transformer pre-trained model to obtain the sample encoding vector;

[0118] Spatial mapping is performed on the sample encoding vector to obtain the spatial mapping result of the sample data;

[0119] The spatial mapping result of the sample data is decoded by the decoding layer of the transformer pre-trained model to obtain the data classification result of the sample data;

[0120] The transformer pre-trained model is iteratively updated based on the data classification results of the sample data to obtain a well-trained data classification model.

[0121] Specifically, before performing data classification, a data classification model needs to be trained in advance. The data classification model in this application is trained based on a transformer pre-trained model. The server retrieves sample data from a pre-set database, along with corresponding metadata descriptions for each sample, resulting in a second metadata description. Based on this description, it performs pedigree analysis on the sample data to obtain pedigree information. Then, it partitions the sample data based on this pedigree information, creating a sample pedigree dataset. This dataset is imported into a pre-trained Transformer model, which includes an encoding layer and a decoding layer. The model extracts features and transforms feature vectors from the sample data in the pedigree dataset, resulting in sample feature vectors. The encoding layer of the Transformer model encodes these feature vectors, creating encoded vectors. These encoded vectors are then spatially mapped to obtain the spatial mapping result. Features extracted from different pedigree datasets are mapped to the same feature space. The decoding layer of the Transformer model decodes the spatial mapping result, yielding the data classification result. Based on this classification result, the Transformer model is iteratively updated to obtain a trained data classification model.

[0122] It's worth noting that the Transformer pre-trained model also includes a self-attention layer and a softmax function layer. The self-attention layer concentrates features to reduce computational complexity. In a standard self-attention mechanism, each token needs to attend to all other tokens. However, it has been observed that the learned attention matrix A for trained Transformers is typically very sparse across most data points. Therefore, computational complexity can be reduced by incorporating structural biases to limit the number of query key pairs attended for each query. Softmax is an activation function that normalizes a numerical vector into a probability distribution vector, where the sum of the probabilities is 1. The softmax function can be used as the final layer of a neural network for multi-class classification problems, and it is often used in conjunction with the cross-entropy loss function.

[0123] Furthermore, the transformer pre-trained model is iteratively updated based on the data classification results of the sample data to obtain a trained data classification model, specifically including:

[0124] Obtain the loss function of the pre-trained transformer model;

[0125] The classification error is obtained by calculating the relative error between the data classification result and the preset standard classification result based on the loss function.

[0126] The classification error is passed into the transformer pre-trained model and compared with a preset error threshold.

[0127] If the classification error is greater than the preset error threshold, the transformer pre-trained model is iteratively updated until the model fits, and a well-trained data classification model is obtained.

[0128] Specifically, the server obtains the loss function of the transformer pre-trained model, calculates the relative error between the data classification result and the preset standard classification result based on the loss function, obtains the classification error, and propagates the classification error in the transformer pre-trained model based on the backpropagation algorithm. The classification error is then compared with a preset error threshold. If the classification error is greater than the preset error threshold, the transformer pre-trained model is iteratively updated until the model is fitted, resulting in a trained data classification model.

[0129] Furthermore, a data map of the data to be processed is constructed based on the mapping relationships between the data to be processed in the data classification results and the metadata descriptions corresponding to each data to be processed. Specifically, this includes:

[0130] Draw an initial data map based on the mapping relationship between the various data to be processed in the data classification results;

[0131] Add metadata descriptions for each piece of data to be processed to the initial data map to form a data map of the data to be processed.

[0132] Specifically, the server first draws an initial data map based on the mapping relationship between each piece of data to be processed in the data classification results. Then, it adds metadata descriptions corresponding to each piece of data to be processed to the initial data map, forming a data map of the data to be processed. The data map makes it easier for developers to understand the data information of each piece of data to be processed more intuitively.

[0133] In the above embodiments, this application discloses a metadata management method based on lineage analysis, belonging to the field of artificial intelligence technology. This application obtains the metadata description corresponding to the data to be processed, performs lineage analysis based on the metadata description to obtain the lineage information of the data to be processed, divides the data according to the lineage information to obtain a lineage dataset, inputs the lineage dataset into a pre-trained data classification model to obtain data classification results, wherein the data classification results record the mapping relationship between each piece of data to be processed, constructs a data map of the data to be processed based on the mapping relationship and the metadata description, and manages the data to be processed based on the data map. This application combines the lineage information and metadata description of the data to perform a simple initial classification of the data to be processed, and then uses a pre-trained data classification model to perform data classification management in one step, improving the reliability of data management and thus enhancing the value of data utilization.

[0134] It should be emphasized that, to further ensure the privacy and security of the data to be processed, the data can also be stored in a node of a blockchain.

[0135] The blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0136] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium, and when executed, they can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0137] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0138] Further reference Figure 3 As a response to the above Figure 2 The implementation of the method shown in this application provides an embodiment of a metadata management device based on lineage analysis, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0139] like Figure 3 As shown, the metadata management device 300 based on lineage analysis described in this embodiment includes:

[0140] The data acquisition module 301 is used to acquire data to be processed from a preset data table, and to acquire metadata descriptions corresponding to each piece of data to be processed, thereby obtaining a first metadata description;

[0141] The first analysis module 302 is used to perform lineage analysis on the data to be processed based on the first metadata description to obtain the lineage information of the data to be processed.

[0142] The first partitioning module 303 is used to partition the data to be processed according to the bloodline information of the data to be processed, so as to obtain the bloodline dataset.

[0143] The data classification module 304 is used to input the kinship dataset into a pre-trained data classification model to obtain the data classification results of the data to be processed. The data classification results record the mapping relationship between each data to be processed.

[0144] The data aggregation module 305 is used to construct a data map of the data to be processed based on the mapping relationship between each data to be processed in the data classification results and the metadata description corresponding to each data to be processed.

[0145] The data management module 306 is used to manage the data to be processed based on the data map.

[0146] Furthermore, the first analysis module 302 specifically includes:

[0147] The code conversion unit is used to convert the data to be processed, which carries the first metadata description, into a script file containing SQL code.

[0148] Syntax tree building units are used to extract regularized SQL statements from script files containing SQL code and to transform the SQL statements into abstract syntax trees.

[0149] The relational abstraction unit is used to traverse the abstract syntax tree and obtain the logical relationships between all tree nodes in the abstract syntax tree;

[0150] The lineage information acquisition unit is used to obtain the lineage information of the data to be processed based on the logical relationship of all tree nodes.

[0151] Furthermore, the relational abstraction unit specifically includes:

[0152] Traversing sub-units is used to traverse the abstract syntax tree from the root node down to the leaf node at the bottom level of the abstract syntax tree;

[0153] The relational abstraction subunit is used to extract the logical relationships between all adjacent tree nodes, thus obtaining the logical relationships between all tree nodes in the abstract syntax tree.

[0154] Furthermore, the data classification model includes an encoding layer and a decoding layer, and the data classification module 304 specifically includes:

[0155] The feature processing unit is used to extract features and transform feature vectors from the data to be processed in the kinship dataset to obtain data feature vectors.

[0156] The encoding unit is used to encode the data feature vector through the encoding layer of the data classification model to obtain the data encoding vector;

[0157] The spatial mapping unit is used to perform spatial mapping on the data encoding vector to obtain the spatial mapping result of the data to be processed.

[0158] The decoding unit is used to decode the spatial mapping result of the data to be processed through the decoding layer of the data classification model to obtain the data classification result of the data to be processed.

[0159] Furthermore, the kinship-based metadata management device 300 also includes:

[0160] The sample acquisition module is used to acquire sample data from a preset database and to acquire the metadata description corresponding to each sample data to obtain a second metadata description.

[0161] The second analysis module is used to perform kinship analysis on the sample data based on the second metadata description to obtain the kinship information of the sample data.

[0162] The second partitioning module is used to partition the sample data according to the bloodline information of the sample data to obtain the sample bloodline dataset.

[0163] The sample import module is used to import the sample lineage dataset into a pre-trained transformer model, which includes an encoding layer and a decoding layer.

[0164] The sample processing module is used to extract features and transform feature vectors from the sample pedigree dataset to obtain sample feature vectors.

[0165] The sample encoding module is used to encode the sample feature vector through the encoding layer of the transformer pre-trained model to obtain the sample encoded vector;

[0166] The sample mapping module is used to perform spatial mapping on the sample encoding vector to obtain the spatial mapping result of the sample data.

[0167] The sample decoding module is used to decode the spatial mapping results of the sample data through the decoding layer of the transformer pre-trained model to obtain the data classification results of the sample data.

[0168] The model iteration module is used to iteratively update the transformer pre-trained model based on the data classification results of the sample data, so as to obtain a trained data classification model.

[0169] Furthermore, the model iteration module specifically includes:

[0170] The loss function acquisition unit is used to obtain the loss function of the transformer pre-trained model;

[0171] The classification error calculation unit is used to calculate the relative error between the data classification result and the preset standard classification result based on the loss function, and obtain the classification error.

[0172] The error comparison unit is used to pass the classification error in the transformer pre-trained model and compare the classification error with a preset error threshold.

[0173] The model iteration unit is used to iteratively update the transformer pre-trained model when the classification error exceeds a preset error threshold, until the model is fitted and a well-trained data classification model is obtained.

[0174] Furthermore, the data aggregation module 305 specifically includes:

[0175] The initial map drawing unit is used to draw an initial data map based on the mapping relationship between each data to be processed in the data classification results;

[0176] The metadata description addition unit is used to add metadata descriptions corresponding to each piece of data to be processed to the initial data map, forming a data map of the data to be processed.

[0177] In the above embodiments, this application discloses a metadata management device based on lineage analysis, belonging to the field of artificial intelligence technology. This application obtains the metadata description corresponding to the data to be processed, performs lineage analysis based on the metadata description to obtain the lineage information of the data to be processed, divides the data according to the lineage information to obtain a lineage dataset, inputs the lineage dataset into a pre-trained data classification model to obtain data classification results, wherein the data classification results record the mapping relationship between each piece of data to be processed, constructs a data map of the data to be processed based on the mapping relationship and the metadata description, and manages the data to be processed based on the data map. This application combines the lineage information and metadata description of the data to perform a simple initial classification of the data to be processed, and then further manages the data to be processed through a pre-trained data classification model, improving the reliability of data management and thus enhancing the value of data utilization.

[0178] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0179] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected via a system bus. It should be noted that only the computer device 4 with components 41-43 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0180] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0181] The memory 41 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 41 may be an internal storage unit of the computer device 4, such as the hard disk or memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 4. Of course, the memory 41 may also include both the internal storage unit and its external storage device of the computer device 4. In this embodiment, the memory 41 is typically used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions for metadata management methods based on lineage analysis. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or will be output.

[0182] In some embodiments, the processor 42 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 42 is typically used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to execute computer-readable instructions stored in the memory 41 or to process data, for example, to execute computer-readable instructions of the lineage analysis-based metadata management method.

[0183] The network interface 43 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 4 and other electronic devices.

[0184] This application discloses a computer device belonging to the field of artificial intelligence technology. This application obtains metadata descriptions corresponding to the data to be processed, performs lineage analysis based on the metadata descriptions to obtain lineage information of the data to be processed, divides the data according to the lineage information to obtain a lineage dataset, inputs the lineage dataset into a pre-trained data classification model to obtain data classification results, wherein the data classification results record the mapping relationships between each piece of data to be processed, constructs a data map of the data to be processed based on the mapping relationships and metadata descriptions, and manages the data to be processed based on the data map. This application combines the lineage information and metadata descriptions of the data to perform a simple initial classification of the data to be processed, and then uses a pre-trained data classification model to perform data classification management in one step, improving the reliability of data management and thus enhancing the value of data utilization.

[0185] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the above-described metadata management method based on lineage analysis.

[0186] This application discloses a storage medium belonging to the field of artificial intelligence technology. This application obtains metadata descriptions corresponding to the data to be processed, performs lineage analysis based on the metadata descriptions to obtain the lineage information of the data to be processed, divides the data according to the lineage information to obtain a lineage dataset, inputs the lineage dataset into a pre-trained data classification model to obtain data classification results, wherein the data classification results record the mapping relationships between each data to be processed, constructs a data map of the data to be processed based on the mapping relationships and metadata descriptions, and manages the data to be processed based on the data map. This application combines the lineage information and metadata descriptions of the data to perform a simple initial classification of the data to be processed, and then uses a pre-trained data classification model to perform data classification management in one step, improving the reliability of data management and thus enhancing the value of data utilization.

[0187] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0188] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0189] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A method for managing metadata based on consanguinity analysis, characterized in that, The method comprises the following steps: obtaining to-be-processed data from a preset data table and obtaining metadata descriptions corresponding to each of the to-be-processed data to obtain first metadata descriptions; performing blood relationship analysis on the to-be-processed data according to the first metadata descriptions to obtain blood relationship information of the to-be-processed data; performing data division on the to-be-processed data according to the blood relationship information of the to-be-processed data to obtain a blood relationship data set; inputting the blood relationship data set into a pre-trained data classification model to obtain a data classification result of the to-be-processed data, wherein the data classification result records a mapping relationship between each of the to-be-processed data; constructing a data map of the to-be-processed data according to the mapping relationship between each of the to-be-processed data and the metadata descriptions corresponding to each of the to-be-processed data in the data classification result; performing data management on the to-be-processed data according to the data map; wherein the data classification model is trained based on a transformer pre-training model, the transformer pre-training model is a model based on an encoder-decoder structure, the encoder and the decoder of the transformer pre-training model are both composed of an attention module and a previous neural network, the data classification model comprises an encoding layer and a decoding layer, the blood relationship data set is inputted into the pre-trained data classification model to obtain the data classification result of the to-be-processed data, and the data classification result of the to-be-processed data is obtained by specifically comprising: performing feature extraction and feature vector conversion on the to-be-processed data in the blood relationship data set to obtain a data feature vector; encoding the data feature vector through the encoding layer of the data classification model to obtain a data encoding vector; performing spatial mapping on the data encoding vector to obtain a spatial mapping result of the to-be-processed data; decoding the spatial mapping result of the to-be-processed data through the decoding layer of the data classification model to obtain the data classification result of the to-be-processed data.

2. The method of claim 1, wherein, The blood relationship information of the to-be-processed data is obtained by specifically comprising: converting the to-be-processed data carrying the first metadata descriptions into a script file of SQL code; extracting a regularized SQL statement from the script file of the SQL code and converting the SQL statement into an abstract syntax tree; traversing the abstract syntax tree to obtain a logical relationship of all tree nodes in the abstract syntax tree; obtaining the blood relationship information of the to-be-processed data based on the logical relationship of all tree nodes.

3. The method of claim 2, wherein the metadata management based on the blood analysis is characterized by, The tree nodes comprise root nodes and leaf nodes, and the logical relationship of all tree nodes in the abstract syntax tree is obtained by specifically comprising: traversing the abstract syntax tree from the root nodes downwardly until the leaf nodes at the bottom layer of the abstract syntax tree; extracting a logical relationship between all adjacent tree nodes to obtain the logical relationship of all tree nodes in the abstract syntax tree.

4. The method of claim 1, wherein the metadata management based on the blood analysis is characterized by, Before the blood relationship data set is inputted into the pre-trained data classification model to obtain the data classification result of the to-be-processed data, the method further comprises the following steps: Obtaining sample data from a preset database, and obtaining metadata descriptions corresponding to each of the sample data, to obtain second metadata descriptions; Performing blood analysis on the sample data according to the second metadata descriptions, to obtain blood information of the sample data; Performing data division on the sample data according to the blood information of the sample data, to obtain a sample blood data set; Importing the sample blood data set into a preset transformer pre-training model, wherein the transformer pre-training model includes an encoding layer and a decoding layer; Performing feature extraction and feature vector conversion on sample data in the sample blood data set, to obtain sample feature vectors; Encoding the sample feature vectors through the encoding layer of the transformer pre-training model, to obtain sample encoding vectors; Performing spatial mapping on the sample encoding vectors, to obtain spatial mapping results of the sample data; Decoding the spatial mapping results of the sample data through the decoding layer of the transformer pre-training model, to obtain data classification results of the sample data; Iteratively updating the transformer pre-training model based on the data classification results of the sample data, to obtain a trained data classification model.

5. The method of claim 4, wherein the metadata management based on the blood analysis is characterized by, Iteratively updating the transformer pre-training model based on the data classification results of the sample data, to obtain a trained data classification model, specifically including: Obtaining a loss function of the transformer pre-training model; Calculating a relative error between the data classification results and preset standard classification results based on the loss function, to obtain a classification error; Passing the classification error in the transformer pre-training model, and comparing the classification error with a preset error threshold; If the classification error is greater than the preset error threshold, iteratively updating the transformer pre-training model until model fitting, to obtain a trained data classification model.

6. The method of metadata management based on blood analysis according to any one of claims 1 to 5, wherein, Constructing a data map of the to-be-processed data according to mapping relationships between each of the to-be-processed data in the data classification results and metadata descriptions corresponding to each of the to-be-processed data, specifically including: Drawing an initial data map according to mapping relationships between each of the to-be-processed data in the data classification results; Adding metadata descriptions corresponding to each of the to-be-processed data in the initial data map, to form a data map of the to-be-processed data.

7. A metadata management apparatus based on consanguinity analysis, characterized by, Including: A data acquisition module, configured to obtain to-be-processed data from a preset data table, and obtain metadata descriptions corresponding to each of the to-be-processed data, to obtain first metadata descriptions; A first analysis module, configured to perform blood analysis on the to-be-processed data according to the first metadata descriptions, to obtain blood information of the to-be-processed data; A first division module, configured to perform data division on the to-be-processed data according to the blood information of the to-be-processed data, to obtain a blood data set; The data classification module is configured to input the blood relationship data set into a pre-trained data classification model to obtain a data classification result of the to-be-processed data, wherein the data classification result records a mapping relationship between each of the to-be-processed data. The data aggregation module is configured to construct a data map of the to-be-processed data according to the mapping relationship between each of the to-be-processed data in the data classification result and the metadata description corresponding to each of the to-be-processed data. The data management module is configured to perform data management on the to-be-processed data according to the data map. The data classification model is trained based on a transformer pre-training model, the transformer pre-training model is a model based on an encoder-decoder structure, the encoder and the decoder of the transformer pre-training model are both composed of an attention module and a previous neural network, the data classification model includes an encoding layer and a decoding layer, and the data classification module specifically includes: The feature processing unit is configured to perform feature extraction and feature vector conversion on the to-be-processed data in the blood relationship data set to obtain a data feature vector. The encoding unit is configured to encode the data feature vector through the encoding layer of the data classification model to obtain a data encoding vector. The spatial mapping unit is configured to perform spatial mapping on the data encoding vector to obtain a spatial mapping result of the to-be-processed data. The decoding unit is configured to decode the spatial mapping result of the to-be-processed data through the decoding layer of the data classification model to obtain the data classification result of the to-be-processed data.

8. A computer device, comprising: The computer readable storage medium stores computer readable instructions, and the computer readable instructions are executed by the processor to implement the steps of the blood relationship analysis-based metadata management method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer readable instructions, and the computer readable instructions are executed by the processor to implement the steps of the blood relationship analysis-based metadata management method according to any one of claims 1 to 6.

Images