Metadata management method and apparatus

Abstract

The application discloses a metadata management method and device. The method comprises the following steps: acquiring all first metadata of all inventories in a target data system before a target time, and generating a metadata database based on the first metadata; classifying all the first metadata in the metadata database by using a pre-trained data classification model, and dividing each first metadata into first hot metadata or first cold metadata; determining first sub-hot metadata meeting a preset data standard and second sub-hot metadata not meeting the preset data standard from the plurality of first hot metadata, and performing data correction on the second sub-hot metadata; and synchronizing the first sub-hot metadata and the second sub-hot metadata after data correction to a blockchain. The application solves the technical problem that the management of metadata in the related art is relatively chaotic, resulting in low quality of metadata and difficulty in application.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a metadata management method and apparatus. Background Technology

[0002] With the rapid development of the big data era, enterprises that have undergone years of IT infrastructure development have accumulated a large number of IT systems and massive amounts of metadata. However, the aggregation of large-scale data has led to problems such as unreasonable allocation and use of data, leakage and abuse of privacy data, and low data quality, which urgently need to be addressed.

[0003] Typically, enterprise-level metadata management architectures employ centralized or distributed architectures. Centralized architectures facilitate standardized and unified management and application of metadata, but to ensure data consistency, they require handling large amounts of data and place high demands on storage and platform requirements. Distributed architectures offer the advantage of keeping metadata up-to-date and valid, and simplifying queries, but they struggle to guarantee data consistency, and data standards from different data sources are difficult to unify.

[0004] Therefore, neither of the above two management architectures can guarantee the accuracy, completeness, and consistency of metadata, resulting in poor metadata quality and difficulty in effectively governing enterprise data through metadata.

[0005] There is currently no effective solution to the above problems. Summary of the Invention

[0006] This application provides a metadata management method and apparatus to at least solve the technical problem in the related art that the management of metadata is relatively chaotic, resulting in low metadata quality and difficulty in application.

[0007] According to one aspect of the embodiments of this application, a metadata management method is provided, comprising: acquiring all existing first metadata in a target data system before a target time, and generating a metadata database based on the first metadata; classifying all first metadata in the metadata database using a pre-trained data classification model, dividing each first metadata into first hot metadata or first cold metadata; determining first sub-hot metadata that meets a preset data standard and second sub-hot metadata that does not meet the preset data standard among a plurality of first hot metadata, and performing data correction on the second sub-hot metadata; and synchronizing the first sub-hot metadata and the data-corrected second sub-hot metadata to a blockchain.

[0008] Optionally, the incremental second metadata in the target data system after the target time is obtained; the second metadata is validated, wherein the data validation includes at least: data accuracy validation, data integrity validation and data consistency validation; when the second metadata passes the data validation, the second metadata is added to the metadata database and synchronized to the blockchain.

[0009] Optionally, after each preset time period, the data classification model is used again to classify all first cold metadata in the metadata database, and each first cold metadata is divided into second hot metadata or second cold metadata; the third sub-hot metadata that meets the preset data standard and the fourth sub-hot metadata that does not meet the preset data standard are identified among the multiple second hot metadata, and the data of the fourth sub-hot metadata is corrected; the third sub-hot metadata and the data-corrected fourth sub-hot metadata are synchronized to the blockchain.

[0010] Optionally, the process involves obtaining the first metadata of all existing data in the target data system prior to the target time, and generating a metadata database based on the first metadata. This includes: determining the target data collection task, wherein the target data collection task includes at least: table creation statements, path information, required permissions, and collection frequency; collecting data from multiple relational and non-relational databases in the target data system based on the target data collection task to obtain the technical metadata of all existing data prior to the target time; and cleaning and transforming the technical metadata to obtain the first metadata, wherein the first metadata includes at least: data source system information, database information, data table information, table field information, index information, and constraint information.

[0011] Optionally, the training process of the data classification model includes: constructing a neural network model based on gated recurrent units to be trained, wherein the neural network model includes an input layer, an output layer, a reset gate, and an update gate; determining the number of operations and the latest operation time of each first metadata in the metadata database within the target time period by parsing the metadata operation log, and sorting the multiple first metadata according to the number of operations and the latest operation time; for each first metadata, determining the training word vector and hyperparameters corresponding to the first metadata, wherein the training word vector includes at least: the metadata name of the first metadata, the number of operations of the first metadata within the target time period, and the latest operation time of the first metadata within the target time period, and the hyperparameter is the weighted average of the ranking of the number of operations and the ranking of the latest operation time of the first metadata; inputting each training word vector and hyperparameter into the neural network model for iterative training, and adjusting the model parameters of the neural network model based on the backpropagation algorithm to obtain the data classification model.

[0012] Optionally, data correction is performed on the second sub-hot data, including: for each second sub-hot data, determining multiple table metadata in the data source system corresponding to the second sub-hot data, and determining multiple field metadata in each table metadata; vectorizing the multiple field metadata using one-hot encoding technology, determining the central word of each table metadata based on the continuous bag-of-words model algorithm, and determining the central theme of the multiple table metadata based on the probabilistic latent semantic analysis algorithm; determining a target sub-data system that matches the central theme from the target data system, and determining the correction value of the second sub-hot data based on the data in the target sub-data system; and correcting the second sub-hot data based on the correction value.

[0013] Optionally, the first sub-hot data and the data-corrected second sub-hot data are synchronized to the blockchain, including: uploading the first sub-hot data and the data-corrected second sub-hot data to the target blockchain platform, wherein the target blockchain platform is used to verify and confirm the rights of the first sub-hot data and the data-corrected second sub-hot data, and to conduct trusted transactions of the first sub-hot data and the data-corrected second sub-hot data based on smart contracts.

[0014] Optionally, data validation is performed on the second metadata, including: inputting the second metadata into a pre-trained data validation model to obtain the target confidence level output by the data validation model, wherein the data validation model is used to perform accuracy validation, integrity validation, and consistency validation on the input data, and outputs the confidence level of the data passing the validation; when the target confidence level is greater than a preset confidence level threshold, the second metadata is determined to have passed the data validation; when the target confidence level is not greater than the preset confidence level threshold, the second metadata is sent to the manual validation module for manual data validation.

[0015] According to another aspect of the embodiments of this application, a metadata management device is also provided, comprising: an acquisition module, configured to acquire all existing first metadata in a target data system before a target time, and generate a metadata database based on the first metadata; a classification module, configured to classify all first metadata in the metadata database using a pre-trained data classification model, dividing each first metadata into first hot metadata or first cold metadata; a correction module, configured to determine first sub-hot metadata that meets a preset data standard and second sub-hot metadata that does not meet the preset data standard among a plurality of first hot metadata, and to correct the second sub-hot metadata; and a synchronization module, configured to synchronize the first sub-hot metadata and the corrected second sub-hot metadata to a blockchain.

[0016] According to another aspect of the embodiments of this application, an electronic device is also provided, the electronic device including: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-described metadata management method through the computer program.

[0017] In this embodiment, all existing first metadata in the target data system before the target time is obtained, and a metadata database is generated based on the first metadata. A pre-trained data classification model is used to classify all first metadata in the metadata database, dividing each first metadata into first hot metadata or first cold metadata. First sub-hot metadata that meets preset data standards and second sub-hot metadata that does not meet the preset data standards are identified among multiple first hot metadata, and the second sub-hot metadata is corrected. The first sub-hot metadata and the corrected second sub-hot metadata are synchronized to the blockchain. Specifically, after classifying the first metadata using the pre-trained data classification model, the classified second sub-hot metadata is accurately corrected, and the corrected second sub-hot metadata is synchronized with the first sub-hot metadata to the blockchain. This enables shared transactions of metadata within the enterprise, ensuring the quality and consistency of the metadata, while also ensuring that the metadata is up-to-date and valid. The query method is also simple, thus solving the technical problem in related technologies where metadata management is chaotic, leading to low metadata quality and difficulty in application. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 This is a flowchart illustrating a metadata management method according to an embodiment of this application;

[0020] Figure 2 This is a flowchart illustrating an optional metadata management method according to an embodiment of this application;

[0021] Figure 3 This is a schematic diagram of an optional GRU neural network model according to an embodiment of this application;

[0022] Figure 4 This is a schematic diagram of the structure of a metadata management device according to an embodiment of this application. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0024] It should be noted that the terms "first," "second," etc., used in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] To better understand the embodiments of this application, the following is a translation and explanation of some nouns or terms that appear in the description of the embodiments of this application:

[0026] GRU (Gated Recurrent Unit): GRU introduces reset and update gates. The reset gate helps capture short-term dependencies in a time series, while the update gate helps capture long-term dependencies. The inputs to both the reset and update gates in GRU are the current time step input and the hidden state from the previous time step. The output is calculated using a fully connected layer with a sigmoid activation function. In other words, GRU can control the flow of information through learned gates, better capturing dependencies with large time step distances in a time series. GRU modifies the calculation method of the hidden state through the reset and update gates, and the output value of the hidden state is calculated by combining the hidden state from the previous time step and the candidate hidden state of the current step using the update gate output of the current step.

[0027] One-Hot encoding, also known as one-bit valid encoding, primarily uses an N-bit state register to encode N states. Each state has its own independent register bit, and only one bit is valid at any given time. One-Hot encoding represents categorical variables as binary vectors. This first requires mapping categorical values ​​to integer values ​​and representing each integer value as a binary vector. Except for the integer index, all other values ​​are zero and are marked as 1.

[0028] CBOW (Continuous Bag of Words) model: CBOW is a variation of the word2vec model. CBOW predicts the center word from the context in a single pass. Therefore, by taking all the words in the context (excluding the middle word) as input, CBOW can obtain the center word that is most likely to be located in the center.

[0029] Probabilistic Latent Semantic Analysis (PLSA) Model: Also known as Probabilistic Latent Semantic Indexing, PLSA is an unsupervised learning method that uses a probabilistic generative model to perform topic analysis on a collection of texts. Given a collection of texts, where each text discusses several topics and each topic is represented by a number of words, PLSA allows the observable discovery of several topics for each text and several words for each topic. The most significant feature of the PLSA model is its use of a probabilistic generative model for topic analysis, which can utilize latent variables to represent topics. This model can transform a collection of texts into text-word co-occurrence data, i.e., a word-text matrix.

[0030] Example 1

[0031] With the rapid development of the big data era, enterprises that have undergone years of IT infrastructure development have accumulated a large number of IT systems and massive amounts of metadata. However, the aggregation of large-scale data has led to problems such as unreasonable allocation and use of data, leakage and abuse of privacy data, and low data quality, which urgently need to be addressed.

[0032] Metadata management is the foundation of data governance. For enterprises, the successful realization of data value relies on high-quality metadata. Only accurate, complete, and consistent metadata can build a high-value metadata catalog, thereby serving data value acquisition, business model innovation, and operational risk control. For telecommunications operators, due to the high level of informatization, each telecommunications company has tens of thousands of systems, resulting in a massive amount of metadata. Although this data has formed metadata and been aggregated into a big data lake, forming a metadata database, the presence of empty fields, garbled characters, lack of Chinese comments, and numerous duplicate tables and fields leads to low metadata quality, failing to provide strong support for business decision-making and data application.

[0033] Typically, enterprise-level metadata management architectures employ centralized or distributed architectures. Centralized architectures facilitate standardized and unified management and application of metadata, but to ensure data consistency, they require handling large amounts of data and place high demands on storage and platform requirements. Distributed architectures offer the advantage of keeping metadata up-to-date and valid, and simplifying queries, but they struggle to guarantee data consistency, and data standards from different data sources are difficult to unify.

[0034] Therefore, the relevant technologies cannot guarantee the accuracy, completeness, and consistency of metadata, resulting in poor metadata quality and difficulty in effectively governing enterprise data through metadata.

[0035] To address the aforementioned issues, this application provides a metadata management method. A pre-trained data classification model categorizes first metadata into first hot metadata or first cold metadata, wherein the first cold metadata is not immediately added to the blockchain. Second sub-hot metadata within the first hot metadata that does not meet preset data standards is precisely corrected, and the corrected second sub-hot metadata is synchronized with the first sub-hot metadata to the blockchain. This enables shared transactions of metadata within the enterprise, ensuring the quality and consistency of metadata, while also guaranteeing that the metadata is up-to-date and valid. The query method is also relatively simple.

[0036] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0037] Figure 1 This is a flowchart illustrating an optional metadata management method according to an embodiment of this application, such as... Figure 1 As shown, the method includes at least steps S102-S108, wherein:

[0038] Step S102: Obtain the first metadata of all existing data in the target data system before the target time, and generate a metadata database based on the first metadata.

[0039] According to an optional embodiment of this application, all metadata in a data system can be encapsulated in the following way: First, a target acquisition task is determined, wherein the target acquisition task includes at least: table creation statement, path information, required permissions, and acquisition frequency; based on the target acquisition task, data is acquired from multiple relational databases and non-relational databases in the target data system to obtain all existing technical metadata up to the target time; the technical metadata is cleaned and transformed to obtain first metadata, wherein the first metadata includes at least: data source system information, database information, data table information, table field information, index information, and constraint information.

[0040] For example, Figure 2 A flowchart illustrating an optional metadata management method is shown, such as... Figure 2As shown, based on the data collection task, data is collected from relational and non-relational databases in data systems such as MSS domains, OSS domains, BSS domains, or network element platforms at the current moment. This yields all technical metadata up to the current moment. This technical metadata can be compiled into a data resource list. The data resource list is then cleaned and transformed, removing duplicates, garbled characters, empty fields, and data without Chinese comments. After standardizing the format of all data, the first metadata is obtained. Finally, all the first metadata is aggregated into a big data lake, forming a metadata database. This metadata database can be used for applications in different scenarios, for managing enterprise-level data asset catalogs, and for big data decision analysis.

[0041] Step S104: Use a pre-trained data classification model to classify all first metadata in the metadata database, and divide each first metadata into first hot metadata or first cold metadata.

[0042] According to an optional embodiment of this application, the training of the data classification model can be completed through steps S1-S4:

[0043] Step S1: Construct the GRU neural network model to be trained (i.e., a neural network model based on gated recurrent units), wherein the GRU neural network model includes an input layer, an output layer, a reset gate, and an update gate.

[0044] Specifically, Figure 3 A schematic diagram of an optional GRU neural network model is shown, such as... Figure 3 As shown, by taking the input x at the current time... t and the hidden state h passed down from the previous node t-1 By inputting the relevant information of previous nodes into the GRU neural network model, the output value of the current hidden layer and the hidden state h passed to the next node can be obtained. t .

[0045] The training word vector x at time t t The hyperparameter value c and the output value h of the hidden layer of the GRU neural network model at the previous time step. t-1 As a new word vector [h t-1 ,x t ,c], to the word vector [h t-1 ,x t ,c] serves as the input vector for the model.

[0046] The word vector [h] t-1 ,x t [c] is input to the reset gate, which determines how much past information needs to be forgotten. Therefore, the reset gate is expressed as follows:

[0047]

[0048] Among them, w r This indicates the weight parameters for resetting the door.

[0049] The new word vector [h t-1 ,x t [c] Input to the update gate, where the update gate determines how much past information should be passed to the future, or how much of the information from the previous and current moments needs to be passed. Therefore, the expression for the update gate is as follows:

[0050]

[0051] Among them, w z To update the weight parameters of the gate.

[0052] It should be noted that both the reset gate and the update gate use the sigmoid function as the activation function; the only difference lies in the parameters and application of the linear transformation. The sigmoid function maps a real number to the interval (0,1), and its expression is as follows:

[0053]

[0054] Reset the gate output r t Input to candidate hidden layer As new information at the current moment, the candidate hidden layer is represented as follows:

[0055]

[0056] Where W represents the weight parameters of the candidate hidden layer, and tanh is the hyperbolic tangent function, expressed as:

[0057]

[0058] Then convert the output values ​​of the candidate hidden layer The input is fed into the hidden layer, and the output value h of the hidden layer is obtained. t Therefore, the hidden layer is represented as follows:

[0059]

[0060] Finally, the output value h of the hidden layer t The input is fed into the output layer to obtain the final output result y. Therefore, the representation formula of the output layer is as follows:

[0061] y = w y *h t+b y

[0062] Among them, w y b represents the weight parameters of the output layer. y This represents the offset of the output layer weight parameters.

[0063] S2, by parsing the metadata operation log, determine the number of operations and the latest operation time of each first metadata in the metadata database within the target time period, and sort the multiple first metadata according to the number of operations and the latest operation time respectively.

[0064] The number of operations can be the number of times the first metadata is accessed or modified.

[0065] Specifically, the recent access or modification count, access time, or modification time of each primary metadata in the metadata database can be determined by parsing the metadata operation log. Then, based on the operation count of each primary metadata in the metadata dictionary, it is sorted in ascending order to obtain the sequence number A for each metadata. Finally, based on the access time or modification time of each primary metadata in the metadata dictionary, it is sorted in ascending order to obtain the sequence number B for each primary metadata.

[0066] For example, if the modification count of the xth piece of first metadata in the metadata database ranks 268th out of all 1000 pieces of metadata, then A equals 268 / 1000. If the access time of the xth piece of first metadata in the metadata database ranks 865th out of all 1000 pieces of metadata, then B equals 865 / 1000. Therefore, the hyperparameter C of the xth piece of first metadata equals 1133 / 2000.

[0067] S3. For each first metadata, determine the training word vector and hyperparameters corresponding to the first metadata. The training word vector includes at least: the metadata name of the first metadata, the number of operations of the first metadata within the target time period, and the latest operation time of the first metadata within the target time period. The hyperparameter is the weighted average of the ranking of the number of operations of the first metadata and the ranking of the latest operation time of the first metadata.

[0068] Specifically, the training word vectors of the first metadata can be determined according to the format [metadata name of the first metadata, number of recent operations on the first metadata, and recent access time or modification time of the first data], as shown in Table 1.

[0069] Table 1

[0070] First metadata name Number of operations Access time or modification time lan_id 55 2022 / 05 / 30 / 16：46 cust_id 34 2021 / 12 / 25 / 09：27

[0071] Meanwhile, to improve the performance and effectiveness of data classification models, the optimal hyperparameters for the model at the beginning of the learning process are usually determined in advance. In this embodiment, the corresponding hyperparameters are determined by the number of operations performed on the first metadata within the target time period and the latest operation time of the first metadata within the target time period, and are used as the optimal hyperparameters for the GRU neural network model.

[0072] Specifically, after sorting multiple primary data elements according to the number of operations and the latest operation time to obtain sequence number A and sequence number B, the hyperparameter C can be calculated based on sequence number A and sequence number B. The specific calculation formula is as follows:

[0073]

[0074] S4. Input each training word vector and hyperparameter into the neural network model in sequence for iterative training, and adjust the model parameters of the neural network model based on the backpropagation algorithm to obtain the data classification model.

[0075] Specifically, the loss function J of the model is first determined based on the training word vectors and hyperparameters corresponding to the first metadata. The partial derivative of the loss function J with respect to the input layer is as follows:

[0076]

[0077] The formula for the partial derivative of the loss function J with respect to the output layer is as follows:

[0078]

[0079] The partial derivative of the loss function J with respect to the update gate is formulated as follows:

[0080]

[0081] The partial derivative of the loss function J with respect to the reset gate is formulated as follows:

[0082]

[0083] Then, the model parameters are updated through the backpropagation algorithm to make the trained GRU neural network model more accurate, thereby ensuring better performance and effectiveness of the data classification model.

[0084] Therefore, the output value y of the GRU neural network model can be used as the output of the final data classification model, and y∈[0,1]. Specifically, when the output y=1 of the GRU neural network model, it indicates that the primary metadata is the first hot metadata; when the output y=0 of the GRU neural network model, it indicates that the primary metadata is the first cold metadata. The data classification model can accurately classify all primary metadata in the metadata database.

[0085] After classifying the first set of metadata, the following steps are required to accurately correct the hot metadata that does not meet the data standards in the first set of hot metadata, and then synchronize the corrected hot metadata to the blockchain.

[0086] Step S106: Determine the first sub-thermal data that meets the preset data standard and the second sub-thermal data that does not meet the preset data standard among the multiple first thermal data, and perform data correction on the second sub-thermal data.

[0087] According to an optional embodiment of this application, data correction of the second sub-hot data can be achieved through the following method: For each second sub-hot data, determine multiple table metadata in the data source system corresponding to the second sub-hot data, and determine multiple field metadata in each table metadata; vectorize the multiple field metadata using one-hot encoding technology, determine the central word of each table metadata based on the continuous bag-of-words model algorithm, and determine the central theme of the multiple table metadata based on the probabilistic latent semantic analysis algorithm; determine a target sub-data system that matches the central theme from the target data system, and determine the correction value of the second sub-hot data based on the data in the target sub-data system; and perform data correction on the second sub-hot data based on the correction value.

[0088] Specifically, the data source system set is first denoted as P = {P1, P2, P3, ..., P}. t ,…,P m}, where P t Let represent the t-th data source system set that does not conform to the preset data standard, and m represent the total number of data source system sets that do not conform to the preset data standard. Since each data source system has multiple table metadata, the table metadata dataset in the data source system can be denoted as P(x) = {T1, T2, T3, ..., T...}. t ,…,T n}, where T t Let represent the metadata of the t-th table, and n represent the total number of table metadata in the data source system. Furthermore, each table metadata includes multiple field metadata; therefore, the dataset of field metadata for a single table metadata can be denoted as T. x ={C1,C2,C3,…,C t ,…,C k}, where C t This represents the metadata of the t-th field, and k represents the total number of field metadata in the table metadata dataset.

[0089] Secondly, one-hot encoding is used to vectorize the metadata of each field in the table metadata, and then the CBOW (Continuous Bag-of-Words) algorithm is used to determine the central word of each field metadata. If the field metadata T(x) is denoted as w... inputThe probability of the predicted central word appearing is denoted as w. t Then the central term of each field's metadata in the table metadata can be represented as:

[0090] p(w t |w input ) = w j =exp(w j ) / ∑exp(w it )

[0091] The preferred approach is to determine the central keywords of each field's metadata in each table's metadata using the maximum conditional probability, which can be expressed by the following formula:

[0092] maxp(w t |w input ) = maxlogw j =w j -log∑exp(w j )

[0093] After determining the central words of each field's metadata in each table's metadata, the PLSA topic model (i.e., probabilistic latent semantic analysis) algorithm can be used, where first, with probability P(P n Select a data source system P from the metadata set P. m Then with probability P(T) n |P n Select a table metadata T from the data source system set P(x). n Finally, with probability P(C) k |T n Select a field metadata C from the table metadata T(x). k It is calculated using the maximum likelihood function, and its formula is as follows:

[0094]

[0095] In the above formula, n(P) m |C k ) indicates (P m |C k The number of times ) appears, m′ represents the number of table metadata, and k′ represents the number of field metadata.

[0096] Taking the logarithm of both sides of the formula, the subject with the highest probability prediction center is ultimately taken as the central subject of the metadata of each table in each data system. The specific calculation formula is as follows:

[0097]

[0098] After determining the central theme of the table metadata, a sub-data system matching the central theme category is identified from the data system. Based on the data in the sub-data system, the correction value of the second sub-hot metadata is determined, thereby enabling precise correction of the second sub-hot metadata to meet the preset data standards of the data system.

[0099] Step S108: Synchronize the first sub-hot data and the corrected second sub-hot data to the blockchain.

[0100] According to an optional embodiment of this application, the first sub-hot data and the data-corrected second sub-hot data are uploaded to a target blockchain platform. The target blockchain platform is used to verify the first sub-hot data and the data-corrected second sub-hot data for verification and confirmation of rights, and to conduct trusted transactions of the first sub-hot data and the data-corrected second sub-hot data based on smart contracts.

[0101] Specifically, the corrected second sub-hot data and the first sub-hot data are uploaded to the blockchain platform through step S106. For the first sub-hot data and the corrected second sub-hot data on the blockchain platform, the valid hot data is verified and confirmed. According to the needs of the enterprise system, the first sub-hot data and the corrected second sub-hot data are queried and retrieved in the form of a data service smart contract. The query and retrieval operation is performed by using the PostgreSQL external table function to perform SQL statements to query and retrieve data in the first metabase database and return the results to achieve trusted transactions.

[0102] After completing the chain entry operation for the existing data (i.e., the first metadata) of the data system at and before the target time through the above process steps S106-S108, it is also necessary to perform the chain entry operation for the incremental data of the data system after the target time.

[0103] According to an optional embodiment of this application, firstly, incremental second metadata after the target time is obtained in the target data system; then, data verification is performed on the second metadata, wherein the data verification includes at least: data accuracy verification, data integrity verification and data consistency verification; when the second metadata passes the data verification, the second metadata is added to the metadata database and synchronized to the blockchain.

[0104] Among them, data accuracy verification provides data uniqueness verification methods, including single indicator value domain uniqueness detection, joint value domain uniqueness detection of multiple indicators in the same group, and multi-group multi-indicator value domain uniqueness detection; data integrity verification provides data integrity verification methods, including single indicator value domain non-empty detection and joint value domain non-empty detection of multiple indicators in the same group; data consistency verification provides data consistency verification methods, including field-based detection, indicator-based detection, and calculation-based detection.

[0105] Optionally, data verification of the second metadata can be implemented as follows: input the second metadata into a pre-trained data verification model to obtain the target confidence level output by the data verification model, wherein the data verification model is used to verify the accuracy, completeness, and consistency of the input data and output the confidence level of the data passing the verification; when the target confidence level is greater than a preset confidence level threshold, the second metadata is determined to have passed the data verification; when the target confidence level is not greater than the preset confidence level threshold, the second metadata is sent to the manual verification module for manual data verification.

[0106] Specifically, based on the data collection task settings in step S102, metadata will be continuously added to the metadata database in the enterprise system. Therefore, the second metadata added to the data system after the target time can be obtained and input into a pre-trained data verification model. The second metadata is then verified according to preset verification rules such as data accuracy, data integrity, and data consistency, and its confidence level is output. By comparing the output confidence level with a preset confidence threshold, it is determined whether the second metadata passes data verification. Second metadata that passes verification is added to the metadata database and synchronized to the blockchain; second metadata that fails verification continues to undergo manual verification.

[0107] Furthermore, after uploading the hot metadata to the blockchain platform through the above steps S106-S108, the first cold metadata can be classified and processed, and the hot metadata that meets the preset data standards in the first cold metadata can be uploaded to the blockchain.

[0108] According to an optional embodiment of this application, after each preset time period, the data classification model is used again to classify all first cold metadata in the metadata database, and each first cold metadata is divided into second hot metadata or second cold metadata; the third sub-hot metadata that meets the preset data standard and the fourth sub-hot metadata that does not meet the preset data standard are determined among the multiple second hot metadata, and the fourth sub-hot metadata is corrected; the third sub-hot metadata and the corrected fourth sub-hot metadata are synchronized to the blockchain.

[0109] For example, every quarter, a pre-trained data classification model is used to classify all first cold metadata in the metadata database to distinguish the latest second hot metadata and second cold metadata. Then, the fourth sub-hot metadata in each second hot metadata that does not meet the system's preset data standards is corrected through step S106. Finally, the third sub-hot metadata in the second hot metadata that meets the system's preset standards and the corrected fourth sub-hot metadata are uploaded to the blockchain simultaneously.

[0110] In this embodiment, all existing first metadata in the target data system before the target time is obtained, and a metadata database is generated based on the first metadata. A pre-trained data classification model is used to classify all first metadata in the metadata database, dividing each first metadata into first hot metadata or first cold metadata. First sub-hot metadata that meets preset data standards and second sub-hot metadata that does not meet the preset data standards are identified among multiple first hot metadata, and the second sub-hot metadata is corrected. The first sub-hot metadata and the corrected second sub-hot metadata are synchronized to the blockchain. Specifically, after classifying the first metadata using the pre-trained data classification model, the classified second sub-hot metadata is accurately corrected, and the corrected second sub-hot metadata is synchronized with the first sub-hot metadata to the blockchain. This enables shared transactions of metadata within the enterprise, ensuring the quality and consistency of the metadata, while also ensuring that the metadata is up-to-date and valid. The query method is also simple, thus solving the technical problem in related technologies where metadata management is chaotic, leading to low metadata quality and difficulty in application.

[0111] Example 2

[0112] According to an embodiment of this application, a metadata management device for implementing the metadata management method in embodiment 1 is also provided, such as... Figure 4 As shown, the metadata management device includes at least an acquisition module 41, a classification module 42, a correction module 43, and a synchronization module 44, wherein:

[0113] The acquisition module 41 is used to acquire the first metadata of all existing data in the target data system before the target time, and generate a metadata database based on the first metadata.

[0114] According to an optional embodiment of this application, all metadata in a data system can be encapsulated in the following way: First, a target acquisition task is determined, wherein the target acquisition task includes at least: table creation statement, path information, required permissions, and acquisition frequency; based on the target acquisition task, data is acquired from multiple relational databases and non-relational databases in the target data system to obtain all existing technical metadata up to the target time; the technical metadata is cleaned and transformed to obtain first metadata, wherein the first metadata includes at least: data source system information, database information, data table information, table field information, index information, and constraint information.

[0115] The classification module 42 is used to classify all primary metadata in the metadata database using a pre-trained data classification model, and divide each primary metadata into primary hot metadata or primary cold metadata.

[0116] According to an optional embodiment of this application, the classification module 42 can complete the training of the data classification model through steps S1-S4:

[0117] Step S1: Construct the GRU neural network model to be trained (i.e., a neural network model based on gated recurrent units), wherein the GRU neural network model includes an input layer, an output layer, a reset gate, and an update gate.

[0118] S2, by parsing the metadata operation log, determine the number of operations and the latest operation time of each first metadata in the metadata database within the target time period, and sort the multiple first metadata according to the number of operations and the latest operation time respectively.

[0119] The number of operations can be the number of times the first metadata is accessed or modified.

[0120] Specifically, the recent access or modification count, access time, or modification time of each primary metadata in the metadata database can be determined by parsing the metadata operation log. Then, based on the operation count of each primary metadata in the metadata dictionary, it is sorted in ascending order to obtain the sequence number A for each metadata. Finally, based on the access time or modification time of each primary metadata in the metadata dictionary, it is sorted in ascending order to obtain the sequence number B for each primary metadata.

[0121] S3. For each first metadata, determine the training word vector and hyperparameters corresponding to the first metadata. The training word vector includes at least: the metadata name of the first metadata, the number of operations of the first metadata within the target time period, and the latest operation time of the first metadata within the target time period. The hyperparameter is the weighted average of the ranking of the number of operations of the first metadata and the ranking of the latest operation time of the first metadata.

[0122] S4. Input each training word vector and hyperparameter into the neural network model in sequence for iterative training, and adjust the model parameters of the neural network model based on the backpropagation algorithm to obtain the data classification model.

[0123] Specifically, the output value y of the GRU neural network model can be used as the output of the final data classification model, where y∈[0,1]. When the output y=1 of the GRU neural network model, it indicates that the primary metadata is the first "hot" metadata; when the output y=0, it indicates that the primary metadata is the first "cold" metadata. The data classification model can accurately classify all primary metadata in the metadata database.

[0124] After classifying the first set of metadata, the following steps are required to accurately correct the hot metadata that does not meet the data standards in the first set of hot metadata, and then synchronize the corrected hot metadata to the blockchain.

[0125] The correction module 43 is used to determine the first sub-thermal data that meets the preset data standard and the second sub-thermal data that does not meet the preset data standard among the multiple first thermal data, and to correct the data of the second sub-thermal data.

[0126] According to an optional embodiment of this application, the correction module 43 can correct the data of the second sub-hot data in the following way: for each second sub-hot data, determine multiple table metadata in the data source system corresponding to the second sub-hot data, and determine multiple field metadata in each table metadata; vectorize the multiple field metadata using one-hot encoding technology, determine the central word of each table metadata based on the continuous bag-of-words model algorithm, and determine the central theme of the multiple table metadata based on the probabilistic latent semantic analysis algorithm; determine the target sub-data system that matches the central theme from the target data system, and determine the correction value of the second sub-hot data based on the data in the target sub-data system; and correct the second sub-hot data based on the correction value.

[0127] Synchronization module 44 is used to synchronize the first sub-hot data and the corrected second sub-hot data to the blockchain.

[0128] According to an optional embodiment of this application, the synchronization module 44 uploads the first sub-hot data and the data-corrected second sub-hot data to the target blockchain platform, wherein the target blockchain platform is used to verify and confirm the rights of the first sub-hot data and the data-corrected second sub-hot data, and to conduct trusted transactions of the first sub-hot data and the data-corrected second sub-hot data based on smart contracts.

[0129] Specifically, the second sub-hot data and the first sub-hot data, corrected by the correction module 43, are uploaded to the blockchain platform. For the first sub-hot data and the corrected second sub-hot data on the blockchain platform, the valid hot data is verified and its rights are confirmed. According to the needs of the enterprise system, the first sub-hot data and the corrected second sub-hot data are queried and retrieved in the form of a data service smart contract. The query and retrieval operation is performed by using the PostgreSQL external table function to perform SQL statements to query and retrieve data in the first metabase database and return the results to achieve trusted transactions.

[0130] After completing the on-chain operation of the existing data (i.e., the first metadata) of the data system at and before the target time through the above modules, it is also necessary to perform the on-chain operation of the incremental data of the data system after the target time.

[0131] According to an optional embodiment of this application, firstly, incremental second metadata after the target time is obtained in the target data system; then, data verification is performed on the second metadata, wherein the data verification includes at least: data accuracy verification, data integrity verification and data consistency verification; when the second metadata passes the data verification, the second metadata is added to the metadata database and synchronized to the blockchain.

[0132] Among them, data accuracy verification provides data uniqueness verification methods, including single indicator value domain uniqueness detection, joint value domain uniqueness detection of multiple indicators in the same group, and multi-group multi-indicator value domain uniqueness detection; data integrity verification provides data integrity verification methods, including single indicator value domain non-empty detection and joint value domain non-empty detection of multiple indicators in the same group; data consistency verification provides data consistency verification methods, including field-based detection, indicator-based detection, and calculation-based detection.

[0133] Optionally, data verification of the second metadata can be implemented as follows: input the second metadata into a pre-trained data verification model to obtain the target confidence level output by the data verification model, wherein the data verification model is used to verify the accuracy, completeness, and consistency of the input data and output the confidence level of the data passing the verification; when the target confidence level is greater than a preset confidence level threshold, the second metadata is determined to have passed the data verification; when the target confidence level is not greater than the preset confidence level threshold, the second metadata is sent to the manual verification module for manual data verification.

[0134] Specifically, based on the data acquisition task settings in module 41, metadata will be continuously added to the metadata database within the enterprise system. Therefore, it is possible to acquire the second metadata added to the data system after the target time, and input this second metadata into a pre-trained data verification model. The second metadata is then verified according to preset verification rules such as data accuracy, data integrity, and data consistency, and its confidence level is output. By comparing the output confidence level with a preset confidence threshold, it is determined whether the second metadata passes data verification. Second metadata that passes verification is added to the metadata database and synchronized to the blockchain; second metadata that fails verification continues to undergo manual verification.

[0135] Furthermore, after uploading the hot metadata to the blockchain platform through the above modules, the first cold metadata can be classified and processed, and the hot metadata that meets the preset data standards in the first cold metadata can be uploaded to the blockchain.

[0136] According to an optional embodiment of this application, after each preset time period, the data classification model is used again to classify all first cold metadata in the metadata database, and each first cold metadata is divided into second hot metadata or second cold metadata; the third sub-hot metadata that meets the preset data standard and the fourth sub-hot metadata that does not meet the preset data standard are determined among the multiple second hot metadata, and the fourth sub-hot metadata is corrected; the third sub-hot metadata and the corrected fourth sub-hot metadata are synchronized to the blockchain.

[0137] It should be noted that each module in the metadata management device in this application embodiment corresponds one-to-one with each implementation step of the metadata management method in embodiment 1. Since embodiment 1 has been described in detail, some details not shown in this embodiment can be referred to embodiment 1, and will not be elaborated further here.

[0138] Example 3

[0139] According to an embodiment of this application, a non-volatile storage medium is also provided, which includes a stored program, wherein the device where the non-volatile storage medium is located executes the metadata management method in Embodiment 1 by running the program.

[0140] Specifically, the device containing the non-volatile storage medium executes the following steps by running the program: acquiring all existing first metadata in the target data system before the target time, and generating a metadata database based on the first metadata; classifying all first metadata in the metadata database using a pre-trained data classification model, dividing each first metadata into first hot metadata or first cold metadata; determining first sub-hot metadata that meets the preset data standard and second sub-hot metadata that does not meet the preset data standard among multiple first hot metadata, and correcting the data of the second sub-hot metadata; synchronizing the first sub-hot metadata and the corrected second sub-hot metadata to the blockchain.

[0141] Optionally, after each preset time period, the data classification model is used again to classify all first cold metadata in the metadata database, and each first cold metadata is divided into second hot metadata or second cold metadata; the third sub-hot metadata that meets the preset data standard and the fourth sub-hot metadata that does not meet the preset data standard are identified among the multiple second hot metadata, and the data of the fourth sub-hot metadata is corrected; the third sub-hot metadata and the data-corrected fourth sub-hot metadata are synchronized to the blockchain.

[0142] Optionally, the training process of the data classification model includes: constructing a neural network model based on gated recurrent units to be trained, wherein the neural network model includes an input layer, an output layer, a reset gate, and an update gate; determining the number of operations and the latest operation time of each first metadata in the metadata database within the target time period by parsing the metadata operation log, and sorting the multiple first metadata according to the number of operations and the latest operation time; for each first metadata, determining the training word vector and hyperparameters corresponding to the first metadata, wherein the training word vector includes at least: the metadata name of the first metadata, the number of operations of the first metadata within the target time period, and the latest operation time of the first metadata within the target time period, and the hyperparameter is the weighted average of the ranking of the number of operations and the ranking of the latest operation time of the first metadata; inputting each training word vector and hyperparameter into the neural network model for iterative training, and adjusting the model parameters of the neural network model based on the backpropagation algorithm to obtain the data classification model.

[0143] Optionally, data correction is performed on the second sub-hot data, including: for each second sub-hot data, determining multiple table metadata in the data source system corresponding to the second sub-hot data, and determining multiple field metadata in each table metadata; vectorizing the multiple field metadata using one-hot encoding technology, determining the central word of each table metadata based on the continuous bag-of-words model algorithm, and determining the central theme of the multiple table metadata based on the probabilistic latent semantic analysis algorithm; determining a target sub-data system that matches the central theme from the target data system, and determining the correction value of the second sub-hot data based on the data in the target sub-data system; and correcting the second sub-hot data based on the correction value.

[0144] Example 4

[0145] According to an embodiment of this application, a processor is also provided for running a program, wherein the program executes the metadata management method in embodiment 1 during runtime.

[0146] Specifically, the program executes the following steps during runtime: acquiring all existing first metadata in the target data system before the target time, and generating a metadata database based on the first metadata; classifying all first metadata in the metadata database using a pre-trained data classification model, dividing each first metadata into first hot metadata or first cold metadata; determining the first sub-hot metadata that meets the preset data standard and the second sub-hot metadata that does not meet the preset data standard among multiple first hot metadata, and correcting the data of the second sub-hot metadata; synchronizing the first sub-hot metadata and the corrected second sub-hot metadata to the blockchain.

[0147] Optionally, after each preset time period, the data classification model is used again to classify all first cold metadata in the metadata database, and each first cold metadata is divided into second hot metadata or second cold metadata; the third sub-hot metadata that meets the preset data standard and the fourth sub-hot metadata that does not meet the preset data standard are identified among the multiple second hot metadata, and the data of the fourth sub-hot metadata is corrected; the third sub-hot metadata and the data-corrected fourth sub-hot metadata are synchronized to the blockchain.

[0148] Optionally, the training process of the data classification model includes: constructing a neural network model based on gated recurrent units to be trained, wherein the neural network model includes an input layer, an output layer, a reset gate, and an update gate; determining the number of operations and the latest operation time of each first metadata in the metadata database within the target time period by parsing the metadata operation log, and sorting the multiple first metadata according to the number of operations and the latest operation time; for each first metadata, determining the training word vector and hyperparameters corresponding to the first metadata, wherein the training word vector includes at least: the metadata name of the first metadata, the number of operations of the first metadata within the target time period, and the latest operation time of the first metadata within the target time period, and the hyperparameter is the weighted average of the ranking of the number of operations and the ranking of the latest operation time of the first metadata; inputting each training word vector and hyperparameter into the neural network model for iterative training, and adjusting the model parameters of the neural network model based on the backpropagation algorithm to obtain the data classification model.

[0149] Optionally, data correction is performed on the second sub-hot data, including: for each second sub-hot data, determining multiple table metadata in the data source system corresponding to the second sub-hot data, and determining multiple field metadata in each table metadata; vectorizing the multiple field metadata using one-hot encoding technology, determining the central word of each table metadata based on the continuous bag-of-words model algorithm, and determining the central theme of the multiple table metadata based on the probabilistic latent semantic analysis algorithm; determining a target sub-data system that matches the central theme from the target data system, and determining the correction value of the second sub-hot data based on the data in the target sub-data system; and correcting the second sub-hot data based on the correction value.

[0150] Example 5

[0151] According to an embodiment of this application, an electronic device is also provided, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the metadata management method of embodiment 1 through the computer program.

[0152] Specifically, the processor is configured to execute the following steps via a computer program: acquire all existing first metadata in the target data system before the target time, and generate a metadata database based on the first metadata; classify all first metadata in the metadata database using a pre-trained data classification model, dividing each first metadata into first hot metadata or first cold metadata; determine the first sub-hot metadata that meets the preset data standard and the second sub-hot metadata that does not meet the preset data standard among multiple first hot metadata, and perform data correction on the second sub-hot metadata; synchronize the first sub-hot metadata and the data-corrected second sub-hot metadata to the blockchain.

[0153] Optionally, after each preset time period, the data classification model is used again to classify all first cold metadata in the metadata database, and each first cold metadata is divided into second hot metadata or second cold metadata; the third sub-hot metadata that meets the preset data standard and the fourth sub-hot metadata that does not meet the preset data standard are identified among the multiple second hot metadata, and the data of the fourth sub-hot metadata is corrected; the third sub-hot metadata and the data-corrected fourth sub-hot metadata are synchronized to the blockchain.

[0154] Optionally, the training process of the data classification model includes: constructing a neural network model based on gated recurrent units to be trained, wherein the neural network model includes an input layer, an output layer, a reset gate, and an update gate; determining the number of operations and the latest operation time of each first metadata in the metadata database within the target time period by parsing the metadata operation log, and sorting the multiple first metadata according to the number of operations and the latest operation time; for each first metadata, determining the training word vector and hyperparameters corresponding to the first metadata, wherein the training word vector includes at least: the metadata name of the first metadata, the number of operations of the first metadata within the target time period, and the latest operation time of the first metadata within the target time period, and the hyperparameter is the weighted average of the ranking of the number of operations and the ranking of the latest operation time of the first metadata; inputting each training word vector and hyperparameter into the neural network model for iterative training, and adjusting the model parameters of the neural network model based on the backpropagation algorithm to obtain the data classification model.

[0155] Optionally, data correction is performed on the second sub-hot data, including: for each second sub-hot data, determining multiple table metadata in the data source system corresponding to the second sub-hot data, and determining multiple field metadata in each table metadata; vectorizing the multiple field metadata using one-hot encoding technology, determining the central word of each table metadata based on the continuous bag-of-words model algorithm, and determining the central theme of the multiple table metadata based on the probabilistic latent semantic analysis algorithm; determining a target sub-data system that matches the central theme from the target data system, and determining the correction value of the second sub-hot data based on the data in the target sub-data system; and correcting the second sub-hot data based on the correction value.

[0156] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0157] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0158] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0159] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0160] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0161] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0162] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A metadata management method, characterized in that, include: Obtain the first metadata of all existing data in the target data system before the target time, and generate a metadata database based on the first metadata; The first metadata in the metadata database is classified using a pre-trained data classification model, and each first metadata is divided into first hot metadata or first cold metadata. First sub-thermal data that meets a preset data standard and second sub-thermal data that does not meet the preset data standard are identified from multiple first thermal data data, and data correction is performed on the second sub-thermal data. Synchronize the first sub-hot data and the corrected second sub-hot data to the blockchain; The data correction of the second sub-hot data includes: for each second sub-hot data, determining multiple table metadata in the data source system corresponding to the second sub-hot data, and determining multiple field metadata in each table metadata; vectorizing the multiple field metadata using one-hot encoding technology, determining the central word of each table metadata based on the continuous bag-of-words model algorithm, and determining the central topic of the multiple table metadata based on the probabilistic latent semantic analysis algorithm; determining a target sub-data system matching the central topic from the target data system, and determining the correction value of the second sub-hot data based on the data in the target sub-data system; and correcting the second sub-hot data based on the correction value.

2. The method according to claim 1, characterized in that, The method further includes: Obtain the second metadata of the increment after the target time in the target data system; The second metadata is validated, wherein the validation includes at least: data accuracy validation, data integrity validation, and data consistency validation. When the second metadata passes the data verification, the second metadata is added to the metadata database and synchronized to the blockchain.

3. The method according to claim 1, characterized in that, The method further includes: After each preset time period, the data classification model is used again to classify all the first cold data in the metadata database, and each first cold data is divided into second hot data or second cold data. Identify third sub-thermal data that meets the preset data standard and fourth sub-thermal data that does not meet the preset data standard among multiple second thermal data, and perform data correction on the fourth sub-thermal data; The third sub-hot data and the corrected fourth sub-hot data are synchronized to the blockchain.

4. The method according to claim 1, characterized in that, Obtain the first metadata of all existing data in the target data system up to the target time, and generate a metadata database based on the first metadata, including: The target data collection task is determined, wherein the target data collection task includes at least: table creation statement, path information, required permissions, and collection frequency; Based on the target acquisition task, data is acquired from multiple relational and non-relational databases in the target data system to obtain all existing technical metadata prior to the target time. The technical metadata is cleaned and transformed to obtain the first metadata, wherein the first metadata includes at least: data source system information, database information, data table information, table field information, index information, and constraint information.

5. The method according to claim 1, characterized in that, The training process of the data classification model includes: Construct a neural network model based on gated recurrent units to be trained, wherein the neural network model includes an input layer, an output layer, a reset gate, and an update gate; The number of operations and the latest operation time of each first metadata in the metadata database within the target time period are determined by parsing the metadata operation log, and the multiple first metadata are sorted according to the number of operations and the latest operation time respectively. For each of the first metadata, determine the training word vector and hyperparameters corresponding to the first metadata, wherein the training word vector includes at least: the metadata name of the first metadata, the number of operations of the first metadata within the target time period, and the latest operation time of the first metadata within the target time period, and the hyperparameter is the weighted average of the ranking of the number of operations of the first metadata and the ranking of the latest operation time of the first metadata. Each of the training word vectors and the hyperparameters is sequentially input into the neural network model for iterative training, and the model parameters of the neural network model are adjusted based on the backpropagation algorithm to obtain the data classification model.

6. The method according to claim 1, characterized in that, Synchronizing the first sub-hot data and the corrected second sub-hot data to the blockchain includes: The first sub-hot data and the data-corrected second sub-hot data are uploaded to the target blockchain platform, wherein the target blockchain platform is used to verify and confirm the rights of the first sub-hot data and the data-corrected second sub-hot data, and to conduct trusted transactions of the first sub-hot data and the data-corrected second sub-hot data based on smart contracts.

7. The method according to claim 2, characterized in that, Data validation is performed on the second metadata, including: The second metadata is input into a pre-trained data validation model to obtain the target confidence score output by the data validation model. The data validation model is used to perform accuracy validation, integrity validation, and consistency validation on the input data and output the confidence score of the data passing the validation. When the target confidence level is greater than a preset confidence level threshold, the second metadata is determined to have passed the data verification. When the target confidence level is not greater than the preset confidence level threshold, the second metadata is sent to the manual verification module for manual data verification.

8. A metadata management device, characterized in that, include: The acquisition module is used to acquire the first metadata of all existing data in the target data system before the target time, and generate a metadata database based on the first metadata. The classification module is used to classify all the first metadata in the metadata database using a pre-trained data classification model, and divide each first metadata into first hot metadata or first cold metadata. A correction module is used to determine first sub-hot data that meets a preset data standard and second sub-hot data that does not meet the preset data standard among a plurality of first hot data, and to correct the second sub-hot data. The correction process includes: for each second sub-hot data, determining a plurality of table data in the data source system corresponding to the second sub-hot data, and determining a plurality of field data in each table data; vectorizing the plurality of field data using one-hot encoding technology, determining the central word of each table data based on the continuous bag-of-words model algorithm, and determining the central topic of the plurality of table data based on the probabilistic latent semantic analysis algorithm; determining a target sub-data system that matches the central topic from the target data system, and determining the correction value of the second sub-hot data based on the data in the target sub-data system; and correcting the second sub-hot data based on the correction value. The synchronization module is used to synchronize the first sub-hot data and the data-corrected second sub-hot data to the blockchain.

9. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the metadata management method according to any one of claims 1 to 7 through the computer program.

Images