Metadata processing method and system

Abstract

The invention provides a metadata processing method. The metadata processing method comprises the following steps: acquiring to-be-processed metadata; the to-be-processed metadata are input into a first neural network model, the first neural network model is obtained through training of multiple sets of training metadata, and the multiple sets of training metadata comprise metadata storage features, metadata consanguinity features and metadata missing features; obtaining a first output result of the first neural network model, wherein the first output result comprises a metadata asset directory, a data blood relationship graph and a data updating list; and monitoring the output process of the first output result. The invention further provides a metadata processing system. According to the method, the automation of metadata management can be realized, the data management efficiency and the processing accuracy are improved, and the method has good expansibility and is suitable for different types of metadata processing scenes.

Description

Technical Field

[0001] The present disclosure relates to the field of data management and information technology, and more specifically to a metadata processing method and system. Background Art

[0002] In the era of big data, data diversity and complexity are increasing day by day, and all kinds of enterprises are facing rapidly growing data volumes and increasingly complex data structures. With the continuous enrichment of data sources, the diversification of data types, and the frequency of data flow, how to efficiently manage these massive amounts of data has become an important issue that enterprises need to solve urgently.

[0003] Data-intensive industries, such as finance, healthcare, and manufacturing, have particularly strict requirements for metadata management. These industries not only require accurate data lineage tracing to clearly understand the source and destination of data, but also need to achieve real-time data updates to keep the data accurate and up-to-date. At the same time, accurate metadata supplementation is also an important basis for supporting business decisions, ensuring that all types of data can be effectively understood and used.

[0004] However, traditional manual management methods are difficult to cope with these complex requirements. Manual operations are not only time-consuming and labor-intensive, but also prone to errors, resulting in incomplete and inaccurate metadata, further affecting the credibility and use value of the data. In addition, manual management is difficult to achieve real-time updates and automated processing, and cannot adapt to the dynamically changing data environment, making data governance work passive. Summary of the invention

[0005] In view of the above problems, the present disclosure provides a metadata processing method and system, which can realize the automation of metadata management, improve data management efficiency and processing accuracy, significantly reduce the need for manual intervention, reduce the probability of human errors, and realize data source tracing, data flow analysis, impact assessment and other functions. At the same time, the method has good scalability and is suitable for different types of metadata processing scenarios.

[0006] According to a first aspect of the present disclosure, a metadata processing method is provided, including: obtaining metadata to be processed; inputting the metadata to be processed into a first neural network model, the first neural network model is obtained by training multiple sets of training metadata, the multiple sets of training metadata include metadata entry features, metadata lineage features and metadata missing features; obtaining a first output result of the first neural network model, the first output result includes a metadata asset catalog, a data lineage relationship diagram and a data update list; and monitoring the output process of the first output result.

[0007] According to the embodiments of the present disclosure, metadata storage features include data structure, data type and data storage location; metadata lineage features include metadata source and data flow path; metadata missing features include data creation time and data modification time.

[0008] According to an embodiment of the present disclosure, based on the first neural network model, a metadata asset catalog of the metadata to be processed is obtained, including: preprocessing the metadata to be processed; using the first neural network model to extract the warehousing features of the metadata to be processed; merging the warehousing features of the metadata to be processed with a preset metadata structure to generate a metadata asset catalog of the metadata to be processed to complete the warehousing of the metadata to be processed.

[0009] According to an embodiment of the present disclosure, based on the first neural network model, a data lineage relationship graph is obtained, including: extracting features from the metadata to be processed that is stored in the database to generate an input feature vector in the same format as that during training; inputting the preprocessed feature vector into the trained first neural network model to obtain the nodes and edges of the metadata to be processed; and constructing a data lineage relationship graph based on the nodes and edges.

[0010] According to an embodiment of the present disclosure, based on the first neural network model, a data update list is obtained, including: using the trained first neural network model to infer the metadata to be processed that has been stored, identifying and predicting missing metadata and duplicate metadata; generating an update list containing missing metadata and an update list containing duplicate metadata.

[0011] According to an embodiment of the present disclosure, the metadata asset catalog includes metadata description information, metadata classification information and metadata management information, wherein; the metadata description information includes the metadata name, metadata type and metadata size; the metadata classification information includes the classification level of the metadata and the business domain to which it belongs; the metadata management information includes the storage location, access rights and life cycle of the metadata.

[0012] According to an embodiment of the present disclosure, monitoring the output process of the first output result includes monitoring the process of outputting the data lineage relationship diagram, including: monitoring the source of the metadata to be processed; monitoring the flow path and processing method of the metadata to be processed at each node; and monitoring the final storage location and status of the metadata to be processed.

[0013] According to an embodiment of the present disclosure, monitoring the output process of the first output result includes monitoring the process of outputting the data update list, including: monitoring the quality of the metadata to be processed to regularly check the integrity and consistency of the metadata; monitoring the changes of the metadata to be processed to record the change history of the metadata to be processed.

[0014] A second aspect of the present disclosure provides a metadata processing system, including: a metadata acquisition module, used to acquire metadata to be processed; a metadata input module, used to input the metadata to be processed into a first neural network model, the first neural network model is obtained by training multiple sets of training metadata, the multiple sets of training metadata include metadata storage features, metadata lineage features and metadata missing features; a result acquisition module, used to obtain a first output result of the first neural network model, the first output result includes a metadata asset catalog, a data lineage relationship diagram and a data update list; a monitoring module, used to monitor the output process of the first output result.

[0015] According to an embodiment of the present disclosure, the result acquisition module further includes an asset catalog acquisition unit, a data lineage relationship graph acquisition unit and a data update list acquisition unit. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0017] Figure 1 A flowchart of a metadata processing method according to an embodiment of the present disclosure is schematically shown;

[0018] Figure 2 The framework diagram of the metadata processing system according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0019] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0020] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0021] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0022] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0023] Some block diagrams and / or flow charts are shown in the accompanying drawings. It should be understood that some blocks or combinations thereof in the block diagrams and / or flow charts can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that these instructions can create a device for implementing the functions / operations described in these block diagrams and / or flow charts when executed by the processor. The technology of the present disclosure can be implemented in the form of hardware and / or software (including firmware, microcode, etc.). In addition, the technology of the present disclosure can take the form of a computer-readable storage medium storing instructions, on which executable instructions are stored, which, when executed by the processor, causes the processor to perform the method of the present disclosure.

[0024] In the technical solution of the present disclosure, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0025] An embodiment of the present disclosure provides a metadata processing method, including: obtaining metadata to be processed; inputting the metadata to be processed into a first neural network model, the first neural network model is obtained by training multiple sets of training metadata, the multiple sets of training metadata include metadata entry features, metadata lineage features and metadata missing features; obtaining a first output result of the first neural network model, the first output result includes a metadata asset catalog, a data lineage relationship diagram and a data update list; and monitoring the output process of the first output result.

[0026] Through the embodiments of the present disclosure, the method can realize the automation of metadata management, improve data management efficiency, significantly reduce the need for manual intervention, and reduce the probability of human error; through the constructed data lineage relationship diagram, it can realize data source tracing, data flow analysis, impact assessment and other functions, helping enterprises optimize data management and decision-making; significantly improve metadata processing accuracy and optimize data management processes. At the same time, the method has good scalability and is suitable for different types of metadata processing scenarios.

[0027] Figure 1 The flowchart of the metadata processing method according to the embodiment of the present disclosure is schematically shown.

[0028] like Figure 1 As shown, in the metadata processing method in the embodiment of the present disclosure, the process may further include S110~S140:

[0029] In operation S110 , metadata to be processed is acquired.

[0030] Collect the required metadata to be processed from different data sources (such as databases, file systems, APIs, etc.). The metadata to be processed can be a large amount of unused data accumulated in enterprise data management, and can be different types of data environments, such as structured data and semi-structured data.

[0031] In operation S120, the metadata to be processed is input into the first neural network model. The first neural network model is obtained by training multiple sets of training metadata. The multiple sets of training metadata include metadata storage features, metadata lineage features, and metadata missing features.

[0032] In operation S130, a first output result of the first neural network model is obtained, and the first output result includes a metadata asset catalog, a data lineage relationship diagram, and a data update list.

[0033] Specifically, after obtaining the metadata to be processed, the metadata storage processing is completed based on the trained first neural network model. It should be noted that when performing metadata storage processing, the first neural network model can be obtained through multiple sets of training metadata such as metadata storage feature training.

[0034] In an embodiment of the present disclosure, based on the trained first neural network model, a metadata asset catalog of the metadata to be processed is obtained to complete the warehousing of the metadata to be processed. The specific process includes: preprocessing the metadata to be processed; extracting the warehousing features of the metadata to be processed by using the first neural network model; merging the warehousing features of the metadata to be processed with the preset metadata structure to generate a metadata asset catalog of the metadata to be processed to complete the warehousing of the metadata to be processed.

[0035] Exemplarily, the metadata to be processed is cleaned to remove redundant information and noise, such as null values ​​and duplicate records, and data in different formats are converted to a unified format standard. For example, the metadata in the data is automatically scanned to check whether it meets the specified standards. If metadata that does not meet the requirements or metadata is missing, a reminder must be issued, and the subsequent process will be automated when it meets the preset rules. During the parsing and matching process, it is necessary to conduct a detailed check of each metadata, including whether its format is correct, whether the content meets the requirements, and whether there are any missing key fields. This ensures that all data entered into the database meets the pre-set standards and improves data quality and consistency. When it is confirmed that all metadata meet the standards, these metadata are then established as a metadata asset catalog based on the neural network model and saved to the specified storage location.

[0036] Furthermore, the preprocessed metadata is input into the first neural network model. After training, the neural network model can automatically learn and extract useful warehousing features, where the metadata warehousing features may include data structure, data type and data storage location, and generate a warehousing feature vector. Then, according to the preset metadata structure (such as data table structure, field definition, etc.), the extracted warehousing feature vector is expanded, the corresponding fields are filled, and a metadata asset catalog that conforms to the preset structure is generated, which contains all necessary metadata attributes, such as metadata name, description, creation time, update time, etc. Finally, the generated metadata asset catalog is stored in a database or data warehouse, completing the warehousing of the metadata to be processed. At the same time, corresponding logs can be generated as needed to record important information and abnormal situations during the warehousing process.

[0037] It should be noted that, in the embodiments of the present disclosure, the metadata asset catalog also includes metadata description information, metadata classification information and metadata management information, wherein; the metadata description information includes the metadata name, metadata type and metadata size; the metadata classification information includes the metadata classification level and the business domain to which it belongs; the metadata management information includes the storage location, access rights and life cycle of the metadata.

[0038] It should be noted that, in the embodiments of the present disclosure, the first neural network model can be a convolutional neural network or a recurrent neural network.

[0039] Through the embodiments of the present disclosure, a metadata asset catalog of metadata to be processed can be generated efficiently and accurately, improving data management efficiency. The establishment of a metadata asset catalog not only includes simple storage of metadata, but also classification and tag management of metadata to facilitate subsequent data query and analysis. By utilizing the powerful feature extraction capability of the first neural network model, the need for manual intervention is significantly reduced, reducing the probability of human error.

[0040] In an embodiment of the present disclosure, after the metadata is stored in the warehouse, a data lineage relationship diagram is also obtained based on the first neural network model, which specifically includes: extracting features from the metadata to be processed that has been stored in the warehouse to generate an input feature vector in the same format as that during training; inputting the preprocessed feature vector into the trained first neural network model to obtain the nodes and edges of the metadata to be processed; and constructing a data lineage relationship diagram based on the nodes and edges.

[0041] Specifically, firstly, the metadata to be processed is extracted from the database, including field name, data type, data source and other information. The metadata to be processed is cleaned and standardized to remove redundant and erroneous data to ensure data quality. The feature extraction algorithm is used to extract features from the metadata to be processed in the database. The extracted features are converted into input feature vectors in the same format as those used during training. The preprocessed feature vectors are input into the trained first neural network model.

[0042] It should be noted that when constructing the data lineage relationship graph, the first neural network model can be obtained by training multiple sets of training metadata such as metadata lineage features, where the metadata lineage features include metadata sources and data flow paths.

[0043] Furthermore, the neural network model performs forward propagation based on the input features to generate nodes and edges of the metadata to be processed. For example, in the output layer of the neural network, the relationship between the metadata is identified and marked as nodes and edges. Among them, the node represents the metadata entity, and the edge represents the dependency or association relationship between the metadata. Based on the identified nodes and edges, a data lineage relationship diagram is constructed using a graph database or graphical tools.

[0044] In this way, you can clearly understand the complete path of each data from the source table to the target table. The data lineage relationship diagram records in detail the time, operator, and specific operation content of the data operation, and shows the flow of data between different tables and columns. These relationship diagrams can not only help data administrators quickly understand the flow path of data, but also be used for data auditing and problem tracking. When data anomalies or errors occur, the data lineage relationship diagram can quickly locate the source of the problem and make corrections.

[0045] Through the embodiments of the present disclosure, by constructing a data lineage relationship diagram, functions such as data source tracing, data flow analysis, and impact assessment can be implemented to help enterprises optimize data management and decision-making.

[0046] In an embodiment of the present disclosure, based on the first neural network model, a data update list can also be obtained, specifically including: using the trained first neural network model to infer the metadata to be processed that has been stored, identifying and predicting missing metadata and duplicate metadata; generating an update list containing missing metadata and an update list of duplicate metadata.

[0047] Specifically, when obtaining the data update list, the first neural network model can be trained through multiple sets of training metadata, such as metadata missing features, where the metadata missing features include data creation time and data modification time. The metadata to be processed is input into the trained first neural network model. The model infers each metadata through the forward propagation algorithm and outputs its missing and repeated prediction results.

[0048] After the model inference is completed, the data processing module will perform the following processing based on the model output: 1) Missing metadata update list: identify all missing metadata and generate an update list containing this information; 2) Duplicate metadata update list: identify all duplicate metadata and generate an update list containing this information. Finally, the generated missing metadata update list and duplicate metadata update list are displayed or saved to a specified storage location for subsequent data maintenance and management.

[0049] Exemplarily, based on the missing metadata update list, when data without specific metadata is found, corresponding supplementary information can be automatically generated according to the preset rule system to ensure that all data has complete metadata information. For example, for data with missing creation time and modification time, this information can be filled in through inference and calculation; for data with missing creators and modifiers, it can be supplemented and improved through operation logs and other auxiliary information, so as to ensure that all data in the database has complete and accurate metadata information.

[0050] Exemplarily, based on the duplicate metadata update list, duplicate or outdated metadata information is deleted and updated.

[0051] Through the embodiments of the present disclosure, the processing efficiency and accuracy of metadata can be significantly improved, manual intervention can be reduced, and the data management process can be optimized. At the same time, the method has good scalability and is applicable to different types of metadata processing scenarios.

[0052] In operation S140, an output process of the first output result is monitored.

[0053] In an embodiment of the present disclosure, monitoring the output process of the first output result includes monitoring the process of outputting the data lineage relationship diagram, including: monitoring the source of the metadata to be processed; monitoring the flow path and processing method of the metadata to be processed at each node; and monitoring the final storage location and status of the metadata to be processed.

[0054] Specifically, the process of outputting the data lineage relationship diagram is monitored. For example, the processing process of metadata at each node can be recorded, and the workflow engine can be used to track the state changes of data in each processing link (such as cleaning, conversion, and loading). A flow path diagram is generated to show the complete path of metadata from the source to the final storage.

[0055] Furthermore, the storage location and status of the metadata can be checked regularly to ensure the security and integrity of the data.

[0056] Furthermore, visual monitoring can be performed. For example, an interface is provided to display metadata lineage diagrams, allowing users to browse interactively to quickly locate the source of the problem and data flow.

[0057] By monitoring the data lineage, it is ensured that it always reflects the latest status of the database. Whenever a new data operation occurs, the corresponding lineage diagram will be automatically updated to ensure the real-time and accuracy of the data lineage. This automated data lineage tracing can not only greatly reduce the manual workload, but also significantly improve the efficiency and accuracy of data management. By establishing automated data lineage tracing, enterprises can better understand the flow of data and provide strong support for data management and analysis. This not only helps to improve data quality, but also improves the traceability and transparency of data, providing guarantees for corporate compliance and risk management.

[0058] Through the embodiments of the present disclosure, the source, flow path and final storage status of metadata to be processed are monitored in real time, which enhances data traceability and optimizes the data management process.

[0059] In an embodiment of the present disclosure, monitoring the output process of the first output result includes monitoring the process of outputting the data update list, including: monitoring the quality of the metadata to be processed to regularly check the integrity and consistency of the metadata; monitoring the changes of the metadata to be processed to record the change history of the metadata to be processed.

[0060] Specifically, the process of outputting data update lists is monitored, such as defining metadata quality specifications, including completeness, consistency, accuracy and other indicators; regularly running quality inspection tasks, automatically identifying and marking problematic metadata through data cleaning and verification rules. When metadata is updated, change records are automatically generated and stored in the change history database. When metadata quality issues or abnormal changes are detected, the system automatically triggers an alarm to notify processing.

[0061] Through the embodiments of the present disclosure, the consistency and integrity of data are ensured by real-time monitoring of the quality and change history of metadata. Such a system can effectively support enterprises to achieve efficient data management and flexible dynamic adjustment in a complex data environment.

[0062] According to the above metadata processing method, the present disclosure also provides a metadata processing system.

[0063] like Figure 2 As shown, Figure 2 The framework diagram of the metadata processing system according to an embodiment of the present disclosure is schematically shown.

[0064] like Figure 2 As shown, the metadata processing system 200 of this embodiment includes a metadata acquisition module 210 , a metadata input module 220 , a result acquisition module 230 and a monitoring module 240 .

[0065] The metadata acquisition module 210 is used to acquire metadata to be processed. In one embodiment, the metadata acquisition module 210 can be used to perform the operation S110 described above, which will not be described in detail here.

[0066] The metadata input module 220 is used to input the metadata to be processed into the first neural network model, and the first neural network model is obtained by training multiple sets of training metadata, and the multiple sets of training metadata include metadata storage features, metadata lineage features, and metadata missing features. In one embodiment, the metadata input module 220 can be used to perform the operation S120 described above, which will not be repeated here.

[0067] The result acquisition module 230 is used to obtain the first output result of the first neural network model, and the first output result includes a metadata asset directory, a data lineage relationship diagram, and a data update list. In one embodiment, the result acquisition module 230 can be used to perform the operation S130 described above, which will not be repeated here.

[0068] The monitoring module 240 is used to monitor the output process of the first output result. In one embodiment, the result acquisition module 240 can be used to perform the operation S140 described above, which will not be described in detail here.

[0069] In the embodiment of the present disclosure, the result acquisition module 230 further includes an asset catalog acquisition unit, a data lineage relationship graph acquisition unit, and a data update list acquisition unit.

[0070] Through the embodiments of the present disclosure, the system can improve data management efficiency, significantly reduce the need for manual intervention, reduce the probability of human error, improve metadata processing accuracy and optimize data management processes.

[0071] The flow chart and block diagram in the accompanying drawings illustrate the system architecture, functions and operations that may be implemented according to the system and method of various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0072] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0073] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A metadata processing method, characterized in that: The method comprises: Get metadata to be processed; Inputting the metadata to be processed into a first neural network model, wherein the first neural network model is obtained by training multiple sets of training metadata, wherein the multiple sets of training metadata include metadata storage features, metadata lineage features, and metadata missing features; Obtaining a first output result of the first neural network model, the first output result comprising a metadata asset catalog, a data lineage relationship diagram, and a data update list; The output process of the first output result is monitored.

2. The metadata processing method according to claim 1, characterized in that: The metadata storage characteristics include data structure, data type and data storage location; The metadata lineage characteristics include metadata source and data flow path; The metadata missing features include data creation time and data modification time.

3. The metadata processing method according to claim 1, characterized in that: Based on the first neural network model, obtaining a metadata asset catalog of the metadata to be processed includes: Preprocessing the metadata to be processed; Extracting the storage features of the metadata to be processed by using the first neural network model; The storage characteristics of the metadata to be processed are combined with a preset metadata structure to generate a metadata asset catalog of the metadata to be processed to complete the storage of the metadata to be processed.

4. The metadata processing method according to claim 1 or 3, characterized in that: Based on the first neural network model, a data blood relationship diagram is obtained, including: Extract features from the metadata to be processed and generate input feature vectors in the same format as that used during training; Inputting the preprocessed feature vector into the trained first neural network model to obtain the nodes and edges of the metadata to be processed; Based on the nodes and edges, a data lineage graph is constructed.

5. The metadata processing method according to claim 1 or 3, characterized in that: Based on the first neural network model, obtaining a data update list includes: The trained first neural network model is used to infer the metadata to be processed in the database, and to identify and predict missing metadata and duplicate metadata; Generates an update list containing missing metadata and an update list of duplicate metadata.

6. The metadata processing method according to claim 1, characterized in that: The metadata asset catalog includes metadata description information, metadata classification information and metadata management information, wherein; The metadata description information includes metadata name, metadata type and metadata size; The metadata classification information includes the classification level of the metadata and the business domain to which it belongs; The metadata management information includes the storage location, access rights, and life cycle of the metadata.

7. The metadata processing method according to claim 4, characterized in that: The monitoring of the output process of the first output result includes monitoring the process of outputting the data blood relationship diagram, including: Monitoring the source of the metadata to be processed; Monitoring the flow path and processing method of the metadata to be processed at each node; The final storage location and status of the metadata to be processed are monitored.

8. The metadata processing method according to claim 5, characterized in that: The monitoring of the output process of the first output result includes monitoring the process of updating the output data list, including: Monitoring the quality of the metadata to be processed to regularly check the integrity and consistency of the metadata; The changes of the metadata to be processed are monitored to record the change history of the metadata to be processed.

9. A metadata processing system, characterized in that: The system comprises: Metadata acquisition module, used to acquire metadata to be processed; A metadata input module, used to input the metadata to be processed into a first neural network model, wherein the first neural network model is obtained by training multiple sets of training metadata, wherein the multiple sets of training metadata include metadata storage features, metadata lineage features, and metadata missing features; A result acquisition module, used to obtain a first output result of the first neural network model, wherein the first output result includes a metadata asset catalog, a data lineage relationship diagram, and a data update list; A monitoring module is used to monitor the output process of the first output result.

10. The metadata processing system according to claim 9, characterized in that: The result acquisition module also includes an asset catalog acquisition unit, a data lineage relationship diagram acquisition unit and a data update list acquisition unit.

Images