Metadata management method, data system and electronic equipment
By collecting and storing metadata in real time in the data warehouse and correcting stored metadata according to preset cycles, the omissions and inconsistencies in metadata management in the existing technology are solved, and more efficient and accurate metadata management is achieved , improves the normal operation and user experience of the data warehouse.
Patent Information
- Application Number
- CN202410485148.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-22
- Publication Date
- 2024-11-12
AI Technical Summary
The existing metadata management methods have missing metadata, and the stored metadata may be inconsistent and incomplete with the metadata in the data warehouse, affecting the management and use of data in the data warehouse.
Provide a metadata management method, including a metadata acquisition module, a metadata storage module and a metadata correction module, collect metadata generated by data tasks in the data warehouse in real time or in time, and perform stored metadata according to preset periods. Correct processing to ensure the integrity and consistency of metadata.
It effectively improves the integrity and consistency of metadata collection, ensures the normal management and use of data in the data warehouse, improves the efficiency and accuracy of metadata query, and enhances the user experience.
Smart Images

Figure CN118939666A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data management, and in particular to a metadata management method, a data system and an electronic device. Background Art
[0002] With the development of science and technology, the use of data is becoming more and more frequent, and a large amount of data will be stored in data warehouses (such as Hive data warehouses) to facilitate data query and analysis. In the process of using these data warehouses to perform data tasks (such as data query, merging, splitting, etc.), a large amount of metadata about the data will be generated. Metadata (Metadata) is mainly used to describe the content, attributes or characteristics of data, and is used to support data management and use. The management of metadata is usually implemented through a metadata management system. The metadata management system is mainly responsible for managing and displaying various metadata entities in the data warehouse, such as databases (DataBase), tables (Hive Table), fields (HiveColumn), table storage (Hive Table Storage), and the relationship between them (lineage graph), etc.
[0003] In the prior art, one way to collect metadata corresponding to data in a data warehouse is for users to manually collect and store it when needed. This method not only wastes manpower and has low collection efficiency, but also causes metadata omissions, affecting the management and use of data in the data warehouse. Another method is that the metadata management system sets a collection time point and only collects and stores metadata generated at a specific time corresponding to the collection time point. In this method, since metadata outside the set collection time point is not collected, a large amount of metadata is lost, which also affects the management and use of data in the data warehouse. In addition, the metadata in the data warehouse often changes (for example, increases), but the stored metadata generally does not change. Therefore, there will be a problem that the stored metadata is inconsistent and incomplete with the metadata in the data warehouse, which will also affect the management and use of data in the data warehouse.
[0004] In summary, the existing metadata management methods have the problem of missing metadata, and the stored metadata may be inconsistent and incomplete compared with the metadata in the data warehouse, which affects the management and use of data in the data warehouse. Summary of the invention
[0005] The purpose of the present invention is to solve the problem that the existing metadata management method has missing metadata, and the stored metadata may be inconsistent and incomplete compared with the metadata in the data warehouse, which affects the management and use of data in the data warehouse. The present invention provides a metadata management method, a data system and an electronic device, which can collect and store metadata generated in the process of executing data tasks based on data in the data warehouse in real time or in a timely manner, and timely correct the stored metadata, effectively improving the integrity and consistency of metadata collection, so as to better ensure the normal management and use of data in the data warehouse.
[0006] To solve the above technical problems, an embodiment of the present invention discloses a metadata management method, which is applied to a data system including a data warehouse and a metadata management system, wherein the metadata management system includes a metadata collection module, a metadata storage module and a metadata correction module, and the method includes: when the data warehouse performs a data task based on data in the data warehouse and generates metadata corresponding to the data, the data warehouse stores the metadata and sends a metadata collection instruction to the metadata collection module in the metadata management system, wherein the metadata collection instruction includes identification information of the data task; in response to receiving the metadata collection instruction, the metadata collection module collects metadata corresponding to the data task according to the identification information in the metadata collection instruction, and sends the metadata to the metadata storage module; the metadata storage module stores the metadata; and the metadata correction module corrects the metadata stored in the metadata storage module based on the metadata stored in the data warehouse according to a preset period.
[0007] By adopting the above technical solution, when the data warehouse performs data tasks based on the data in the data warehouse and generates metadata corresponding to the data, it will send a metadata collection instruction including identification information corresponding to the data task to the metadata storage module in real time. In response to the received metadata collection instruction, the metadata collection module can timely collect metadata corresponding to the data task and send it to the metadata storage module for storage. Therefore, as long as the data warehouse performs data tasks based on the data in the data warehouse, the metadata collection module can timely collect metadata corresponding to the data generated by the data warehouse and send it to the metadata storage module for storage. Compared with the manual collection method or the metadata management system setting time point for metadata collection in the prior art, the integrity of metadata collection is effectively guaranteed and metadata omission will not be caused. In addition, the metadata correction module can correct the metadata stored in the metadata storage module based on the metadata stored in the data warehouse according to the preset period, and can timely correct the stored metadata, further ensuring the consistency and integrity of the stored metadata and the metadata in the data warehouse, so as to better ensure the normal management and use of data in the data warehouse.
[0008] According to another specific embodiment of the present invention, an embodiment of the present invention discloses a metadata management method, wherein a metadata storage module includes a first database and a second database, the first database is a graph database, and the second database is a relational database, the metadata management system also includes a storage middleware and a metadata determination module, and the metadata acquisition module sends the metadata to the metadata storage module, including: the metadata acquisition module determines the entity information and identification information of the metadata, sends the entity information to the first database for storage, and sends the identification information to the storage middleware; the metadata determination module obtains the identification information in the storage middleware, and when it is determined that the entity information exists in the first database according to the identification information, obtains the entity information from the first database, and sends the entity information to the second database for storage.
[0009] By adopting the above technical solution, in the process of metadata storage, the entity information and identification information of the metadata are first determined by the metadata acquisition module, and then the entity information is sent to the first database (i.e., the graph database) for storage, and the identification information is sent to the storage middleware; the metadata determination module obtains the identification information in the storage middleware, and when it is determined that the entity information exists in the first database according to the identification information, the entity information is sent to the second database for storage. Thus, the entity information stored in the graph database is further verified according to the identification information of the metadata, which can effectively screen whether the data is missing, ensure the integrity of the metadata storage, and will not cause metadata omissions. Furthermore, the metadata entity information is stored in the graph database and the relational database respectively, so that subsequent queries can be queried not only from the graph database, but also from the relational database, which effectively solves the problem of weak parallel query capabilities caused by only using the graph database to store metadata, and improves the efficiency of metadata queries.
[0010] According to another specific embodiment of the present invention, an embodiment of the present invention discloses a metadata management method, which also includes: when the metadata determination module determines that entity information does not exist in the first database based on identification information, the record information of the entity information not existing in the first database is stored in the information exception table corresponding to the second database.
[0011] By adopting the above technical solution, when the metadata determination module determines that the entity information does not exist in the first database based on the identification information, the record of the entity information not existing in the first database is stored in the information exception table corresponding to the second database. This allows the module to obtain the record of the entity information not existing in the first database from the information exception table each time the identification information is subsequently read from the message middleware, and continuously query the first database for the entity information corresponding to the record according to the record until the corresponding entity information is found. This can effectively ensure the integrity of the metadata storage and prevent metadata omissions.
[0012] According to another specific implementation of the present invention, an implementation of the present invention discloses a metadata management method, wherein the identification information includes identifier information and type information corresponding to the metadata.
[0013] By adopting the above technical solution, the entity information corresponding to the metadata can be determined more accurately through the identifier information and type information corresponding to the metadata.
[0014] According to another specific embodiment of the present invention, an embodiment of the present invention discloses a metadata management method, in which a metadata correction module corrects metadata stored in a metadata storage module based on metadata stored in a data warehouse according to a preset period, including: the metadata correction module corrects entity information of metadata stored in a first database based on metadata stored in a data warehouse according to a preset period to obtain corrected first entity information, and corrects entity information of metadata stored in a second database based on the corrected first entity information to obtain corrected second entity information.
[0015] By adopting the above technical solution, the metadata correction module can correct the entity information of the metadata stored in the first database based on the metadata stored in the data warehouse to obtain the corrected first entity information, and correct the entity information of the metadata stored in the second database based on the corrected first entity information to obtain the corrected second entity information. That is, the metadata correction module can compare the metadata information stored by the three parties and correct the metadata information in the first database and the second database that is inconsistent with the data warehouse, thereby ensuring the consistency and integrity of the stored metadata and the metadata in the data warehouse and improving the accuracy of metadata query.
[0016] According to another specific embodiment of the present invention, an embodiment of the present invention discloses a metadata management method, in which a metadata correction module corrects entity information of metadata stored in a first database based on metadata stored in a data warehouse to obtain corrected first entity information, including: the metadata correction module obtains first information of metadata stored in the data warehouse, and compares it with second information of entity information of metadata stored in the first database, and when it is determined that metadata that does not belong to the first database exists in the data warehouse according to the first information and the second information, the entity information of metadata that does not belong to the first database in the data warehouse is added to the first database to obtain the corrected first entity information, the first information includes organization and structure information of the metadata stored in the data warehouse, and the second information includes organization and structure information of the entity information.
[0017] By adopting the above technical solution, by comparing the organization and structure information of the metadata stored in the data warehouse with the organization and structure information of the entity information of the metadata stored in the first database, it is possible to accurately determine whether there is metadata in the data warehouse that does not belong to the first database. Therefore, the entity information of the metadata stored in the first database can be accurately corrected according to the metadata information in the data warehouse to obtain the corrected first entity information, so that the metadata in the first database is more accurate.
[0018] According to another specific embodiment of the present invention, an embodiment of the present invention discloses a metadata management method, in which a metadata correction module corrects entity information of metadata stored in a second database based on the corrected first entity information to obtain corrected second entity information, including: the metadata correction module determines third information of the corrected first entity information, and compares it with fourth information corresponding to the entity information of metadata stored in the second database, and when it is determined that entity information of metadata that does not belong to the second database exists in the first database according to the third information and the fourth information, the entity information of metadata that does not belong to the second database in the first database is added to the second database to obtain the corrected second entity information, the third information includes identification information corresponding to the corrected first entity information, and the fourth information includes identification information corresponding to the entity information of metadata stored in the second database.
[0019] By adopting the above technical solution, by comparing the identification information of the corrected first entity information with the identification information corresponding to the entity information of the metadata stored in the second database, it is possible to accurately determine whether there is entity information of the metadata that does not belong to the second database in the first database. Therefore, the entity information of the metadata stored in the second database can be accurately corrected according to the corrected first entity information to obtain the corrected second entity information, so that the metadata in the second database is more accurate.
[0020] According to another specific embodiment of the present invention, an embodiment of the present invention discloses a metadata management method, wherein the metadata management system also includes a query module, and the method also includes: the query module responds to receiving a first operation from the user, and displays the metadata lineage graph corresponding to the entity information of the metadata stored in the second database through a first interface corresponding to the query module, and the first operation includes the user's query operation on the metadata lineage graph of the target time period.
[0021] By adopting the above technical solution, the query module can respond to the user's query operation on the metadata lineage graph of a set time period, and display the metadata lineage graph corresponding to the entity information of the metadata stored in the second database on the first interface corresponding to the query module. As a result, the metadata lineage graph of a specific time period can be displayed to the user based on the time period input by the user, thereby improving the user experience.
[0022] According to another specific embodiment of the present invention, an embodiment of the present invention discloses a metadata management method, which also includes: the query module modifies the metadata lineage graph in response to receiving a second operation from the user, and the second operation includes a filtering operation and / or a dragging operation of the metadata lineage graph by the user through the first interface.
[0023] By adopting the above technical solution, the query module can modify the metadata lineage graph in response to the user's screening operation and / or dragging operation on the metadata lineage graph through the first interface. That is, the user can operate the displayed metadata lineage graph through the first interface, which effectively improves the user experience.
[0024] According to another specific embodiment of the present invention, an embodiment of the present invention discloses a metadata management method, which also includes: the query module displays code information corresponding to the target blood relationship in response to receiving a third operation from the user, the third operation includes the user selecting a target node corresponding to the metadata blood relationship through the first interface, and the target node corresponds to the target blood relationship.
[0025] By adopting the above technical solution, the query module can display the code information corresponding to the target blood relationship in response to the user's selection operation on the target node corresponding to the target blood relationship in the metadata blood relationship graph through the first interface, provide the query capability of specific codes, and effectively improve the user experience.
[0026] An embodiment of the present invention further discloses a data system, including a data warehouse and a metadata management system, wherein the metadata management system includes a metadata collection module, a metadata storage module and a metadata correction module, wherein the data warehouse is used to store metadata when a data task is executed based on data in the data warehouse and metadata corresponding to the data is generated, and to send a metadata collection instruction to the metadata collection module in the metadata management system, wherein the metadata collection instruction includes identification information of the data task; the metadata collection module is used to respond to receiving the metadata collection instruction, collect metadata corresponding to the data task according to the identification information in the metadata collection instruction, and send the metadata to the metadata storage module; the metadata storage module is used to store metadata; and the metadata correction module is used to correct the metadata stored in the metadata storage module based on the metadata stored in the data warehouse according to a preset period.
[0027] By adopting the above technical solution, when the data warehouse in the data system performs data tasks based on the data in the data warehouse and generates metadata corresponding to the data, it will send a metadata collection instruction including identification information corresponding to the data task to the metadata storage module in real time. In response to the received metadata collection instruction, the metadata collection module can timely collect metadata corresponding to the data task and send it to the metadata storage module for storage. Therefore, as long as the data warehouse performs data tasks based on the data in the data warehouse, the metadata collection module can timely collect metadata corresponding to the data generated by the data warehouse and send it to the metadata storage module for storage. Compared with the manual collection method or the metadata management system setting time point for metadata collection in the prior art, the integrity of metadata collection is effectively guaranteed, and no metadata omission will be caused. In addition, the metadata correction module can correct the metadata stored in the metadata storage module based on the metadata stored in the data warehouse according to the preset period, further ensuring the consistency and integrity of the stored metadata with the metadata in the data warehouse, thereby ensuring the normal management and use of data in the data warehouse.
[0028] An embodiment of the present invention further discloses an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the electronic device implements the aforementioned metadata management method.
[0029] The embodiment of the present invention further discloses a computer-readable storage medium, which stores a computer program. The computer program includes program instructions, and the program instructions are executed by an electronic device to execute the aforementioned metadata management method.
[0030] The embodiment of the present invention further discloses a computer program product, including a computer program / instruction, and the computer program / instruction implements the aforementioned metadata management method when executed by a processor.
[0031] Beneficial effects of this application:
[0032] The metadata management method provided by the present application is that when the data warehouse performs a data task based on the data in the data warehouse and generates metadata corresponding to the data, it will send a metadata collection instruction including identification information corresponding to the data task to the metadata storage module in real time. The metadata collection module can timely collect metadata corresponding to the data task and send it to the metadata storage module for storage in response to the received metadata collection instruction. Therefore, as long as the data warehouse performs a data task based on the data in the data warehouse, the metadata collection module can timely collect metadata corresponding to the data generated by the data warehouse and send it to the metadata storage module for storage. Compared with the manual collection method in the prior art or the method of metadata collection at a set time point by the metadata management system, the integrity of metadata collection is effectively guaranteed and metadata omission will not be caused. In addition, the metadata correction module can correct the metadata stored in the metadata storage module based on the metadata stored in the data warehouse according to the preset period, further ensuring the consistency and integrity of the stored metadata with the metadata in the data warehouse, thereby ensuring the normal management and use of data in the data warehouse.
[0033] Furthermore, based on the query module provided by the metadata management system, the metadata lineage map of a specific time period can be displayed to the user. Through the first interface, the user can operate the displayed metadata lineage map and display the code information corresponding to the target lineage relationship, providing the ability to query specific codes and effectively improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 is a schematic diagram showing the structure of a data system according to some implementation modes of the present application;
[0035] Figure 2 is a flowchart showing a metadata management method according to some implementation modes of the present application;
[0036] Figure 3 According to some implementations of the present application, a schematic diagram of a process of a metadata collection module sending metadata to a metadata storage module is shown;
[0037] Figure 4 is a flowchart showing another metadata management method according to some implementation modes of the present application;
[0038] Figure 5 According to some embodiments of the present application, a schematic diagram of a process in which a metadata storage module reads metadata information stored in a graph database and converts the metadata information stored in the graph data into relational data and stores it in a metadata repository is shown;
[0039] Figure 6is a schematic diagram showing a query module querying metadata information in parallel according to some implementation modes of the present application;
[0040] Fig. 7A is a schematic diagram showing metadata information displayed on a UI interface corresponding to a query module according to some implementation modes of the present application;
[0041] Figure 7B is another schematic diagram showing metadata information displayed on a UI interface corresponding to a query module according to some implementation modes of the present application;
[0042] Figures 8A-8B is a schematic diagram showing metadata detail information displayed on a UI interface corresponding to a query module according to some implementation modes of the present application;
[0043] Figure 8C-8D is another schematic diagram showing metadata detail information displayed on a UI interface corresponding to a query module according to some implementation modes of the present application;
[0044] Fig.9A is a schematic diagram showing a metadata lineage graph displayed on a UI interface corresponding to a query module according to some implementation modes of the present application;
[0045] Fig. 9B According to some embodiments of the present application, another schematic diagram of a metadata lineage graph displayed on a UI interface corresponding to a query module is shown;
[0046] Fig.10 A schematic diagram of the structure of an electronic device is shown according to some embodiments of the present application. DETAILED DESCRIPTION
[0047] As mentioned above, existing metadata management methods have the problem of missing metadata, and the stored metadata may be inconsistent and incomplete compared with the metadata in the data warehouse, which affects the management and use of data in the data warehouse.
[0048] Specifically, metadata is data that describes data, and can describe the organization, data domain, and relationship information of data. By mastering the metadata of data, the management and use of data in the data warehouse can be effectively guaranteed. For example, the description of information about the location of data resources in metadata, such as data storage location, path, etc., can help users quickly find data resources; or the metadata description of data can enable users to have an intuitive understanding of the data without browsing the data object; or the description of the version, management, and use rights of the data object through metadata can facilitate the management and use of information objects.
[0049] At present, the main pain points of metadata management systems based on data warehouses (for example, Hive data warehouses. Hive is a data warehouse (DataAarehouse, DW for short) based on Hadoop, which can map structured data files into a database table and provide Structured Query Language (SQL) query functions) include: as mentioned above, it is impossible to collect metadata change information in real time, and most of the collection is done by scheduled collection or manual task collection; graph databases are generally used for storage, and the parallel query capability is weak and the query performance is poor; users cannot perform various filtering, dragging and other operations on the bloodline map displayed on the front-end display page of the bloodline map; the front-end display page of the bloodline map displays only one bloodline map, and it is impossible to query the bloodline map within a specific time period; and it is impossible to query the specific code that generates the bloodline relationship.
[0050] In response to the above problems, the present application provides a metadata management method. When the data warehouse executes data tasks based on the data in the data warehouse and generates metadata corresponding to the data, the metadata collection module in the metadata management system can collect the metadata corresponding to the data tasks in real time or in a timely manner and send it to the metadata storage module for storage, which effectively ensures the integrity of metadata collection and does not cause metadata omissions. In addition, the metadata correction module can correct the metadata stored in the metadata storage module based on the metadata stored in the data warehouse according to a preset period, ensuring the consistency and integrity of the stored metadata with the metadata in the data warehouse, thereby better ensuring the normal management and use of data in the data warehouse.
[0051] Next, the implementation process of the metadata management method provided by the present application is described in detail with reference to the accompanying drawings.
[0052] In one embodiment, the metadata management method provided by the present application is applied to Figure 1 The data system shown includes a data warehouse and a metadata management system, wherein the data warehouse can be specifically a Hive data warehouse, which can be referred to as a Hive data warehouse for short, and is used to store metadata when a data task is executed based on the data in the data warehouse and metadata corresponding to the data is generated, and to send a metadata collection instruction including identification information of the data task to a metadata collection module in the metadata management system. The data management system includes a metadata collection module, a metadata storage module and a metadata correction module. The metadata collection module is used to collect metadata corresponding to the data task according to the identification information in the metadata collection instruction in response to receiving the metadata collection instruction, and send the metadata to the metadata storage module. The metadata storage module is used to store metadata. The metadata correction module is used to correct the metadata stored in the metadata storage module based on the metadata stored in the data warehouse according to a preset period.
[0053] like Figure 2 As shown, the metadata management method includes the following steps:
[0054] S100: When the data warehouse executes a data task based on the data in the data warehouse and generates metadata corresponding to the data, the data warehouse stores the metadata and sends a metadata collection instruction to a metadata collection module in a metadata management system. The metadata collection instruction includes identification information of the data task.
[0055] The data tasks performed by the data warehouse based on the data in the data warehouse can specifically be data tasks such as merging, splitting, and querying data. In the process of executing data tasks, the data warehouse will generate metadata corresponding to the data tasks. After generating metadata, the data warehouse stores it and sends a metadata collection instruction including identification information corresponding to the data task to the metadata collection module. The data task here refers to the data task currently being executed.
[0056] S200: In response to receiving the metadata collection instruction, the metadata collection module collects metadata corresponding to the data task according to identification information in the metadata collection instruction, and sends the metadata to the metadata storage module.
[0057] The metadata collection module can find the metadata generated by the data tasks being executed in the data warehouse according to the identification information in the metadata collection instruction, and collect the metadata. In this way, the metadata generated in real time when the data warehouse executes data tasks based on the stored data can be collected, avoiding the omission of metadata.
[0058] The identification information of the data task is used to identify the currently executed data task, so that the metadata collection module can accurately identify the metadata corresponding to the currently executed data task through the identification information. The identification information can be determined according to the name of the executed data task, or it can be a determined string of character codes, etc.
[0059] S300: The metadata storage module stores metadata.
[0060] S400: The metadata correction module corrects the metadata stored in the metadata storage module based on the metadata stored in the data warehouse according to a preset period.
[0061] It should be noted that the correction processing of the metadata stored in the metadata storage module based on the metadata stored in the data warehouse according to the preset period recorded in step S400 can be performed every time the metadata storage module stores metadata, or it can be performed at a preset time point (for example, at 8 o'clock in the morning every day, or at fixed time intervals). The preset period can be set as needed, and the present application is not limited to executing step S400 after the metadata storage module stores metadata in step S300.
[0062] In one embodiment, if Figure 1 As shown, the metadata storage module includes a first database and a second database, the first database is a graph database, the second database is a relational database, and the metadata management system also includes a storage middleware and a metadata determination module. Figure 3 As shown, the metadata collection module sends the metadata to the metadata storage module, including the following steps:
[0063] S210: The metadata collection module determines entity information and identification information of the metadata, sends the entity information to the first database for storage, and sends the identification information to the storage middleware.
[0064] The storage middleware can be a high-throughput distributed publish-subscribe messaging system, Apache Kafka, which can process all action stream data of consumers on the website. It can also be other software systems that can perform message transfer and storage.
[0065] In one implementation, the identification information of the metadata includes identifier information and type information corresponding to the metadata.
[0066] S220: The metadata determination module obtains identification information in the storage middleware, and when it is determined according to the identification information that entity information exists in the first database, the process proceeds to step S230.
[0067] S230 acquires entity information from the first database, and sends the entity information to the second database for storage.
[0068] In one embodiment, if Figure 3 As shown, the method further includes: when the metadata determination module determines according to the identification information that there is no entity information in the first database, entering step S240.
[0069] S240: Storing the record information of entity information not existing in the first database into the information exception table corresponding to the second database.
[0070] In one embodiment, the metadata correction module corrects the metadata stored in the metadata storage module based on the metadata stored in the data warehouse according to a preset period, including: the metadata correction module corrects the entity information of the metadata stored in the first database based on the metadata stored in the data warehouse according to the preset period to obtain the corrected first entity information, and corrects the entity information of the metadata stored in the second database based on the corrected first entity information to obtain the corrected second entity information.
[0071] In one embodiment, a metadata correction module corrects entity information of metadata stored in a first database based on metadata stored in a data warehouse to obtain corrected first entity information, including: the metadata correction module obtains first information of metadata stored in the data warehouse, and compares it with second information of entity information of metadata stored in the first database; when it is determined that metadata that does not belong to the first database exists in the data warehouse according to the first information and the second information, the entity information of metadata that does not belong to the first database in the data warehouse is added to the first database to obtain the corrected first entity information, the first information includes organization and structure information of the metadata stored in the data warehouse, and the second information includes organization and structure information of the entity information.
[0072] The metadata correction module can use Hive SDK to read all metadata organization and structure information (i.e. Schema, which can include Hive libraries, tables, fields, etc.) from the data warehouse, and then query and compare the organization and structure information of the entity information of the metadata stored in the first database (i.e. graph database) one by one. For example, it can use the string of the type "library name@table name@field name" as an index to query and compare. If the metadata of the first database is missing, then add relevant metadata information to the first database. Since the metadata management system usually saves the metadata information deleted in Hive by default, there is generally no situation where the metadata exists in the first database but not in Hive.
[0073] The metadata correction module may also perform a comparative query of metadata through other information that can identify metadata.
[0074] In one embodiment, the metadata correction module corrects the entity information of the metadata stored in the second database based on the corrected first entity information to obtain the corrected second entity information, including: the metadata correction module determines third information of the corrected first entity information, and compares it with fourth information corresponding to the entity information of the metadata stored in the second database, and when it is determined that there is entity information of the metadata that does not belong to the second database in the first database according to the third information and the fourth information, the entity information of the metadata that does not belong to the second database in the first database is added to the second database to obtain the corrected second entity information, the third information includes identification information corresponding to the corrected first entity information, and the fourth information includes identification information corresponding to the entity information of the metadata stored in the second database.
[0075] The metadata correction module can query all relational data and entity information from the first database, and compare the identification information (such as identifier information and type information) corresponding to the entity information (i.e., the corrected first entity information) with the identification information of the entity information of the metadata stored in the second database one by one, that is, use the identification information (such as identifier information and type information) as an index to query and compare. If the metadata in the second database is missing, the relevant metadata information (metadata entity information) is added to the second database.
[0076] In one embodiment, if Figure 1 As shown, the metadata management system also includes a query module, and the method also includes: the query module responds to receiving a first operation from the user, and displays the metadata lineage graph corresponding to the entity information of the metadata stored in the second database through a first interface corresponding to the query module, the metadata lineage graph is composed of edges and nodes, and the first operation includes a user query operation on the metadata lineage graph of the target time period. That is, the user can query the metadata lineage graph of any desired target time period through the query module, thereby increasing the user experience.
[0077] In one embodiment, the method further includes: the query module modifies the metadata lineage graph in response to receiving a second operation from the user, the second operation including a screening operation and / or a dragging operation of the metadata lineage graph by the user through the first interface.
[0078] Of course, in other implementations, the second operation may also include other operations performed by the user on the metadata lineage graph through the first interface.
[0079] In one embodiment, the method also includes: the query module displays code information corresponding to the target blood relationship in response to receiving a third operation from the user, the third operation including the user's selection operation of the target node corresponding to the metadata blood relationship through the first interface, and the target node corresponds to the target blood relationship.
[0080] Next, the metadata management method provided by this application is explained in conjunction with specific application scenarios.
[0081] In one embodiment, the metadata management method provided by the present application is applied to Figure 4 The data system shown includes a Hive data warehouse (as an example of a data warehouse) and a metadata management system, and the metadata management system includes a Hive metadata plug-in (as an example of a metadata collection module), a graph database (as an example of a first database), a metadata repository (such as a MySQL database, as an example of a second database), a metadata storage module (as an example of a metadata determination module), a metadata periodic comparison module (as an example of a metadata correction module), and a UI interface (as an example of a first interface of a query module).
[0082] The metadata management method provided in this application includes:
[0083] The execution of Hive data warehouse tasks (i.e., executing data tasks) is triggered based on the Hook interface provided by the open source Hive database. The Hive metadata plug-in is triggered by using the ability to use Hive extensions and integrate external functions provided by the Hive interface. The metadata plug-in collects the metadata generated during the execution of the Hive data warehouse tasks and pushes the metadata information to the graph database for storage in the form of graph data.
[0084] The metadata storage module then reads the metadata information (such as metadata entity information) stored in the graph database, and converts the metadata information stored in the graph data into relational data and stores it in the MySQL database. Since the MySQL database stores metadata information in the form of relational data, users have good parallel query capabilities during the query process, which effectively improves the metadata query efficiency.
[0085] In one embodiment, the metadata periodic comparison module periodically (for example, at time T+1) reads and verifies the metadata in the Hive data warehouse, the MySQL database, and the metadata information in the graph database, and corrects the inconsistent metadata information (i.e., metadata entity information) in the MySQL database and the graph database.
[0086] In one embodiment, when a user needs to query metadata information, the user can query through the query module provided by the metadata management system. The query module corresponds to a front-end UI page. The user can query the metadata lineage graph through the UI interface and perform functions such as filtering and dragging on the lineage graph.
[0087] In one embodiment, if Figure 5As shown, the metadata management system also includes a message middleware Kafka (as an example of storage middleware), and the metadata storage module reads the metadata information stored in the graph database, and converts the metadata information stored in the graph data into relational data and stores it in the metadata repository, including:
[0088] When the Hive data warehouse in any big data scenario generates new metadata information when executing data tasks, the Hive metadata plug-in pushes the specific entity information (as an example of the first information) of the metadata information to the graph database for graph data storage, and simultaneously submits a copy of the entity information's identifier information (guid) and type information (as an example of the second information) to the message middleware (Kafka). The identifier information and type information of each metadata correspond to the entity information one by one. Among them, the identifier information can be a random string of the system, and the type information can mainly include hive_db, hive_table, hive_column, hive_db_ddl, hive_table_ddl, hive_storagedesc, hive_relationship, etc.
[0089] The metadata storage module will read (also called real-time consumption) the identifier information and type information in Kafka in real time, and use the obtained identifier information and type information to query the graph database for specific information of related entities, and then convert the entity specific information into relational data and write it into the relational database. In the relational database, corresponding storage tables can be built in advance according to the type, and each type can correspond to a table.
[0090] If the specific information of the relevant entity is queried in the graph database based on the identifier information and type information, and it is determined that the corresponding entity information does not exist in the graph database, the metadata storage module will record this information in the exception table (as an example of the information exception table corresponding to the second database). Before each subsequent consumption of Kafka, read the record information in the exception table and request the graph database until the corresponding entity information is queried, thereby effectively preventing data loss. The exception table can have one or more records about the non-existence of corresponding entity information in the graph database.
[0091] In the metadata storage module, the metadata information collected by the Hive metadata plug-in is converted from a graph data structure to a relational data structure and stored in a relational database (i.e., MySQL). Since the graph data structure only has two data structures, node and edge, this storage structure combines all types of entities together, which is not conducive to fast query. The metadata storage module reads the node and edge information and converts them into entity tables of various types (i.e., relational data). When querying by type, the Cartesian product can be reduced, and the query performance is higher. Figure 5 As shown, entity information may include:
[0092] hive_db: basic information about the database in Hive.
[0093] hive_table: basic information about the table in Hive.
[0094] hive_column: basic information about the Hive field (column).
[0095] hive_db_ddl: basic information about the database creation statement (db_ddl) in Hive.
[0096] hive_table_ddl: basic information about the table creation statement (table_ddl) in Hive.
[0097] hive_storagedesc: basic information about table storage (storagedesc) in Hive.
[0098] hive_relationship: the relationship information between the above entities.
[0099] In one embodiment, the query module provided by the metadata management system of the present application is as follows: Figure 6 As shown in the figure, a series of query interfaces can be provided according to different types of Hive entity classifications, such as Hive DB interface, Hive Table interface, Hive Column interface, and Hive Storage interface. When users query the lineage graph through the front-end UI, the query module uses JAVA thread pool parallel query technology in the background to query metadata corresponding to different types from the metadata repository, which can realize efficient query of metadata lineage graphs. The queried metadata information can be displayed in the form of a lineage graph in the UI interface corresponding to the query module.
[0100] In one embodiment, the metadata periodic comparison module periodically (for example, at time T+1) compares the metadata in the Hive data warehouse, the MySQL database, and the metadata in the graph database, and corrects inconsistent metadata information in the MySQL database and the graph database, including:
[0101] The metadata periodic comparison module can read the metadata information of three parties every day (or every few hours or days, the specific time period can be set as needed), including the metadata information in the Hive data warehouse, the metadata information in the graph database, and the metadata information in MySQL, and correct the inconsistent metadata information in MySQL and the graph database to ensure the accuracy of metadata queries. The metadata information can specifically be the entity information of the metadata.
[0102] The specific steps for comparing the metadata in the Hive data warehouse and the metadata in the graph database include:
[0103] Every morning (can also be configured to other time points), the metadata regular comparison module can use the Hive SDK to read all Schemas (Hive libraries, tables, fields, etc.) from the Hive data warehouse (as an example of the first information), and then query the graph database one by one to compare whether all the Schemas corresponding to the metadata information (as an example of the second information) are consistent. Specifically, the graph database can be queried based on a string of the type "library name@table name@field name" as a unique index. If the metadata information of the graph database is missing, metadata-related information is added to the graph database.
[0104] Then the metadata periodic comparison module queries all relationship information and entity information (i.e., all nodes and edges, as an example of the third information) from the graph database, and then compares the relationship information and entity information (as an example of the fourth information) corresponding to the metadata information in MySQL one by one to see if they are consistent. Specifically, the MySQL query can be performed based on the identifier information and type information as the unique index. If the metadata information in MySQL is missing, the relevant metadata information is added to MySQL.
[0105] In one embodiment, the UI interface corresponding to the query module of the present application can display the basic information corresponding to the metadata according to the user's query needs, such as Figure 7A-7B As shown, Fig. 7A A schematic diagram of metadata information displayed on the UI interface corresponding to the query module provided in this application. Figure 7BThe actual interface diagram of the metadata information displayed on the UI interface corresponding to the query module provided in this application. The basic information is shown in the figure, including the number of views 97, the number of favorites 0, the status ACTIVE, the person in charge is bigdataprod, the creation time, the last access time, the table type MANAGED_TABLE, whether it is a partitioned table is yes, whether it is a hudi table is yes, the partition field is dt, and information such as table comments, table parameters, and table creation statements. Of course, the basic information can also include other information, and the corresponding specific basic information value can also be other data. This is just an example.
[0106] The UI interface corresponding to the query module of this application can also display detailed information corresponding to the metadata according to the user's query needs, such as Figures 8A-8D The metadata shown corresponds to the field information, partition information, and change information. Figures 8A-8B A schematic diagram of metadata details displayed on the UI interface corresponding to the query module provided in this application. Figure 8C-8D This is an actual interface diagram of the metadata details displayed on the UI interface corresponding to the query module provided in this application.
[0107] The specific field information may include the corresponding serial number (such as 0, 1, ..., n), field name (such as hh, Vin, etl_updata_time), type (such as String, Int), description, creator (such as bigdataprod), status (such as ACTIVE), etc. The partition field may also be displayed separately, that is, the detailed information of each metadata field is displayed. The specific partition information may include the partition name (such as >dt=20230222, >dt=20230202, >dt=20230215), partition size (such as 5.65KB, 5.83KB, 8.62KB), number of partition directories (such as 3), number of partition files (such as 3), partition path (such as hdfs.XXX), etc.
[0108] The UI interface corresponding to the query module of the present application can also display the metadata lineage graph according to the user's query needs. As shown in Figure 9, the lineage graph is composed of nodes and edges (i.e., lines). The figure shows the table lineage graph corresponding to the metadata. The table lineage graph may include lineage legends, such as the current node (not shown in the figure), clicked nodes, lineage lines, and lines after time filtering (not shown in the figure), etc. It may also include a time input control. The user can query the metadata lineage graph corresponding to a specific time period through the time input control corresponding to the UI interface, such as a lineage control that displays lineage by time period or a date quick selection control. Compared with the prior art solution that can only query one lineage graph, the lineage graph query method provided by the present application can display the lineage graph corresponding to the time period that the user wants to view according to the user's needs, so that the user experience is better.
[0109] Specifically, by using the UI interface of the query module provided by the present application, providing a data model and developing corresponding rendering logic based on the function of D3.js, the blood relationship canvas can be rendered on the UI interface, and the drag and zoom functions on the blood relationship canvas can be realized based on the SDK function provided by D3.js. This allows users to perform operations such as filtering and dragging on the blood relationship map through the UI interface.
[0110] In one embodiment, when storing metadata, the metadata storage module can store the specific code of the blood relationship map (i.e., blood relationship) corresponding to the metadata. When the user clicks on the node corresponding to the blood relationship map through the UI interface, the specific code corresponding to the clicked node can be displayed for the user to view. Figures 9A-9B By clicking a node in the dialog box, the specific code of the target blood relationship corresponding to the node can be displayed.
[0111] The specific code (such as SQL statement) corresponding to the generated blood relationship may include:
[0112] When data is written, modified, or deleted (i.e., data tasks are executed) in the Hive data warehouse, metadata is generated. The Hive metadata plug-in will parse the relationship (i.e., the entity of hive_relationship) corresponding to the collected metadata and store it in the graph database (including the specific code SQL), and then enter the MySQL database after conversion by the metadata storage module.
[0113] When executing data tasks in Hive Data Warehouse, Hive Data Warehouse may repeatedly execute the same data (or the same table) at different times (for example, the same task is executed once a day or once an hour, where only the date is different and other execution logics are the same). In this case, the metadata storage module will determine the source and target of the hive_relationship, and mark the hive_relationships with the same source and target with the same tag (that is, group them by tags), indicating that these hive_relationships belong to the same blood relationship.
[0114] When querying the specific code SQL of a blood relationship in the UI interface, it will be grouped and sorted according to the labels marked on the above content, and the most recent specific code SQL of the blood relationship will be selected and displayed by time. It should be noted that in actual application, a blood relationship may have thousands of hive_relationships with the same source and target. Their SQLs are highly identical and have consistent business logic, so it is impossible to display them all. Therefore, this application can select the latest one.
[0115] The metadata management method provided in this application realizes the real-time collection of metadata. Compared with the existing technology that uses graph databases to store metadata and has the problem of weak query performance of bloodline graphs, the use of multi-threading technology (that is, relational databases for storing metadata information) effectively improves the query performance of bloodline graphs. Based on the self-developed front-end UI page of D3.js technology, it provides users with operation functions such as filtering and dragging of bloodline graphs, and provides bloodline graph query capabilities within a specific time period, as well as specific code (SQL) query capabilities corresponding to bloodline relationships. Through regular comparison, the consistency of the stored metadata information and the metadata information in the Hive data warehouse is ensured.
[0116] The functional components and modules used in the metadata management system provided in this application can be applied to a variety of business scenarios and Hive data warehouses in any big data scenario. The calling interface can be adaptively modified and can be deployed in the cloud. Each functional component and module can be customized according to the actual application, flexibly adapted, and deployed.
[0117] See also Fig.10 , Fig.10The electronic device may include one or more processors 1002, a system control logic 1008 connected to at least one of the processors 1002, a system memory 1004 connected to the system control logic 1008, a non-volatile memory (NVM) 1006 connected to the system control logic 1008, and a network interface 1010 connected to the system control logic 1008.
[0118] The processor 1002 may include one or more single-core or multi-core processors. The processor 1002 may include any combination of a general-purpose processor and a dedicated processor (e.g., a graphics processor, an application processor, a baseband processor, etc.). In the implementation of this document, the processor 1002 may be configured to execute the aforementioned metadata management method.
[0119] In some implementations, the system control logic 1008 may include any suitable interface controller to provide any suitable interface to at least one of the processors 1002 and / or any suitable device or component in communication with the system control logic 1008 .
[0120] In some implementations, the system control logic 1008 may include one or more memory controllers to provide an interface to the system memory 1004. The system memory 1004 may be used to load and store data and / or instructions. In some implementations, the system memory 1004 of the electronic device may include any suitable volatile memory, such as a suitable dynamic random access memory (DRAM).
[0121] NVM / memory 1006 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some implementations, NVM / memory 1006 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as a hard disk drive (HDD), a compact disc (CD) drive, and a digital versatile disc (DVD) drive.
[0122] NVM / storage 1006 may include a portion of storage resources installed on a device of an electronic device, or it may be accessible by the device but not necessarily a portion of the device. For example, NVM / storage 1006 may be accessed over a network via network interface 1010 .
[0123] In particular, the system memory 1004 and the NVM / storage 1006 may include: a temporary copy and a permanent copy of the instruction 1020, respectively. The instruction 1020 may include: instructions that, when executed by at least one of the processors 1002, cause the electronic device to implement the aforementioned metadata management method. In some implementations, the instruction 1020, hardware, firmware, and / or software components thereof may be additionally / alternatively placed in the system control logic 1008, the network interface 1010, and / or the processor 1002.
[0124] The network interface 1010 may include a transceiver for providing a radio interface for the electronic device, thereby communicating with any other suitable device (such as a front-end module, an antenna, etc.) through one or more networks. In some implementations, the network interface 1010 may be integrated with other components of the electronic device. For example, the network interface 1010 may be integrated with at least one of the processor 1002, the system memory 1004, the NVM / storage 1006, and a firmware device (not shown) having instructions. When at least one of the processors 1002 executes the instructions, the electronic device implements the aforementioned metadata management method.
[0125] The network interface 1010 may further include any suitable hardware and / or firmware to provide a multiple-input multiple-output radio interface. For example, the network interface 1010 may be a network adapter, a wireless network adapter, a telephone modem and / or a wireless modem.
[0126] In one implementation, at least one of the processors 1002 may be packaged with logic for one or more controllers of the system control logic 1008 to form a system in a package (SiP). In one implementation, at least one of the processors 1002 may be integrated with logic for one or more controllers of the system control logic 1008 on the same die to form a system on chip (SoC).
[0127] The electronic device may further include: an input / output (I / O) device 1012. The I / O device 1012 may include a user interface that enables a user to interact with the electronic device; the design of the peripheral component interface enables the peripheral component to also interact with the electronic device. In some implementations, the electronic device further includes a sensor for determining at least one of an environmental condition and location information related to the electronic device.
[0128] In some implementations, the user interface may include, but is not limited to, a display (e.g., an LCD display, a touch screen display, etc.), a speaker, a microphone, one or more cameras (e.g., a still image camera and / or a video camera), a flashlight (e.g., an LED flash), and a keyboard.
[0129] In some implementations, the peripheral component interface can include, but is not limited to, a non-volatile memory port, an audio jack, and a power interface.
[0130] In some implementations, the sensors may include, but are not limited to, gyroscope sensors, accelerometers, proximity sensors, ambient light sensors, and positioning units. The positioning unit may also be part of or interact with the network interface 1010 to communicate with components of a positioning network (e.g., Global Positioning System (GPS) satellites).
[0131] It is to be understood that the structure illustrated in the implementation of the present invention does not constitute a specific limitation on the electronic device. In other implementations of the present application, the electronic device may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
[0132] Program code can be applied to input instructions to perform the functions described herein and generate output information. The output information can be applied to one or more output devices in a known manner. For the purpose of the present application implementation, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.
[0133] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described herein is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.
[0134] One or more aspects of at least one implementation may be implemented by representative instructions stored on a computer-readable storage medium, which represent various logic within a processor, and which, when read by a machine, cause the machine to fabricate logic for performing the techniques described herein. These representations, known as "IP cores," may be stored on a tangible computer-readable storage medium and supplied to multiple customers or manufacturing facilities to load into fabrication machines that actually manufacture the logic or processor.
[0135] It should be noted that, in addition to the implementation methods of the present invention described in the above-mentioned specific embodiments, those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Although the description of the present invention is introduced in conjunction with the preferred embodiment, this does not mean that the features of this invention are limited to this implementation method. On the contrary, the purpose of introducing the invention in conjunction with the implementation method is to cover other options or modifications that may be extended based on the claims of the present invention. In order to provide a deep understanding of the present invention, the above description contains many specific details, and the present invention can also be implemented without using these details. In addition, in order to avoid confusion or blurring the focus of the present invention, some specific details will be omitted in the description. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.
[0136] Although the present invention has been illustrated and described with reference to certain preferred embodiments of the present invention, it should be understood by those skilled in the art that the above is a further detailed description of the present invention in conjunction with specific embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. Those skilled in the art may make various changes in form and details, including making several simple deductions or substitutions, without departing from the spirit and scope of the present invention.
Claims
1. A metadata management method, characterized in that: Applied to a data system including a data warehouse and a metadata management system, the metadata management system including a metadata collection module, a metadata storage module and a metadata correction module, the method includes: When the data warehouse performs a data task based on the data in the data warehouse and generates metadata corresponding to the data, the data warehouse stores the metadata and sends a metadata collection instruction to the metadata collection module in the metadata management system, wherein the metadata collection instruction includes identification information of the data task; In response to receiving the metadata collection instruction, the metadata collection module collects metadata corresponding to the data task according to the identification information in the metadata collection instruction, and sends the metadata to the metadata storage module; The metadata storage module stores the metadata; The metadata correction module corrects the metadata stored in the metadata storage module based on the metadata stored in the data warehouse according to a preset period.
2. The metadata management method according to claim 1, characterized in that: The metadata storage module includes a first database and a second database, the first database is a graph database, and the second database is a relational database. The metadata management system also includes a storage middleware and a metadata determination module. The metadata collection module sends the metadata to the metadata storage module, including: The metadata collection module determines entity information and identification information of the metadata, sends the entity information to the first database for storage, and sends the identification information to the storage middleware; The metadata determination module obtains the identification information in the storage middleware, and when it is determined according to the identification information that the entity information exists in the first database, obtains the entity information from the first database, and sends the entity information to the second database for storage.
3. The metadata management method according to claim 2, characterized in that: The method further comprises: When the metadata determination module determines according to the identification information that the entity information does not exist in the first database, the metadata determination module stores record information that the entity information does not exist in the first database in an information exception table corresponding to the second database.
4. The metadata management method according to claim 2 or 3, characterized in that: The identification information includes identifier information and type information corresponding to the metadata.
5. The metadata management method according to any one of claims 2 to 4, characterized in that: The metadata correction module corrects the metadata stored in the metadata storage module based on the metadata stored in the data warehouse according to a preset period, including: The metadata correction module corrects the entity information of the metadata stored in the first database based on the metadata stored in the data warehouse according to a preset period to obtain corrected first entity information, and corrects the entity information of the metadata stored in the second database based on the corrected first entity information to obtain corrected second entity information.
6. The metadata management method according to claim 5, characterized in that: The metadata correction module corrects the entity information of the metadata stored in the first database based on the metadata stored in the data warehouse to obtain the corrected first entity information, including: The metadata correction module obtains first information of metadata stored in the data warehouse, and compares it with second information of entity information of the metadata stored in the first database. When it is determined that metadata that does not belong to the first database exists in the data warehouse according to the first information and the second information, the entity information of the metadata that does not belong to the first database in the data warehouse is added to the first database to obtain corrected first entity information, wherein the first information includes organization and structure information of the metadata stored in the data warehouse, and the second information includes organization and structure information of the entity information.
7. The metadata management method according to claim 6, characterized in that: The metadata correction module corrects the entity information of the metadata stored in the second database based on the corrected first entity information to obtain the corrected second entity information, including: The metadata correction module determines third information of the corrected first entity information, and compares it with fourth information corresponding to the entity information of the metadata stored in the second database. When it is determined that entity information of the metadata that does not belong to the second database exists in the first database according to the third information and the fourth information, the entity information of the metadata that does not belong to the second database in the first database is added to the second database to obtain the corrected second entity information, the third information includes identification information corresponding to the corrected first entity information, and the fourth information includes identification information corresponding to the entity information of the metadata stored in the second database.
8. The metadata management method according to any one of claim 7, characterized in that: The metadata management system further includes a query module, and the method further includes: In response to receiving a first operation from a user, the query module displays a metadata lineage graph corresponding to the entity information of the metadata stored in the second database through a first interface corresponding to the query module, wherein the first operation includes a query operation by the user on the metadata lineage graph of a target time period.
9. The metadata management method according to claim 8, characterized in that: The method further comprises: The query module modifies the metadata lineage graph in response to receiving a second operation from the user, where the second operation includes a screening operation and / or a dragging operation on the metadata lineage graph by the user through the first interface.
10. The metadata management method according to claim 9, characterized in that: The method further comprises: The query module displays code information corresponding to the target blood relationship in response to receiving a third operation from the user, wherein the third operation includes a selection operation of the user on a target node corresponding to the metadata blood relationship through the first interface, and the target node corresponds to the target blood relationship.
11. A data system, characterized in that: It includes a data warehouse and a metadata management system, wherein the metadata management system includes a metadata collection module, a metadata storage module and a metadata correction module, wherein: The data warehouse is used to store the metadata when a data task is executed based on the data in the data warehouse and metadata corresponding to the data is generated, and to send a metadata collection instruction to the metadata collection module in the metadata management system, wherein the metadata collection instruction includes identification information of the data task; The metadata collection module is used for collecting metadata corresponding to the data task according to the identification information in the metadata collection instruction in response to receiving the metadata collection instruction, and sending the metadata to the metadata storage module; The metadata storage module is used to store the metadata; The metadata correction module is used to correct the metadata stored in the metadata storage module based on the metadata stored in the data warehouse according to a preset period.
12. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the electronic device implements the metadata management method as described in any one of claims 1-10.
Citation Information
Patent Citations
Metadata updating method, device and system, electronic equipment and storage medium
CN110019267A
Data processing method and device
CN111984686A
Data synchronization method and device, electronic equipment and storage medium
CN114328741A
Metadata management method and system
CN116991931A
Meta data generation method and device
JP2003296342A
