Metadata acquisition method and device and electronic equipment

Abstract

The invention provides a metadata acquisition method and device and electronic equipment, and relates to the technical field of big data, cloud computing and the like. According to the specific implementation scheme, a metadata collection task sent by a server side is received, and metadata of a first data table in at least one data source is obtained according to the collection task; according to metadata of a first data table, a first abstract set is generated, and the first abstract set comprises first abstract information of the first data table; a second summary set is obtained, the second summary set comprises second summary information of a second data table in which metadata is stored, and the second summary information is generated by the metadata of the second data table; and storing the collected metadata of the first data table according to the first abstract set and the second abstract set. Therefore, the metadata of the collected first data table is stored according to the first abstract set and the second abstract set, and the efficiency and the automation degree of metadata collection are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and more specifically to the field of big data and cloud computing, and more particularly to a metadata collection method, device and electronic device. Background Art

[0002] In the modern information society, data in various industries are of various types and from a wide range of sources, which often leads to a series of problems in the process of data management and data analysis. Among them, metadata can be used to describe data and is the key to achieving effective data management and utilization. Through the collection of metadata, comprehensive support can be provided for various aspects such as data storage, retrieval, analysis, and sharing.

[0003] In the related art, the automated collection tool interacts with various data sources through pre-configured adapters and connectors. The automated collection tool automatically identifies and extracts the corresponding metadata by parsing the structure and content of the data source. However, the above method often leads to low efficiency and automation in the metadata collection process. Therefore, how to improve the efficiency and automation of metadata collection has become an urgent problem to be solved. Summary of the invention

[0004] The present disclosure provides a metadata collection method, device, electronic device, storage medium and program product.

[0005] According to one aspect of the present disclosure, a metadata collection method is provided, which is executed by a computing node in a distributed system, and includes: receiving a metadata collection task sent by a server, and acquiring metadata of a first data table in at least one data source according to the collection task; generating a first summary set according to the metadata of the first data table, the first summary set including first summary information of the first data table; acquiring a second summary set, the second summary set including second summary information of a second data table in which metadata has been stored, the second summary information being generated from the metadata of the second data table; and storing the collected metadata of the first data table according to the first summary set and the second summary set.

[0006] According to another aspect of the present disclosure, a metadata collection device is provided, which is used for a computing node in a distributed system, and includes: a first acquisition module, which is used to receive a metadata collection task sent by a server, and obtain metadata of a first data table in at least one data source according to the collection task; a generation module, which is used to generate a first summary set according to the metadata of the first data table, wherein the first summary set includes first summary information of the first data table; a second acquisition module, which is used to obtain a second summary set, wherein the second summary set includes second summary information of a second data table in which metadata has been stored, wherein the second summary information is generated from the metadata of the second data table; and a storage module, which is used to store the collected metadata of the first data table according to the first summary set and the second summary set.

[0007] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the metadata collection method described in one aspect of the present disclosure.

[0008] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the metadata collection method according to one aspect of the present disclosure.

[0009] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the computer program implements the steps of the metadata collection method according to one aspect of the present disclosure.

[0010] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0012] Figure 1 is a flowchart of a metadata collection method according to the first embodiment of the present disclosure;

[0013] Figure 2 is a schematic diagram of a preset task state transition diagram according to the present disclosure;

[0014] Figure 3 is a flowchart of a metadata collection method according to a second embodiment of the present disclosure;

[0015] Figure 4 is a schematic diagram of the overall architecture of a metadata collection method according to the present disclosure;

[0016] Figure 5 is a schematic diagram of an execution process of a metadata collection task according to the present disclosure;

[0017] Figure 6 It is a schematic diagram of a metadata collection task execution process according to the present disclosure;

[0018] Figure 7 is a schematic diagram of an application system of a metadata collection method according to the present disclosure;

[0019] Figure 8 is a block diagram of a metadata collection device for implementing an embodiment of the present disclosure;

[0020] Fig. 9 is a block diagram of an electronic device for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0022] Data processing, including data collection, storage, retrieval, processing, change and transmission, aims to extract and derive valuable and meaningful data for certain specific users from large amounts of data that may be disorganized and difficult to understand.

[0023] Big Data, or massive data, refers to the amount of data involved that is so large that it cannot be captured, managed, processed, and organized into information that helps companies make more proactive business decisions within a reasonable time through mainstream software tools.

[0024] Cloud computing is a type of distributed computing. It refers to breaking down huge data computing programs into countless small programs through the network "cloud". These small programs are then processed and analyzed by a system composed of multiple servers to obtain results and return them to users.

[0025] The following describes the metadata collection method of the embodiment of the present disclosure in conjunction with the accompanying drawings.

[0026] Figure 1 1 is a flowchart of a metadata collection method according to the first embodiment of the present disclosure. It should be noted that the execution subject of the metadata collection method of this embodiment is a metadata collection device, which can be a hardware device or software in a hardware device. The hardware device is, for example, a terminal device, a server, etc.

[0027] like Figure 1 As shown, the metadata collection method provided by the embodiment of the present disclosure is executed by a computing node in a distributed system, and the specific method may include the following steps:

[0028] S101: Receive a metadata collection task sent by a server, and obtain metadata of a first data table in at least one data source according to the collection task.

[0029] Among them, metadata, also known as intermediary data or relay data, is data that describes data, mainly information that describes data attributes, and is used to support functions such as indicating storage location, historical data, resource search, and file records.

[0030] It should be noted that the server can receive parameter information and a scheduling method for a metadata collection task from a user, and create a metadata collection task according to the task information and the scheduling method.

[0031] The parameter information of the collection task includes but is not limited to: (1) basic parameter information of the collection task, such as the name and description of the collection task; (2) data source information of the collection task, such as the type of data source; (3) data table information, such as the range information of the data table.

[0032] The scheduling methods of the collection tasks include, but are not limited to, periodic scheduling and timed scheduling.

[0033] It should be noted that in order to improve the stability and efficiency during the execution of the collection task, in the embodiment of the present disclosure, the execution of the collection task can be monitored based on the agent in the computing node to obtain the monitoring data of the collection task, and the status of the collection task can be updated based on the preset task state transition diagram and the monitoring data.

[0034] Optionally, in response to the collection task being in an abnormal state, the collection task is re-executed; and / or, the abnormal state of the collection task is sent to the server to instruct the server to reschedule the computing node for the collection task.

[0035] For example, the default task state transition diagram is as follows Figure 2As shown, when the collection task is not running, the status of the collection task is not running. After the collection task starts running, the status of the collection task is updated to Running in this case. The collection task can be stopped according to the actual situation. In this case, the status of the collection task is updated to Stopped. If the collection task is re-executed after stopping, the status of the collection task is updated to Running in this case. It can be judged whether the collection task runs successfully based on the monitoring data. If it is determined that the collection task is in an abnormal state (unsuccessful) according to the monitoring data, the status of the collection task is updated to Failed in this case. By re-executing the collection task, the status of the collection task is updated to Running in this case. If it is determined that the collection task is executed successfully according to the monitoring data, the status of the collection task is updated to Successful in this case.

[0036] It should be noted that the present disclosure does not limit the type of data source, which can be selected according to actual conditions.

[0037] Optionally, the data source can be Hive, MySQL, or Oracle.

[0038] It should be noted that the present disclosure does not limit the setting of the first data table in the data source, and it can be selected according to actual conditions.

[0039] Optionally, the first data table may be a partial data table in the data source; optionally, the first data table may be all data tables in the data source.

[0040] In an embodiment of the present disclosure, the server sends a metadata collection task to a computing node in a distributed system. Accordingly, the computing node in the distributed system receives the metadata collection task sent by the server, and obtains metadata of a first data table of at least one data source according to the collection task.

[0041] S102: Generate a first summary set according to metadata of the first data table, where the first summary set includes first summary information of the first data table.

[0042] In an embodiment of the present disclosure, after obtaining the metadata of the first data table, the metadata of the first data table can be preprocessed to obtain summary information corresponding to the metadata of the first data table, and first summary information of the first data table can be obtained based on the summary information corresponding to the metadata of the first data table, so as to generate a first summary set based on the first summary information of the first data table.

[0043] The first summary information includes but is not limited to the identifier, content, keywords, data structure, etc. of the first data table.

[0044] S103: Acquire a second summary set, where the second summary set includes second summary information of a second data table that has stored metadata, and the second summary information is generated from the metadata of the second data table.

[0045] It should be noted that, through the metadata Metadata component, the second summary information of the second data table that has stored (historical) metadata can be obtained in advance, and the obtained second summary information can be stored in the metadata collection library.

[0046] Optionally, a call instruction may be initiated to the Metadata component, and the Metadata component obtains second summary information of the second data table that has stored metadata from the metadata collection library in response to the call instruction. Accordingly, a second summary set may be generated based on the second summary information of the second data table.

[0047] S104: Store the collected metadata of the first data table according to the first summary set and the second summary set.

[0048] In the embodiment of the present disclosure, after the first summary set and the second summary set are acquired, the metadata of the collected first data table may be stored according to the first summary set and the second summary set.

[0049] Optionally, the target set to which the first data table belongs can be determined according to the first summary set and the second summary set, at least part of the metadata of the first data table can be added to the target set, and the metadata of the first data table can be stored in the target database in the form of the target set.

[0050] It should be noted that the target set includes but is not limited to: update set, add set, and delete set.

[0051] For example, the first summary set and the second summary set can be matched to obtain the first summary information and the second summary information belonging to the same first data table. In response to the first summary information and the second summary information of the same first data table being different, it is determined that the same first data table belongs to an update set, at least part of the metadata of the first data table is added to the update set, and the metadata of the first data table is stored in the target database in the form of a target set.

[0052] In summary, the metadata collection method of the embodiment of the present disclosure receives a metadata collection task sent by a server, and obtains metadata of a first data table in at least one data source according to the collection task, generates a first summary set according to the metadata of the first data table, the first summary set includes first summary information of the first data table, obtains a second summary set, the second summary set includes second summary information of a second data table in which metadata has been stored, the second summary information is generated by the metadata of the second data table, and stores the collected metadata of the first data table according to the first summary set and the second summary set. Therefore, the present disclosure stores the collected metadata of the first data table according to the first summary set and the second summary set, thereby improving the efficiency and automation of metadata collection.

[0053] Figure 3 FIG. 2 is a flowchart of a method for collecting metadata according to the second embodiment of the present disclosure. Figure 2 As shown, based on the above embodiment, the metadata collection method of the embodiment of the present disclosure may specifically include the following steps:

[0054] S301: Receive a metadata collection task sent by a server, and obtain metadata of a first data table in at least one data source according to the collection task.

[0055] S302: Generate a first summary set according to metadata of the first data table, where the first summary set includes first summary information of the first data table.

[0056] S303: Acquire a second summary set, where the second summary set includes second summary information of a second data table that has stored metadata, and the second summary information is generated from the metadata of the second data table.

[0057] The steps S301 to S303 may be implemented in any of the implementation methods in the embodiments of the present disclosure and will not be described in detail here.

[0058] In the above embodiment, step S104 “storing the collected metadata of the first data table according to the first summary set and the second summary set” may specifically include the following steps S304 and S305 .

[0059] S304: Determine the target set to which the first data table belongs according to the first summary set and the second summary set.

[0060] In the embodiment of the present disclosure, after the first summary set and the second summary set are acquired, the target set to which the first data table belongs may be determined according to the first summary set and the second summary set.

[0061] Optionally, the first summary set and the second summary set may be matched to obtain first summary information and second summary information belonging to the same first data table. In response to the first summary information and the second summary information of the same first data table being different, it is determined that the same first data table belongs to the update set {U}.

[0062] For example, for the first summary set {A1} and the second summary set {A2}, the first summary information M1 and the second summary information M2 belonging to the same first data table table1 are obtained. If the first summary information M1 of table1 is different from the first summary information M2, it is determined that table1 belongs to the update set {U}.

[0063] It should be noted that, in response to the first summary information and the second summary information of the same first data table being the same, it is determined that the metadata of the same first data table is repeated, and the first data table is skipped.

[0064] For example, for the first summary set {A1} and the second summary set {A2}, obtain the first summary information M1 and the second summary information M2 belonging to the same first data table table1. If the first summary information M1 of table1 is the same as the first summary information M1, it is determined that the metadata of table1 is repeated, and table1 is skipped.

[0065] Optionally, the first summary set and the second summary set may be matched to obtain first summary information newly added to the first summary set relative to the second summary set, determine the target first data table corresponding to the newly added first summary information, and determine that metadata of the target first data table belongs to the newly added set.

[0066] For example, for the first summary set {A1} and the second summary set {A2}, by matching the first summary set {A1} and the second summary set {A2}, the first summary information M3 newly added to the first summary set relative to the second summary set is obtained, and the target first data table table2 corresponding to the newly added first summary information M3 is determined, and it is determined that the metadata of table2 belongs to the newly added set {N}.

[0067] S305: Add at least part of the metadata of the first data table to the target set, and store the metadata of the first data table in the target database in the form of the target set.

[0068] It should be noted that the present disclosure does not limit the type of target database, which can be selected according to actual conditions.

[0069] Optionally, the target database may be MySQL; optionally, the target database may be Elasticsearch.

[0070] In an embodiment of the present disclosure, metadata adding restriction conditions for the target set can be determined, and based on the metadata adding restriction conditions, target metadata to be added to the first data table can be determined from the metadata of the first data table, and the target metadata of the first data table can be added to the target set.

[0071] In an embodiment of the present disclosure, for an update set, one or more update fields of a data table pre-configured for the update set can be obtained, and metadata associated with the update fields can be determined from the metadata of the same first data table as target metadata for the same first data table, and the target metadata of the same first data table can be added to the update set.

[0072] In an embodiment of the present disclosure, for a newly added collection, the amount of metadata of a single data table allowed to be added to the newly added collection can be determined, and based on the amount of metadata, the target metadata of the target first data table can be determined from the metadata of the target first data table, and the target metadata of the target first data table can be added to the newly added collection.

[0073] It should be noted that the present disclosure does not limit the specific method of determining the target metadata of the target first data table from the metadata of the target first data table, and the method may be selected according to actual conditions.

[0074] Optionally, the target metadata of the target first data table may be determined from the metadata of the target first data table according to the priority of the metadata.

[0075] In the embodiment of the present disclosure, it is determined that each metadata of the target first data table is the target metadata of the target first data table, and the target metadata of the target first data table is added to the newly added set.

[0076] In the embodiment of the present disclosure, whenever a first data table is collected, the collected first data table is added to the processed first table set, a second table set generated based on the second data table is obtained, the first table set and the second table set are matched to obtain the data table that is missing from the first table set relative to the second table set, and the metadata of the missing data table is added to the deletion set {D}.

[0077] For example, for the first table set {E1} and the second table set {E2}, by matching {E1} and {E2}, it is obtained that the data table missing from {E1} relative to {E2} is table3, and the metadata of table3 is added to the deletion set {D}.

[0078] In an embodiment of the present disclosure, after the metadata of the first data table is stored in the target database in the form of a target set, third summary information of the first data table can be generated according to the target metadata of the first data table added in the update set and the newly added set, and the second summary set can be updated based on the third summary information.

[0079] In summary, the metadata collection method of the embodiment of the present disclosure receives a metadata collection task sent by a server, and obtains metadata of a first data table in at least one data source according to the collection task, generates a first summary set according to the metadata of the first data table, the first summary set includes first summary information of the first data table, obtains a second summary set, the second summary set includes second summary information of a second data table that has stored metadata, and the second summary information is generated by the metadata of the second data table, determines the target set to which the first data table belongs according to the first summary set and the second summary set, adds at least part of the metadata of the first data table to the target set, and stores the metadata of the first data table in the target database in the form of a target set, thereby, the present disclosure can collect metadata when the amount of data is large, improves the efficiency of metadata collection, can adapt to a variety of data sources and different data formats, improves the scalability of the metadata collection process, can intelligently schedule the collection tasks, and automatically adjust according to the load of the computing node in the distributed system, thereby improving the efficiency and automation of metadata collection.

[0080] The overall architecture of the metadata collection method proposed in the present disclosure is explained below.

[0081] For example, Figure 4As shown, for the server (di-server), the task information of the metadata collection task can be configured to create the metadata collection task. The collection task can be divided into periodic tasks and manual tasks. Among them, the manual task needs to be manually triggered and generates a scheduling instance of the manual task. The periodic task will directly start the scheduling after reaching the pre-configured periodic time, and generate a scheduling instance of the periodic task. The scheduling instances of the manual task and the scheduling instances of the periodic task are summarized to generate a metadata collection task instance. The server sends the collection task instance to the computing node in the distributed system. For the computing node in the distributed system, the metadata collection task sent by the server is received. Based on the agent (Remote The collection task is monitored by an agent. When it is determined that the collection task is in an abnormal state, the collection task is re-executed and / or the abnormal state of the collection task is sent to the server to instruct the server to reschedule the computing node for the collection task. Based on the working node (di-worker) in the computing node, metadata of a first data table in at least one data source is obtained according to the collection task. The data source includes but is not limited to: Hive, MySQL, Oracle, and a summary set generated by metadata of a second data table in which metadata has been stored is obtained based on the working node (di-worker) in the computing node. For example, through the Easy Data Lake Management and Analysis Platform An EDAP platform (abbreviated as EDAP) collects data from various data sources and stores it in an EDAP data lake, models the data collected in the EDAP data lake to generate an EDAP data model, obtains a summary set generated from metadata of a second data table in which metadata has been stored according to the EDAP data lake and the EDAP data model, and stores the collected metadata of the first data table according to a first summary set generated from the metadata of the first data table and a summary set generated from the metadata of the second data table in which metadata has been stored, wherein the metadata of the first data table may be stored in a target database (distributed storage system), for example: the distributed storage system includes but is not limited to MySQL and Elasticsearch.

[0082] Optionally, for the target database MySQL, the task information can be stored in MySQL. Through the scheduling of periodic tasks, the scheduled tasks can be polled and the collection task instances can be created, and the polling information of the scheduled tasks and the creation of the collection task instance information can be stored in MySQL.

[0083] The following is an explanation of the execution process of the metadata collection task proposed in the present disclosure.

[0084] For example, Figure 5As shown, according to the metadata collection method proposed in the present disclosure, first, the collection task starts to be executed, and the execution process of the metadata collection task is divided into three stages: (i) a metadata preprocessing stage, that is, after the second data table that has stored metadata is obtained from the Metadata component based on the working node di-worker, the second summary information of the second data table of the metadata is obtained through the metadata preprocessing stage, and the second summary set is obtained according to the second summary information; (ii) a metadata collection stage, that is, according to the collection task, at least one data source is accessed to collect metadata of the first data table in the data source, and a first summary set is generated according to the metadata of the first data table, and the metadata collection result of the metadata is determined according to the first summary set and the second summary set; (iii) a metadata storage stage, that is, the collection result is returned to the Metadata component, and the collection result is stored in the target database, and then the collection task is completed.

[0085] It should be noted that the execution of the collection task can be monitored in real time to update the status of the collection task. For example: when the collection task starts to execute, the status of the collection task is updated to Running; when it is monitored that the collection task fails to execute in the preprocessing stage, the status of the collection task is updated to Failed; when it is monitored that the collection task is successfully executed in the preprocessing stage, the metadata collection stage is entered; when it is monitored that the collection task fails to execute in the metadata collection stage, the status of the collection task is updated to Failed; when it is monitored that the collection task is successfully executed in the metadata collection stage, the metadata storage stage is entered; when it is monitored that the collection task fails to execute in the metadata storage stage, the status of the collection task is updated to Failed; when it is monitored that the collection task is successfully executed in the metadata storage stage, the collection task ends and the status of the collection task is updated to Succeed.

[0086] The specific execution process of the metadata collection task proposed in the present disclosure is explained below.

[0087] For example, Figure 6As shown, the metadata collection task starts to execute, and a request is made to call the Metadata component to obtain the second summary information of the second data table whose metadata has been stored in the metadata collection library, traverse the second data table of the metadata, and mark the second data table with the collection task configuration label to determine whether the second data table has been collected. If it is determined that the second data table has not been collected, the second data table is added to the newly added collection. If it is determined that the second data table has been collected, the first summary information of the newly collected first data table is obtained, and the first summary information is compared with the second summary information to determine whether the first summary information is the same as the second summary information. If it is determined that the first summary information and the second summary information of the same first data table are different, the metadata associated with the update field can be determined from the metadata of the same first data table according to one or more update fields of the data table pre-configured in the update set as the target metadata of the same first data table, and the target metadata of the same first data table is added to the update set. If it is determined that the first summary information and the second summary information of the same first data table are the same, it is determined that the metadata of the same first data table is repeated, and the first data table is skipped. Whenever a first data table is collected, the collected first data table is added to the processed first table set Processed. List, determine whether the second data table of metadata has been traversed. If not, continue to traverse the second data table of metadata. If traversal is completed, match the first table set and the second table set generated by the second data table, add the data tables missing from the first table set relative to the second table set to the deletion set, request to call the Metadata component, and return the metadata collection results to the Metadata component.

[0088] In summary, the metadata collection method of the embodiment of the present disclosure can support periodic scheduling and timed scheduling for metadata collection tasks according to a pre-set distributed scheduling strategy, which can maximize the scheduling speed and throughput of the collection tasks. Through load balancing, it is ensured that the collection tasks run efficiently on multiple computing nodes, and the data sources are accessed in the form of components, thereby realizing rapid access to multiple data sources. According to the summary information of the metadata, repeated collection is reduced, and the efficiency and automation of metadata collection are improved. At the same time, the execution of the collection tasks can be monitored in real time. If the collection task is executed abnormally, it can be automatically retried and scheduled to other computing nodes to ensure stability and reliability in the metadata collection process. According to the summary information of the metadata, the target set (update, addition and deletion) to which the data table belongs is determined, and metadata is added to the target set, and the metadata is stored in the target database in the form of a target set, thereby ensuring the persistence of metadata.

[0089] The following is an explanation of the application scenarios of the metadata collection method proposed in the present disclosure.

[0090] For example, Figure 7 As shown, according to the metadata collection method proposed in the present disclosure, a metadata application system can be constructed, which includes an application layer, a service layer, a computing layer, and a storage layer, wherein the application layer includes an Open API module, an indicator and an alarm module, and The API module can realize interaction with the system. The indicator and alarm module can collect the system's operating indicator data. The operating indicator data can be used to monitor and alarm the system in real time. The service layer includes a search module, a metadata details module, a collection task management module and a user behavior statistics module. The search module can be used to retrieve the data stored in the system. The metadata details module can be used to obtain detailed information about metadata. The collection task management module can be used to manage metadata collection tasks. The user behavior statistics module can be used to count the user's behavior data when using the system. The computing layer includes a task scheduling system, a task execution engine and a collection result processing module. The task scheduling system can effectively schedule metadata collection tasks on multiple computing nodes. The task execution engine can be used to execute metadata collection tasks to obtain the collection results (execution results) of the metadata collection tasks. The collection result processing module can store the collection results in a distributed storage system. The storage layer includes a MySQL module, an Elasticsearch module and a Doris module. The MySQL module, the Elasticsearch module and the Doris module can be used to store collection results.

[0091] In summary, the metadata application system of the disclosed embodiment can provide a visual metadata management interface, support users to retrieve, analyze and use metadata, enable better understanding and utilization of data assets, provide a solid foundation for data-driven decision-making, and provide necessary support for data governance, data compliance and data security. It can be applied in the scenario of enterprise data integration, that is, through automated metadata collection, manual operations can be reduced, and metadata consistency can be improved. It can also be applied in the scenario of data asset management, and metadata information scattered in multiple heterogeneous data sources can be uniformly managed through collection to build data assets. It can also be applied in the scenario of data lake analysis, that is, to provide real-time and accurate metadata support for the data lake and improve the analysis effect.

[0092] Figure 8 The metadata collection device 800 proposed in the present disclosure is applied to a computing node in a distributed system, and includes: a first acquisition module 801, a generation module 802, a second acquisition module 803 and a storage module 804.

[0093] The first acquisition module 801 is used to receive a metadata collection task sent by the server, and obtain metadata of a first data table in at least one data source according to the collection task;

[0094] A generating module 802, configured to generate a first summary set according to metadata of the first data table, wherein the first summary set includes first summary information of the first data table;

[0095] A second acquisition module 803 is used to acquire a second summary set, where the second summary set includes second summary information of a second data table in which metadata has been stored, and the second summary information is generated from the metadata of the second data table;

[0096] The storage module 804 is configured to store the collected metadata of the first data table according to the first summary set and the second summary set.

[0097] Furthermore, the storage module 804 is also used to: determine the target set to which the first data table belongs based on the first summary set and the second summary set; add at least part of the metadata of the first data table to the target set, and store the metadata of the first data table in the target database in the form of the target set.

[0098] Furthermore, the storage module 804 is also used to: determine metadata adding restriction conditions for the target set; determine target metadata to be added to the first data table from the metadata of the first data table based on the metadata adding restriction conditions; and add the target metadata of the first data table to the target set.

[0099] Furthermore, the storage module 804 is also used to: match the first summary set and the second summary set to obtain first summary information and second summary information belonging to the same first data table; in response to the first summary information and the second summary information of the same first data table being different, determine that the same first data table belongs to an update set.

[0100] Furthermore, the device 800 is also used to: obtain one or more update fields of the data table pre-configured in the update set; determine metadata associated with the update field from the metadata of the same first data table as target metadata of the same first data table; and add the target metadata of the same first data table to the update set.

[0101] Furthermore, the device 800 is further configured to: in response to the first summary information and the second summary information of the same first data table being the same, determine that the metadata of the same first data table is repeated, and skip the first data table.

[0102] Furthermore, the storage module 804 is also used to: match the first summary set and the second summary set to obtain first summary information newly added to the first summary set relative to the second summary set; determine a target first data table corresponding to the newly added first summary information; and determine that metadata of the target first data table belongs to the newly added set.

[0103] Furthermore, the device 800 is also used to: determine the amount of metadata of a single data table allowed to be added to the newly added collection; determine the target metadata of the target first data table from the metadata of the target first data table based on the metadata amount; and add the target metadata of the target first data table to the newly added collection.

[0104] Furthermore, the storage module 804 is further configured to determine target metadata of the target first data table from the metadata of the target first data table according to the priority of the metadata.

[0105] Furthermore, the device 800 is further configured to: determine that each metadata of the target first data table is target metadata of the target first data table, and add the target metadata of the target first data table to the newly added set.

[0106] Furthermore, the device 800 is also used to: whenever a first data table is collected, add the collected first data table to the processed first table set; obtain a second table set generated based on the second data table; match the first table set and the second table set to obtain the data table that is missing from the first table set relative to the second table set; and add the metadata of the missing data table to the deletion set.

[0107] Furthermore, the device 800 is further configured to: generate third summary information of the first data table according to target metadata of the first data table added in the update set and the new set; and update the second summary set based on the third summary information.

[0108] Furthermore, the device 800 is also used to: monitor the execution of the collection task based on the agent in the computing node to obtain monitoring data of the collection task; and update the status of the collection task based on a preset task state transition diagram and the monitoring data.

[0109] Furthermore, the device 800 is also used to: in response to the collection task being in an abnormal state, re-execute the collection task; and / or, send the abnormal state of the collection task to the server to instruct the server to reschedule the computing node for the collection task.

[0110] In summary, the metadata collection device of the embodiment of the present disclosure receives a metadata collection task sent by a server, and obtains metadata of a first data table in at least one data source according to the collection task, generates a first summary set according to the metadata of the first data table, the first summary set includes first summary information of the first data table, obtains a second summary set, the second summary set includes second summary information of a second data table in which metadata has been stored, the second summary information is generated by the metadata of the second data table, and stores the collected metadata of the first data table according to the first summary set and the second summary set. Therefore, the present disclosure stores the collected metadata of the first data table according to the first summary set and the second summary set, thereby improving the efficiency and automation of metadata collection.

[0111] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0112] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0113] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0114] like Fig. 9 As shown, the electronic device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0115] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0116] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as Figures 1 to 7 The metadata collection method shown in the figure. For example, in some embodiments, the metadata collection method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the metadata collection method described above can be performed. Alternatively, in other embodiments, the computing unit 901 can be configured to perform the metadata collection method in any other suitable manner (for example, by means of firmware).

[0117] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0118] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable metadata collection device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0119] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0120] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0121] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0122] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0123] According to an embodiment of the present disclosure, the present disclosure further provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, the steps of the metadata collection method shown in the above embodiment of the present disclosure are implemented.

[0124] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0125] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for collecting metadata, wherein: Executed by a computing node in a distributed system, the method includes: Receive a metadata collection task sent by the server, and acquire metadata of a first data table in at least one data source according to the collection task; Generate a first summary set according to the metadata of the first data table, where the first summary set includes first summary information of the first data table; Acquire a second summary set, where the second summary set includes second summary information of a second data table that has stored metadata, where the second summary information is generated from the metadata of the second data table; According to the first summary set and the second summary set, the collected metadata of the first data table is stored.

2. The method according to claim 1, wherein: The storing the collected metadata of the first data table according to the first summary set and the second summary set includes: determining, according to the first summary set and the second summary set, a target set to which the first data table belongs; At least part of the metadata of the first data table is added to the target set, and the metadata of the first data table is stored in the target database in the form of the target set.

3. The method according to claim 2, wherein: The adding at least part of the metadata of the first data table to the target set includes: Determining metadata addition restriction conditions for the target set; According to the metadata adding restriction condition, determining target metadata to be added of the first data table from the metadata of the first data table; Add target metadata of the first data table to the target set.

4. The method according to claim 2, wherein: The determining, according to the first summary set and the second summary set, the target set to which the first data table belongs includes: Matching the first summary set and the second summary set to obtain first summary information and second summary information belonging to the same first data table; In response to the first summary information and the second summary information of the same first data table being different, it is determined that the same first data table belongs to an update set.

5. The method according to claim 3 or 4, wherein: The method further comprises: Obtain one or more update fields of the data table pre-configured by the update set; Determining metadata associated with the update field from metadata of the same first data table as target metadata of the same first data table; Add the target metadata of the same first data table to the update set.

6. The method according to claim 4, wherein: The method further comprises: In response to the first summary information and the second summary information of the same first data table being the same, it is determined that metadata of the same first data table is repeated, and the first data table is skipped.

7. The method according to claim 2, wherein: The determining, according to the first summary set and the second summary set, a target set to which the metadata of the first data table belongs includes: matching the first summary set and the second summary set to obtain first summary information newly added to the first summary set relative to the second summary set; Determine a target first data table corresponding to the newly added first summary information; It is determined that the metadata of the target first data table belongs to a newly added set.

8. The method according to claim 3 or 7, wherein: The method comprises: Determine the metadata amount of a single data table that is allowed to be added to the newly added collection; determining target metadata of the target first data table from metadata of the target first data table according to the metadata amount; Add the target metadata of the target first data table to the newly added collection.

9. The method according to claim 8, wherein: The step of determining target metadata of the target first data table from metadata of the target first data table according to the metadata amount includes: According to the priority of the metadata, target metadata of the target first data table is determined from the metadata of the target first data table.

10. The method according to claim 3 or 7, wherein: The method comprises: It is determined that each metadata of the target first data table is target metadata of the target first data table, and the target metadata of the target first data table is added to the newly added set.

11. The method according to claim 1, wherein: The method further comprises: Whenever a first data table is collected, the collected first data table is added to the processed first table set; Acquire a second table set generated based on the second data table; Match the first table set and the second table set to obtain data tables that are missing from the first table set relative to the second table set; The metadata of the missing data table is added to the deletion set.

12. The method according to claim 2, wherein: After storing the metadata of the first data table in the target database in the form of the target set, the method further includes: generating third summary information of the first data table according to target metadata of the first data table added in the update set and the newly added set; The second summary set is updated based on the third summary information.

13. The method according to claim 1, wherein: The method further comprises: Monitoring the execution of the collection task based on the agent in the computing node to obtain monitoring data of the collection task; Based on a preset task state transition diagram and the monitoring data, the state of the acquisition task is updated.

14. The method according to claim 7, wherein: The method further comprises: In response to the acquisition task being in an abnormal state, re-executing the acquisition task; and / or, The abnormal state of the acquisition task is sent to the server to instruct the server to reschedule the computing node for the acquisition task.

15. A metadata collection device, applied to a computing node in a distributed system, comprising: A first acquisition module, configured to receive a metadata acquisition task sent by a server, and acquire metadata of a first data table in at least one data source according to the acquisition task; A generating module, configured to generate a first summary set according to the metadata of the first data table, wherein the first summary set includes first summary information of the first data table; A second acquisition module, configured to acquire a second summary set, wherein the second summary set includes second summary information of a second data table in which metadata has been stored, and the second summary information is generated from the metadata of the second data table; A storage module is used to store the collected metadata of the first data table according to the first summary set and the second summary set.

16. The metadata collection device according to claim 15, wherein: The storage module is further used for: determining, according to the first summary set and the second summary set, a target set to which the first data table belongs; At least part of the metadata of the first data table is added to the target set, and the metadata of the first data table is stored in the target database in the form of the target set.

17. The metadata collection device according to claim 16, wherein: The storage module is further used for: Determining metadata addition restriction conditions for the target set; According to the metadata adding restriction condition, determining target metadata to be added of the first data table from the metadata of the first data table; Add target metadata of the first data table to the target set.

18. The metadata collection device according to claim 16, wherein: The storage module is further used for: Matching the first summary set and the second summary set to obtain first summary information and second summary information belonging to the same first data table; In response to the first summary information and the second summary information of the same first data table being different, it is determined that the same first data table belongs to an update set.

19. The metadata collection device according to claim 17 or 18, wherein: The device is also used for: Obtain one or more update fields of the data table pre-configured by the update set; Determining metadata associated with the update field from metadata of the same first data table as target metadata of the same first data table; Add the target metadata of the same first data table to the update set.

20. The metadata collection device according to claim 18, wherein: The device is also used for: In response to the first summary information and the second summary information of the same first data table being the same, it is determined that metadata of the same first data table is repeated, and the first data table is skipped.

21. The metadata collection device according to claim 16, wherein: The storage module is further used for: matching the first summary set and the second summary set to obtain first summary information newly added to the first summary set relative to the second summary set; Determine a target first data table corresponding to the newly added first summary information; It is determined that the metadata of the target first data table belongs to a newly added set.

22. The metadata collection device according to claim 17 or 21, wherein: The device is also used for: Determine the metadata amount of a single data table that is allowed to be added to the newly added collection; determining target metadata of the target first data table from metadata of the target first data table according to the metadata amount; Add the target metadata of the target first data table to the newly added collection.

23. The metadata collection device according to claim 22, wherein: The storage module is further used for: According to the priority of the metadata, target metadata of the target first data table is determined from the metadata of the target first data table.

24. The metadata collection device according to claim 17 or 21, wherein: The device is also used for: It is determined that each metadata of the target first data table is target metadata of the target first data table, and the target metadata of the target first data table is added to the newly added set.

25. The metadata collection device according to claim 15, wherein: The device is also used for: Whenever a first data table is collected, the collected first data table is added to the processed first table set; Acquire a second table set generated based on the second data table; Match the first table set and the second table set to obtain data tables that are missing from the first table set relative to the second table set; The metadata of the missing data table is added to the deletion set.

26. The metadata collection device according to claim 16, wherein: The device is also used for: generating third summary information of the first data table according to target metadata of the first data table added in the update set and the newly added set; The second summary set is updated based on the third summary information.

27. The metadata collection device according to claim 15, wherein: The device is also used for: Monitoring the execution of the collection task based on the agent in the computing node to obtain monitoring data of the collection task; Based on a preset task state transition diagram and the monitoring data, the state of the acquisition task is updated.

28. The metadata collection device according to claim 21, wherein: The device is also used for: In response to the acquisition task being in an abnormal state, re-executing the acquisition task; and / or, The abnormal state of the acquisition task is sent to the server to instruct the server to reschedule the computing node for the acquisition task.

29. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 14.

30. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-14.

31. A computer program product comprising a computer program which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 14.

Images