Metadata acquisition method and device, electronic equipment and storage medium

Abstract

The invention discloses a metadata collection method and device, electronic equipment and a storage medium, and relates to the technical field of big data or other related technical fields, and the method comprises the steps: determining a to-be-collected heterogeneous data source; deploying an adapter to connect different heterogeneous data sources, and performing offline collection on metadata in each heterogeneous data source through the adapter; the method comprises the following steps: identifying a heterogeneous data source with frequent data updating, integrating an incremental metadata acquisition component for the heterogeneous data source with frequent data updating, monitoring data change in the heterogeneous data source through the incremental metadata acquisition component, and acquiring incremental metadata for the changed data; and performing format conversion on the acquired metadata and incremental metadata according to a predefined metadata format to obtain converted metadata information, and storing the converted metadata in a metadatabase. According to the method and the device, the technical problem that the format of the metadata collected from the heterogeneous data source is not uniform in the prior art is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data or other related technical fields, in particular, to a metadata collection method and device, electronic equipment and storage medium. BACKGROUND

[0002] Under the background of digital transformation today, enterprises are increasingly dependent on data, and the diversity and complexity of data have also increased. Data is no longer limited to traditional structured databases, but is widely distributed in relational databases, non-relational databases, file systems, API interfaces, log records and other heterogeneous data sources. These data sources have their own characteristics, such as relational databases are good at transaction processing and structured data storage, non-relational databases are more suitable for large-scale, high-speed and high-variety of unstructured data storage, and file systems are often used to store massive file data.

[0003] With the growth of data volume and the increase of data sources, metadata management has become a core link of enterprise data governance. Metadata, as data describing data, contains various attributes of data, such as data structure, content, format, quality, etc., which is crucial for understanding data meaning, tracking data bloodline and ensuring data quality.

[0004] In related technologies, when collecting metadata from heterogeneous data sources, different types of data sources need to use different collection technologies and customize different collection tools, not only the collection cost is high, but also the collected metadata has the problem of non-uniform format, which not only limits the effective use of internal data of enterprises, but also affects the information sharing between enterprises.

[0005] In view of the above problems, no effective solutions have been proposed so far. SUMMARY

[0006] The embodiments of the present application provide a metadata collection method and device, electronic equipment and storage medium, to at least solve the technical problem of non-uniform format of metadata collected from heterogeneous data sources in related technologies.

[0007] According to an aspect of the embodiments of the present application, a metadata collection method is provided, comprising: determining a collection range of metadata to obtain a heterogeneous data source to be collected; deploying an adapter to connect different heterogeneous data sources, and collecting metadata in each of the heterogeneous data sources offline through the adapter according to a metadata collection task; identifying a heterogeneous data source with frequent data updates, integrating an incremental metadata collection component for the heterogeneous data source with frequent data updates, monitoring data changes in the heterogeneous data source through the incremental metadata collection component, and collecting incremental metadata of the changed data, wherein the heterogeneous data source with frequent data updates is determined based on a data update frequency in the heterogeneous data source; and converting the collected metadata and the incremental metadata into a pre-defined metadata format to obtain converted metadata information, and storing the converted metadata in a metadata database.

[0008] Further, the step of deploying an adapter to connect different heterogeneous data sources comprises: in a case where data stored in the heterogeneous data source is structured data, deploying an application programming interface as an adapter to connect the heterogeneous data source; and in a case where data stored in the heterogeneous data source is unstructured data or semi-structured data, integrating an artificial intelligence algorithm into the adapter according to a data type, and connecting the heterogeneous data source through the adapter with the integrated artificial intelligence algorithm, wherein the artificial intelligence algorithm comprises at least one of an image recognition algorithm and a natural language processing algorithm.

[0009] Further, the step of collecting metadata in a heterogeneous data source offline through an adapter according to a metadata collection task comprises: receiving a metadata collection task defined by a user end, and extracting a target heterogeneous data source, a collection time, and a metadata collection priority from the metadata collection task; connecting the target heterogeneous data source based on an adapter corresponding to the target heterogeneous data source, and collecting the metadata from the heterogeneous data source based on the collection time and the collection priority.

[0010] Further, the step of identifying a heterogeneous data source with frequent data updates comprises: monitoring the heterogeneous data source to obtain a data update frequency in the heterogeneous data source; and in a case where the data update frequency of the heterogeneous data source is greater than a preset frequency threshold, determining the heterogeneous data source as a heterogeneous data source with frequent data updates.

[0011] Further, the step of monitoring data changes in the heterogeneous data source by the incremental metadata collection component and collecting incremental metadata of changed data comprises: monitoring the heterogeneous data source by the incremental metadata collection component to identify a data change event in the heterogeneous data source; collecting a running log of the heterogeneous data source, and verifying the data change event based on the running log; determining changed data and collecting incremental metadata corresponding to the changed data when the data change event passes the verification.

[0012] Further, the step of converting the collected metadata and the incremental metadata into a pre-defined metadata format comprises: data cleaning of the collected metadata and the incremental metadata, wherein the data cleaning comprises at least one of the following: data deduplication, data standardization; and converting the cleaned metadata and the incremental metadata into a unified format according to the pre-defined metadata format.

[0013] Further, the metadata collection method further comprises: configuring a data transmission interface for each of the heterogeneous data sources; connecting a meta model of the heterogeneous data source through the data transmission interface, and pushing metadata in the meta model to the meta database through the data transmission interface, wherein the meta model is used for managing metadata.

[0014] According to another aspect of the embodiment of the present application, a metadata collection device is also provided, comprising: a determination unit configured to determine a collection range of metadata to obtain heterogeneous data sources to be collected; a collection unit configured to deploy an adapter to connect different heterogeneous data sources, and collect metadata in each of the heterogeneous data sources offline through the adapter according to a metadata collection task; an integration unit configured to identify a heterogeneous data source with frequent data updates, integrate an incremental metadata collection component for the heterogeneous data source with frequent data updates, monitor data changes in the heterogeneous data source through the incremental metadata collection component, and collect incremental metadata of changed data, wherein the heterogeneous data source with frequent data updates is determined based on a data update frequency in the heterogeneous data source; and a storage unit configured to convert the collected metadata and the incremental metadata into a pre-defined metadata format to obtain converted metadata information, and store the converted metadata into a meta database.

[0015] Further, the collection unit comprises: a first deployment module, configured to, in a case where the data stored in the heterogeneous data source is structured data, deploy an application program interface as an adapter to connect the heterogeneous data source; a first integration module, configured to, in a case where the data stored in the heterogeneous data source is unstructured data or semi-structured data, integrate an artificial intelligence algorithm according to a data type into the adapter, and connect the heterogeneous data source through the adapter integrated with the artificial intelligence algorithm, wherein the artificial intelligence algorithm comprises at least one of: an image recognition algorithm, a natural language processing algorithm.

[0016] Further, the collection unit further comprises: a first extraction module, configured to receive a metadata collection task defined by a user end, and extract a target heterogeneous data source, a collection time, and a metadata collection priority from the metadata collection task; and a first collection module, configured to connect the target heterogeneous data source based on an adapter corresponding to the target heterogeneous data source, and collect the metadata from the heterogeneous data source based on the collection time and the collection priority.

[0017] Further, the integration unit comprises: a first monitoring module, configured to monitor the heterogeneous data source, and acquire a data update frequency in the heterogeneous data source; and a first determination module, configured to, in a case where the data update frequency of the heterogeneous data source is greater than a preset frequency threshold, determine that the heterogeneous data source is a frequently-updated heterogeneous data source.

[0018] Further, the integration unit further comprises: a first identification module, configured to monitor the heterogeneous data source through the incremental metadata collection component, and identify a data change event in the heterogeneous data source; a first verification module, configured to collect a running log of the heterogeneous data source, and perform authenticity verification on the data change event based on the running log; and a second collection module, configured to, in a case where the data change event passes the authenticity verification, determine changed data, and collect incremental metadata corresponding to the changed data.

[0019] Further, the storage unit comprises: a first cleaning module, configured to perform data cleaning on the collected metadata and the incremental metadata, wherein the data cleaning comprises at least one of: data deduplication, data standardization; and a first conversion module, configured to convert the cleaned metadata and the incremental metadata into a unified format according to a pre-defined metadata format.

[0020] Further, the metadata collection apparatus further comprises a first configuration module configured to configure a data transmission interface for each of the heterogeneous data sources; and a first connection module configured to connect a meta-model of the heterogeneous data sources through the data transmission interface, and push metadata in the meta-model to the meta-database through the data transmission interface, wherein the meta-model is used to manage the metadata.

[0021] According to a further aspect of the embodiments of the present application, a computer readable storage medium is provided, which includes a stored computer program, wherein the computer readable storage medium controls a device where the computer readable storage medium is located to perform any of the above metadata collection methods when the computer program is executed.

[0022] According to a further aspect of the embodiments of the present application, an electronic device is provided, which includes one or more processors and a memory, the memory being configured to store one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement any of the above metadata collection methods.

[0023] According to a further aspect of the embodiments of the present application, a computer program product is provided, which includes a computer program, wherein the computer program, when executed by a processor, implements any of the above metadata collection methods.

[0024] In the present application, the collection range of metadata is determined, the heterogeneous data sources to be collected are obtained, the adapters are deployed to connect different heterogeneous data sources, the metadata in each of the heterogeneous data sources is collected offline by the adapters according to a metadata collection task, the heterogeneous data source with frequently updated data is identified, the incremental metadata collection component is integrated for the heterogeneous data source with frequently updated data, the data change in the heterogeneous data source is monitored by the incremental metadata collection component, and the incremental metadata of the changed data is collected, wherein the heterogeneous data source with frequently updated data is determined based on the data update frequency in the heterogeneous data source, finally, the collected metadata and the incremental metadata are converted into the converted metadata information according to a pre-defined metadata format, and the converted metadata is stored into the meta-database.

[0025] In the present application, the metadata from multiple heterogeneous data sources is unified by deploying multiple adapters to connect the heterogeneous data sources, collecting metadata from the heterogeneous data sources, and converting the metadata information into a unified format according to a predefined metadata format, thereby improving the data governance level and the efficiency and quality of metadata management. Meanwhile, in order to cope with the heterogeneous data sources with frequent data updates, an incremental metadata collection component is integrated to monitor the change events of the data sources in real time, collect and process the changed data, thereby avoiding repeated collection and resource waste, and ensuring the timeliness and accuracy of the metadata. Further, the technical problem of the metadata collected from the heterogeneous data sources being in different formats is solved. BRIEF DESCRIPTION OF DRAWINGS

[0026] The drawings described herein are used to provide a further understanding of the present application, form a part of the present application, and the illustrative embodiments of the present application and their descriptions serve to explain the present application, and do not constitute an improper limitation on the present application. In the drawings:

[0027] Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing the metadata collection method is shown;

[0028] Figure 2 A flowchart of an optional metadata collection method according to an embodiment of the present application is shown;

[0029] Figure 3 A schematic diagram of an optional metadata collection process according to an embodiment of the present application is shown;

[0030] Figure 4 A schematic diagram of an optional metadata collection device according to an embodiment of the present application is shown;

[0031] Figure 5 A hardware structure block diagram of an electronic device (or mobile device) for executing the metadata collection method according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of the present application.

[0033] It should be noted that the terms "first", "second", and the like in the description and in the claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] To facilitate the understanding of the present application for those skilled in the art, the following explanations are made for some terms or names involved in the embodiments of the present application:

[0035] JDBC adapter, Java Database Connectivity, a database connection technology, aims to provide a unified interface, so that developers can use code to write database applications, without having to worry about the specific implementation details of the underlying database.

[0036] SQL, full name Structured Query Language, is a standard programming language for managing and operating databases.

[0037] It should be noted that the metadata acquisition method and device in the present application can be used in the field of big data technology when collecting metadata, and can also be used in any field other than the field of big data technology when collecting metadata. The application field of the metadata acquisition method and device in the present application is not limited.

[0038] It should be noted that the information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for display, analyzed data, etc.) involved in the present application are information and data authorized by the user or authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of related data comply with relevant laws, regulations and standards, necessary security measures are taken, do not violate public order and good customs, and provide corresponding operation portal for user to choose authorization or refusal. For example, the system and the interface between the related users or institutions provide the user with a corresponding operation portal for the user to choose to agree or refuse the automatic decision result; if the user chooses to refuse, the expert decision process is entered.

[0039] The following embodiments of the present application can be applied to various metadata collection systems / applications / devices. The present application automatically identifies and collects metadata in various types of data sources by configuring adapters for heterogeneous data sources, including but not limited to relational databases, non-relational databases, file systems, API interfaces, etc. At the same time, the collected data can be format-converted and standardized, so that the metadata format is unified and centralized management is facilitated.

[0040] The present application will be described in detail below in conjunction with various embodiments.

[0041] Embodiment One

[0042] According to an embodiment of the present application, an embodiment of a metadata collection method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown.

[0043] The method embodiment provided by Embodiment One of the present application can be executed in a mobile terminal, a computer terminal, or a similar computing device. Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing the metadata collection method is shown. As shown in Figure 1 , the computer terminal 10 (or mobile device) can include one or more processors 102 (the processor 102 can include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it can also include a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which can be included as one of the ports in the BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art can understand that Figure 1 The structure shown is only schematic, and does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 10 can include more or fewer components than those shown in Figure 1 , or have a different configuration than that shown in Figure 1 .

[0044] It should be noted that the one or more processors 102 and / or other data processing circuitry described above can be referred to herein generally as "data processing circuitry." The data processing circuitry can be embodied in whole or in part as software, hardware, firmware, or any combination thereof. Furthermore, the data processing circuitry can be a single standalone processing module, or incorporated in whole or in part within any of the other elements of the computer terminal 10 (or mobile device). As referred to in embodiments of the present application, the data processing circuitry functions as a processor to control, for example, the selection of the variable resistance terminal path in connection with the interface.

[0045] The memory 104 can be used to store software programs of application software and modules, such as program instructions / data storage means corresponding to the metadata acquisition method in embodiments of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, i.e., implements the metadata acquisition method described above. The memory 104 can include a high-speed random access memory, and can further include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 can further include a memory disposed remotely with respect to the processor 102, which can be connected to the computer terminal 10 through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0046] The transmission device 106 is used to receive or send data via a network. Specific examples of the network can include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet in a wireless manner.

[0047] The display can be, for example, a touch screen type liquid crystal display (LCD), which can enable a user to interact with the user interface of the computer terminal 10 (or mobile device).

[0048] In the above operating environment, the present application provides a metadata acquisition method as shown in Figure 2 The implementation subject of the method is a metadata acquisition system.

[0049] Figure 2 is a flowchart of an optional metadata acquisition method according to an embodiment of the present application, as shown in Figure 2 The method includes the following steps:

[0050] Metadata collection involves extracting metadata from various data sources such as production systems, IT design platforms, and more. After necessary conversion processing, these data are imported into the metadata center. This process is an integral part of data management and data governance, aiming to identify, collect, and record key metadata information, thereby achieving a deep understanding and effective management of data resources. As a continuous task, metadata collection needs to be constantly updated and optimized to adapt to changing data needs and environments. It plays a crucial role in supporting data management, data governance, and data analysis.

[0051] Step S201, determine the collection range of metadata, and obtain the heterogeneous data sources to be collected.

[0052] Before starting the metadata collection work, the target metadata and its support for business operations and decision-making processes need to be clearly defined. Specifically, it involves defining the breadth and depth of metadata collection, including the production systems and their data sources that must be involved, i.e., determining the heterogeneous data sources to be collected, which include but are not limited to: application code, relational databases, distributed file storage systems, object storage facilities, and data warehouses, etc. diverse and different structured data sources.

[0053] Step S202, deploy adapters to connect different heterogeneous data sources, and collect metadata in each heterogeneous data source through adapters according to the metadata collection task.

[0054] In the above step S202, according to the type of each heterogeneous data source, such as relational database, non-relational database, file system, data warehouse, etc., select and deploy the corresponding adapter. For example, for structured data, a basic relational database adapter can be used to access the metadata table of the data source through standard SQL query statements; for unstructured or semi-structured data, artificial intelligence technologies such as image recognition and natural language processing can be combined to build a special metadata collection adapter.

[0055] After deploying the adapter, metadata collection tasks can be defined through a visual interface or configuration file. Task definition includes but is not limited to the selection of target heterogeneous data sources, the type of metadata collected, the collection frequency, and the priority setting. The task scheduling system will schedule the adapter to execute the collection task according to the preset collection time and priority, ensuring the automation and efficiency of metadata collection.

[0056] After receiving the collection instruction, the adapter will collect metadata offline according to the characteristics of the data source. The collection process involves extracting table structure, field information, data type, index, constraint and other key metadata from the data source. The collected metadata is then cleaned and converted by the standard processing module of the adapter to ensure that its format is compatible with the unified metadata model, and finally the converted metadata information is imported into the metadata database.

[0057] Through the intelligent connection and task scheduling of the adapter, offline collection of metadata of multiple heterogeneous data sources is realized, ensuring the automation, standardization and efficiency of the collection process.

[0058] Further, the step of deploying the adapter to connect different heterogeneous data sources includes: in the case that the data stored in the heterogeneous data source is structured data, deploying an application programming interface as an adapter to connect the heterogeneous data source; in the case that the data stored in the heterogeneous data source is unstructured data or semi-structured data, integrating an artificial intelligence algorithm into the adapter according to the data type, and connecting the heterogeneous data source through the adapter integrated with the artificial intelligence algorithm, wherein the artificial intelligence algorithm includes at least one of the following: image recognition algorithm, natural language processing algorithm.

[0059] Specifically, when the heterogeneous data source stores structured data, such as in common relational databases, an application programming interface (API) can be selected as an adapter to connect and collect metadata. The design of the API adapter can be based on JDBC (Java Database Connectivity) or other database connection standards. JDBC provides a general interface for interacting with databases, allowing metadata information about table structure, field definition, relationship, constraint, etc. to be extracted by executing SQL queries. The advantage of this adapter is that it can access the data source in a standardized and efficient manner, reducing the coding workload caused by differences in database types, while also ensuring the quality of metadata collection.

[0060] For heterogeneous data sources that store unstructured or semi-structured data, such as documents, pictures, videos or log texts, the traditional metadata collection method of relational databases is no longer applicable. Artificial intelligence algorithms can be integrated into the adapter to achieve intelligent analysis and metadata collection of such complex data. Specifically, the adapter can integrate image recognition algorithms and natural language processing algorithms and other artificial intelligence algorithms according to the type of data stored. By integrating the above algorithms into the adapter, unstructured or semi-structured data can be intelligently understood and parsed, and key metadata features can be automatically identified and extracted without manually writing complex parsing logic, greatly improving the efficiency and accuracy of metadata collection.

[0061] Further, the step of collecting metadata in the heterogeneous data source offline by the adapter according to the metadata collection task comprises: receiving a metadata collection task defined by a user end, and extracting a target heterogeneous data source, a collection time and a metadata collection priority from the metadata collection task; connecting the target heterogeneous data source based on an adapter corresponding to the target heterogeneous data source, and collecting metadata from the heterogeneous data source based on the collection time and the collection priority.

[0062] Specifically, most of the metadata collection is in an offline collection manner. First, the system receives a metadata collection task from a user end. These tasks are usually submitted in the form of a visual interface or a configuration file, which contains key information for collection, such as the type and location of the target heterogeneous data source, the specific time of collection (such as periodic time, such as once a week, once a day, etc.), and the priority of the task (for handling task scheduling conflicts). The receiving function ensures that the user's requirements can be accurately captured by the system, and provides basic data for subsequent collection and scheduling. The system parses the metadata collection task submitted by the user, and extracts key parameters such as the target heterogeneous data source, the collection time and the metadata collection priority from the task. According to the type of the target heterogeneous data source, the system automatically selects and deploys the corresponding adapter to establish a connection with the data source. For example, for a relational database, the system will start the JDBC adapter; and for unstructured or semi-structured data, an adapter integrating AI algorithms such as image recognition or natural language processing is called.

[0063] After the adapter establishes a connection with the data source to be collected, it starts to execute metadata collection according to the collection time and priority settings. If the collection time is set to be periodic, the adapter will automatically run at the preset time interval without human intervention. The setting of the collection priority helps to prioritize important or urgent collection tasks in the case of limited resources. The adapter extracts metadata from the data source, such as table structure, field information, data type, index, constraint, etc. for structured data, or image content, text theme, keyword, etc. for unstructured / semi-structured data, by executing queries or algorithm analysis.

[0064] The metadata collection process realizes high automation and intelligence, improves the collection efficiency and accuracy. The collection task defined by the user end can be accurately parsed by the system, and the selection and connection process of the adapter is automatically completed without the need for manual writing of specific collection scripts. At the same time, based on the flexible setting of the collection time and priority, the system can intelligently schedule the collection task, even in the case of resource shortage, it can also prioritize the collection of high-priority or time-sensitive metadata, to meet the diversified data governance needs of enterprises.

[0065] In step S203, the heterogeneous data source with frequent data updates is identified, and an incremental metadata collection component is integrated for the heterogeneous data source with frequent data updates, the incremental metadata collection component is used to monitor the data changes in the heterogeneous data source, and incremental metadata is collected for the changed data.

[0066] In step S203, the heterogeneous data source with frequent data updates is identified, and an incremental metadata collection component is integrated for the heterogeneous data source with frequent data updates, the incremental metadata collection component is used to monitor the data changes in the heterogeneous data source, and incremental metadata is collected for the changed data.

[0067] After the incremental metadata collection component is integrated into the heterogeneous data source with frequent data updates, the component actively monitors the data change events in the data source. When the data changes, such as adjustment of table structure, addition, deletion or modification of field, the component immediately responds, collects the metadata related to the changed data, and transmits the metadata to the unified metadata database for storage. This ensures that the data in the metadata database is always up-to-date, and full collection is not required, thereby saving a lot of time and processing capacity.

[0068] By intelligently monitoring the data update frequency and directionally integrating the incremental collection component, the real-time performance and accuracy of metadata collection are improved, the resource consumption of metadata collection is greatly reduced, and the overall data collection efficiency is improved.

[0069] Further, the step of identifying the heterogeneous data source with frequent data updates includes: monitoring the heterogeneous data source to obtain the data update frequency in the heterogeneous data source; and determining that the heterogeneous data source is a heterogeneous data source with frequent data updates when the data update frequency of the heterogeneous data source is greater than a preset frequency threshold.

[0070] Specifically, in identifying the heterogeneous data with frequent data updates, first, the system monitors each heterogeneous data source in real time or periodically to obtain the update frequency of the internal data of the data source. The monitoring process can be achieved by deploying a lightweight monitoring probe on the data source or using the change log function provided by the data source. The system continuously records the update records of each data source, including the modification time of the data, the modification type (such as insertion, update, and deletion operations), and the data objects (such as tables and fields) involved. According to the obtained update records, the system calculates the data update frequency of each heterogeneous data source, which is usually measured by the average number of updates per unit time (such as per hour or per day). Then, the calculated update frequency is compared with the preset frequency threshold. The preset frequency threshold is defined in the system design stage and is used to distinguish whether the data update is frequent. The setting of this threshold needs to consider factors such as enterprise data environment, data change requirements, and system processing capacity to balance the collection efficiency and system resource consumption. If the update frequency of a certain heterogeneous data source exceeds the preset threshold, the system will automatically mark the data source as a heterogeneous data source with frequent data updates. This identification result will be used for subsequent incremental metadata collection decisions, that is, the system will start the incremental collection component for this type of data source instead of full collection, thereby ensuring real-time update of metadata and efficient use of system resources.

[0071] Further, the step of monitoring the data changes in the heterogeneous data source and collecting incremental metadata of the changed data by the incremental metadata collection component includes: monitoring the heterogeneous data source by the incremental metadata collection component to identify data change events in the heterogeneous data source; collecting the running log of the heterogeneous data source, and verifying the authenticity of the data change events based on the running log; and determining the changed data and collecting the incremental metadata corresponding to the changed data if the data change events pass the authenticity verification.

[0072] Specifically, the incremental metadata collection component first monitors the identified heterogeneous data source with frequent data updates in real time. This monitoring function is based on an event listening mechanism and can capture any data change event in the data source, such as modification of table structure, change of field definition, data insertion or deletion, etc. Through close integration with the data source, the component can quickly respond to these events and lay the foundation for subsequent incremental metadata collection. In order to verify the authenticity of the data change events, the system will also collect the running log of the heterogeneous data source. The running log contains detailed operation records of the data source, including but not limited to operation timestamp, operation type, data objects involved, and operation user, etc. By comparing the data change events with the running log, the system can verify the validity and integrity of the events, eliminate any possible false positives or abnormal situations, and ensure that the incremental collection of metadata is based on real change events.

[0073] If the data change event passes the authenticity check, the system will determine the specific changed data and start the collection process of the incremental metadata corresponding to the changed data. This includes metadata information of the data object extracted from the change event, such as table name, field name, data type, data permission, etc., as well as the comparison difference of the metadata before and after the change. The incremental metadata collection component only needs to focus on these change points, without the need to reacquire the metadata of the entire data source, thereby significantly reducing the data processing amount and collection time, and improving the response speed and resource utilization of the system.

[0074] In step S204, the collected metadata and incremental metadata are format-converted according to the pre-defined metadata format to obtain converted metadata information, and the converted metadata is stored in the metadata database.

[0075] In the above step S204, the collected metadata and incremental metadata come from multiple heterogeneous data sources, and the metadata format of each data source is different. In order to uniformly manage these metadata, the system needs to perform format conversion to convert them into a pre-defined metadata format. The pre-defined metadata format can cover all key metadata attributes of the data sources, such as data type, data ownership, data quality rules, data flow, etc., to eliminate the format differences between the data sources and enable the metadata to be stored and accessed in a unified manner.

[0076] After format conversion, the uniformly formatted metadata information is stored in the metadata database. The metadata database can provide efficient data query, data indexing, and data permission control services. Storing the converted metadata in the metadata database not only facilitates subsequent metadata management, query, and analysis, but also ensures the security and consistency of the metadata.

[0077] Further, the step of format-converting the collected metadata and incremental metadata according to the pre-defined metadata format includes data cleaning of the collected metadata and incremental metadata, wherein the data cleaning includes at least one of the following: data deduplication, data standardization; and converting the cleaned metadata and incremental metadata into a unified format according to the pre-defined metadata format.

[0078] Specifically, after metadata collection, the collected metadata is first cleaned to ensure data quality. This process aims to eliminate or correct various data problems that may arise during collection, including but not limited to data deduplication and data standardization. Data deduplication ensures that the same metadata information is not repeatedly stored, avoiding waste of storage resources and reducing query efficiency; data standardization focuses on unifying the expression form and data type of metadata to eliminate differences in the expression of the same information by different data sources. The cleaned metadata and incremental metadata will then be converted to a unified format according to the pre-defined metadata format. This format conversion step is crucial because it ensures that metadata from different data sources can be integrated and stored in a consistent format. During the conversion process, the system maps the cleaned metadata to a unified metadata model, which pre-defines the structure, field type, and data relationship of the metadata, allowing the metadata to be presented in a standardized manner.

[0079] Through format conversion, even metadata with completely different structures and formats, such as metadata from a relational database and metadata from an unstructured file system, can be managed and utilized under a unified framework, greatly improving the interoperability and accessibility of metadata.

[0080] Further, the metadata collection method further comprises: configuring a data transmission interface for each heterogeneous data source; connecting the meta-model of the heterogeneous data source through the data transmission interface, and pushing the metadata in the meta-model to the meta-database through the data transmission interface, wherein the meta-model is used to manage the metadata.

[0081] Specifically, to ensure that metadata from different heterogeneous data sources can be successfully transmitted to the meta-database, a specific data transmission interface needs to be configured for each data source. The role of this interface is to establish a communication bridge between the data source and the meta-database, allowing the metadata in the meta-model to be actively transmitted in a specified format and protocol. The configuration of the data transmission interface needs to take into account the characteristics of the data source, such as its data structure, transmission rate, and security requirements, to ensure that the data transmission process is both efficient and secure. After configuring the data transmission interface, the system can push the metadata in the meta-model of the heterogeneous data source to the meta-database in real time or on demand through the interface. The meta-model, as a management framework for metadata, contains information such as the definition, classification, and relationship of metadata, and is an abstract representation of the data source metadata. Through the data transmission interface, the meta-model can interact with the meta-database and seamlessly deliver the latest metadata information to the latter.

[0082] Through the above steps, the collection range of the metadata is determined, the heterogeneous data source to be collected is obtained, the adapter is deployed to connect different heterogeneous data sources, the metadata in each heterogeneous data source is collected offline by the adapter according to the metadata collection task, then the heterogeneous data source with frequent data updates is identified, the incremental metadata collection component is integrated for the heterogeneous data source with frequent data updates, the data changes in the heterogeneous data source are monitored through the incremental metadata collection component, and the incremental metadata of the changed data is collected, wherein the heterogeneous data source with frequent data updates is determined based on the data update frequency in the heterogeneous data source, and finally the collected metadata and incremental metadata are format-converted according to the pre-defined metadata format to obtain the converted metadata information, and the converted metadata is stored in the metadata database.

[0083] In the embodiment, the metadata is collected from the heterogeneous data sources by deploying multiple adapters to connect the heterogeneous data sources, and is converted into metadata information in a unified format according to the pre-defined metadata format, thereby realizing the unification of metadata from multiple heterogeneous data sources, improving the data governance level, and improving the efficiency and quality of metadata management. Meanwhile, in order to cope with the heterogeneous data source with frequent data updates, the incremental metadata collection component is integrated to monitor the change events of the data source in real time, collect and process the changed data, thereby avoiding repeated collection and resource waste, and ensuring the timeliness and accuracy of the metadata. Further, the technical problem of the metadata collected from the heterogeneous data sources in the related art being in a non-uniform format is solved.

[0084] The following will be described in detail in combination with another optional specific embodiment.

[0085] Figure 3 is a schematic diagram of an optional metadata collection process according to an embodiment of the application, as shown in Figure 3 the metadata information of the relevant data is collected from the heterogeneous data sources (the heterogeneous data sources are schematically shown as data source 1, data source 2, …, and data source N) Figure 3 in the form of offline collection, real-time collection, API collection, and manual uploading, and is stored in the metadata database after being format-converted into a unified format. Specifically,

[0086] The offline collection collects the metadata through various adapters connected to the heterogeneous data sources. Different adapters are configured for different data source systems and different types of metadata. For structured metadata, such as the metadata of a relational database, the metadata table of the database is queried using SQL through the JDBC adapter connected to the metadata of the data source. For semi-structured and unstructured metadata, image recognition, natural language processing, and other artificial intelligence technologies are needed to build professional metadata collection adapters to identify and collect the metadata.

[0087] A visual interface is provided to facilitate user definition of metadata collection tasks, while supporting real-time monitoring of the running state of task instances. In addition, the metadata collection system also has a collection task directory management function, which facilitates effective organization and management of tasks. For offline collection tasks, the task scheduling module can be set to achieve efficient and automatic operation at regular intervals.

[0088] When offline collection cannot meet the timeliness requirements of metadata management, real-time collection should be used, and only incremental metadata should be collected. Real-time collection is usually completed with the help of an integrated incremental metadata collection component, which can capture changes in metadata according to the characteristics of different data sources and achieve real-time synchronization of metadata when changes occur.

[0089] In order to ensure efficient management and full utilization of metadata, in addition to automatically extracting metadata from source systems, the metadata database also needs to provide a flexible data transmission interface. This interface can support source systems actively pushing metadata to the metadata database. Through the data transmission interface, the source system can define a dedicated meta model according to the characteristics of its own data source. Once the source system defines the meta model, it can register the model through the data transmission interface. After successful registration, metadata can be collected through interface calls, thereby achieving effective management and utilization of metadata.

[0090] For special scenarios that cannot be collected by automated tools, such as when business metadata is not standardized during system construction, it is often difficult to accurately interpret the business logic contained in tables, views, stored procedures, and data structures, even if technical metadata from the business system is imported into the metadata database using a metadata adapter. In such cases, manual means must be used to supplement and improve the business metadata in existing data, thereby achieving unified metadata management. For manually collected data, use the manual upload module for data transmission. The manual upload module provides a standard metadata upload template, which can be filled in and uploaded according to the template.

[0091] The collected data is cleaned, parsed, and converted into a standard format. The standard format supports customization and provides uploading. The standardized data is stored in a unified metadata database.

[0092] The embodiments of the present application automatically identify and collect metadata in various types of data sources, including but not limited to relational databases, non-relational databases, file systems, API interfaces, etc., by configuring adapters for heterogeneous data sources. At the same time, these collected data can be format-converted and standardized, so that the metadata format is unified and easy to manage centrally.

[0093] Another embodiment will be described in detail below.

[0094] Embodiment Two

[0095] The metadata collection device provided in this embodiment comprises a plurality of implementation units, each of which corresponds to each implementation step in Embodiment One described above, and the specific implementation and beneficial effects thereof can refer to the foregoing method embodiment, which will not be described here again.

[0096] Figure 4 is a schematic diagram of an optional metadata collection device according to an embodiment of the present application, as shown in Figure 4 The metadata collection device can comprise a determination unit 41, a collection unit 42, an integration unit 43, and a storage unit 44, wherein

[0097] The determination unit 41 is configured to determine the collection range of metadata, and obtain a heterogeneous data source to be collected.

[0098] The collection unit 42 is configured to deploy an adapter to connect different heterogeneous data sources, and collect metadata in each heterogeneous data source offline through the adapter according to a metadata collection task.

[0099] The integration unit 43 is configured to identify a frequently-updated heterogeneous data source, integrate an incremental metadata collection component for the frequently-updated heterogeneous data source, monitor data changes in the frequently-updated heterogeneous data source through the incremental metadata collection component, and collect incremental metadata of the changed data, wherein the frequently-updated heterogeneous data source is determined based on a data update frequency in the heterogeneous data source.

[0100] The storage unit 44 is configured to convert the collected metadata and the incremental metadata into a pre-defined metadata format, obtain converted metadata information, and store the converted metadata into a metadata database.

[0101] The metadata collection device determines the collection range of metadata through the determination unit 41, obtains a heterogeneous data source to be collected, deploys an adapter to connect different heterogeneous data sources through the collection unit 42, collects metadata in each heterogeneous data source offline through the adapter according to a metadata collection task, identifies a frequently-updated heterogeneous data source through the integration unit 43, integrates an incremental metadata collection component for the frequently-updated heterogeneous data source, monitors data changes in the frequently-updated heterogeneous data source through the incremental metadata collection component, and collects incremental metadata of the changed data, wherein the frequently-updated heterogeneous data source is determined based on a data update frequency in the heterogeneous data source, and the storage unit 44 converts the collected metadata and the incremental metadata into a pre-defined metadata format, obtains converted metadata information, and stores the converted metadata into a metadata database.

[0102] In this embodiment, by deploying multiple adapters to connect heterogeneous data sources, metadata is collected from the heterogeneous data sources, and the metadata is converted into unified format metadata information according to the pre-defined metadata format, thereby realizing the unification of metadata from multiple heterogeneous data sources, improving the data governance level, and improving the efficiency and quality of metadata management. At the same time, in order to cope with the frequently updated heterogeneous data sources, an incremental metadata collection component is integrated to monitor the change events of the data source in real time, collect and process the changed data, thereby avoiding repeated collection and resource waste, and ensuring the timeliness and accuracy of the metadata. Further, the technical problem of the metadata collected from the heterogeneous data sources in the related art is solved.

[0103] Further, the collection unit comprises: a first deployment module configured to, in a case where the data stored in the heterogeneous data source is structured data, deploy an application program interface as an adapter to connect the heterogeneous data source; and a first integration module configured to, in a case where the data stored in the heterogeneous data source is unstructured data or semi-structured data, integrate an artificial intelligence algorithm into the adapter according to the data type, and connect the heterogeneous data source through the adapter integrated with the artificial intelligence algorithm, wherein the artificial intelligence algorithm comprises at least one of an image recognition algorithm and a natural language processing algorithm.

[0104] Further, the collection unit further comprises: a first extraction module configured to receive a metadata collection task defined by a user end, and extract a target heterogeneous data source, a collection time, and a metadata collection priority from the metadata collection task; and a first collection module configured to connect the target heterogeneous data source based on an adapter corresponding to the target heterogeneous data source, and collect metadata from the heterogeneous data source based on the collection time and the collection priority.

[0105] Further, the integration unit comprises: a first monitoring module configured to monitor the heterogeneous data source to obtain a data update frequency in the heterogeneous data source; and a first determination module configured to determine that the heterogeneous data source is a frequently updated heterogeneous data source in a case where the data update frequency of the heterogeneous data source is greater than a preset frequency threshold.

[0106] Further, the integration unit further comprises: a first identification module configured to monitor the heterogeneous data source through an incremental metadata collection component to identify a data change event in the heterogeneous data source; a first verification module configured to collect a running log of the heterogeneous data source, and perform authenticity verification on the data change event based on the running log; and a second collection module configured to determine changed data in a case where the data change event passes the authenticity verification, and collect incremental metadata corresponding to the changed data.

[0107] Furthermore, the storage unit includes: a first cleaning module for cleaning the collected metadata and incremental metadata, wherein the data cleaning includes at least one of the following: data deduplication and data standardization; and a first conversion module for converting the cleaned metadata and incremental metadata into a unified format according to a predefined metadata format.

[0108] Furthermore, the metadata acquisition device also includes: a first configuration module for configuring data transmission interfaces for each heterogeneous data source; and a first connection module for connecting to the metamodel of the heterogeneous data source through the data transmission interface, and pushing the metadata in the metamodel to the metadatabase through the data transmission interface, wherein the metamodel is used to manage the metadata.

[0109] It should be noted that the aforementioned determining unit 41, acquisition unit 42, integration unit 43, and storage unit 44 correspond to steps S201 to S204 in Embodiment 1. The instances and application scenarios implemented by the aforementioned units and corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should be noted that the aforementioned modules or units may be hardware or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The aforementioned modules or units may also be part of a device and can run in the computer terminal 10 provided in Embodiment 1.

[0110] The invention will now be described in conjunction with another alternative embodiment.

[0111] Example 3

[0112] The present invention can also provide an electronic device. Figure 5 This is a hardware structure block diagram of an electronic device (or mobile device) for performing an optional metadata collection method according to an embodiment of the present invention, such as... Figure 5 As shown, the electronic device may include: one or more ( Figure 5 (Only one is shown) processor 502, memory 504, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0113] The memory can be used to store software programs and modules, such as program instructions / modules corresponding to the method and device in the embodiments of the present application. The processor executes various functions and data processing by running the software programs and modules stored in the memory, that is, implements the above method. The memory can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory can further include a memory remotely arranged with respect to the processor, which can be connected to the terminal through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0114] The processor can call information and applications stored in the memory through the transmission device to perform the following steps: determining the collection range of the metadata to obtain the heterogeneous data source to be collected; deploying an adapter to connect different heterogeneous data sources, and collecting metadata in each heterogeneous data source offline through the adapter according to the metadata collection task; identifying a heterogeneous data source with frequent data updates, integrating an incremental metadata collection component for the heterogeneous data source with frequent data updates, monitoring data changes in the heterogeneous data source through the incremental metadata collection component, and collecting incremental metadata for the changed data, wherein the heterogeneous data source with frequent data updates is determined based on the data update frequency in the heterogeneous data source; and converting the collected metadata and incremental metadata into a pre-defined metadata format to obtain converted metadata information, and storing the converted metadata into a metadata database.

[0115] The processor can call information and applications stored in the memory through the transmission device to perform the following steps: in the case that the data stored in the heterogeneous data source is structured data, deploying an application programming interface as an adapter to connect the heterogeneous data source; in the case that the data stored in the heterogeneous data source is unstructured data or semi-structured data, integrating an artificial intelligence algorithm into the adapter according to the data type, and connecting the heterogeneous data source through the adapter integrated with the artificial intelligence algorithm, wherein the artificial intelligence algorithm includes at least one of the following: an image recognition algorithm, a natural language processing algorithm.

[0116] The processor can call information and applications stored in the memory through the transmission device to perform the following steps: receiving a metadata collection task defined by a user terminal, and extracting a target heterogeneous data source, a collection time, and a metadata collection priority from the metadata collection task; connecting the target heterogeneous data source based on an adapter corresponding to the target heterogeneous data source, and collecting metadata from the heterogeneous data source based on the collection time and the collection priority.

[0117] The processor can call information and application programs stored in the memory through the transmission device to perform the following steps: monitoring the heterogeneous data source, obtaining the data update frequency in the heterogeneous data source; in the case that the data update frequency of the heterogeneous data source is greater than the preset frequency threshold, determining that the heterogeneous data source is a data update frequent heterogeneous data source.

[0118] The processor can call information and application programs stored in the memory through the transmission device to perform the following steps: monitoring the heterogeneous data source through the incremental metadata acquisition component, identifying the data change event in the heterogeneous data source; acquiring the running log of the heterogeneous data source, and verifying the authenticity of the data change event based on the running log; in the case that the data change event passes the authenticity verification, determining the changed data and acquiring the incremental metadata corresponding to the changed data.

[0119] The processor can call information and application programs stored in the memory through the transmission device to perform the following steps: performing data cleaning on the acquired metadata and incremental metadata, wherein the data cleaning includes at least one of the following: data deduplication, data standardization; converting the cleaned metadata and incremental metadata into a unified format according to a pre-defined metadata format.

[0120] The processor can call information and application programs stored in the memory through the transmission device to perform the following steps: configuring a data transmission interface for each heterogeneous data source; connecting the meta-model of the heterogeneous data source through the data transmission interface, and pushing the metadata in the meta-model to the meta-database through the data transmission interface, wherein the meta-model is used for managing the metadata.

[0121] By deploying multiple adapters to connect the heterogeneous data sources, the embodiment of the present application provides a metadata acquisition scheme. The metadata is acquired from the heterogeneous data sources, and the metadata information in a unified format is converted according to a pre-defined metadata format, thereby realizing the unification of the metadata from the multiple-source heterogeneous data sources, improving the data governance level, and improving the efficiency and quality of the metadata management. Meanwhile, in order to cope with the data update frequent heterogeneous data sources, the incremental metadata acquisition component is integrated, the change event of the data source is monitored in real time, and the changed data is acquired and processed, thereby avoiding repeated acquisition and resource waste, and ensuring the timeliness and accuracy of the metadata. Further, the technical problem that the metadata acquired from the heterogeneous data sources is not in a unified format in the related art is solved.

[0122] Those skilled in the art can understand that, Figure 5 The structure shown is only schematic, and the electronic device can also be a terminal device such as a smart phone, a tablet computer, a palm computer, a mobile Internet device (MID), a PAD, and the like. Figure 5It does not cause limitation to the structure of the electronic device. For example, the electronic device can further include more or less components (such as a network interface, a display device, etc.) or have a different configuration from that shown in the drawings. Figure 5 Figure 5

[0123] Those skilled in the art can understand that all or part of the steps in the above-mentioned various methods of the embodiments can be completed by instructing the terminal device related hardware through a program, and the program can be stored in a computer readable storage medium, which can include a flash disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0124] The application will be described in detail below in combination with another alternative embodiment.

[0125] Embodiment Four

[0126] The embodiments of the application further provide a computer readable storage medium. Optionally, in the embodiments of the application, the computer readable storage medium can be used to save the program code executed by the metadata acquisition method provided in the embodiment one.

[0127] Optionally, in the embodiments of the application, the storage medium can be located in any one of the computer terminals in the computer terminal group in the computer network, or in any one of the mobile terminals in the mobile terminal group.

[0128] The embodiments of the application further provide a computer program product, when executed on a data processing device, is suitable for executing the steps of the metadata acquisition method: determining the acquisition range of metadata, obtaining the heterogeneous data source to be acquired; deploying an adapter to connect different heterogeneous data sources, and performing offline acquisition of metadata in each heterogeneous data source through the adapter according to the metadata acquisition task; identifying the heterogeneous data source with frequent data updates, integrating an incremental metadata acquisition component for the heterogeneous data source with frequent data updates, monitoring the data changes in the heterogeneous data source through the incremental metadata acquisition component, and acquiring incremental metadata for the changed data, wherein the heterogeneous data source with frequent data updates is determined based on the data update frequency in the heterogeneous data source; performing format conversion on the acquired metadata and the incremental metadata according to a pre-defined metadata format, obtaining the converted metadata information, and storing the converted metadata into a metadata database.

[0129] The above-mentioned serial numbers of the embodiments of the application are only for description, and do not represent the advantages or disadvantages of the embodiments.

[0130] ​​In the above-mentioned embodiments of the present application, the description of each embodiment is focused on, and the part not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0131] In several embodiments provided in the present application, it should be understood that the disclosed technical contents can be implemented by other ways. Among them, the above-mentioned device embodiments are only schematic, for example, the division of the units can be a logical function division, and in actual implementation, there can be another division way, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or modules shown or discussed can be indirect coupling or communication connection through some interfaces, and can be electrical or other forms.

[0132] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed to a plurality of units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0133] In addition, each functional unit in each embodiment of the present application can be integrated in a processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The above integrated unit can be realized in the form of hardware or in the form of software functional unit.

[0134] The integrated unit, if realized in the form of software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part of the prior art or the whole or part of the technical solutions can be embodied in the form of software product, and the computer software product stored in a storage medium includes a plurality of instructions for making a computer device (which can be a personal computer, a server or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and various program code storage media.

[0135] The above-mentioned is only the preferred embodiment of the present application, and it should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should be regarded as the protection scope of the present application.

Claims

1. A method for collecting metadata, characterized in that, include: Determine the scope of metadata collection to obtain the heterogeneous data sources to be collected; Deploy adapters to connect to different heterogeneous data sources, and collect metadata from each heterogeneous data source offline through the adapters according to the metadata collection task; Identify heterogeneous data sources with frequent data updates, integrate an incremental metadata collection component for the heterogeneous data sources with frequent data updates, monitor data changes within the heterogeneous data sources through the incremental metadata collection component, and collect incremental metadata for the changed data. The heterogeneous data sources with frequent updates are determined based on the data update frequency within the heterogeneous data sources. The collected metadata and incremental metadata are converted according to a predefined metadata format to obtain converted metadata information, and the converted metadata is stored in the metadata database.

2. The method according to claim 1, characterized in that, The steps for deploying an adapter to connect to different heterogeneous data sources include: When the data stored in the heterogeneous data source is structured data, an application programming interface (API) is deployed as an adapter to connect to the heterogeneous data source. When the data stored in the heterogeneous data source is unstructured or semi-structured data, an artificial intelligence algorithm is integrated into the adapter according to the data type, and the heterogeneous data source is connected through the adapter that integrates the artificial intelligence algorithm. The artificial intelligence algorithm includes at least one of the following: image recognition algorithm and natural language processing algorithm.

3. The method according to claim 1, characterized in that, The steps for offline collection of metadata from heterogeneous data sources using the adapter according to the metadata collection task include: Receive metadata collection tasks defined by the user, and extract the target heterogeneous data source, collection time, and metadata collection priority from the metadata collection tasks; Connect to the target heterogeneous data source using the adapter corresponding to the target heterogeneous data source, and collect the metadata from the heterogeneous data source based on the collection time and the collection priority.

4. The method according to claim 1, characterized in that, The steps to identify heterogeneous data sources that are frequently updated include: Monitor the heterogeneous data source and obtain the data update frequency within the heterogeneous data source; If the data update frequency of the heterogeneous data source is greater than a preset frequency threshold, the heterogeneous data source is determined to be a heterogeneous data source with frequent data updates.

5. The method according to claim 1, characterized in that, The steps of monitoring data changes within the heterogeneous data source using the incremental metadata collection component and collecting incremental metadata for the changed data include: The incremental metadata collection component monitors the heterogeneous data source and identifies data change events within the heterogeneous data source. Collect the operation logs of the heterogeneous data source, and verify the authenticity of the data change events based on the operation logs; If the data change event passes the authenticity verification, the changed data is determined, and the incremental metadata corresponding to the changed data is collected.

6. The method according to claim 1, characterized in that, The steps of converting the collected metadata and incremental metadata into a predefined metadata format include: The collected metadata and incremental metadata are cleaned, wherein the data cleaning includes at least one of the following: data deduplication and data standardization; The cleaned metadata and the incremental metadata are converted into a unified format according to the predefined metadata format.

7. The method according to claim 1, characterized in that, Also includes: Configure data transmission interfaces for each of the heterogeneous data sources; The metadata in the heterogeneous data source is connected through the data transmission interface, and the metadata in the metadata is pushed to the metadata database through the data transmission interface. The metadata is used to manage the metadata.

8. A metadata acquisition device, characterized in that, include: The determination unit is used to determine the scope of metadata collection and obtain the heterogeneous data sources to be collected; The acquisition unit is used to deploy adapters to connect to different heterogeneous data sources, and to collect metadata from each heterogeneous data source offline through the adapters according to the metadata acquisition task. An integration unit is used to identify heterogeneous data sources that are frequently updated, integrate an incremental metadata collection component for the heterogeneous data sources that are frequently updated, monitor data changes within the heterogeneous data sources through the incremental metadata collection component, and collect incremental metadata on the changed data. The heterogeneous data sources that are frequently updated are determined based on the data update frequency within the heterogeneous data sources. The storage unit is used to convert the collected metadata and the incremental metadata according to a predefined metadata format to obtain the converted metadata information, and to store the converted metadata in the metadata database.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device containing the computer-readable storage medium to perform the metadata acquisition method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the metadata acquisition method according to any one of claims 1 to 7.

11. A computer program product, characterized in that, The computer program product includes a computer program, wherein the computer program, when executed by a processor, implements the metadata acquisition method according to any one of claims 1 to 7.

Images