Metadata processing method and apparatus, electronic device, storage medium, and product

Abstract

The application provides a metadata collection method and device, electronic equipment, storage medium and product, the method comprises the following steps: collecting target metadata in a data source, transmitting the target metadata to a target adapter matched with a target application proxy program in an access layer through the target application proxy program, performing data conversion processing on the target metadata by the target adapter to obtain converted target metadata, determining the type of the converted target metadata, and storing the converted target metadata into a corresponding storage medium. The metadata processing method provided by the application can ensure the accuracy of metadata processing, reduce maintenance costs, and improve user experience through the sequential processing of metadata collection, adaptation and storage.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a metadata processing method, apparatus, electronic device, storage medium, and product. Background Technology

[0002] According to the "Data Asset Management Practice White Paper 4.0" released by the China Academy of Information and Communications Technology in June 2019: Data, as an increasingly important factor of production, will become a more core production resource than land, oil, and coal mines. How to process and utilize data, release its value, and achieve digital transformation of enterprises is an important issue facing all enterprises.

[0003] Currently, enterprises face numerous problems in data asset management. These problems hinder the interconnection and efficient utilization of data, becoming bottlenecks that make it difficult to effectively release the value of data. These problems mainly include the following: lack of a unified data view, widespread data silos, low data quality, lack of a secure data environment, and lack of a data value management system.

[0004] In existing technologies, data asset management addresses numerous challenges in unlocking data value, systematically ensuring data availability, usability, and ease of use, thereby achieving significant data benefits with relatively low costs. The entry point for data asset management is a comprehensive data inventory, creating a data map to solidify the foundation for business applications and data acquisition. Metadata, as the core foundation of data asset management, is crucial for achieving a complete understanding of the current state of data assets. However, existing metadata collection systems generally suffer from tight coupling between collection and adaptation, a large number of adapters, and high maintenance costs, resulting in a poor user experience. Summary of the Invention

[0005] This invention provides a metadata processing method, apparatus, electronic device, storage medium, and product to solve the technical problems in the prior art where data acquisition and adaptation are coupled, there are many adapters, and maintenance costs are high, resulting in a poor user experience. This invention aims to achieve the goal of separating data acquisition and adaptation processing, ensuring the accuracy of metadata processing, reducing maintenance costs, and improving user experience.

[0006] In a first aspect, the present invention provides a metadata processing method, comprising:

[0007] Collect target metadata from the data source;

[0008] The target metadata is transmitted to a target adapter that matches the target application proxy through the target application proxy in the access layer. The target adapter performs data transformation processing on the target metadata to obtain the transformed target metadata.

[0009] Determine the type of the transformed target metadata and store the transformed target metadata in the corresponding storage medium.

[0010] Furthermore, according to the metadata processing method provided by the present invention, the target metadata collected from the data source includes:

[0011] When the data source is disconnected from the metadata management product network or has restricted access, the target metadata is collected in real time via message queue.

[0012] When the target data in the data source is in real-time incremental mode, the target metadata is collected in real time through a message queue.

[0013] Furthermore, according to the metadata processing method provided by the present invention, the target metadata collected from the data source further includes:

[0014] When the target metadata is being used in offline batch applications, the target metadata is collected in batches via an application programming interface.

[0015] Furthermore, according to the metadata processing method provided by the present invention, the step of transmitting the target metadata data to a target adapter matching the target application proxy through the target application proxy in the access layer includes:

[0016] The mapping relationship between the hierarchical definition of the target application agent in the access layer and the hierarchical definition of the target adapter is pre-determined based on Atlas technology.

[0017] Based on the mapping relationship, a target adapter corresponding to the target application agent is determined, and the target metadata is transferred from the target application agent to the target adapter.

[0018] Furthermore, according to the metadata processing method provided by the present invention, the target adapter performs data transformation processing on the target metadata to obtain transformed target metadata, including:

[0019] The target metadata is converted using the target adapter to obtain the converted target metadata.

[0020] In a second aspect, the present invention also provides a metadata processing apparatus, comprising:

[0021] The acquisition module is used to collect target metadata from the data source;

[0022] The access and adaptation module is used to transmit the target metadata to a target adapter that matches the target application proxy through the target application proxy in the access layer. The target adapter performs data conversion processing on the target metadata to obtain the converted target metadata.

[0023] The storage module is used to determine the type of the converted target metadata and store the converted target metadata in the corresponding storage medium.

[0024] Furthermore, according to the metadata processing apparatus provided by the present invention, the acquisition module is also used for:

[0025] When the data source is disconnected from the metadata management product network or has restricted access, the target metadata is collected in real time via message queue.

[0026] When the target data in the data source is in real-time incremental mode, the target metadata is collected in real time through a message queue.

[0027] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the metadata processing methods described above.

[0028] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the metadata processing methods described above.

[0029] Fifthly, the present invention also provides a computer program product comprising computer-executable instructions, which, when executed, implement the steps of the metadata processing method as described in any of the preceding claims.

[0030] This invention provides a metadata collection method, apparatus, electronic device, storage medium, and product. The method includes: collecting target metadata from a data source; transmitting the target metadata to a target adapter matched with the target application agent through a target application agent in the access layer; the target adapter performing data transformation processing on the target metadata to obtain transformed target metadata; determining the type of the transformed target metadata; and storing the transformed target metadata in a corresponding storage medium. The metadata processing method provided by this invention, through the sequential processing of metadata collection, adaptation, and storage, can ensure the accuracy of metadata processing, reduce maintenance costs, and improve user experience. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0032] Figure 1 This is a flowchart illustrating the metadata processing method provided by the present invention;

[0033] Figure 2 This is a schematic diagram of the overall process of the metadata processing method provided by the present invention;

[0034] Figure 3 This is a schematic diagram of the overall structure of the metadata processing system provided by the present invention;

[0035] Figure 4 This is a schematic diagram of the metadata processing device provided by the present invention;

[0036] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0038] Figure 1 A flowchart illustrating the metadata processing method provided by this invention is shown below. Figure 1 As shown, the metadata processing method provided by this invention specifically includes the following steps:

[0039] Step 101: Collect target metadata from the data source.

[0040] In this embodiment, target data needs to be collected from the data source. The collection methods are divided into two types: real-time collection via a message queue and offline batch processing of target data via an application programming interface (API). Metadata, also known as intermediary data or relay data, describes data attributes and is equivalent to an electronic catalog. It should be noted that the target data collection method can be selected according to the user's actual needs, and no specific limitations are made here.

[0041] It should be noted that this embodiment uses a predefined metadata model for collecting target metadata. This metadata model is granular to the physical data source level, and its definition depends on the specific data source; user participation is not required in its definition. Any metadata that the data source can technically provide can be included as part of the metadata model, and its subsequent evolution follows the evolution of the data source technology. When collecting metadata, key factors such as the data source's existing capabilities, the deployed network environment, and real-time and batch requirements are considered. Metadata push is accomplished by connecting to a message queue or application programming interface (API).

[0042] Step 102: The target metadata is transmitted to the target adapter that matches the target application proxy through the target application proxy in the access layer. The target adapter performs data conversion processing on the target metadata to obtain the converted target metadata.

[0043] In this embodiment, the target metadata needs to be transmitted to the target adapter that matches the target application proxy through the target application proxy in the access layer. The target adapter performs data conversion processing on the target metadata to obtain the converted target metadata.

[0044] It should be noted that the access layer, as the unified access layer of the metadata collection system, is responsible for the access of metadata pushed by the data source metadata collection application agent, and adheres to the collection time-based metadata model. The target application agent can be a message queue or an application programming interface (API). When real-time collection of target metadata is required, the message queue approach is used; when batch collection of target metadata is required, the API approach is used.

[0045] It should be noted that the access layer is only used to transfer metadata during collection. It does not perform any operations or processing on the target metadata. It simply connects the target metadata from the data source to the corresponding target adapter.

[0046] Step 103: Determine the type of the converted target metadata and store the converted target metadata in the corresponding storage medium.

[0047] In this embodiment, it is necessary to determine the type of target metadata processed by the target adapter, and then store the target data in the corresponding storage medium according to the determined type of target metadata.

[0048] It's worth noting that the storage layer is responsible for persisting the target metadata. Designed based on the JanusGraph graph database provided by Atlas, it primarily implements the conversion from the adaptation time-based metadata model to the storage time-based metadata model, completes the storage of metadata and index data, and supports various functions such as metadata analysis and metadata lookup. Atlas, in particular, is a data governance project open-sourced by the Hadoop community under the Apache License 2.0, designed to address metadata governance issues within the Hadoop ecosystem.

[0049] According to the metadata collection method provided by this invention, target metadata is collected from a data source, and then transmitted to a target adapter matching the target application proxy program in the access layer. The target adapter performs data transformation processing on the target metadata to obtain transformed target metadata. The type of the transformed target metadata is determined, and the transformed target metadata is stored in the corresponding storage medium. The metadata processing method provided by this invention, through the sequential processing of metadata collection, adaptation, and storage, can ensure the accuracy of metadata processing, reduce maintenance costs, and improve user experience.

[0050] Based on any of the above embodiments, in this embodiment, the target metadata in the data source includes:

[0051] When the data source is disconnected from the metadata management product network or has restricted access, the target metadata is collected in real time via message queue.

[0052] When the target data in the data source is in real-time incremental mode, the target metadata is collected in real time through a message queue.

[0053] In this embodiment, the access layer provides a unified metadata access point. This is achieved through a data source metadata collection application proxy, such as a message queue or API (REST / SDK). This embodiment considers factors such as the data source's existing capabilities, the network environment, and real-time and batch processing requirements, combining message queues or APIs to implement the application proxy functionality.

[0054] It should be noted that when the data source and the metadata management product are not connected to the network or access is restricted, the target metadata can be pushed and collected in real time through a message queue. Among them, Apache Kafka is an optional message queue.

[0055] When the target metadata in the data source is in a real-time incremental scenario, the target metadata is also pushed in real time through a message queue, thus realizing the real-time collection of target metadata.

[0056] According to the metadata processing method provided by this invention, when the data source and the metadata management product are not connected to the network or access is restricted, the real-time push and collection of target metadata is achieved through a message queue. Similarly, when the target metadata in the data source is in a real-time incremental scenario, the real-time push of target metadata is also achieved through a message queue. This ensures the timeliness of target metadata collection and improves the efficiency of metadata processing.

[0057] Based on any of the above embodiments, in this embodiment, the target metadata in the data source to be collected further includes:

[0058] When the target metadata is being used in offline batch applications, the target metadata is collected in batches via an application programming interface.

[0059] In this embodiment, when the target metadata is in an offline batch application scenario, the target metadata is scheduled to be pushed in batches through the application programming interface (API), and the target metadata is transmitted to the corresponding target adapter via the API. Here, the application programming interface (API) refers to a set of predefined interfaces (such as functions or HTTP interfaces), or conventions for connecting different components of a software system. These interfaces provide applications and developers with a set of routines that can be accessed by developers based on certain software or hardware, without requiring access to the source code or understanding of the details of the internal working mechanism.

[0060] According to the metadata processing method provided by the present invention, when the target metadata is in an offline batch application scenario, the target metadata is scheduled to be pushed in batches through the application programming interface, and the target metadata is transmitted to the corresponding target adapter through the API, which can ensure the timeliness of target metadata collection and improve the efficiency of metadata processing.

[0061] Based on any of the above embodiments, in this embodiment, the step of transmitting the target metadata to a target adapter matching the target medium via a target application proxy in the access layer includes:

[0062] The mapping relationship between the hierarchical definition of the target application agent in the access layer and the hierarchical definition of the target adapter is pre-determined based on Atlas technology.

[0063] Based on the mapping relationship, a target adapter corresponding to the target application agent is determined, and the target metadata is transferred from the target application agent to the target adapter.

[0064] In this embodiment, it is necessary to pre-determine the mapping relationship between the hierarchical definition of the target application agent and the hierarchical definition of the target adapter in the access layer based on Atlas technology. Then, the target adapter matching the target application agent is determined according to the pre-determined mapping relationship to realize the processing of target metadata. It should be noted that Atlas technology aims to exchange metadata with other tools and processes inside and outside the Hadoop stack, thereby achieving platform-agnostic governance control and effectively meeting compliance requirements.

[0065] It's important to note that Apache Atlas provides scalable governance for metadata-driven enterprise Hadoop. At its core, Atlas facilitates the easy modeling of new business processes and data assets through agility. This flexible typing system allows for the exchange of metadata with other tools and processes both inside and outside the Hadoop stack. Furthermore, Apache Atlas is developed around the following two guiding principles:

[0066] 1) Metadata authenticity in Hadoop

[0067] Atlas provides true visibility within Hadoop. By utilizing native connectors for Hadoop components, Atlas offers technology and operational tracking capabilities rich in business-classified metadata. Atlas allows any metadata user to share a common metadata storage medium, facilitating easier metadata exchange and enhancing interoperability between multiple metadata generators.

[0068] 2) Open development model

[0069] A team of engineers from Aetna, Merck, SAS, Schlumberger, and Target collaborated to ensure that Atlas is specifically designed to solve real-world data governance problems across various industries using Hadoop. This approach serves as a model of innovation within the open-source community, helping to accelerate product maturity and time-to-value for data-driven enterprises.

[0070] It should be noted that Apache Atlas enables enterprises to effectively and efficiently meet their compliance requirements through a suite of scalable core governance services. These services include:

[0071] Data tracing: Capturing data tracing across Hadoop components at the platform level.

[0072] Agile data modeling: The type system allows the use of custom metadata structures in hierarchical classifications.

[0073] REST API: Provides more modern and flexible access to Atlas services, HDP components, UI, and external tools.

[0074] Metadata exchange: Make full use of existing metadata / models by importing them from current tools, and export metadata to downstream systems.

[0075] It should be noted that the target adapter conforms to the adaptation time meta model, and the hierarchical definition of the target adapter is implemented based on the type system provided by Atlas, as shown in Table 1 below.

[0076] Table 1

[0077]

[0078]

[0079] According to the metadata processing method provided by the present invention, the mapping relationship between the hierarchical definition of the target application proxy and the hierarchical definition of the target adapter in the access layer is first determined in advance based on Atlas technology; then, the target adapter corresponding to the target application proxy is determined according to the mapping relationship, and the target metadata is transferred from the target application proxy to the target adapter, thereby improving the efficiency and accuracy of metadata processing and enhancing the user experience.

[0080] Based on any of the above embodiments, in this embodiment, the target adapter performs data transformation processing on the target metadata to obtain transformed target metadata, including:

[0081] The target metadata is converted using the target adapter to obtain the converted target metadata.

[0082] In this embodiment, a defined target adapter is used to transform the target metadata, converting the metadata type from DataSet to SQL Server. This allows for better retrieval and data storage, ultimately ensuring the target metadata is better matched to the target user's retrieval and query services. It should be noted that the target adapter is selected using existing technology, and the specific model of the target adapter can be chosen according to the user's actual needs; no limitation is imposed here.

[0083] According to the metadata processing method provided by the present invention, the target metadata is converted by a target adapter to obtain the converted target metadata, which can improve the efficiency of target metadata processing and enhance the user experience.

[0084] Based on any of the above embodiments, in this embodiment, as Figure 2 As shown, the metadata processing method provided in this embodiment is subdivided into four core process points: data source metadata collection Agent, access layer, adapter, and storage layer. In the access layer, the application agent provides a unified metadata access point. The application agent can be a message queue or an application programming interface (API) (REST / SDK). It should be noted that the data source metadata collection application agent, considering factors such as the data source's existing capabilities, the network environment, and real-time and batch requirements, utilizes an application agent message queue or an application programming interface (API) to implement the Agent function.

[0085] It should be noted that this embodiment proposes three independent meta-models: "Acquisition Time Meta-model," "Adaptation Time Meta-model," and "Storage Time Meta-model." The metadata acquisition process is defined as a three-step process of "Acquisition—Adaptation—Storage," which is analogous to ETL (Extract-Transform-Load) data warehouse technology. Each step has its own function, reducing coupling. ETL describes the process of extracting, transforming, and loading data from the source to the destination.

[0086] 1. The metadata collection time model serves the metadata collection process and is granular to the physical data source. The definition of the metadata collection time model depends on the specific data source and does not require user intervention. Any metadata that the data source can technically provide can be included as part of the metadata model, and its subsequent evolution follows the evolution of the data source technology. During metadata collection, key factors such as the data source's existing capabilities, the deployed network environment, and real-time and batch requirements are considered, and metadata is pushed through message queues or APIs.

[0087] 2. The adaptation time-based meta-model serves the metadata adaptation process and is granular at the logical data source level. The adaptation time-based meta-model is defined independently of specific data sources and is entirely user-defined. It should be noted that the meta-model mentioned in metadata collection, unless otherwise specified, refers to the adaptation time-based meta-model. During metadata adaptation, the adaptation time-based meta-model is matched to the data source type marked in the collection time-based meta-model to complete the metadata adaptation.

[0088] 3. The storage time-based model serves the metadata storage process and is granular at the global level. It is a derivative of the adaptation time-based model, independent of specific data sources, and does not require user definition. During metadata storage, the mapping from the adaptation time-based model to the storage time-based model is completed based on the requirements of the selected metadata storage technology.

[0089] It should be noted that the application of the access layer application proxy program needs to consider the following three situations.

[0090] 1) When the data source and the metadata management product are not connected to the network or access is restricted, the target metadata can be pushed and collected in real time through a message queue. The message queue can be Apache Kafka.

[0091] 2) When the target metadata in the data source is in a real-time incremental scenario, the target metadata is also pushed in real time through a message queue, thus realizing the real-time collection of target metadata.

[0092] 3) When the target metadata is in an offline batch application scenario, the batch push is scheduled through the application programming interface (API) to realize the collection of target metadata.

[0093] Based on any of the above embodiments, this embodiment also provides a metadata management system, such as... Figure 3 The diagram illustrates the overall structure of a metadata management system, an Atlas-based metadata management product. It interfaces with various data sources and serves multiple user types. Internally, it comprises four layers from bottom to top: a collection layer, a storage layer, a functional layer, and a service layer. The system provided in this embodiment is distributed across the collection and storage layers. The collection layer handles unified metadata access and adaptation, while the storage layer handles metadata persistence. The collection layer is further subdivided into two layers: an access layer and an adaptation layer. The access layer provides a unified metadata access point, while the adaptation layer provides a set of adapters to convert metadata from the collection-time metadata model to the adaptation-time metadata model. Adapters support plug-in dynamic integration, allowing users to flexibly define them according to their needs.

[0094] 1. Data Source Metadata Collection Agent: Responsible for collecting metadata from different physical data sources, adhering to the metadata collection model. Based on the Atlas system, two metadata collection methods are provided: message queues and application programming interfaces (APIs). Each metadata collection application agent considers key factors such as the data source's existing capabilities, the deployment network environment, and real-time and batch requirements, combining message queues or APIs to implement agent functionality.

[0095] 2. The access layer, serving as the unified access layer for the metadata collection system, is responsible for receiving metadata pushed by the data source metadata collection agent, adhering to the metadata model during collection. The system completes metadata access based on message queues or APIs; the access process only involves moving the metadata during collection, without any transformation.

[0096] 3. The adaptation layer is responsible for metadata transformation during data collection, adhering to the adaptation-time metamodel. Based on the type system hierarchy provided by Atlas, the system provides the definition and management of the adaptation-time metamodel. The adaptation-time metamodel is independent of the data source; the data source is the finest granularity of the definition, and higher granularity metamodels can be defined to cover multiple data sources.

[0097] 4. Storage Layer: Responsible for metadata persistence, adhering to the storage time-based metadata model. Based on the JanusGraph graph database provided by Atlas, the system internally implements the conversion from the adaptation time-based model to the storage time-based metadata model, completing the storage of metadata and index data, and supporting functions such as metadata analysis and metadata lookup.

[0098] In this embodiment, when it is determined that the target metadata after the target adapter conversion is technical metadata, the target metadata is stored in the corresponding storage medium; when it is determined that the target metadata after the target adapter conversion is business metadata, the target metadata is stored in the corresponding storage medium, wherein the storage medium is a medium for storing business metadata, which refers to metadata of the business type handled by the user, such as the handling of a passbook or a bank card.

[0099] When it is determined that the target metadata after the target adapter conversion is management metadata, the target metadata is stored in the corresponding storage medium. This storage medium is specifically used for storing management metadata. Specifically, it can be Redis or a message queue. The appropriate storage medium can be selected according to the user's actual needs, and there is no limitation here.

[0100] Figure 4 The present invention provides a metadata processing device, such as... Figure 4 As shown, the metadata processing apparatus provided by the present invention includes:

[0101] The acquisition module is used to collect target metadata from the data source;

[0102] The access and adaptation module is used to transmit the target metadata to a target adapter that matches the target application proxy through the target application proxy in the access layer. The target adapter performs data conversion processing on the target metadata to obtain the converted target metadata.

[0103] The storage module is used to determine the type of the converted target metadata and store the converted target metadata in the corresponding storage medium.

[0104] According to the metadata processing apparatus provided by this invention, target metadata is collected from a data source, and the target metadata is transmitted to a target adapter matching the target application proxy program through a target application proxy program in the access layer. The target adapter performs data transformation processing on the target metadata to obtain transformed target metadata, determines the type of the transformed target metadata, and stores the transformed target metadata in the corresponding storage medium. The metadata processing apparatus provided by this invention, through the sequential processing of metadata collection, adaptation, and storage, can ensure the accuracy of metadata processing, reduce maintenance costs, and improve user experience.

[0105] Since the device described in this embodiment of the invention is based on the same principle as the method described in the above embodiments, more detailed explanations will not be repeated here.

[0106] Figure 5 This is a schematic diagram of the physical structure of the electronic device provided in the embodiments of the present invention, such as... Figure 5 As shown, the present invention provides an electronic device, including: a processor 501, a memory 502, and a bus 503;

[0107] The processor 501 and the memory 502 communicate with each other via the bus 503.

[0108] The processor 501 is used to call program instructions in the memory 502 to execute the methods provided in the above method embodiments, such as: collecting target metadata from a data source; transmitting the target metadata to a target adapter that matches the target application agent through a target application agent in the access layer, wherein the target adapter performs data conversion processing on the target metadata to obtain converted target metadata; determining the type of the converted target metadata; and storing the converted target metadata in a corresponding storage medium.

[0109] Furthermore, the logical instructions in the aforementioned memory 503 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0110] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the methods provided by the above methods, the method comprising: collecting target metadata from a data source; transmitting the target metadata to a target adapter matching the target application agent through a target application agent in an access layer, the target adapter performing data conversion processing on the target metadata to obtain converted target metadata; determining the type of the converted target metadata; and storing the converted target metadata in a corresponding storage medium.

[0111] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods provided above, the method comprising: acquiring target metadata from a data source; transmitting the target metadata to a target adapter matched with the target application agent through a target application agent in an access layer, the target adapter performing data conversion processing on the target metadata to obtain converted target metadata; determining the type of the converted target metadata; and storing the converted target metadata in a corresponding storage medium.

[0112] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0113] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A metadata processing method, characterized in that, include: Collect target metadata from the data source; The target metadata is transmitted to a target adapter that matches the target application proxy through the target application proxy in the access layer. The target adapter performs data transformation processing on the target metadata to obtain the transformed target metadata. Determine the type of the converted target metadata and store the converted target metadata in the corresponding storage medium; The step of transmitting the target metadata to a target adapter that matches the target application proxy via the target application proxy in the access layer includes: The mapping relationship between the hierarchical definition of the target application agent in the access layer and the hierarchical definition of the target adapter is pre-determined based on Atlas technology. Based on the mapping relationship, a target adapter corresponding to the target application agent is determined, and the target metadata is transferred from the target application agent to the target adapter; The method uses three independent meta-models: the acquisition meta-model, the adaptation meta-model, and the storage meta-model; The acquisition time-based metadata model serves the metadata acquisition process and is geared towards the physical data source granularity. The adaptation time-based model serves the metadata adaptation process, is oriented towards the logical data source granularity, and is independent of specific data sources; During metadata adaptation, the metadata adaptation is completed by matching and adapting the metadata model according to the data source type marked in the data collection time-metamodel. The storage time-meta model serves the metadata storage stage, is oriented towards global granularity, and is a derivative of the adaptation time-meta model, independent of specific data sources; When storing the metadata, the mapping from the adaptation time-meta model to the storage time-meta model is completed.

2. The metadata processing method according to claim 1, characterized in that, The target metadata in the data source includes: When the data source is disconnected from the metadata management product network or has restricted access, the target metadata is collected in real time via message queue. When the target data in the data source is in real-time incremental mode, the target metadata is collected in real time through a message queue.

3. The metadata processing method according to claim 1, characterized in that, The target metadata in the data source also includes: When the target metadata is being used in offline batch applications, the target metadata is collected in batches via an application programming interface.

4. The metadata processing method according to claim 1, characterized in that, The target adapter performs data transformation processing on the target metadata to obtain transformed target metadata, including: The target metadata is converted using the target adapter to obtain the converted target metadata.

5. A metadata processing apparatus, characterized in that, include: The acquisition module is used to collect target metadata from the data source; The access and adaptation module is used to transmit the target metadata to a target adapter that matches the target application proxy through the target application proxy in the access layer. The target adapter performs data conversion processing on the target metadata to obtain the converted target metadata. The storage module is used to determine the type of the converted target metadata and store the converted target metadata in the corresponding storage medium. The step of transmitting the target metadata to a target adapter that matches the target application proxy via the target application proxy in the access layer includes: The mapping relationship between the hierarchical definition of the target application agent in the access layer and the hierarchical definition of the target adapter is pre-determined based on Atlas technology. Based on the mapping relationship, a target adapter corresponding to the target application agent is determined, and the target metadata is transferred from the target application agent to the target adapter; The device uses three independent meta-models: the acquisition meta-model, the adaptation meta-model, and the storage meta-model; The acquisition time-based metadata model serves the metadata acquisition process and is geared towards the physical data source granularity. The adaptation time-based model serves the metadata adaptation process, is oriented towards the logical data source granularity, and is independent of specific data sources; During metadata adaptation, the metadata adaptation is completed by matching and adapting the metadata model according to the data source type marked in the data collection time-metamodel. The storage time-meta model serves the metadata storage stage, is oriented towards global granularity, and is a derivative of the adaptation time-meta model, independent of specific data sources; When storing the metadata, the mapping from the adaptation time-meta model to the storage time-meta model is completed.

6. The metadata processing apparatus according to claim 5, characterized in that, The acquisition module is also used for: When the data source is disconnected from the metadata management product network or has restricted access, the target metadata is collected in real time via message queue. When the target data in the data source is in real-time incremental mode, the target metadata is collected in real time through a message queue.

7. An electronic device, characterized in that, include: Processor, memory, and bus, among which, The processor and the memory communicate with each other via the bus; The memory stores program instructions that can be executed by the processor, which can invoke the program instructions to perform the steps of the metadata processing method as described in any one of claims 1 to 4.

8. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions that cause the computer to perform the steps of the metadata processing method as described in any one of claims 1 to 4.

9. A computer program product, said computer program product comprising computer-executable instructions, characterized in that, When executed, the instructions are used to implement the steps of the metadata processing method as described in any one of claims 1 to 4.

Images