A metadata capture method, apparatus, system, and medium

Abstract

The application provides a metadata capture method, device, system and medium, the method comprising: controlling a simple storage service to send request information to a distributed message queue; when the distributed message queue returns response information corresponding to the request information, then controlling the distributed message queue to store a metadata information stream sent by the simple storage service; controlling the distributed message queue to send the metadata information stream to a data governance system; and when the data governance system writes the metadata information stream, returning write completion information. Thus, the automatic collection and capture of metadata information can be realized, and after the data governance system writes the metadata information stream, the write completion information is returned, which can prevent the loss of metadata information and improve the reliability of metadata information transmission.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, system and medium for capturing metadata. Background Technology

[0002] Metadata, also known as intermediary data or relay data, is data about data. It mainly describes the properties of data and is used to support functions such as indicating storage location, historical data, resource lookup, and file records.

[0003] In the process of metadata governance, the real-time nature of data is crucial. The traditional approach is to actively crawl metadata information at regular intervals using methods similar to web crawlers, which requires manual operation and cannot meet the real-time requirements. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a metadata capture method, apparatus, system and medium that can realize the automated collection and capture of metadata information, effectively prevent the loss of metadata information and improve the reliability of metadata information transmission.

[0005] To achieve the above objectives, this application provides the following technical solution:

[0006] In a first aspect, embodiments of this application provide a metadata capture method, including:

[0007] Control the simple storage service to send request information to the distributed message queue;

[0008] When the distributed message queue returns the response information corresponding to the request information, the distributed message queue is controlled to store the metadata information stream sent by the simple storage service.

[0009] The distributed message queue is controlled to send the metadata information stream to the data governance system;

[0010] After the data governance system writes the metadata information stream, it returns a write completion message.

[0011] In one possible implementation, before the simple storage service sends the metadata information stream, it further includes:

[0012] Inject the information version number of the metadata information stream into the metadata information stream.

[0013] In one possible implementation, before the data governance system writes the metadata information stream, the following is also included:

[0014] Determine whether version number metadata exists in the data governance system that matches the information version number of the metadata information flow;

[0015] If it exists, then the metadata information stream is rejected from being written into the data governance system.

[0016] In one possible implementation, the metadata information stream includes:

[0017] Data updates, creation, and / or deletion of information.

[0018] Secondly, embodiments of this application provide a metadata capture device, including:

[0019] The request unit is used to control the simple storage service to send request information to the distributed message queue.

[0020] A storage unit is used to control the distributed message queue to store the metadata information stream sent by the simple storage service when the distributed message queue returns the response information corresponding to the request information;

[0021] The sending unit is used to control the distributed message queue to send the metadata information stream to the data governance system;

[0022] The writing unit is used to return a writing completion message after the data governance system writes the metadata information stream.

[0023] One possible implementation also includes:

[0024] An injection unit is used to inject the information version number of the metadata information stream into the metadata information stream.

[0025] One possible implementation also includes:

[0026] The judgment unit is used to determine whether there is version number metadata in the data governance system that is consistent with the information version number of the metadata information flow;

[0027] The rejection unit is used to reject the writing of the metadata information stream into the data governance system when there is version number metadata in the data governance system that matches the information version number of the metadata information stream.

[0028] In one possible implementation, the metadata information flow includes: data update, creation, and / or deletion information.

[0029] Thirdly, embodiments of this application provide a metadata capture system, including:

[0030] Memory, used to store computer programs;

[0031] A processor for implementing the steps of the metadata capture method described above when executing the computer program.

[0032] Fourthly, embodiments of this application provide a computer-readable medium storing a computer program, which, when processed and executed, implements the steps of the metadata capture method described above.

[0033] Compared with the prior art, the embodiments of this application have the following beneficial effects:

[0034] This application provides a metadata capture method, apparatus, system, and medium. The method includes: controlling a simple storage service to send request information to a distributed message queue; when the distributed message queue returns response information corresponding to the request information, controlling the distributed message queue to store the metadata information stream sent by the simple storage service; controlling the distributed message queue to send the metadata information stream to a data governance system; and returning a write completion message after the data governance system writes the metadata information stream. This enables automated collection and capture of metadata information, and by returning a write completion message after the data governance system writes the metadata information stream, it prevents metadata loss and improves the reliability of metadata information transmission. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 A flowchart of a metadata capture method provided in an embodiment of this application is shown;

[0037] Figure 2 A schematic diagram of a metadata capture system provided in an embodiment of this application is shown;

[0038] Figure 3 A schematic diagram of an interceptor provided in an embodiment of this application is shown;

[0039] Figure 4 This illustration shows a schematic diagram of the solution to the idempotency problem on the consumer side provided by an embodiment of this application;

[0040] Figure 5 A schematic diagram of a metadata capture device provided in an embodiment of this application is shown. Detailed Implementation

[0041] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the specific embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0042] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0043] As described in the background section, metadata, also known as intermediary data or relay data, is data about data. It mainly describes the properties of data and is used to support functions such as indicating storage location, historical data, resource lookup, and file records.

[0044] In the process of metadata governance, the real-time nature of data is crucial. The traditional approach is to actively crawl metadata information at regular intervals using methods similar to web crawlers, which requires manual operation and cannot meet the real-time requirements.

[0045] For example, currently, embedding Atlas hooks into various data sources within the Hadoop ecosystem enables real-time metadata extraction. However, S3 (Simple Storage Service), as a product, cannot be intrusively modified; it can only actively crawl information periodically in a web crawler-like manner, resulting in a significant loss of real-time performance.

[0046] In this context, simple storage services are represented by a very large hard drive where users can store and retrieve their data assets. These data assets are called objects, and the storage segment where the objects reside is called a bucket.

[0047] Apache Atlas is an open-source project addressing metadata governance issues within the Hadoop ecosystem, essentially a data governance system. It provides core metadata governance capabilities for Hadoop clusters, including data classification, a centralized policy engine, data lineage, security, and lifecycle management. This enables enterprises to effectively and efficiently meet compliance requirements within Hadoop and allows for integration with the entire enterprise data ecosystem.

[0048] Hadoop is a distributed system infrastructure. Users can develop distributed programs without understanding the underlying details of distributed systems, fully leveraging the power of clusters for high-speed computing and storage.

[0049] To address the aforementioned technical problems, embodiments of this application provide a metadata capture method, apparatus, system, and medium. The method includes: controlling a simple storage service to send request information to a distributed message queue; when the distributed message queue returns response information corresponding to the request information, controlling the distributed message queue to store the metadata information stream sent by the simple storage service; controlling the distributed message queue to send the metadata information stream to a data governance system; and returning a write completion message after the data governance system writes the metadata information stream. This enables automated collection and capture of metadata information, and by returning a write completion message after the data governance system writes the metadata information stream, it prevents metadata loss and improves the reliability of metadata information transmission.

[0050] Exemplary methods

[0051] See Figure 1 The diagram shown is a flowchart of a metadata capture method provided in an embodiment of this application, including:

[0052] S101: Controls the simple storage service to send request information to the distributed message queue.

[0053] S102: When the distributed message queue returns the response information corresponding to the request information, the distributed message queue is controlled to store the metadata information stream sent by the simple storage service.

[0054] In this embodiment of the application, the simple storage service can be controlled to send request information to the distributed message queue.

[0055] For details, see Figure 2 The diagram shown is a schematic of a metadata capture system provided in an embodiment of this application. In the message production stage, the Simple Message Notification (SMN) function of the Simple Storage Service can be used to send messages to specific terminals when certain events occur in the storage bucket, that is, to send request information to the distributed message queue.

[0056] For example, the distributed message queue provided in this application embodiment may include a Kafka queue. Kafka is a high-throughput distributed publish-subscribe messaging system that can process all action stream data of consumers on a website.

[0057] If you receive a confirmation response from Kafka, it means the message was sent successfully. At the same time, you should also handle any abnormal responses properly.

[0058] When the distributed message queue returns the response information corresponding to the request information, it means that the metadata transmission channel is unobstructed, and the flow of metadata information sent by the distributed message queue storage simple storage service can be controlled.

[0059] Specifically, Kafka can be used as a message middleware during the message storage phase to store metadata information streams from objects in S3 buckets, while multiple replicas need to be configured to ensure message reliability.

[0060] Optionally, the metadata information flow provided in this application embodiment may include: data update, create, and / or delete information. It can capture changed data (CDC, Change DataCapture), extracting data simultaneously with changes (insertion / update / deletion) to the source data, allowing the changed data to be captured in real-time or near real-time.

[0061] S103: Control the distributed message queue to send the metadata information stream to the data governance system;

[0062] S104: After the data governance system writes the metadata information stream, it returns a write completion message.

[0063] In this embodiment of the application, a distributed message queue can be controlled to send metadata information streams to the data governance system. After the data governance system writes the metadata information stream, it returns a writing completion message.

[0064] Specifically, for example, the data governance system provided in this application embodiment can be an information consumer Atlas. After receiving a message, the information consumer Atlas should not immediately send a confirmation consumption request, but should execute the relevant storage logic before sending a consumption confirmation to ensure that the message is not lost.

[0065] In one possible implementation, to further prevent message loss during the metadata capture process, the information version number of the metadata stream can be injected into the metadata stream before the simple storage service sends the metadata stream.

[0066] For details, see Figure 3 The above is a schematic diagram of an interceptor provided in an embodiment of this application. Before each event notification is sent by the storage bucket, the interceptor needs to inject the message version number into the message (the interceptor needs to ensure the global uniqueness of the version number). Before the consumer receives the message, the interceptor performs version verification on the ID, which can prevent data loss and facilitate subsequent data verification.

[0067] In one possible implementation, the method provided in this application embodiment may further include determining whether version number metadata with the same information version number as the metadata information stream exists in the data governance system before the data governance system writes the metadata information stream; if it exists, then rejecting the writing of the metadata information stream into the data governance system.

[0068] For details, please refer to Figure 4 As shown, in order to solve the idempotency problem on the consumer side, when a certain metadata information stream fails to be consumed, the system will notify the sender to resend it through a compensation mechanism. This may lead to the problem of duplicate consumption, which needs to be solved by designing to ensure the idempotency of the consumer side (that consuming the same message multiple times will produce the same effect as the first consumption).

[0069] For example, a message log table can be created using Redis (Remote Dictionary Server) or another database to record the consumption status of messages corresponding to each ID. Before Atlas receives data and actually writes it, it checks the message status to prevent the same message from being written multiple times.

[0070] See Figure 4 As shown, when version number 001 has been executed in the message log table, it indicates that version number metadata with the same information version number as the metadata information flow exists in the data governance system, and therefore the metadata information flow is rejected from being written to the data governance system. When version number 002 has not been executed in the message log table, it indicates that version number metadata with the same information version number as the metadata information flow does not exist in the data governance system, and therefore the metadata information flow is written to the data governance system.

[0071] This application provides a metadata capture method, which includes: controlling a simple storage service to send request information to a distributed message queue; when the distributed message queue returns response information corresponding to the request information, controlling the distributed message queue to store the metadata information stream sent by the simple storage service; controlling the distributed message queue to send the metadata information stream to a data governance system; and returning a write completion message after the data governance system writes the metadata information stream. This enables automated collection and capture of metadata information, and returning a write completion message after the data governance system writes the metadata information stream, preventing metadata loss and improving the reliability of metadata information transmission.

[0072] Exemplary device

[0073] See Figure 5 The diagram shown is a schematic representation of a metadata capture device provided in an embodiment of this application, comprising:

[0074] Request unit 201 is used to control the simple storage service to send request information to the distributed message queue;

[0075] Storage unit 202 is used to control the distributed message queue to store the metadata information stream sent by the simple storage service when the distributed message queue returns the response information corresponding to the request information;

[0076] Sending unit 203 is used to control the distributed message queue to send the metadata information stream to the data governance system;

[0077] The writing unit 204 is used to return a writing completion message after the data governance system writes the metadata information stream.

[0078] One possible implementation also includes:

[0079] An injection unit is used to inject the information version number of the metadata information stream into the metadata information stream.

[0080] One possible implementation also includes:

[0081] The judgment unit is used to determine whether there is version number metadata in the data governance system that is consistent with the information version number of the metadata information flow;

[0082] The rejection unit is used to reject the writing of the metadata information stream into the data governance system when there is version number metadata in the data governance system that matches the information version number of the metadata information stream.

[0083] In one possible implementation, the metadata information flow includes: data update, creation, and / or deletion information.

[0084] This application provides a metadata capture device. A method using this device includes: controlling a simple storage service to send request information to a distributed message queue; when the distributed message queue returns response information corresponding to the request information, controlling the distributed message queue to store the metadata information stream sent by the simple storage service; controlling the distributed message queue to send the metadata information stream to a data governance system; and returning a write completion message after the data governance system writes the metadata information stream. This enables automated collection and capture of metadata information, and by returning a write completion message after the data governance system writes the metadata information stream, it prevents metadata loss and improves the reliability of metadata information transmission.

[0085] Based on the above embodiments, this application provides a metadata capture system, including:

[0086] Memory, used to store computer programs;

[0087] A processor for implementing the steps of the metadata capture method described above when executing the computer program.

[0088] Based on the above embodiments, this application also provides a computer-readable medium storing a computer program, which, when processed and executed, implements the steps of the metadata capture method described above.

[0089] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0090] The aforementioned computer-readable medium may be included in the aforementioned system, or it may exist independently and not assembled into the system.

[0091] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts.

[0092] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0093] The above description is merely a preferred embodiment of this application. Although this application has disclosed preferred embodiments above, it is not intended to limit this application. Any person skilled in the art can make many possible variations and modifications to the technical solutions of this application using the methods and techniques disclosed above, or modify them into equivalent embodiments with equivalent changes, without departing from the scope of the technical solutions of this application. Therefore, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of this application without departing from the content of the technical solutions of this application shall still fall within the protection scope of the technical solutions of this application.

Claims

1. A method for capturing metadata, characterized in that, include: On the simple storage service side, an interceptor is used to inject the information version number of the metadata information stream into the metadata information stream, and the information version number has global uniqueness; The simple storage service is controlled to send request information to a distributed message queue when an event occurs in the storage bucket using the event notification function; wherein, the simple storage service is a non-intrusive service used to store user data assets, and the storage bucket is the storage segment where the data assets are located; When the distributed message queue returns the response information corresponding to the request information, the distributed message queue is controlled to store the metadata information stream sent by the simple storage service. The distributed message queue is controlled to send the metadata information stream to the data governance system; Establish a message log table, which records the consumption status of the metadata information stream corresponding to each version number; Before the data governance system writes the metadata information stream, the version number in the metadata information stream is verified by the interceptor; If the version number has been executed in the message log table, then the metadata information stream is rejected from being written to the data governance system; if the version number has not been executed in the message log table, then the metadata information stream is written to the data governance system. After the data governance system writes the metadata information stream, it returns a write completion message.

2. The method according to claim 1, characterized in that, The metadata information flow includes: data update, creation and / or deletion information.

3. A metadata capture device, characterized in that, include: An injection unit is used on the simple storage service side to inject the information version number of the metadata information stream into the metadata information stream using an interceptor. The information version number has global uniqueness. The request unit is used to control the simple storage service to send request information to the distributed message queue when an event occurs in the storage bucket using the event notification function; wherein, the simple storage service is a non-intrusive service used to store user data assets, and the storage bucket is the storage segment where the data assets are located; A storage unit is configured to control the distributed message queue to store the metadata information stream sent by the simple storage service when the distributed message queue returns response information corresponding to the request information; The sending unit is used to control the distributed message queue to send the metadata information stream to the data governance system; The table creation unit is used to create a message log table, which records the consumption status of the metadata information stream corresponding to each version number. The judgment unit is used to verify the version number in the metadata information stream through the interceptor before the data governance system writes the metadata information stream; A rejection unit is used to reject the writing of the metadata information stream into the data governance system when the version number has already been executed in the message log table; A writing unit is used to write the metadata information stream into the data governance system when the version number is not executed in the message log table; The writing unit is also used to return writing completion information after the data governance system writes the metadata information stream.

4. The apparatus according to claim 3, characterized in that, The metadata information flow includes: data update, creation, and / or deletion information.

5. A metadata capture system, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the metadata capture method as described in any one of claims 1-2 when executing the computer program.

6. A computer-readable medium, characterized in that, The computer-readable medium stores a computer program that, when processed and executed, implements the steps of the metadata capture method as described in any one of claims 1-2.

Images