Method, system and storage medium for managing metadata

Abstract

Some embodiments of the present invention also provide a system for data synchronization, including: a data acquisition agent module, a message middleware, a data consumption landing module and a metadata registration center. The data acquisition agent module can generate metadata according to the table structure of the source database table and register it to the registration center, wherein the generated metadata includes an identifier and a version number. The data consumption landing module can parse the identifier and version number of the metadata from the consumed data, and query the registration center for metadata corresponding to the parsed identifier and version number. The present invention can manage multiple versions of business data and hot update metadata configuration during real-time data synchronization.

Description

Technical Field

[0001] The present disclosure relates to the field of big data technology, and in particular to a data synchronization device, method and storage medium. Background Art

[0002] With the development of big data technology, more and more systems require real-time management. For example, data warehouse technology has also developed from traditional offline data warehouses to real-time data warehouses. Real-time data warehouses are built on traditional offline data warehouses, while adding more data source import storage, more diversified data processing methods and timeliness, and supporting T+0 days timeliness, that is, data real-time. Data real-time means that data from generation (updated to business databases or logs) to final consumption (data reports, dashboards, analysis, mining, data applications, etc.) supports milliseconds / seconds / minutes of delay (strictly speaking, seconds / minutes belong to quasi-real-time, here collectively referred to as real-time). Data real-time involves how to extract data from data sources in real time, how to transfer and connect to data centers in real time, how to store data in real time, and how to provide subsequent consumption in real time. Data real-time is also called real-time data synchronization. Real-time data synchronization usually refers to end-to-end synchronization from multiple sources to multiple targets, and is one of the most important components of real-time data warehouses.

[0003] Data synchronization often involves the use of metadata of database tables. As is well known in the art, the metadata of tables of many different databases is also called schema, which describes the organization and structure of database tables; for example, the three currently popular relational databases MySQL, SQL Server, and Oracle all have their own defined schemas. In order to achieve real-time data synchronization, message middleware (usually a distributed message queue system, such as the commonly used ActiveMQ, RabbitMQ, RocketMQ, and Kafka such as the Kafka system) is usually required. However, message middleware usually does not have metadata management capabilities, let alone multi-version management of metadata. If the synchronized data changes during the period, the current real-time synchronization task must be stopped and the task configuration must be reissued, which will inevitably affect the real-time performance of the data, increase operation and maintenance operations, and there is also the risk of data loss.

[0004] There are some existing technologies for implementing real-time data synchronization, but these technologies cannot achieve multi-version management of metadata, nor support hot updates of metadata configurations. Summary of the invention

[0005] Some embodiments of the present invention provide a method for managing metadata of a database, comprising: obtaining an identifier and a version number of metadata of a source database; attempting to obtain metadata corresponding to the identifier and the version number; and generating metadata of the source database if the corresponding metadata content cannot be successfully obtained.

[0006] Some embodiments of the present invention also provide a method for managing metadata of a database, comprising: consuming data from a message middleware; parsing an identifier and a version number of the metadata from the data; attempting to obtain metadata corresponding to the identifier and the version number; and if the metadata corresponding to the identifier and the version number cannot be obtained, querying and obtaining the metadata corresponding to the identifier and the version number from a registration center.

[0007] Some embodiments of the present invention also provide a system for data synchronization, including: a data acquisition agent module, for acquiring data from a source database;

[0008] A message middleware for receiving data collected from a source database; a data consumption landing module for consuming data from the message middleware; and a registration center for storing metadata and providing query services for metadata, wherein the data collection agent module can generate metadata according to the table structure of the source database table and register it to the registration center, wherein the generated metadata includes an identifier and a version number, and wherein the data consumption landing module can parse the identifier and version number of the metadata from the consumed data, and query the registration center for metadata corresponding to the parsed identifier and version number.

[0009] The technical solution provided by the embodiment of the present disclosure can manage multiple versions of business data in the process of real-time data synchronization. The technical solution provided by the embodiment of the present disclosure can also realize hot update of metadata configuration.

[0010] It is to be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0012] Figure 1 is a block diagram of a data synchronization system according to an exemplary embodiment;

[0013] Figure 2 is a block diagram of a data synchronization device according to an exemplary embodiment;

[0014] Figure 3 is an example of Schema metadata according to an exemplary embodiment;

[0015] Figure 4 is a schematic diagram showing a data protocol format according to an exemplary embodiment;

[0016] Figure 5 The figure is a flow chart of a data synchronization method according to an exemplary embodiment. DETAILED DESCRIPTION

[0017] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0018] It should be understood that the "several" mentioned in this article refers to one or more, and "multiple" refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0019] Figure 1 is a block diagram of a data synchronization system according to an exemplary embodiment. The system may be a real-time data warehouse. For a real-time data warehouse, Figure 1 The multiple modules or processes in the figure are known in the prior art and will not be described in detail here. The data in the relational database at the data source end is sent to the message middleware through the data bus module. The streaming processing module shown in the figure can manage metadata. At the consumer end, each computing engine can store, index, cache and other operations on the data. After the computing service module calculates the data, it can be displayed through the visualization application module.

[0020] Figure 2 This is a schematic diagram of a data synchronization device according to an exemplary embodiment. The device mainly includes two modules: a unified task scheduling module and a data access module. Through the unified task scheduling module, users can start, stop, and manage real-time data synchronization tasks with a visual interface. The data access module may include a task receiving module, a data acquisition agent module, a data receiving module, a data consumption landing module (connector), a message middleware, a sink destination source, and a registration center module.

[0021] After receiving the task sent by the unified task scheduling module, the task receiving module encapsulates the task into a specific protocol and sends it to the data collection agent module and the data consumption landing module respectively. The task receiving module can also report the overall status of the real-time data synchronization task to the unified task scheduling module.

[0022] The data acquisition agent module reads data from a relational database and sends it to a data receiving module. Common relational databases include MySQL, Oracle, DB2, Microsoft SQL Server, Microsoft Access, etc. The data acquisition agent module includes a serialization unit, which is used to serialize metadata, wherein the specific operation of serializing metadata can adopt existing technologies.

[0023] The data receiving module receives the data sent by the data acquisition agent module and writes the data into the message middleware. The message middleware can publish / subscribe message queues on a large scale according to the distributed transaction log architecture.

[0024] The data consumption landing module consumes data from the message middleware and writes the data to the sink destination source. The sink destination source is the end storage data source for real-time data synchronization, which can be Hive, Kudu, jdbc, etc. The data consumption landing module also includes a deserialization unit, which is used to deserialize metadata. The deserialization operation is the inverse operation of the aforementioned serialization operation.

[0025] The registration center module is responsible for managing (adding, modifying, deleting) the metadata information corresponding to each "topic" of the message middleware.

[0026] Figure 3 It is various information of a schema metadata shown according to an exemplary embodiment. At the implementation level, the metadata information can be stored in a JSON string with a fixed format that describes the names and types of multiple fields. In the exemplary information shown, schemaId is the identifier of the metadata. And the metadata has three versions v1, v2 and v3, and the specific content displayed is all of version v3. The format of the aforementioned schema information provides a basis for the embodiments of the present invention to implement multi-version management of metadata.

[0027] Figure 4 Shows a Figure 3 The schema metadata format shown in the figure is the key data protocol format of the message queue that cooperates with it. Figure 4 In the example shown: protocol is the data protocol format, such as json or avro; schemaId is the metadata identifier; schemaVersion is the metadata version number; the number of k / v is the number of additional configuration items, such as taskId or instanceId, etc., to distinguish tasks;

[0028] keyLength is the length of key; key is the content bytes of key; valueLength is the length of the specified value. The metadata (schema) content can be queried based on the schemaId and schemaVersion in the key to serialize the data at the source.

[0029] The data collection agent module can generate schema content according to the table structure of the source database table and register it to the metadata registration center. Then, it reads the data in the database table, and then serializes the data according to the schema content to obtain the serialized value data and send it to the data receiving service. Next, the data receiving service generates the key data of the message middleware, and writes the key data and the received serialized value data together into the message middleware. Finally, the data consumption landing service consumes the data in the message middleware, automatically identifies and obtains the schema content according to the schemaId and schemaVersion in the key data, deserializes the value data according to the schema content, and then writes the deserialized data into the Sink destination source.

[0030] Figure 5 1 is a schematic diagram of a data synchronization method according to an exemplary embodiment. The data acquisition proxy module (Proxy) collects data from the source database, and attempts to obtain metadata content (schemacontext) from the cache (usually the cache of the local system) according to the metadata identifier (schemaId) and the metadata version number (schemaVersion). Saving metadata in the cache can improve access efficiency. In some embodiments, if the corresponding schema does not exist in the cache, the registration center is queried to see if the corresponding schema exists. If the corresponding schema exists in the cache or the registration center, the data is serialized according to the metadata content and the data protocol, and sent to the data receiving module through the TCP protocol after the data serialization is successful. If the corresponding schema does not exist in the cache, the structural information (including table structure, etc.) of the source database is re-queried, new metadata is generated and registered with the metadata registration center module, that is, the hot update of metadata is realized. If the registration fails (for example, a compatibility error occurs), the metadata corresponding to the task is deleted from the metadata registration center module, and new metadata is registered and stored separately, wherein the storage content includes the new schemaId, the new schemaVersion and the new metadata content. If the registration is successful, a new version number (schemaVersion) is added to the metadata, and the schemaId, schemaVersion, and metadata content are stored in the cache for use in data serialization.

[0031] The data receiving module generates key data for the message middleware and writes the key data and the received serialized value data to the message middleware.

[0032] Then, the data consumption landing module consumes the data in the message middleware, parses the schemaId and schemaVersion according to the format of the data protocol in the key data, and attempts to obtain the corresponding metadata content from the cache (usually the cache of the local system). If the corresponding metadata content exists in the cache, the value data is deserialized according to the corresponding metadata content and data protocol, and the deserialized data is written to the Sink destination source. If it does not exist, the corresponding metadata content is queried and obtained from the metadata registration center and stored in the cache for data deserialization. In an embodiment of the present invention, both serialization and deserialization in data synchronization utilize the version information of metadata to achieve multi-version management of metadata.

[0033] One point that needs to be explained is that the device provided in the above embodiment only uses the division of the above-mentioned functional modules as an example to implement its functions. In actual applications, the above-mentioned functions can be assigned to different functional modules according to actual needs, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0034] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present disclosure can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer device readable medium or transmitted as one or more instructions or codes on a computer device readable medium. Computer device readable media include computer device storage media and communication media, wherein the communication medium includes any medium that facilitates the transmission of computer device programs from one place to another. The storage medium can be any available medium that a general or special computer device can access.

[0035] The embodiment of the present disclosure also provides a computer device storage medium for storing computer device software instructions used for the above-mentioned testing device, which includes a program designed for executing the above-mentioned data synchronization method.

[0036] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0037] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for managing metadata of a database during real-time data synchronization, comprising: Obtain the metadata identifier and version number of the source database; Attempting to obtain metadata corresponding to the identifier and the version number, including querying a cache or a registry whether there is metadata corresponding to the identifier and the version number; If the corresponding metadata content cannot be successfully obtained, generating metadata of the source database, and registering the generated metadata with a registration center; If the registration fails, registering the generated metadata and storing it in the cache after deleting the corresponding metadata from the registration center; Using the generated metadata, serialize the data collected from the source database; Generate key data of the message middleware, and write the key data and the serialized data into the message middleware; Consume the data of the message middleware; Parsing the metadata identifier and version number from the data; Attempting to obtain metadata corresponding to the identifier and the version number, including attempting to obtain metadata corresponding to the identifier and the version number from the cache; If the metadata corresponding to the identifier and the version number cannot be obtained, querying and obtaining the metadata corresponding to the identifier and the version number from the registration center, the metadata queried and obtained from the registration center is the metadata generated above; as well as The data is deserialized using the acquired metadata.

2. According to the method of claim 1, generating metadata of the source database further comprises: The structural information of the source database is queried to generate the metadata of the source database.

3. The method according to claim 1, wherein registering the generated metadata with a registration center comprises: The identification and version number of the generated metadata are stored in the registration center.

4. The method according to claim 1, further comprising: Store the generated metadata in cache.

5. The method according to claim 1, wherein the generated metadata includes the following information items: Data protocol format; Logo; Version number; Information used to distinguish tasks; The length of the key used for the message middleware; The content of the key; The length of the data collected; and Optional additional configuration items.

6. A system for managing metadata of a database during real-time data synchronization, comprising: processor; and A memory, wherein the memory stores computer executable instructions, and when the computer executable instructions are executed by the processor, the method according to any one of claims 1 to 5 can be executed.

7. A computer-readable medium storing computer-executable instructions, wherein the computer-executable instructions can perform the method according to any one of claims 1 to 5 when executed.

8. A system for the method according to any one of claims 1 to 5, comprising: The data collection agent module is used to collect and serialize data from the source database; Message middleware, used to receive data collected from the source database; Data consumption landing module, used to consume and deserialize data from the message middleware; and The registration center is used to store metadata and provide query services for metadata. The data acquisition agent module can generate metadata according to the table structure of the source database table and register it with the registration center, wherein the generated metadata includes an identifier and a version number, and The data consumption landing module can parse out the metadata identifier and version number from the consumed data, and query the registration center for metadata corresponding to the parsed identifier and version number.

9. The system according to claim 8, wherein the registration center is capable of adding, modifying and deleting metadata.

Images