A distributed database and big data component metadata sharing system

Abstract

The present invention provides a distributed database and big data component metadata sharing system, comprising: an execution plan generation module, which parses SQL queries submitted by users and generates an execution plan that can be executed by a database; an external metadata management module, which identifies an external metadata name FQN; the external metadata management module cooperates with the execution plan generation module to identify and manage an integer identifier OID of the external metadata; an external metadata translator module, which parses the integer identifier OID of the external metadata and converts it into internally identifiable metadata; the external metadata translator module provides the execution plan generation module with metadata corresponding to the external metadata OID; an external metadata connection parameter storage, which stores connection parameters of each external metadata name FQN; and an external metadata connector, which obtains connection parameters from the external metadata connection parameter storage according to the external metadata FQN.

Description

Technical Field

[0001] The present invention relates to the field of databases, and more specifically, to a distributed database and big data component metadata sharing system. Background Art

[0002] A distributed database is a database that runs and stores data on multiple computers. It usually has better availability and stronger performance, but due to the existence of multiple nodes, ensuring consistency between nodes is the primary challenge.

[0003] Metadata refers to data that describes and manages data. In a database, it usually includes information about table definitions, view definitions, index information, permissions, and access control.

[0004] Generally, different databases do not share metadata. In the context of multiple databases, in order to distinguish them, internal metadata and external metadata are defined based on relative relationships. Internal metadata refers to metadata maintained by the database itself, while external metadata refers to metadata maintained by other databases other than itself. External metadata is key information for accessing and modifying data in other databases.

[0005] The premise for using external metadata is that the external database can provide metadata.

[0006] Currently, databases such as Spark and StarRocks support defining external metadata to access other databases. However, traditional databases, such as PostgreSQL, do not support external metadata and only maintain an internal metadata.

[0007] Databases that support external metadata, such as Spark and StarRocks, are emerging databases in recent years. Their architecture is very different from traditional databases, such as PostgreSQL. Therefore, the method of implementing external metadata in the former is difficult to implement in the latter.

[0008] Figure 1 The following is the execution flow without considering external metadata. Each metadata item will be persistently stored in the database and have a globally unique integer identifier (OID). In the running process, it is called and passed by its integer identifier. In this case, the main difficulties in implementing the external metadata function are as follows:

[0009] 1. How to dynamically manage (allocate, recycle) the integer identifier of external metadata; internal metadata is persisted in the database, so the integer identifier has been allocated and is unique; but external metadata is not persisted in the database, so there is no integer identifier;

[0010] 2. How to parse the external metadata integer identifier into metadata. Summary of the invention

[0011] The purpose of the present invention is to provide a distributed database and big data component metadata sharing system.

[0012] The present invention aims to solve the problems existing in the prior art.

[0013] Compared with the prior art, the technical solution of the present invention and its beneficial effects are as follows:

[0014] A distributed database and big data component metadata sharing system, comprising: an execution plan generation module, parsing an SQL query submitted by a user and generating an execution plan that can be executed by a database; an external metadata management module, identifying an external metadata name FQN; the external metadata management module cooperates with the execution plan generation module to identify and manage an integer identifier OID of the external metadata; an external metadata translator module, parsing the integer identifier OID of the external metadata and converting it into internally identifiable metadata; the external metadata translator module provides the execution plan generation module with metadata corresponding to the external metadata OID; an external metadata connection parameter storage, storing connection parameters for each external metadata name FQN; an external metadata connector, obtaining connection parameters from the external metadata connection parameter storage according to the external metadata FQN; the external metadata connector uses the connection parameters to connect to an external metadata provider and obtains raw external metadata corresponding to the FQN; the external metadata connector cooperates with the external metadata connection parameter storage to establish a connection; the external metadata connector provides the external metadata translator module with raw external metadata.

[0015] As a further improvement, the execution plan generation module exists in the database and generates an execution plan based on internal metadata; the metadata name involved in the SQL exists in the form of FQN, and the execution plan generation module parses the FQN into a unique integer identifier OID, obtains the real metadata through the OID, and finally generates an execution plan using the metadata.

[0016] As a further improvement, for unrecognizable FQNs, the execution plan generation module will be unable to generate an execution plan and will terminate with an error; if the execution plan generation module does not support external metadata, the execution plan generation module will be unable to recognize such FQNs and will terminate with an error.

[0017] As a further improvement, when encountering an unrecognizable FQN, the execution plan generation module will attempt to obtain the OID corresponding to the external metadata from the external metadata management module; when encountering an unrecognizable OID, the execution plan generation module will obtain the external metadata from the external metadata parsing module.

[0018] As a further improvement, the external metadata translator module obtains the FQN corresponding to the OID through the external metadata management module; the external metadata translator module interacts with the external metadata provider through the connection parameters and connector corresponding to the FQN to obtain raw external metadata; the external metadata translator module translates the raw external metadata into an internal metadata format.

[0019] As a further improvement, the external metadata translator module cooperates with the external metadata connector to obtain raw external metadata; the execution plan generation module provides an OID and needs to obtain the corresponding FQN from the external metadata management module.

[0020] As a further improvement, the data processing sequence includes: the user submits an SQL query to the execution plan generation module; the execution plan generation module translates the FQN into an OID, and if the internal metadata cannot be translated, the external metadata management module is asked to translate it, and if the external metadata management module cannot translate it, an error is reported and the process is terminated; the execution plan generation module generates an execution plan using the OID, and if it is external metadata, the OID is handed over to the external metadata translator to obtain the external metadata; after the query is executed, the corresponding resources are released, including recycling the integer identifier of the external metadata.

[0021] As a further improvement, the external metadata translator module parses the integer identifier OID of the external metadata and converts it into internally identifiable metadata, including: receiving the OID of the execution plan generation module; obtaining the FQN corresponding to the OID through the external metadata management module; establishing a connection through the external metadata connector; the external metadata connector obtains connection parameters from the external metadata connection parameter storage through the FQN; the external metadata connector establishes a connection after obtaining the connection parameters; the external metadata connector obtains raw external metadata from the external metadata provider through the FQN and returns it to the external metadata translator; the external metadata translator translates the raw external metadata into a recognizable format and returns it to the execution plan generator.

[0022] The beneficial effects of the present invention are:

[0023] The present invention realizes the parsing of external metadata. The user submits an SQL query to an execution plan generation module; the execution plan generation module translates FQD into OID. If the internal metadata cannot be translated, the external metadata management module is asked to translate it. If the external metadata management module cannot translate it either, an error is reported and the module is terminated; the execution plan generation module generates an execution plan using OID, which involves the work of translating OID into metadata. If it is an external metadata OID, the external metadata translator is used to obtain the external metadata; after the query execution is completed, the corresponding resources are released, including recycling the integer identifier of the external metadata. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is the execution flow of the background technology without considering external metadata.

[0025] Figure 2 It is a logical schematic diagram of a distributed database and big data component metadata sharing system provided by an embodiment of the present invention.

[0026] Figure 3 It is a schematic diagram of a distributed database and big data component metadata sharing system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0028] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0029] Reference Figure 2 and Figure 3 As shown, a distributed database and big data component metadata sharing system includes:

[0030] The execution plan generation module parses the SQL query submitted by the user and generates an execution plan that can be executed by the database;

[0031] An external metadata management module identifies an external metadata name FQN; the external metadata management module cooperates with the execution plan generation module to identify and manage an integer identifier OID of the external metadata;

[0032] The external metadata translator module parses the integer identifier OID of the external metadata and converts it into internally identifiable metadata; the external metadata translator module provides the execution plan generation module with metadata corresponding to the external metadata OID;

[0033] External metadata connection parameter storage, which stores the connection parameters (domain name, IP, port, protocol, user, password, etc.) of each external metadata name FQN;

[0034] The external metadata connector obtains connection parameters from the external metadata connection parameter storage according to the external metadata FQN; the external metadata connector uses the connection parameters to connect to the external metadata provider and obtains the raw external metadata corresponding to the FQN; the external metadata connector cooperates with the external metadata connection parameter storage to establish a connection; the external metadata connector provides the raw external metadata to the external metadata translator module.

[0035] The external metadata management module dynamically manages the integer identifier OID of the external metadata, including allocation and recycling; and establishes an external metadata name OID-integer identifier FQN mapping table.

[0036] Usually, internal metadata and external metadata have different formats and are not compatible with each other. The external metadata translator module needs to translate the raw external metadata into the format of internal metadata.

[0037] The execution plan generation module exists in the database and generates an execution plan based on internal metadata;

[0038] The metadata name involved in SQL exists in the form of FQN. The execution plan generation module parses the FQN into a unique integer identifier OID, obtains the real metadata through the OID, and finally generates an execution plan using the metadata.

[0039] For an unrecognizable FQN, the execution plan generation module will be unable to generate an execution plan and terminate with an error;

[0040] If the execution plan generation module does not support external metadata, the execution plan generation module cannot identify such FQN and terminates with an error.

[0041] When encountering an unrecognizable FQN, the execution plan generation module will try to obtain the OID corresponding to the external metadata from the external metadata management module;

[0042] When encountering an unrecognizable OID, the execution plan generation module will obtain external metadata from the external metadata parsing module.

[0043] The external metadata translator module obtains the FQN corresponding to the OID through the external metadata management module;

[0044] The external metadata translator module interacts with the external metadata provider through the connection parameters and connector corresponding to the FQN to obtain the raw external metadata;

[0045] The external metadata translator module translates raw external metadata into an internal metadata format.

[0046] The external metadata translator module cooperates with the external metadata connector to obtain raw external metadata;

[0047] The execution plan generation module provides an OID and needs to obtain the corresponding FQN from the external metadata management module.

[0048] The data processing sequence includes: the user submits an SQL query to the execution plan generation module; the execution plan generation module translates the FQN into an OID, if the internal metadata cannot be translated, the external metadata management module is asked to translate, if the external metadata management module cannot translate, an error is reported and the module is terminated; the execution plan generation module generates an execution plan using the OID, if it is external metadata, the OID is handed over to the external metadata translator to obtain the external metadata; the query execution is completed, and the corresponding resources are released, including recycling the integer identifier of the external metadata.

[0049] The external metadata translator module parses the integer identifier OID of the external metadata and converts it into internally identifiable metadata, including: receiving the OID of the execution plan generation module; obtaining the FQN corresponding to the OID through the external metadata management module; establishing a connection through the external metadata connector; the external metadata connector obtains connection parameters from the external metadata connection parameter storage through the FQN; the external metadata connector establishes a connection after obtaining the connection parameters; the external metadata connector obtains raw external metadata from the external metadata provider through the FQN and returns it to the external metadata translator; the external metadata translator translates the raw external metadata into a recognizable format and returns it to the execution plan generator.

[0050] Identify the external metadata name:

[0051] Usually the database uses a fully qualified name (FQN) to name each metadata, in the form of "database.namespace.object name". We expand the meaning of FQN to also include "external metadata namespace.subnamespace.object name".

[0052] Determine whether a metadata name is an external metadata name by maintaining a set of all external metadata namespaces.

[0053] Specific approach: Extract the first domain of the FQN and determine whether it is in the external metadata namespace set. If it is in the set, it belongs to the external metadata name, otherwise it is an internal metadata name.

[0054] Dynamically manage the integer identifier of external metadata and define three data structures:

[0055] 1. The reserved integer identifier pool is a set of integer identifiers. The integer identifiers in this set are called reserved integer identifiers and meet the following two properties:

[0056] First, this integer identifier has not been assigned to internal metadata; second, this integer identifier cannot be assigned to internal metadata;

[0057] This integer identifier pool can be changed dynamically if the above requirements are met. When an integer identifier is deleted from the reserved integer identifier pool, it can be allocated to other internal metadata.

[0058] 2. The reserved integer identifier has two states:

[0059] First, use; ("use" means that this reserved integer identifier has been assigned to external metadata)

[0060] Second, free; ("free" means that this reserved integer identifier can be assigned to external metadata)

[0061] 3. External metadata name-integer identifier mapping table. This mapping table is a bidirectional mapping. The external metadata name and integer identifier that have a mapping relationship are bound to each other.

[0062] When SQL starts to execute, a reserved integer identifier pool and an external metadata name-integer identifier mapping table are created;

[0063] When trying to map an external metadata name to an integer identifier, select a reserved integer identifier in the "free" state and assign it to this external metadata name, update its state to "in use", and then record their relationship in the external metadata name-integer identifier mapping table.

[0064] When the SQL execution is completed, the reserved integer identifier pool and the external metadata name-integer identifier mapping table are destroyed.

[0065] Parse the external metadata integer identifier into metadata and define three components:

[0066] 1) External metadata connection parameter storage

[0067] The external metadata connection parameter storage stores the unique connection parameters (such as domain name, IP, port, protocol, user, password, etc.) for each element in the external metadata namespace collection.

[0068] 2) External metadata connector

[0069] The external metadata connector obtains connection parameters from the external metadata connection parameter storage according to the external metadata namespace of the external metadata name, and uses these parameters to connect to the external metadata provider (which can be an external database or a metadata service of an external database).

[0070] 3) External metadata translator

[0071] The external metadata translator can translate metadata obtained from an external metadata provider into metadata that can be recognized internally.

[0072] The process of parsing the external metadata integer identifier is:

[0073] S1 first checks whether the integer identifier is in the reserved integer identifier pool. If so, it jumps to S2. If not, it does not process and jumps to S7.

[0074] S2, obtaining the external metadata name through the external metadata name-integer identifier mapping table;

[0075] S3,extracts the external metadata namespace from the external metadata name (i.e., FQD);

[0076] S4, obtaining connection parameters in the external metadata connection parameter storage according to the external metadata namespace;

[0077] S5, connects to the external metadata provider using the connection parameters;

[0078] S6, using an external metadata translator to parse the external metadata into internally recognizable metadata;

[0079] S7, end.

[0080] The above embodiments are only used to explain the technical solutions of the present invention rather than to limit them. Those skilled in the art should understand that any modification and equivalent substitution that does not depart from the spirit and scope of the present invention should fall within the protection scope of the claims of the present invention.

Claims

1. A distributed database and big data component metadata sharing system, characterized in that: include: The execution plan generation module parses the SQL query submitted by the user and generates an execution plan that can be executed by the database; External metadata management module, identifying the external metadata name FQN; The external metadata management module cooperates with the execution plan generation module to identify and manage the integer identifier OID of the external metadata; The external metadata translator module parses the integer identifier OID of the external metadata and converts it into internally identifiable metadata; The external metadata translator module provides the execution plan generation module with metadata corresponding to the external metadata OID; External metadata connection parameter storage, storing connection parameters for each external metadata name FQN; The external metadata connector obtains connection parameters from the external metadata connection parameter storage according to the external metadata FQN; The external metadata connector uses the connection parameters to connect to the external metadata provider and obtain the raw external metadata corresponding to the FQN; The external metadata connector cooperates with the external metadata connection parameter storage to establish a connection; The external metadata connector provides raw external metadata to the external metadata translator module; The execution plan generation module exists in the database and generates an execution plan based on internal metadata. The metadata name involved in SQL exists in the form of FQN. The execution plan generation module parses the FQN into a unique integer identifier OID, obtains the real metadata through the OID, and finally generates an execution plan using the metadata. For an unrecognizable FQN, the execution plan generation module will be unable to generate an execution plan and terminate with an error; if the execution plan generation module does not support external metadata, the execution plan generation module will be unable to recognize such FQN and terminate with an error; When encountering an unrecognizable FQN, the execution plan generation module will try to obtain the OID corresponding to the external metadata from the external metadata management module; when encountering an unrecognizable OID, the execution plan generation module will obtain the external metadata from the external metadata parsing module; The external metadata translator module obtains the FQN corresponding to the OID through the external metadata management module; the external metadata translator module interacts with the external metadata provider through the connection parameters and connector corresponding to the FQN to obtain the raw external metadata; the external metadata translator module translates the raw external metadata into the internal metadata format; The external metadata translator module cooperates with the external metadata connector to obtain the unprocessed external metadata; the execution plan generation module provides the OID, and the corresponding FQN needs to be obtained from the external metadata management module; The data processing sequence includes: the user submits an SQL query to the execution plan generation module; the execution plan generation module translates the FQN into an OID, if the internal metadata cannot be translated, the external metadata management module is asked to translate, if the external metadata management module cannot translate, an error is reported and the module is terminated; the execution plan generation module generates an execution plan using the OID, if it is external metadata, the OID is handed over to the external metadata translator to obtain the external metadata; the query execution is completed, and the corresponding resources are released, including recycling the integer identifier of the external metadata.

2. A distributed database and big data component metadata sharing system according to claim 1, characterized in that: The external metadata translator module parses the integer identifier OID of the external metadata and converts it into internally identifiable metadata, including: Receive the OID of the execution plan generation module; Obtain the FQN corresponding to the OID through the external metadata management module; Establish connections through external metadata connectors; The external metadata connector obtains the connection parameters from the external metadata connection parameter storage through the FQN; The external metadata connector obtains the connection parameters and establishes a connection; The external metadata connector obtains the raw external metadata from the external metadata provider through the FQN and returns it to the external metadata translator; The external metadata translator translates the raw external metadata into a recognizable format and returns it to the execution plan generator.

Images