Cross-library joint query method, device and equipment and storage medium

Abstract

The invention discloses a cross-library joint query method, device and equipment and a storage medium, which are applied to a distributed key value storage system ectd. Comprising the steps of obtaining a query request based on a data node, and determining a local cache condition corresponding to the query request; when the local cache condition is that the metadata does not exist, generating library table metadata information according to a pre-constructed global mapping relationship; and obtaining a global identifier corresponding to the library table metadata information, generating a physical execution plan according to the global identifier, and generating and executing an actuator based on the physical execution plan. Data joint query can be realized among different databases, the query efficiency is remarkably improved, the data management and maintenance cost is reduced, and data sharing and intercommunication are realized. Metadata between different databases can be effectively managed through global metadata management, and consistency and integration of the data are ensured. The efficiency of enterprise database management and data processing can be greatly improved, and the development process of a distributed system is promoted.

Description

Technical Field

[0001] The present invention relates to the field of distributed databases, and in particular to a cross-database joint query method, device, equipment and storage medium. Background Art

[0002] With the rapid development of information technology and the continuous growth of data volume, enterprises are facing more and more severe challenges in data management. In distributed systems, data interaction and query between different databases are becoming increasingly complex and time-consuming. Traditional data management methods have many problems in dealing with cross-database joint queries.

[0003] However, the existing cross-database joint query technology still has many problems, such as difficult to control consistency, high synchronization delay, excessive network communication overhead and insufficient scalability, which seriously restrict the efficiency of enterprise database management and data processing and hinder the development of distributed systems. Summary of the invention

[0004] The present invention provides a cross-database joint query method, device, equipment and storage medium to efficiently manage and access metadata of different databases and ensure accurate positioning of the corresponding back-end database according to the library and table name.

[0005] According to one aspect of the present invention, a cross-database joint query method is provided, which is applied to a distributed key-value storage system ectd. The method comprises:

[0006] Obtaining the query request based on the data node, determining the local cache status corresponding to the query request, wherein the local cache status includes whether the metadata exists or not;

[0007] When the local cache does not contain metadata, the library table metadata information is generated according to the pre-built global mapping relationship;

[0008] Obtain the global identifier corresponding to the metadata information of the library table, generate a physical execution plan based on the global identifier, generate an executor based on the physical execution plan, and execute it.

[0009] Optionally, determining the local cache status corresponding to the query request includes: parsing the query request to obtain a resolution request, wherein the resolution request includes a target library table name; determining whether there is target library table metadata corresponding to the target library table name in the local database, and if so, determining that the local cache status is that metadata exists; otherwise, determining that the local cache status is that metadata does not exist.

[0010] Optionally, the method also includes: when the local cache situation shows that metadata exists, obtaining the metadata status corresponding to each target library table metadata; judging whether the metadata status is available, and if so, directly using the target library table metadata as the library table metadata information; otherwise, generating an unavailable prompt message.

[0011] Optionally, the library table metadata information is generated according to a pre-built global mapping relationship, including: matching the target library table name based on the global mapping relationship to determine a match; when the match is a match, generating a global identifier according to the target library table name, and generating the library table metadata information according to the global identifier.

[0012] Optionally, the target library table name is matched based on the global mapping relationship to determine the matching situation, including: matching the target library table name based on the global mapping relationship, wherein the global mapping relationship includes the correspondence between each library table name and the backend data source; judging whether the target data source corresponding to the target library table name is matched, and if so, determining that the matching situation is a match; otherwise, determining that the matching situation is a mismatch.

[0013] Optionally, a global identifier is generated according to the target library table name, and library table metadata information is generated according to the global identifier, including: determining the physical library identifier corresponding to the target data source, and determining the library table identifier corresponding to the target library table name; generating a global identifier according to the physical library identifier and the library table identifier; and updating the cache in the local database according to the global identifier to generate library table metadata information.

[0014] Optionally, a physical execution plan is generated based on the global identifier, including: determining whether the global identifier is 0, if so, obtaining the local identifier, obtaining the latest metadata cache in the local database based on the local identifier, and generating a physical execution plan based on the latest metadata cache; otherwise, locating the latest metadata cache based on the global identifier, and generating a physical execution plan based on the latest metadata cache.

[0015] According to another aspect of the present invention, a cross-database joint query device is provided, the device comprising:

[0016] A local cache status determination module is used to obtain a query request based on a data node and determine a local cache status corresponding to the query request, wherein the local cache status includes whether metadata exists or not;

[0017] The library table metadata information generation module is used to generate the library table metadata information according to the pre-built global mapping relationship when the local cache situation is that the metadata does not exist;

[0018] The global identifier acquisition and execution module is used to obtain the global identifier corresponding to the library table metadata information, generate a physical execution plan according to the global identifier, and generate an executor based on the physical execution plan and execute it.

[0019] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0020] at least one processor;

[0021] and a memory communicatively coupled to the at least one processor;

[0022] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute a cross-database joint query method described in any embodiment of the present invention.

[0023] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement a cross-database joint query method described in any embodiment of the present invention when executed.

[0024] The technical solution of the embodiment of the present invention can realize data joint query between different databases, significantly improve query efficiency, reduce the cost of data management and maintenance, and realize data sharing and intercommunication. Through global metadata management, metadata between different databases can be effectively managed to ensure data consistency and integration. It can greatly improve the efficiency of enterprise database management and data processing and promote the development of distributed systems.

[0025] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0027] Figure 1 is a flowchart of a cross-database joint query method provided according to the first embodiment of the present invention;

[0028] Figure 2 This is a schematic diagram of a joint query architecture provided according to the first embodiment of the present invention;

[0029] Figure 3 is a flowchart of another cross-database joint query method provided according to the second embodiment of the present invention;

[0030] Figure 4 This is a schematic diagram of the structure of a cross-database joint query device provided according to Embodiment 3 of the present invention;

[0031] Figure 5 It is a structural schematic diagram of an electronic device for implementing a cross-database joint query method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] Embodiment 1

[0035] Figure 1 A flowchart of a cross-database joint query method is provided for the first embodiment of the present invention. This embodiment is applicable to a distributed key-value storage system ectd. The method can be executed by a cross-database joint query device. The cross-database joint query device can be implemented in the form of hardware and / or software. The cross-database joint query device can be configured in a computer controller. Figure 1 As shown, the method includes:

[0036] S110. Obtaining a query request based on a data node, determining a local cache status corresponding to the query request, wherein the local cache status includes whether metadata exists or not.

[0037] It should be noted that cross-database joint query refers to the process of querying data between multiple different databases. Since the data of an enterprise or project is often stored in various heterogeneous databases, cross-database joint query can break the limitation of a single database, integrate multiple data resources, obtain comprehensive and relevant results at one time, and meet the needs of complex business analysis.

[0038] Among them, data node refers to the basic unit in the distributed key-value storage system ectd. Figure 2 A schematic diagram of a joint query architecture is provided for the first embodiment of the present invention. etcd is a distributed key-value storage system for storing metadata information and maintaining metadata consistency. etcd is connected to the three management nodes below through the watch mechanism. The middle layer is three management nodes, and the management nodes communicate with each other through "metadata interaction". A data node is connected to each management node. The bottom layer is three data nodes, which are connected to database 1, database 2 and database 3 respectively. The data node is responsible for performing import, update or query operations. The entire architecture shows a hierarchical data management system, in which etcd serves as the metadata storage center, the management node is responsible for metadata interaction and management, and the data node is responsible for data operations with specific databases. By combining the cluster deployment of etcd and multiple management nodes, as well as multiple executors under the management nodes, efficient metadata management and synchronization are achieved. A single data node can access multiple backend databases at the same time to update or query data.

[0039] Furthermore, after the data node is successfully started, it will automatically start the information collection process, that is, the data node will pull all metadata information and configuration management information. A query request refers to an instruction initiated by an external client, application, etc. to a distributed database to obtain specific data, usually entered in the form of an SQL statement. Local cache refers to data information stored in the local database. Metadata refers to data descriptions, which do not directly contain business data content, but include database architecture information, such as library table names, field names, field types, and relationships between tables, to provide support for the query system to understand data structure and logic.

[0040] Specifically, in a distributed data storage architecture, data nodes are scattered across different physical or virtual servers, each responsible for managing a specific database. For example, in the big data architecture of an e-commerce platform, user information data is stored in the user database, and order data is stored in the order database. When an operator initiates a request to "query the first order information of new users in the past month", the system can push it to the relevant user and order data nodes according to the query request.

[0041] Furthermore, when the query submitted by the user involves cross-database access, the data node will capture the latest metadata snapshot saved locally. The metadata snapshot can reflect the current data distribution and structure, provide a basic reference for subsequent parsing and processing requests, and allow the system to quickly locate the range of libraries and tables that may be involved.

[0042] Optionally, determining the local cache status corresponding to the query request includes: parsing the query request to obtain a resolution request, wherein the resolution request includes a target library table name; determining whether there is target library table metadata corresponding to the target library table name in the local database, and if so, determining that the local cache status is that metadata exists; otherwise, determining that the local cache status is that metadata does not exist.

[0043] Specifically, the system will parse the incoming query request, that is, the SQL statement, to sort out which database tables are involved in this query, and then try to extract the metadata corresponding to the database tables. The system can perform lexical and grammatical analysis on the incoming query statement, extract key elements from it, and form a parsing request. Among them, the target database table name clearly points to the specific storage location where the user expects to obtain the data. Then, the system will determine whether there is target database table metadata corresponding to the target database table name in the local database. If matching target database table metadata can be found, it means that there have been similar queries before, and the local cache situation can be determined as metadata exists. Subsequent queries can quickly reuse this ready-made information to reduce unnecessary overhead; if the target database table metadata cannot be found, it means that the local cache does not retain relevant data. At this time, the system will determine that the local cache situation does not have metadata.

[0044] S120: When the local cache situation shows that no metadata exists, generate library table metadata information according to the pre-built global mapping relationship.

[0045] Among them, the global mapping relationship is a set of logical relationships that are pre-sorted and constructed, integrating the database instances of each backend data source in the entire distributed data environment and the corresponding relationships between database tables, spanning the differences between heterogeneous databases, allowing the system to know how different database tables are related, and assisting cross-database queries to locate target data. The metadata information of the database table refers to a set of metadata that comprehensively describes the detailed architecture and relationships of the database and database table, and clearly specifies which databases are involved, the table structure, field details, and the logic of mutual association.

[0046] Optionally, the method also includes: when the local cache situation shows that metadata exists, obtaining the metadata status corresponding to each target library table metadata; judging whether the metadata status is available, and if so, directly using the target library table metadata as the library table metadata information; otherwise, generating an unavailable prompt message.

[0047] Among them, the metadata status, as an identifier, reflects the current availability of the corresponding library table metadata. If the metadata status is in an available state, the target library table metadata can be directly used as the library table metadata information, and the subsequent query process can be seamlessly connected. On the contrary, if it is determined that the metadata status is not available, it means that the cached metadata can no longer adapt to the current query requirements. In order to avoid the generation of erroneous query results, the system will generate an unavailable prompt message. On the one hand, the prompt message will be fed back to the query initiator to inform the user that the cached data is unavailable. On the other hand, it will also trigger relevant operation and maintenance or data management processes, prompting the staff to update the cached metadata in time to ensure that subsequent queries are not hindered. For example, the library table may be locked by an exclusive lock and is in an unavailable state. At this time, the system can only return an error message to inform the user that the query cannot be executed; if the status is normal and available, the system will directly feedback the corresponding library table metadata.

[0048] Optionally, the library table metadata information is generated according to a pre-built global mapping relationship, including: matching the target library table name based on the global mapping relationship to determine a match; when the match is a match, generating a global identifier according to the target library table name, and generating the library table metadata information according to the global identifier.

[0049] It should be noted that if all the required library table metadata cannot be found according to the above steps, the system will expand the search scope and search according to the target library table name in the global mapping relationship. If there is no relevant record in the global mapping relationship, it means that the target library table to be found does not exist, and the system will return an error prompt truthfully. If the corresponding target library table name is found, the system can generate a global identifier based on the target library table name and update the local cache to facilitate subsequent quick calls.

[0050] Optionally, the target library table name is matched based on the global mapping relationship to determine the matching situation, including: matching the target library table name based on the global mapping relationship, wherein the global mapping relationship includes the correspondence between each library table name and the backend data source; judging whether the target data source corresponding to the target library table name is matched, and if so, determining that the matching situation is a match; otherwise, determining that the matching situation is a mismatch.

[0051] Among them, the global mapping relationship is the mapping relationship between the backend data source configured by the user and the database table. The system will search according to the database table name in the global mapping relationship. If the corresponding target data source is found, the matching situation is determined to be a match. If the corresponding target data source cannot be found, it means that the target database table name is not recorded in the global mapping relationship. At this time, the system will determine that the matching situation is a mismatch. When the matching situation is a mismatch, the user can make corresponding adjustments, or update the global mapping relationship, or adopt other remedial strategies to promote the continuation of cross-database query. The user refers to the developer who designs cross-database queries.

[0052] Optionally, a global identifier is generated according to the target library table name, and library table metadata information is generated according to the global identifier, including: determining the physical library identifier corresponding to the target data source, and determining the library table identifier corresponding to the target library table name; generating a global identifier according to the physical library identifier and the library table identifier; and updating the cache in the local database according to the global identifier to generate library table metadata information.

[0053] It can be known that different data sources correspond to different physical databases, such as MySQL, Oracle, SQLServer, etc. Each physical database needs a unique identifier to distinguish it. The physical library identifier can be a combination of the database instance name, IP address and port number, or a custom number. In a physical database, each library table will also have a unique identifier. The library table identifier can be generated based on the name of the library table, the database mode to which it belongs, and other related attributes. Furthermore, the system can generate a global identifier by combining the physical library identifier and the library table identifier. For example, use a specific delimiter to connect the two. Through the global identifier, you can accurately locate the specified library table among many data sources and library tables.

[0054] It should be noted that in the data processing process, in order to improve performance, a cache is usually maintained in the local database to store commonly used metadata information. When the global identifier is generated, the system will query or update the local cache based on the global identifier. If the metadata information corresponding to the global identifier already exists in the local cache, the system will directly obtain the relevant information from the cache. If not, it is necessary to obtain the detailed metadata information of the library table from the data source and store it in the local cache.

[0055] S130: Obtain a global identifier corresponding to the metadata information of the library table, generate a physical execution plan according to the global identifier, generate an executor based on the physical execution plan, and execute the executor.

[0056] Among them, the global identifier refers to the unique code assigned to the metadata information of the library table in the entire distributed system, which can help accurately locate the corresponding metadata in a complex distributed environment and ensure the consistency of the query. The physical execution plan refers to a detailed query execution plan customized according to the library table metadata and the global identifier. For example, it plans which library to check first, which index to use, and in what order to associate tables, and clarifies the specific data retrieval and processing operations at each step to ensure efficient query execution. The executor refers to the functional module responsible for putting the physical execution plan into practice. The executor can drive the underlying database connection, initiate query instructions, monitor the execution process, handle exceptions, and finally aggregate data according to the plan to output results that meet the query request.

[0057] Optionally, you can use the etcd node data version number as the version identifier for global metadata management, implement global metadata version control and synchronization in a lock-free list manner, and improve the concurrent processing capability of the Data Definition Language (DDL).

[0058] It can be seen that by generating a global identifier, the corresponding metadata snapshot is accurately located, breaking the design of binding users to a single database table. Cross-database queries are identified in the parsing and preprocessing phase, and the global identifier is used in the optimization and execution phase to generate a physical execution plan bound to a specific metadata snapshot. During execution, the corresponding backend database table is accessed with the bound metadata snapshot to implement cross-database joint query.

[0059] The technical solution of the embodiment of the present invention can realize data joint query between different databases, significantly improve query efficiency, reduce the cost of data management and maintenance, and realize data sharing and intercommunication. Through global metadata management, metadata between different databases can be effectively managed to ensure data consistency and integration. It can greatly improve the efficiency of enterprise database management and data processing and promote the development of distributed systems.

[0060] Embodiment 2

[0061] Figure 3 This is a flowchart of a cross-database joint query method provided in the second embodiment of the present invention. This embodiment adds a specific process of generating a physical execution plan based on a global identifier on the basis of the above-mentioned first embodiment. Among them, the specific contents of steps S210-S220 are roughly the same as steps S110-S120 in the first embodiment, so they will not be repeated in this embodiment. Figure 3 As shown, the method includes:

[0062] S210: Obtain a query request based on a data node, and determine a local cache status corresponding to the query request, wherein the local cache status includes whether metadata exists or not.

[0063] Optionally, determining the local cache status corresponding to the query request includes: parsing the query request to obtain a resolution request, wherein the resolution request includes a target library table name; determining whether there is target library table metadata corresponding to the target library table name in the local database, and if so, determining that the local cache status is that metadata exists; otherwise, determining that the local cache status is that metadata does not exist.

[0064] S220: When the local cache situation shows that metadata does not exist, generate library table metadata information according to the pre-built global mapping relationship.

[0065] Optionally, the method also includes: when the local cache situation shows that metadata exists, obtaining the metadata status corresponding to each target library table metadata; judging whether the metadata status is available, and if so, directly using the target library table metadata as the library table metadata information; otherwise, generating an unavailable prompt message.

[0066] Optionally, the library table metadata information is generated according to a pre-built global mapping relationship, including: matching the target library table name based on the global mapping relationship to determine a match; when the match is a match, generating a global identifier according to the target library table name, and generating the library table metadata information according to the global identifier.

[0067] Optionally, the target library table name is matched based on the global mapping relationship to determine the matching situation, including: matching the target library table name based on the global mapping relationship, wherein the global mapping relationship includes the correspondence between each library table name and the backend data source; judging whether the target data source corresponding to the target library table name is matched, and if so, determining that the matching situation is a match; otherwise, determining that the matching situation is a mismatch.

[0068] Optionally, a global identifier is generated according to the target library table name, and library table metadata information is generated according to the global identifier, including: determining the physical library identifier corresponding to the target data source, and determining the library table identifier corresponding to the target library table name; generating a global identifier according to the physical library identifier and the library table identifier; and updating the cache in the local database according to the global identifier to generate library table metadata information.

[0069] S230: Obtain a global identifier corresponding to the library table metadata information.

[0070] It should be noted that the global identifier is used for the entire distributed database system and is globally unique. The global identifier can span multiple heterogeneous databases and accurately locate the metadata of the database table in the distributed storage environment. The local identifier is only effective for the local database and is a local index mark. The local identifier is used to organize and manage data in the local database storage area and assist in quickly locating the latest status of local data.

[0071] S240, determine whether the global flag is 0, if so, execute S250, otherwise, execute S260.

[0072] It should be noted that this implementation is an optimization phase, and the system will first verify the value of the global identifier. If the global identifier is not 0, it means that the previous process has generated a valid global identifier. At this time, the system can quickly locate the metadata cache with the global identifier and generate a physical execution plan. In addition, each table read and update plan in the physical execution plan will be firmly bound to the metadata cache to ensure that the execution plan fits the actual data situation.

[0073] S250: Obtain a local identifier, obtain the latest metadata cache in the local database according to the local identifier, and generate a physical execution plan according to the latest metadata cache.

[0074] Specifically, when the global identifier is 0, the system will obtain the local identifier, and then search for the corresponding latest metadata in the cache of the local database according to the identifier. Then, the physical execution plan is generated according to the latest metadata.

[0075] S260: Locate the latest metadata cache according to the global identifier, and generate a physical execution plan according to the latest metadata cache.

[0076] Specifically, when the global identifier is not 0, the control can use the global identifier to locate the corresponding latest metadata in the metadata cache. Similar to the case where the global identifier is 0, after obtaining the latest metadata cache, the system will also generate a physical execution plan.

[0077] S270: Generate an executor based on the physical execution plan and execute it.

[0078] Specifically, in the execution phase, the system generates and executes the executor based on the previously generated physical execution plan. Since each table read execution plan is closely related to the metadata cache, the executor can access the corresponding backend physical database based on the latest metadata cache, complete the cross-database query action, and extract the data required by the user.

[0079] The technical solution of the embodiment of the present invention can realize data joint query between different databases, significantly improve query efficiency, reduce the cost of data management and maintenance, and realize data sharing and intercommunication. Through global metadata management, metadata between different databases can be effectively managed to ensure data consistency and integration. It can greatly improve the efficiency of enterprise database management and data processing and promote the development of distributed systems.

[0080] Embodiment 3

[0081] Figure 4 This is a schematic diagram of the structure of a cross-database joint query device provided by Embodiment 3 of the present invention. Figure 4 As shown, the device includes: a local cache status determination module 310, which is used to obtain a query request based on a data node and determine a local cache status corresponding to the query request, wherein the local cache status includes the presence of metadata and the absence of metadata;

[0082] The library table metadata information generation module 320 is used to generate the library table metadata information according to the pre-built global mapping relationship when the local cache situation is that the metadata does not exist;

[0083] The global identifier acquisition and execution module 330 is used to acquire the global identifier corresponding to the library table metadata information, generate a physical execution plan according to the global identifier, and generate and execute an executor based on the physical execution plan.

[0084] Optionally, the local cache status determination module 310 is specifically used to: parse the query request to obtain a resolution request, wherein the resolution request includes the target library table name; determine whether there is target library table metadata corresponding to the target library table name in the local database, and if so, determine that the local cache status is that metadata exists; otherwise, determine that the local cache status is that metadata does not exist.

[0085] Optionally, the device also includes: a local cache metadata module, which is used to: when the local cache situation shows that metadata exists, obtain the metadata status corresponding to each target library table metadata; determine whether the metadata status is available, and if so, directly use the target library table metadata as the library table metadata information; otherwise, generate an unavailable prompt message.

[0086] Optionally, the library table metadata information generation module 320 specifically includes: a matching situation determination unit, used to match the target library table name based on the global mapping relationship to determine the matching situation; a library table metadata information generation unit, used to generate a global identifier according to the target library table name when the matching situation is a match, and generate the library table metadata information according to the global identifier.

[0087] Optionally, a matching determination unit is specifically used to: match the target library table name based on a global mapping relationship, wherein the global mapping relationship includes the correspondence between each library table name and the backend data source; determine whether the target data source corresponding to the target library table name is matched, and if so, determine that the matching situation is a match; otherwise, determine that the matching situation is a mismatch.

[0088] Optionally, the library table metadata information generation unit is specifically used to: determine the physical library identifier corresponding to the target data source, and determine the library table identifier corresponding to the target library table name; generate a global identifier based on the physical library identifier and the library table identifier; update the cache in the local database based on the global identifier to generate library table metadata information.

[0089] Optionally, the global identifier acquisition and execution module 330 is specifically used to: determine whether the global identifier is 0, if so, obtain the local identifier, obtain the latest metadata cache in the local database according to the local identifier, and generate a physical execution plan according to the latest metadata cache; otherwise, locate the latest metadata cache according to the global identifier, and generate a physical execution plan according to the latest metadata cache.

[0090] The technical solution of the embodiment of the present invention can realize data joint query between different databases, significantly improve query efficiency, reduce the cost of data management and maintenance, and realize data sharing and intercommunication. Through global metadata management, metadata between different databases can be effectively managed to ensure data consistency and integration. It can greatly improve the efficiency of enterprise database management and data processing and promote the development of distributed systems.

[0091] A cross-database joint query device provided in an embodiment of the present invention can execute a cross-database joint query method provided in any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

[0092] Embodiment 4

[0093] Figure 5 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0094] like Figure 5 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0095] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0096] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as a cross-library joint query method.

[0097] In some embodiments, a cross-library joint query method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the cross-library joint query method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform a cross-library joint query method in any other appropriate manner (for example, by means of firmware).

[0098] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0099] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0100] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0101] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0102] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0103] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0104] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0105] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A cross-database joint query method, applied to a distributed key-value storage system ectd, characterized in that: include: Obtaining a query request based on a data node, determining a local cache status corresponding to the query request, wherein the local cache status includes whether metadata exists or not; When the local cache situation is that metadata does not exist, generating library table metadata information according to the pre-built global mapping relationship; Obtain a global identifier corresponding to the metadata information of the library table, generate a physical execution plan according to the global identifier, and generate an executor based on the physical execution plan and execute it.

2. The method according to claim 1, characterized in that: The determining the local cache status corresponding to the query request includes: Parsing the query request to obtain a parsing request, wherein the parsing request includes a target library table name; Determine whether there is target library table metadata corresponding to the target library table name in the local database, and if so, determine that the local cache status is that metadata exists; Otherwise, it is determined that the local cache situation is that no metadata exists.

3. The method according to claim 2, characterized in that The method further comprises: When the local cache situation shows that metadata exists, the metadata status corresponding to the metadata of each target library table is obtained; Determine whether the metadata status is available, and if so, directly use the target library table metadata as the library table metadata information; Otherwise, an unavailable prompt message is generated.

4. The method according to claim 2, characterized in that: The generating of the library table metadata information according to the pre-built global mapping relationship includes: Matching the target library table name based on the global mapping relationship to determine a matching situation; When the matching condition is a match, a global identifier is generated according to the target library table name, and library table metadata information is generated according to the global identifier.

5. The method according to claim 4, characterized in that The matching of the target library table name based on the global mapping relationship to determine the matching situation includes: Matching the target library table name based on the global mapping relationship, wherein the global mapping relationship includes a correspondence between each library table name and a backend data source; Determine whether a target data source corresponding to the target library table name is matched, and if so, determine that the matching situation is a match; Otherwise, the matching condition is determined to be a mismatch.

6. The method according to claim 5, characterized in that The step of generating a global identifier according to the target library table name, and generating library table metadata information according to the global identifier includes: Determine the physical library ID corresponding to the target data source, and determine the library table ID corresponding to the target library table name; Generate a global identifier according to the physical library identifier and the library table identifier; The cache in the local database is updated according to the global identifier to generate library table metadata information.

7. The method according to claim 1, characterized in that The generating a physical execution plan according to the global identifier includes: Determine whether the global identifier is 0, if so, obtain the local identifier, obtain the latest metadata cache in the local database according to the local identifier, and generate a physical execution plan according to the latest metadata cache; Otherwise, the latest metadata cache is located according to the global identifier, and a physical execution plan is generated according to the latest metadata cache.

8. A cross-database joint query device, characterized in that: include: A local cache status determination module, used to obtain a query request based on a data node, and determine a local cache status corresponding to the query request, wherein the local cache status includes whether metadata exists or not; A library table metadata information generation module, used to generate library table metadata information according to a pre-built global mapping relationship when the local cache situation is that no metadata exists; The global identifier acquisition and execution module is used to acquire the global identifier corresponding to the library table metadata information, generate a physical execution plan according to the global identifier, and generate and execute an executor based on the physical execution plan.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that: The computer storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method according to any one of claims 1 to 7 when executed.

Images