Data query methods for distributed databases and distributed databases

Abstract

This invention relates to a data query method for a distributed database and a distributed database. The data query method includes: a computing node querying metadata information corresponding to a received query request; the computing node searching for data page information based on business information and regional information in the metadata information; the computing node determining the IP address corresponding to the data page information based on the correspondence between the data page information and IP addresses; the computing node sending the query request to the data node corresponding to the IP address; and the data node retrieving hot data corresponding to the query request from the data page corresponding to the IP address in its corresponding synchronization cache pool and returning it. This embodiment avoids retrieving hot data across shards or IPs, reducing the scale of distributed access, shortening access latency, and distributing concurrency pressure.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a data query method for a distributed database and a distributed database. Background Technology

[0002] In distributed database systems in fields such as finance and telecommunications, data is stored in shards using different sharding strategies. Query requests are sent to compute nodes, which then perform the data retrieval.

[0003] However, different services in different regions query data at different frequencies. Some data may be accessed more frequently, but if this data is distributed across different shards and IPs, the access efficiency will be greatly reduced when the access latency between IPs is long. Summary of the Invention

[0004] To address the technical problem of reduced access efficiency due to latency issues arising from cross-IP access requirements for distributed database data query services, this application provides a data query method for distributed databases and a distributed database.

[0005] Firstly, this application provides a data query method for a distributed database, the method comprising:

[0006] The compute node queries the metadata information corresponding to the received query request;

[0007] The computing node searches for data page information based on the business information and geographic information in the metadata information;

[0008] The computing node searches for the IP address corresponding to the data page information based on the correspondence between data page information and IP address;

[0009] The computing node sends the query request to the data node corresponding to the IP address;

[0010] The data node retrieves the hot data corresponding to the query request from the data page corresponding to the IP address in its corresponding synchronization cache pool and returns it.

[0011] Optionally, the method further includes:

[0012] If the computing node does not find the IP address corresponding to the data page information, the computing node sends the query request to the data node corresponding to the query request;

[0013] The data node retrieves the hot data corresponding to the query request from the data page corresponding to the IP address in the storage engine buffer pool and returns it.

[0014] Optionally, the method further includes:

[0015] If the data node fails to obtain the hot data corresponding to the query request, or if the dynamic migration module in the data node detects that the computing node has not found the IP address corresponding to the data page information, the dynamic migration module records the hot data. The dynamic migration module counts the access frequency of the hot data within the statistical time interval. If the access frequency is greater than a preset threshold, the dynamic migration module migrates the hot data to the data page corresponding to the IP address in the synchronization buffer pool according to the business information and the regional information, and records the metadata information corresponding to the data page.

[0016] Optionally, the method further includes:

[0017] The dynamic migration module obtains statistical time intervals and auxiliary parameter values ​​from the artificial intelligence module. The statistical time interval is obtained by acquiring metadata information corresponding to multiple historically received query requests, determining the access frequency of the corresponding data page within a preset time period based on at least one of statistical business information, regional information, and IP address based on the multiple metadata information, and determining the frequency change trend based on the multiple access frequencies. The auxiliary parameter values ​​are pre-configured.

[0018] The dynamic migration module adjusts the hot data in each data page of the synchronization buffer pool corresponding to each data node based on the statistical time interval and the auxiliary parameter value.

[0019] Secondly, this application provides a distributed database, including: computing nodes and multiple data nodes, each of the data nodes having a corresponding synchronization cache pool;

[0020] The computing node is used to query the metadata information corresponding to the received query request, find the data page information based on the business information and regional information in the metadata information, determine the IP address corresponding to the data page information based on the correspondence between the data page information and the IP address, and send the query request to the data node corresponding to the IP address.

[0021] The data node is used to retrieve the hot data corresponding to the query request from the data page corresponding to the IP address in its corresponding synchronous cache pool, and return it.

[0022] Optionally, the computing node is further configured to send the query request to the data node corresponding to the query request if no IP address corresponding to the data page information is found;

[0023] The data node is also used to retrieve hot data corresponding to the query request from the data page corresponding to the IP address in the storage engine buffer pool, and return it.

[0024] Optionally, the dynamic migration module in the data node is further configured to record the hot data if the data node fails to obtain hot data corresponding to the query request, or if the computing node fails to find the IP address corresponding to the data page information.

[0025] The dynamic migration module is also used to count the access frequency of hot data within a statistical time interval. If the access frequency is greater than a preset threshold, the hot data is migrated to the data page corresponding to the IP address in the synchronization buffer pool according to the business information and the regional information, and the metadata information corresponding to the data page is recorded.

[0026] Optionally, the dynamic migration module is further configured to obtain statistical time intervals and auxiliary parameter values ​​from the artificial intelligence module, and adjust the hot data in each data page in the synchronization buffer pool corresponding to each data node based on the statistical time intervals and the auxiliary parameter values;

[0027] The artificial intelligence module is also used to obtain metadata information corresponding to multiple historically received query requests, determine the access frequency of the corresponding data page within a preset time period based on at least one of the statistical business information, regional information, and IP address according to the multiple metadata information, determine the frequency change trend based on the multiple access frequencies, determine the statistical time interval, and obtain the pre-configured auxiliary parameter value.

[0028] Thirdly, this application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0029] Memory, used to store computer programs;

[0030] When a processor executes a program stored in memory, it implements the data query method for a distributed database as described in any of the first aspects.

[0031] Fourthly, this application provides a computer-readable storage medium storing a program for a data query method for a distributed database, wherein when the program for the data query method for a distributed database is executed by a processor, it implements the steps of the data query method for a distributed database as described in any of the first aspects.

[0032] The technical solutions provided in this application have the following advantages compared with the prior art:

[0033] This application embodiment can locate the data node where the hot data is located based on business information, regional information, and IP address, and then obtain the hot data to be obtained from the data page in the synchronous cache pool corresponding to the data node. Since the hot data is stored in the same synchronous cache pool data page according to business information, regional information, and IP address, the retrieval of hot data across shards or across IPs is avoided, which reduces the scale of distributed access, shortens access latency, and distributes concurrency pressure. Attached Figure Description

[0034] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 A structural diagram of a distributed database provided in an embodiment of this application;

[0037] Figure 2 A flowchart illustrating a data query method for a distributed database provided in this application embodiment;

[0038] Figure 3 A flowchart illustrating another data query method for a distributed database provided in this application embodiment;

[0039] Figure 4 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0041] Because different services in different regions query data at varying frequencies, access to certain data may be frequent, but this data is distributed across different shards and IPs. When access latency between IPs is high, access efficiency will be significantly reduced. Therefore, this application provides a data query method for a distributed database and a distributed database.

[0042] like Figure 1 As shown, the distributed database of this application adds a new driver to the upper layer of the database storage engine, which consists of three parts: a metadata management module, a dynamic migration module, and a synchronization buffer pool. These three parts are responsible for management, computation, and storage functions, respectively. The dynamic migration module identifies data segments concentrated in a specific region or business based on factors such as region and business information, and dynamically migrates these data segments to the synchronization buffer pool corresponding to the IP address. The synchronization buffer pool centrally stores data for a specific region or business, and uses the same mechanism as the storage engine buffer pool, storing data in the form of data pages.

[0043] The detailed processing logic of the three modules inside the newly added driver is as follows:

[0044] (1) Dynamic migration module

[0045] This module is mainly responsible for data statistics and data migration. Data statistics are divided into three layers: the first layer is regional information, the second layer is IP information, and the third layer is business information. When a client accesses a piece of data, after finding the corresponding data in the data page through the storage engine, the module tags the data page containing this data, including the accessed region, time, accessed IP, and accessed business information. Then, by analyzing the access frequency of the data page by different regions and IPs within a certain period of time, the module copies the data page to the data page of the data node under the synchronization buffer pool of the region IP with the highest access frequency, and records the relevant metadata information of the data page at the same time.

[0046] Another important function is data synchronization, ensuring data consistency. This driver is mainly designed for query scenarios. If the business side issues an update request to update the migrated data, the module will set a timer to periodically read the metadata and synchronize the migrated data recorded in the metadata, ensuring data consistency.

[0047] (2) Metadata Management Module

[0048] The metadata management module primarily records business information, geographic information, and the binding relationship between IP addresses and data pages. The compute nodes analyze the client's source information, including geographic and business information, locate its corresponding data page, and then find the corresponding IP address based on the IP address's binding relationship with the data page.

[0049] (3) Synchronization buffer pool

[0050] The synchronization buffer pool uses the same storage mechanism as the storage engine buffer. This includes the data page structure, data page management, LUR algorithm optimization, and checkpointing technology.

[0051] This application provides a distributed database, including: computing nodes and multiple data nodes, each of the data nodes having a corresponding synchronization cache pool;

[0052] The computing node is used to query the metadata information corresponding to the received query request, find the data page information based on the business information and regional information in the metadata information, determine the IP address corresponding to the data page information based on the correspondence between the data page information and the IP address, and send the query request to the data node corresponding to the IP address.

[0053] The data node is used to retrieve the hot data corresponding to the query request from the data page corresponding to the IP address in its corresponding synchronous cache pool, and return it.

[0054] This application embodiment can locate the data node where the hot data is located based on business information, regional information, and IP address, and then obtain the hot data to be obtained from the data page in the synchronous cache pool corresponding to the data node. Since the hot data is stored in the same synchronous cache pool data page according to business information, regional information, and IP address, the retrieval of hot data across shards or across IPs is avoided, which reduces the scale of distributed access, shortens access latency, and distributes concurrency pressure.

[0055] In another embodiment of this application, the computing node is further configured to send the query request to the data node corresponding to the query request if no IP address corresponding to the data page information is found;

[0056] The data node is also used to retrieve hot data corresponding to the query request from the data page corresponding to the IP address in the storage engine buffer pool, and return it.

[0057] In this embodiment of the application, for hot data that is not located in the synchronous cache pool, it can be obtained directly from the storage engine buffer pool through the data node according to the original logic, ensuring that the query request can obtain the corresponding response.

[0058] In another embodiment of this application, the dynamic migration module in the data node is further configured to record the hot data if the data node fails to obtain hot data corresponding to the query request, or if it is detected that the computing node fails to find the IP address corresponding to the data page information.

[0059] The dynamic migration module is also used to count the access frequency of hot data within a statistical time interval. If the access frequency is greater than a preset threshold, the hot data is migrated to the data page corresponding to the IP address in the synchronization buffer pool according to the business information and the regional information, and the metadata information corresponding to the data page is recorded.

[0060] This application embodiment can perform statistical calculations based on business hotspots, regions, business types, etc., redistribute data, and dynamically migrate data to corresponding IPs. Query services can be directly sent to data nodes with the same region and IP, reducing the scale of distributed access, shortening access latency, and distributing concurrency pressure.

[0061] In another embodiment of this application, the dynamic migration module is further configured to obtain statistical time intervals and auxiliary parameter values ​​from the artificial intelligence module, and adjust the hot data in each data page in the synchronization buffer pool corresponding to each data node based on the statistical time intervals and the auxiliary parameter values.

[0062] The artificial intelligence module is also used to obtain metadata information corresponding to multiple historically received query requests, determine the access frequency of the corresponding data page within a preset time period based on at least one of the statistical business information, regional information, and IP address according to the multiple metadata information, determine the frequency change trend based on the multiple access frequencies, determine the statistical time interval, and obtain the pre-configured auxiliary parameter value.

[0063] The algorithm for statistical calculations in the AI ​​module involves counting the number of times different IPs in different regions access data pages within a certain time period and plotting the data page access frequency trend. First, the statistical time interval needs to be initialized, and this interval needs to be adjusted according to real-time business conditions. Second, business hotspot data may be accessed by different IPs in different regions at different times, requiring the addition of optimization parameters to make the data page access data statistics more accurate. The AI ​​module mainly calculates the statistical time interval and optimization parameter values: auxiliary parameters (migration network, persistence). As business increases, the AI ​​module needs to optimize and adjust these two parameters.

[0064] The data statistics algorithm in this application embodiment is continuously and dynamically optimized using artificial intelligence. It considers the statistical time interval and optimization parameters added to account for the fact that business hotspot data may belong to different geographical IP access hotspots at different times, and dynamically adjusts these parameters as business increases. While optimizing the statistical algorithm, it's necessary to address the fact that due to differences in business areas among different branches of each bank, the algorithm may not be suitable for all scenarios. Therefore, an artificial intelligence module is needed to dynamically adjust the statistical algorithm based on actual business scenarios, automatically optimizing it during use to make it applicable to more business scenarios.

[0065] like Figure 2 As shown, the data query method for a distributed database provided in this application embodiment may include the following steps:

[0066] Step S101: The compute node queries the metadata information corresponding to the received query request;

[0067] Step S102: The computing node searches for data page information based on the business information and regional information in the metadata information;

[0068] Step S103: The computing node determines the IP address corresponding to the data page information based on the correspondence between the data page information and the IP address;

[0069] Step S104: The computing node sends the query request to the data node corresponding to the IP address;

[0070] In step S105, the data node retrieves the hot data corresponding to the query request from the data page corresponding to the IP address in its corresponding synchronization cache pool and returns it.

[0071] In another embodiment of this application, the method further includes:

[0072] If the computing node does not find the IP address corresponding to the data page information, the computing node sends the query request to the data node corresponding to the query request;

[0073] The data node retrieves the hot data corresponding to the query request from the data page corresponding to the IP address in the storage engine buffer pool and returns it.

[0074] In another embodiment of this application, the method further includes:

[0075] If the data node fails to obtain the hot data corresponding to the query request, or if the dynamic migration module in the data node detects that the computing node has not found the IP address corresponding to the data page information, the dynamic migration module records the hot data. The dynamic migration module counts the access frequency of the hot data within the statistical time interval. If the access frequency is greater than a preset threshold, the dynamic migration module migrates the hot data to the data page corresponding to the IP address in the synchronization buffer pool according to the business information and the regional information, and records the metadata information corresponding to the data page.

[0076] In another embodiment of this application, the method further includes:

[0077] The dynamic migration module obtains statistical time intervals and auxiliary parameter values ​​from the artificial intelligence module. The statistical time interval is obtained by acquiring metadata information corresponding to multiple historically received query requests, determining the access frequency of the corresponding data page within a preset time period based on at least one of statistical business information, regional information, and IP address based on the multiple metadata information, and determining the frequency change trend based on the multiple access frequencies. The auxiliary parameter values ​​are pre-configured.

[0078] The dynamic migration module adjusts the hot data in each data page of the synchronization buffer pool corresponding to each data node based on the statistical time interval and the auxiliary parameter value.

[0079] For ease of understanding, such as Figure 3 As shown, this application also provides an embodiment in practical application.

[0080] 1. After the select statement is issued, the compute node does not directly send it to each shard to query data. Instead, it queries the metadata, finds the data page based on the region and business information, finds the corresponding IP based on the binding relationship between the data page and the IP, and sends the statement to the data node on the corresponding IP.

[0081] 2. After receiving a select statement, the data node does not directly use the storage engine, but instead obtains data from the synchronization buffer pool by adding a new driver.

[0082] 3. If the queried data is not found in the synchronization buffer pool, or if the corresponding IP cannot be found based on the metadata information, the statement will be sent to the corresponding data node according to the original logic. The corresponding data page will be queried through the storage engine. On the one hand, the data will be returned to the upper-layer computing node, and on the other hand, the dynamic migration module will be driven to perform statistical calculations on the data, dynamically migrate the data to the data page of the synchronization buffer pool of the specified IP, and record the global metadata information.

[0083] Compared with existing research and technology, the beneficial effects of this invention are mainly reflected in the following aspects:

[0084] A new driver is added at the underlying data node to complete the dynamic data migration. This process is not client-facing, does not disrupt the original data distribution, and only affects query operations, without impacting update-related operations. Data migration is performed on a per-data-page basis. The synchronization buffer pool uses the same mechanism as the storage engine buffer pool. Due to the locality of data storage in the newly added synchronization buffer pool, data queries are more efficient than before.

[0085] In another embodiment of this application, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus.

[0086] Memory, used to store computer programs;

[0087] When a processor executes a program stored in memory, it implements the data query method for a distributed database as described in any of the foregoing method embodiments.

[0088] The electronic device provided in this embodiment of the invention enables the processor to execute a program stored in the memory to locate the data node where hot data is located based on business information, regional information, and IP address. Then, the processor retrieves the hot data from the data page in the synchronous cache pool corresponding to the data node. Since the hot data is stored in the same synchronous cache pool data page based on business information, regional information, and IP address, the retrieval of hot data across shards or IPs is avoided, reducing the scale of distributed access, shortening access latency, and distributing concurrency pressure.

[0089] The communication bus 1140 mentioned in the above-mentioned electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus 1140 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0090] The communication interface 1120 is used for communication between the above-mentioned electronic device and other devices.

[0091] The memory 1130 may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0092] The processor 1110 mentioned above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0093] In another embodiment of this application, a computer-readable storage medium is provided, on which a program for a data query method for a distributed database is stored. When the program for the data query method for a distributed database is executed by a processor, it implements the steps of the data query method for a distributed database as described in any of the foregoing method embodiments.

[0094] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0095] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A data query method for a distributed database, characterized in that, The method includes: The compute node queries the metadata information corresponding to the received query request; The computing node searches for data page information based on the business information and geographic information in the metadata information; The computing node searches for the IP address corresponding to the data page information based on the correspondence between data page information and IP address; The computing node sends the query request to the data node corresponding to the IP address; The data node retrieves the hot data corresponding to the query request from the data page corresponding to the IP address in its corresponding synchronization cache pool and returns it. The method further includes: If the data node fails to obtain the hot data corresponding to the query request, or if the dynamic migration module in the data node detects that the computing node has not found the IP address corresponding to the data page information, the dynamic migration module records the hot data. The dynamic migration module counts the access frequency of the hot data within the statistical time interval. If the access frequency is greater than a preset threshold, the dynamic migration module migrates the hot data to the data page corresponding to the IP address in the synchronization buffer pool according to the business information and the regional information, and records the metadata information corresponding to the data page. The dynamic migration module obtains statistical time intervals and auxiliary parameter values ​​from the artificial intelligence module. The statistical time interval is obtained by acquiring metadata information corresponding to multiple historically received query requests, determining the access frequency of the corresponding data page within a preset time period based on at least one of statistical business information, regional information, and IP address based on the multiple metadata information, and determining the frequency change trend based on the multiple access frequencies. The auxiliary parameter values ​​are pre-configured. The dynamic migration module adjusts the hot data in each data page of the synchronization buffer pool corresponding to each data node based on the statistical time interval and the auxiliary parameter value.

2. The method according to claim 1, characterized in that, The method further includes: If the computing node does not find the IP address corresponding to the data page information, the computing node sends the query request to the data node corresponding to the query request; The data node retrieves the hot data corresponding to the query request from the data page corresponding to the IP address in the storage engine buffer pool and returns it.

3. A distributed database, characterized in that, include: The system comprises a compute node and multiple data nodes, each of which has a corresponding synchronization cache pool. The computing node is used to query the metadata information corresponding to the received query request, find the data page information based on the business information and regional information in the metadata information, determine the IP address corresponding to the data page information based on the correspondence between the data page information and the IP address, and send the query request to the data node corresponding to the IP address. The data node is used to retrieve the hot data corresponding to the query request from the data page corresponding to the IP address in its corresponding synchronous cache pool, and return it. The dynamic migration module in the data node is also used to record the hot data if the data node does not obtain the hot data corresponding to the query request, or if the computing node does not find the IP address corresponding to the data page information. The dynamic migration module is also used to count the access frequency of hot data within the statistical time interval. If the access frequency is greater than a preset threshold, the hot data is migrated to the data page corresponding to the IP address in the synchronization buffer pool according to the business information and the regional information, and the metadata information corresponding to the data page is recorded. The dynamic migration module is also used to obtain statistical time intervals and auxiliary parameter values ​​from the artificial intelligence module, and adjust the hot data in each data page in the synchronization buffer pool corresponding to each data node based on the statistical time intervals and auxiliary parameter values. The artificial intelligence module is also used to obtain metadata information corresponding to multiple historically received query requests, determine the access frequency of the corresponding data page within a preset time period based on at least one of the statistical business information, regional information, and IP address according to the multiple metadata information, determine the frequency change trend based on the multiple access frequencies, determine the statistical time interval, and obtain the pre-configured auxiliary parameter value.

4. The distributed database according to claim 3, characterized in that, The computing node is also configured to send the query request to the data node corresponding to the query request if no IP address corresponding to the data page information is found. The data node is also used to retrieve hot data corresponding to the query request from the data page corresponding to the IP address in the storage engine buffer pool, and return it.

5. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the data query method for a distributed database as described in any one of claims 1 to 2.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program for a data query method for a distributed database, which, when executed by a processor, implements the steps of the data query method for a distributed database as described in any one of claims 1-2.

Images