Database query method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a database query method, and relates to the technical field of database query, and the method comprises the following steps: obtaining a query request, generating a target query instance corresponding to the query request, querying in a database by adopting the target query instance to obtain initial query data, distributing the initial query data to a data processing node, and under the condition that the data volume of the distributed initial query data in the data processing node is greater than a preset processing threshold value, redistributing the distributed initial query data in the data processing node, and processing the redistributed initial query data by adopting the data processing node to obtain target query data, in the data query process, the data is distributed for multiple times, so that the load of the query task is more balanced, the data skew is reduced, and the query response speed is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of database query, and in particular to a method and device for database query, electronic equipment, and storage medium. Background Art

[0002] With the rapid development of big data technology, the demand for data analysis and processing has exploded. Enterprises not only need to process massive amounts of data, but also need to complete complex query and analysis tasks in a short period of time. In order to meet these challenges, modern distributed database systems have emerged, and distributed database systems are particularly outstanding in processing massive amounts of data.

[0003] Distributed database systems process query requests in parallel by distributing data to multiple nodes. However, in practical applications, the problem of uneven data distribution remains a difficult problem that needs to be solved. Uneven data distribution can lead to a "long tail effect" in the query execution process, that is, the execution time of some query instances is significantly extended, thus affecting the overall query performance.

[0004] When dealing with the problem of uneven data distribution, existing technologies mainly rely on predefined bucketing strategies, which divide data into multiple buckets and assign each bucket to different nodes to achieve uniform data distribution. However, predefined bucketing strategies are usually static, that is, they are determined when data is loaded, and cannot adapt to dynamic changes in data distribution. Summary of the invention

[0005] In view of the above problems, a method and device, electronic device, and storage medium for database query are proposed to overcome the above problems or at least partially solve the above problems, including:

[0006] A method for querying a database, the method comprising:

[0007] Obtaining a query request and generating a target query instance corresponding to the query request;

[0008] Using the target query instance to query the database to obtain initial query data;

[0009] Distributing the initial query data to data processing nodes;

[0010] When the amount of the initial query data allocated in the data processing node is greater than a preset processing threshold, reallocating the initial query data allocated in the data processing node;

[0011] The data processing node is used to process the redistributed initial query data to obtain target query data.

[0012] In an optional embodiment of the present application, the query request includes an aggregate query request, and generating a target query instance corresponding to the query request includes:

[0013] Determining multiple initial query instances according to the aggregate query request;

[0014] Scanning the database using the multiple initial query instances to obtain data volumes corresponding to the multiple initial query instances in the database;

[0015] The multiple initial query instances are adjusted according to the data amounts respectively corresponding to the multiple initial query instances to obtain multiple target query instances.

[0016] In an optional embodiment of the present application, the adjusting the multiple initial query instances includes:

[0017] Determining data to be retrieved from the first initial query instance in which the amount of data is greater than a preset instance threshold;

[0018] The to-be-called data of the first initial instance is assigned to a second initial query instance whose data volume is smaller than the preset instance threshold.

[0019] In an optional embodiment of the present application, the adjusting the multiple initial query instances includes:

[0020] Determining data to be retrieved from the first initial query instance in which the amount of data is greater than a preset instance threshold;

[0021] A new query instance is generated, and the to-be-called data of the first initial instance is assigned to the new query instance.

[0022] In an optional embodiment of the present application, the allocating the initial query data to the data processing node includes:

[0023] Determine the first identification information of the initial query data according to a first preset algorithm;

[0024] The initial query data is distributed to a plurality of data processing nodes according to the first identification information.

[0025] In an optional embodiment of the present application, when the amount of the initial query data allocated in the data processing node is greater than a preset processing threshold, the initial query data allocated in the data processing node is redistributed, including:

[0026] When the amount of the allocated initial query data in the data processing node is greater than the preset processing threshold, determining the second identification information of the initial query data according to a second preset algorithm;

[0027] The initial query data allocated to the data processing node is reallocated according to the second identification information.

[0028] In an optional embodiment of the present application, the adopting the data processing node to process the redistributed initial query data to obtain target query data includes:

[0029] Determining the type of aggregation operation in the aggregation query request;

[0030] Calculating the redistributed initial query data according to the aggregation operation type to obtain multiple aggregation calculation results;

[0031] The multiple aggregate calculation results are combined to obtain the target query data.

[0032] A device for querying a database, comprising:

[0033] A first acquisition module is configured to acquire a query request and generate a target query instance corresponding to the query request;

[0034] An example query module is configured to use the target query instance to query the database to obtain initial query data;

[0035] A first allocation module, configured to allocate the initial query data to a data processing node;

[0036] A second allocation module is configured to reallocate the initial query data allocated in the data processing node when the amount of the initial query data allocated in the data processing node is greater than a preset processing threshold;

[0037] The result processing module is configured to process the redistributed initial query data using the data processing node to obtain target query data.

[0038] An electronic device comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the above-mentioned database query method when executed by the processor.

[0039] A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by the processor, the database query method as described above is implemented.

[0040] The embodiments of the present invention have the following advantages:

[0041] By obtaining a query request and generating a target query instance corresponding to the query request, the target query instance is used to query in the database to obtain initial query data, and the initial query data is distributed to a data processing node. When the amount of the initial query data distributed in the data processing node is greater than a preset processing threshold, the initial query data distributed in the data processing node is redistributed, and the data processing node is used to process the redistributed initial query data to obtain target query data. In the process of data query, the data is distributed multiple times to ensure that the load of the query task is more balanced, reduce data skew, and improve the query response speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0043] Figure 1 is a flowchart of the steps of a database query method provided by some embodiments of the present application;

[0044] Figure 2a It is a data original distribution effect diagram provided by some embodiments of the present application;

[0045] Figure 2b It is a data optimization distribution effect diagram provided by some embodiments of the present application;

[0046] Figure 3 It is an implementation flow chart provided by some embodiments of the present application;

[0047] Figure 4 This is a structural block diagram of a database query device provided in some embodiments of the present application. DETAILED DESCRIPTION

[0048] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] Reference Figure 1 , shows a flowchart of a method for querying a database provided by some embodiments of the present invention, which may specifically include the following steps:

[0050] Step 101: Obtain a query request and generate a target query instance corresponding to the query request.

[0051] Among them, a database is an organized collection of data, usually stored in a computer system, which can be accessed, managed and operated through a database management system. It is used to store, manage and retrieve data to support various applications and business needs. A database management system is a software system used to create, maintain and manage databases. It provides an interface between users and databases and supports operations such as data storage, query, update and deletion. A query request refers to a request issued by a user or a system to obtain specific information or data. A query request usually contains information such as query conditions and target data type.

[0052] A query instance in a database management system refers to a specific query operation or query request. A query instance may include the following aspects: 1. Query statement, which is an SQL statement submitted by a user or application to the database to retrieve data from the database. For example, a simple query statement may be SELECT*FROM employees WHEREdepartment='Sales'; before executing a query, the database management system will generate a query plan, which describes how the database will execute the query, including which indexes to use, how to connect tables, and the order of execution. The execution result of the query instance is a data set returned by the database. This result set may be a table, a record set, or a scalar value, depending on the type and structure of the query statement. The target query instance is a specific query object or instance generated based on the query request, which is used to perform the actual data query operation.

[0053] In some examples of this embodiment, the database may be, but is not limited to, Apache Doris, which is a distributed high-performance database system mainly used for real-time analysis and large-scale data processing. Apache Doris adopts a distributed architecture and can process large-scale data sets.

[0054] First, a query request sent by a user or an external system is received, which contains a description of the information or data that the user wants to obtain. Based on the received query request, a specific query object or instance is generated. This instance contains all the information required to execute the query, such as query conditions, data sources, return formats, etc. The generated target query instance is used to perform actual data query operations and obtain the required data from the database or other data sources.

[0055] Suppose there is an online book management system where users can query information about specific books.

[0056] The user enters a query request in the library management system, such as "query information on books with the title "ABCD"".

[0057] Generate a target query instance corresponding to the query request. After receiving the user's query request, the system generates a target query instance. This instance may contain the following information:

[0058] Query conditions: Book title = "ABCD", Data source: Book database, Return format: Detailed information of the book (such as author, publication date, ISBN number, etc.).

[0059] Execute the query. The system uses the generated target query instance to query the book information with the title "ABCD" from the book database and returns the result to the user.

[0060] In some embodiments of the present application, the query request includes an aggregate query request, and the above step 101 may include the following sub-steps:

[0061] Sub-step 11: determining multiple initial query instances according to the aggregate query request;

[0062] Among them, the aggregate query request is a special query request used to summarize and statistically analyze data. Aggregate query usually involves operations such as grouping, summing, average calculation, and counting of data.

[0063] In the specific implementation, different business needs correspond to different query requests. Users can input aggregate query requests that meet business needs according to business needs. For example, "query the total sales of each product in the first quarter of 2023", then the corresponding query condition is: time range = first quarter of 2023, and the aggregation operation corresponding to the query request includes grouping multiple products by product identifier and summing the sales of each product. According to the aggregate query request, multiple initial query instances are generated, and the generated initial query instances are distributed to multiple nodes. For example, if the database system has 2 nodes, then according to the aggregate query request, 2 initial query instances are determined to execute the product information query with the time range = first quarter of 2023 on the 2 nodes respectively. For another example, if the data source corresponding to the aggregate query request is 3 data partitions, then according to the aggregate query request, 3 initial query instances are determined to execute the product information query with the time range = first quarter of 2023 on the 3 data partitions respectively.

[0064] Sub-step 12: Scanning the database using the multiple initial query instances to obtain data volumes corresponding to the multiple initial query instances in the database;

[0065] Sub-step 13: adjusting the multiple initial query instances according to the data volumes respectively corresponding to the multiple initial query instances to obtain multiple target query instances.

[0066] In some examples of the present embodiment, multiple initial query instances are used to scan the database. During the scanning process, if it is found that the amount of data in a partition corresponding to one or several initial query instances is too large or too small, these initial query instances can be adjusted. Alternatively, during the scanning process, if it is found that the load of one or several initial query instances is too high, these initial query instances also need to be adjusted to obtain multiple adjusted query instances, namely, target query instances.

[0067] In some embodiments of the present application, sub-step 13 may include the following sub-steps:

[0068] Sub-step 111: determining data to be retrieved from the first initial query instance whose data volume is greater than a preset instance threshold;

[0069] Sub-step 112: assigning the to-be-called data of the first initial instance to a second initial query instance whose data volume is smaller than the preset instance threshold.

[0070] The preset instance threshold refers to the amount of data that a query instance can carry, and the preset instance threshold can be adjusted according to the development of the business and the adjustment of hardware resources and system environment. It is understandable that with the development of hardware or system environment, the processing speed of hardware resources and system environment is getting faster and faster, so the preset instance threshold can be adjusted to a larger preset value accordingly.

[0071] In some examples of the present embodiment, a data block scanning strategy may be adopted to divide the data source into multiple data blocks according to a fixed size or logical rules, and each initial query instance reads data by block. During the scanning process, if the amount of data corresponding to a certain initial query instance is greater than a preset instance threshold, or the load of a certain initial query instance is too high, the number of data blocks processed by it will be reduced, and some data blocks will be transferred to other initial query instances with lower loads. That is, the data to be called out is determined from the first initial query instance whose data amount is greater than the preset instance threshold, and the data to be called out is adjusted to the second initial query instance whose data amount is less than the preset instance threshold.

[0072] In some embodiments of the present application, sub-step 13 may also include the following sub-steps:

[0073] Sub-step 211: determining data to be retrieved from the first initial query instance whose data volume is greater than a preset instance threshold;

[0074] Sub-step 212: Generate a new query instance, and assign the to-be-called data of the first initial instance to the new query instance.

[0075] In some examples of the present embodiment, a partition scanning strategy may also be adopted. According to the partition information of the data source, the data source is divided into multiple small blocks by partition. Each initial query instance reads data from one or more partitions. During the scanning process, if it is found that the data volume of a partition is greater than the preset instance threshold, the system will dynamically adjust the partition allocation between the initial query instances. For example, a larger partition can be divided into smaller sub-partitions, which are processed in parallel by multiple initial query instances. That is, the data to be called out is determined from the query instances whose data volume is greater than the preset instance threshold, and some new query instances are generated, and the data to be adjusted is assigned to the newly added query instances.

[0076] In other examples of this embodiment, during the scanning process, it may be found that some partitions have a small amount of data, and the initial query instance is usually designed to process a large amount of data, so it may start multiple threads or processes to process data in parallel. However, for partitions with smaller data volumes, these additional parallel processing capabilities may not be fully utilized, resulting in a waste of CPU resources; in addition, the initial query instance may use batch reading or writing in I / O operations. For small partitions, this batch operation may not be efficient, but will increase unnecessary I / O overhead. Therefore, in the specific implementation of this application, smaller partitions are merged into other initial query instances for processing, thereby reducing unnecessary resource consumption.

[0077] Step 102: Use the target query instance to query in the database to obtain initial query data;

[0078] Among them, the target query instance is the adjusted query instance. As can be seen from the above content, the balanced distribution of data volume is fully considered when adjusting the initial query instance to the target query instance. This means that the amount of data in each partition or data block is roughly equal, avoiding the situation where the amount of data in some partitions is too large or too small. The balanced distribution of data volume helps to ensure that each query instance can process data with similar efficiency, thereby avoiding resource waste and performance bottlenecks. On this basis, the target query instance is used to read data from the database to obtain the initial query data, which improves query efficiency and reduces response time.

[0079] Step 103: Distribute the initial query data to the data processing nodes;

[0080] The data processing node may be a Pipeline Task, which is the basic unit for executing data processing tasks in Apache Doris. Apache Doris manages and schedules data processing tasks through Pipeline Task to achieve efficient data query and analysis.

[0081] In some examples of this embodiment, the initial query data may be distributed through a certain algorithm, so that the data distribution of each data processing node is more uniform.

[0082] In some embodiments of the present application, step 103 may include the following sub-steps:

[0083] Sub-step 311: determining first identification information of the initial query data according to a first preset algorithm;

[0084] Sub-step 312: Distribute the initial query data to multiple data processing nodes according to the first identification information.

[0085] The first preset algorithm may be a hash function, and the first identification information of the data calculated by the first preset algorithm is directly distributed to each processing Pipeline Task, that is, the data is directly distributed after being calculated by the first preset algorithm.

[0086] In a specific implementation, the hash function corresponding to the first preset algorithm may be SHA-1 (Secure Hash Algorithm 1), which is a cryptographic hash function that generates a 160-bit hash value, usually expressed as 40 hexadecimal digits. The working principle of SHA-1 can be divided into the following steps: The input data is first padded so that its length satisfies a multiple of 512 bits minus 64 bits. The padding data includes a 1 bit followed by several 0 bits until the length requirement is met. The last 64 bits are used to store the length of the original input data. Five 32-bit registers are initialized, which are used to store intermediate and final hash values. The input data is divided into 512-bit blocks, each of which is processed in turn, and the processing of each block includes 80 rounds of loop operations, and each round of operation updates the value of the register. After all blocks are processed, the values ​​of the five registers are connected together to form the final 160-bit hash value.

[0087] Step 104: when the amount of the initial query data allocated to the data processing node is greater than a preset processing threshold, the initial query data allocated to the data processing node is redistributed;

[0088] The process of determining the first identification information using the first preset algorithm and allocating the initial query data according to the identification information can distribute the data more evenly to different data processing nodes.

[0089] However, there may be several major problems: some specific hash values ​​may correspond to a large number of data records. For example, the number of records for some key values ​​is much greater than that for other key values, resulting in a large number of data records corresponding to the hash values ​​corresponding to these key values. For another example, using an inappropriate partition key or a partition key that is not selective enough will result in too much data in some partitions. The large number of data records corresponding to certain specific hash values ​​will further cause some data processing nodes to carry too much data, making it impossible to balance the load, and thus causing data skew again. Due to the randomness of the hash function, when data distribution is extremely uneven, a simple hash distribution method cannot ensure load balancing among nodes, causing some nodes to be overloaded while other nodes may be idle, wasting system resources.

[0090] In view of the above problems, in some embodiments of the present application, step 104 includes the following sub-steps:

[0091] Sub-step 411: when the amount of the allocated initial query data in the data processing node is greater than the preset processing threshold, determining the second identification information of the initial query data according to a second preset algorithm;

[0092] Sub-step 412: reallocate the initial query data allocated to the data processing node according to the second identification information.

[0093] In some examples of this embodiment, the initial query data is redistributed by Local Shuffle (local shuffle technology). In the distributed computing and big data processing framework, Local Shuffle is an optimization technology. The core idea is to complete the redistribution of data on the local node as much as possible, rather than transmitting it through the network, thereby reducing the transmission of data in the network, thereby improving computing efficiency and reducing latency. In the specific implementation, the data is first subjected to a conventional SHA-1 hash calculation to obtain a preliminary hash value (first identification information), and then the data distribution strategy is dynamically adjusted by analyzing the distribution results corresponding to the preliminary hash value. If the amount of data corresponding to a hash value is too large, the data corresponding to the hash value is subjected to a SHA-256 hash function calculation again, and the data is redistributed to multiple Pipeline Tasks.

[0094] SHA-256 (Secure Hash Algorithm 256-bit) is a hash function in the SHA-2 family. The working principle of SHA-256 can be divided into the following steps: The input data is first padded so that its length meets the multiple of 512 bits minus 64 bits. The padding consists of a 1 bit followed by several 0 bits until the length requirement is met, and the last 64 bits are used to store the length of the original input data. Eight 32-bit registers (A, B, C, D, E, F, G, H) are initialized, which are used to store intermediate and final hash values. The input data is divided into 512-bit blocks, and each block is processed in turn. The processing of each block includes 64 rounds of loop operations. Each round of operation updates the value of the register. Each round of operation uses a constant K and a message word W, which are calculated by the input data and the output of the previous round. After processing all blocks, the values ​​of the eight registers are concatenated together to form the final 256-bit hash value. SHA-256 is designed to be more collision-resistant and secure, and no effective collision attack method has been found so far.

[0095] This strategy of determining the second identification information through the second preset algorithm and redistributing the data according to the second identification information can achieve dynamic adjustment of data distribution based on load conditions to ensure that the load of each Task is more balanced. Figure 2a As shown, it is a diagram of the original distribution effect of data provided by some embodiments of the present application, from Figure 2a It can be seen from the data distribution of the first preset algorithm that the corresponding data volume at data processing node 1 is 20 million, while the data volume of nodes 2 and 3 is only 5 million. The load of node 1 is much higher than that of other nodes, which may cause node 1 to become a performance bottleneck. The processing speed of node 1 may be much lower than that of other nodes, resulting in increased latency of the overall computing task and decreased system performance. In addition, the amount of data in nodes 2 and 3 is small, and the utilization rate of resources (such as central processing unit, memory, and disk read and write operations) is low. System resources are not fully utilized, resulting in resource waste and reduced overall computing efficiency. Due to the high load and slow processing speed of node 1, the completion time of the entire task may be extended, the time users wait for query results may increase, and the user experience may decrease. Excessive load on node 1 may cause node overload or even crash, resulting in decreased system stability and may even lead to data loss or task failure. Figure 2b As shown, it is a data optimization distribution effect diagram provided by some embodiments of the present application, from Figure 2b It can be seen that by combining the first preset algorithm with the data redistribution technology of the second preset algorithm, part of the data of node 1 is migrated to node 2 and node 3, reducing the load of node 1 while ensuring the load balance of each node, thereby improving the resource utilization of each node and further improving the performance and stability of the system.

[0096] In other embodiments of the present application, key indicators such as the processing speed and resource usage of each data processing node can also be monitored in real time. According to the monitoring results of these key indicators, if a data processing node is found to be overloaded, the system will automatically adjust the amount of data it receives, perform SHA-256 hash calculation on some data again, and redistribute it to other data processing nodes with lighter loads. This dynamic adjustment mechanism can continuously optimize data distribution during data processing, reduce data skew problems, and further improve system performance and stability.

[0097] Step 105: Use a data processing node to process the redistributed initial query data to obtain target query data.

[0098] After the data is locally shuffled and redistributed, the data distribution has been optimized, and the data distribution of each task is uniform enough to perform aggregation calculations efficiently and evenly. Each Pipeline Task performs aggregation calculations on the assigned data to obtain the aggregation results. The aggregation results are initially merged locally to generate intermediate results, which are then aggregated to one or more BE nodes for final merging calculations. The merged final results are returned to the user to complete the query.

[0099] In some embodiments of the present application, step 105 may include the following sub-steps:

[0100] Sub-step 511: determining the type of aggregation operation in the aggregation query request;

[0101] Sub-step 512: Calculating the redistributed initial query data according to the aggregation operation type to obtain a plurality of aggregation calculation results;

[0102] Sub-step 513: Merge the multiple aggregate calculation results to obtain the target query data.

[0103] Aggregation operation types refer to operations that summarize and calculate a set of data during data processing and analysis. Common aggregation operation types include sum (SUM), count (COUNT), average (AVG), minimum (MIN), maximum (MAX), etc. These operations generate one or more summary results by performing statistics and calculations on the data for further analysis and decision-making.

[0104] For example, the aggregate query request is to calculate the average value of all records in a table. The data of the table is distributed on multiple nodes, and each node is responsible for storing and processing a part of the data. Generate the target query instance corresponding to the aggregate query request. After obtaining the initial query data in the database using the target query instance, the initial query data is distributed to the data processing node (Pipeline Task). In the case of data skew, the initial query data in the data processing node (Pipeline Task) is secondary distributed. After secondary distribution, the data processing node (Pipeline Task) performs further aggregate calculations. For example, the target query instance obtains the initial query data on nodes 1 and 2 in the database, where the data of node 1 includes record 1, record 2, and record 3, where record 1 is (key=1, value=100), record 2 is (key=1, value=200), and record 3 is (key=1, value=300); the data of node 2 includes record 4 and record 5, record 4: (key=1, value=400), record 5: (key=1, value=500), the data processing node (Pipeline Task) is Task 1 receives the initial query data from nodes 1 and 2 and processes them into intermediate results: the result in node 1 is (SUM=600, COUNT=3), the result in node 2 is (SUM=900, COUNT=2), and the aggregate calculation sum (SUM) is performed: 600+900=1500, the number of records (COUNT) is 3+2=5, and the intermediate result (SUM=1500, COUNT=5) is generated. The intermediate results are aggregated to the BE node (SUM=1500, COUNT=5), the BE node receives the intermediate results (SUM=1500, COUNT=5), and calculates the average value: AVG=1500 / 5=300. The BE node returns the final average value result AVG=300 to the user, completing the query.

[0105] like Figure 3The figure shows an implementation flow chart provided by some embodiments of the present application, which illustrates the embodiments of the present application. First, the user submits an aggregate query, and multiple initial query instances are obtained through parsing. In the process of scanning and reading the database, the initial query instance is adjusted according to the data volume to obtain a target query instance. When the initial query data obtained by querying the target query instance is distributed to each data processing node (pipeline task), it is necessary to determine whether there is data skew. If there is no data skew, the aggregate output result is completed directly in each node, where data skew refers to uneven data distribution in a distributed database system; if there is data skew, the data is first redistributed using the first preset algorithm SHA-1 algorithm. The data redistributed by the SHA-1 algorithm may still have data imbalance in each node, so it is necessary to continuously monitor the distribution of data at each node. In the case of data overload at a data node, the data of the data overload node is hashed using a second preset algorithm (SHA-256 algorithm) to obtain a second data identifier (SHA-256 hash value), and the data is redistributed again based on the obtained second data identifier. Even after redistribution, the data distribution of each node can be monitored in real time. According to the monitoring results, if a data node is still overloaded, part of the data can be subjected to multiple SHA-256 hash calculations and redistributed multiple times until no node is overloaded. The intermediate data obtained by each node is aggregated and the final result is output.

[0106] The embodiments of the present invention have the following advantages:

[0107] By obtaining a query request and generating a target query instance corresponding to the query request, the target query instance is used to query in the database to obtain initial query data, and the initial query data is distributed to a data processing node. When the amount of the initial query data distributed in the data processing node is greater than a preset processing threshold, the initial query data distributed in the data processing node is redistributed, and the data processing node is used to process the redistributed initial query data to obtain the target query data. By distributing the data multiple times, it is ensured that the load of the query task is more balanced, data skewness is reduced, and the query response speed is improved.

[0108] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0109] Reference Figure 4 , shows a schematic diagram of a database query device structure provided by some embodiments of the present invention, which may specifically include the following modules:

[0110] The first generating module 401 is configured to obtain a query request and generate a target query instance corresponding to the query request;

[0111] An example query module 402 is configured to use the target query example to query the database to obtain initial query data;

[0112] A first allocation module 403 is configured to allocate the initial query data to a data processing node;

[0113] The second allocation module 404 is configured to reallocate the initial query data allocated in the data processing node when the amount of the initial query data allocated in the data processing node is greater than a preset processing threshold;

[0114] The result processing module 405 is configured to process the redistributed initial query data using the data processing node to obtain target query data.

[0115] In an optional embodiment of the present application, the first generating module 401 includes:

[0116] A first instance determination submodule is configured to determine a plurality of initial query instances according to the aggregate query request;

[0117] A first scanning submodule is configured to use the multiple initial query instances to scan the database to obtain the data volumes corresponding to the multiple initial query instances in the database;

[0118] The second instance determination submodule is configured to adjust the multiple initial query instances according to the data volumes respectively corresponding to the multiple initial query instances to obtain multiple target query instances.

[0119] In an optional embodiment of the present application, the second instance determination submodule includes:

[0120] A first adjustment unit is configured to determine the data to be retrieved from the first initial query instance whose data volume is greater than a preset instance threshold;

[0121] The second adjustment unit is configured to assign the to-be-queried data of the first initial instance to a second initial query instance whose data volume is smaller than the preset instance threshold.

[0122] In an optional embodiment of the present application, the second instance determination submodule further includes:

[0123] A third adjustment unit is configured to determine the data to be retrieved from the first initial query instance whose data volume is greater than a preset instance threshold;

[0124] The fourth adjustment unit is configured to generate a new query instance and assign the to-be-queried data of the first initial instance to the new query instance.

[0125] In an optional embodiment of the present application, the first allocation module 403 includes:

[0126] A first identification determination submodule, configured to determine first identification information of the initial query data according to a first preset algorithm;

[0127] The first allocation submodule is configured to allocate the initial query data to multiple data processing nodes according to the first identification information.

[0128] In an optional embodiment of the present application, the second allocation module 404 includes:

[0129] A second identification determination submodule is configured to determine second identification information of the initial query data according to a second preset algorithm when the amount of data of the allocated initial query data in the data processing node is greater than the preset processing threshold;

[0130] The second allocation submodule is configured to reallocate the initial query data allocated to the data processing node according to the second identification information.

[0131] In an optional embodiment of the present application, the result processing module 405 includes:

[0132] An operation type determination submodule, configured to determine the type of aggregation operation in the aggregation query request;

[0133] an aggregation submodule, configured to calculate the redistributed initial query data according to the aggregation operation type to obtain a plurality of aggregation calculation results;

[0134] The merging submodule is configured to merge the multiple aggregate calculation results to obtain the target query data.

[0135] Some embodiments of the present invention further provide an electronic device, which may include a processor, a memory, and a computer program stored in the memory and capable of running on the processor, and the computer program implements the above database query method when executed by the processor.

[0136] Some embodiments of the present invention further provide a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the above database query method is implemented.

[0137] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0138] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0139] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0140] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0141] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0142] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0143] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0144] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0145] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the above elements.

[0146] The method and device for database query, electronic device, and storage medium provided above are introduced in detail. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for querying a database, characterized in that: The method comprises: Obtaining a query request and generating a target query instance corresponding to the query request; Using the target query instance to query the database to obtain initial query data; Distributing the initial query data to data processing nodes; When the amount of the initial query data allocated in the data processing node is greater than a preset processing threshold, reallocating the initial query data allocated in the data processing node; The data processing node is used to process the redistributed initial query data to obtain target query data.

2. The method according to claim 1, characterized in that The query request includes an aggregate query request, and generating a target query instance corresponding to the query request includes: Determining multiple initial query instances according to the aggregate query request; Scanning the database using the multiple initial query instances to obtain data volumes corresponding to the multiple initial query instances in the database; The multiple initial query instances are adjusted according to the data amounts respectively corresponding to the multiple initial query instances to obtain multiple target query instances.

3. The method according to claim 2, characterized in that The adjusting the multiple initial query instances includes: Determining data to be retrieved from the first initial query instance in which the amount of data is greater than a preset instance threshold; The to-be-called data of the first initial instance is assigned to a second initial query instance whose data volume is smaller than the preset instance threshold.

4. The method according to claim 2, characterized in that: The adjusting the multiple initial query instances includes: Determining data to be retrieved from the first initial query instance in which the amount of data is greater than a preset instance threshold; A new query instance is generated, and the to-be-called data of the first initial instance is assigned to the new query instance.

5. The method according to claim 1, characterized in that The allocating the initial query data to the data processing nodes comprises: Determine the first identification information of the initial query data according to a first preset algorithm; The initial query data is distributed to a plurality of data processing nodes according to the first identification information.

6. The method according to claim 5, characterized in that When the amount of the initial query data allocated in the data processing node is greater than a preset processing threshold, the initial query data allocated in the data processing node is redistributed, including: When the amount of the allocated initial query data in the data processing node is greater than the preset processing threshold, determining the second identification information of the initial query data according to a second preset algorithm; The initial query data allocated to the data processing node is reallocated according to the second identification information.

7. The method according to claim 2, characterized in that The adopting the data processing node to process the redistributed initial query data to obtain target query data includes: Determining the type of aggregation operation in the aggregation query request; Calculating the redistributed initial query data according to the aggregation operation type to obtain multiple aggregation calculation results; The multiple aggregate calculation results are combined to obtain the target query data.

8. A database query device, characterized in that: The device comprises: A first acquisition module is configured to acquire a query request and generate a target query instance corresponding to the query request; An example query module is configured to use the target query instance to query the database to obtain initial query data; A first allocation module, configured to allocate the initial query data to a data processing node; A second allocation module is configured to reallocate the initial query data allocated in the data processing node when the amount of the initial query data allocated in the data processing node is greater than a preset processing threshold; The result processing module is configured to process the redistributed initial query data using the data processing node to obtain target query data.

9. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the method for querying a database as claimed in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for querying a database according to any one of claims 1 to 7 is implemented.

Images