Query task processing method, device, electronic device and readable storage medium

Abstract

The embodiment of the present application provides a method, device, electronic device and storage medium for processing a query task, and relates to the field of database query technology. The method includes: receiving a query task to be processed sent by a second processing node; the query task is generated by the second processing node based on a query request; determining the timestamp of the query task; the timestamp includes the request time of the query request; based on the request time, calling the thread resources in the thread pool of the first processing node, executing the query task, and generating a query result. The first processing node in the embodiment of the present application performs task scheduling and task execution based on the timestamp, thereby improving query efficiency.

Description

Technical Field

[0001] The present application relates to the technical field of database query, and in particular to a method, device, electronic device and storage medium for processing a query task. Background Art

[0002] The MPP (Massively Parallel Processing) system is composed of many loosely coupled processing units. Each processing unit has an independent operating system and an instance copy of the management database. The MPP system has good data volume and flexibility support.

[0003] The database based on MPP architecture runs on multiple machines, namely processing units. When querying data, each machine exchanges data through a network connection. These machines are divided into two roles, workers (i.e., the first processing node) and coordinators (i.e., the second processing node); there are multiple machines configured for each role; in the prior art, query tasks are usually globally scheduled based on the coordinator, and the tasks are distributed to each worker for execution; and in the global scheduling process of the coordinator, a resource synchronization mechanism needs to be introduced. The implementation of this resource synchronization mechanism will cause the query task to take too long, and there is a problem of low query efficiency. Summary of the invention

[0004] The embodiments of the present application provide a query task processing method, device, electronic device and computer-readable storage medium, which can solve the problem of low query efficiency of MPP database based on multiple processing nodes in the prior art. The technical solution is as follows:

[0005] According to one aspect of an embodiment of the present application, a method for processing a query task is provided, which is applied to a first processing node, and the method includes:

[0006] receiving a query task to be processed sent by the second processing node; the query task is generated by the second processing node based on the query request;

[0007] Determine the timestamp of the query task; the timestamp includes the request time of the query request;

[0008] The thread resources in the thread pool of the first processing node are called based on the request time to execute the query task and generate the query result.

[0009] Optionally, the calling of thread resources in the thread pool of the first processing node based on the request time to execute the query task includes:

[0010] When the request time is earlier than the first time, and based on the first capacity threshold of the thread pool, it is determined that there are free thread resources in the thread pool; or when the request time is later than the first time, and based on the second capacity threshold of the thread pool, it is determined that there are free thread resources in the thread pool;

[0011] Add the query task to the work queue of the first processing node, and execute the query task based on the free thread resources; wherein the first capacity threshold is greater than the second capacity threshold; wherein the first time is the minimum time value of the request time corresponding to all query tasks in the waiting queue and the work queue of the first processing node.

[0012] Optionally, the above method further includes:

[0013] Determine a first difference between a first capacity threshold and a number of thread occupancy of a work queue of a first processing node;

[0014] If the first difference is not less than the number of threads required by the query task, it is determined that there are free thread resources in the thread pool based on the first capacity threshold of the thread pool.

[0015] Optionally, the above method further includes:

[0016] Determine a second difference between a second capacity threshold and the number of threads occupied by the work queue of the first processing node;

[0017] When the second difference is not less than the required number of threads for the query task, it is determined that there are free thread resources in the thread pool based on the second capacity threshold of the thread pool.

[0018] Optionally, the above method further includes:

[0019] When the request time is earlier than the first time, and it is determined based on the first capacity threshold of the thread pool that there are no free thread resources in the thread pool, the query task is not executed and an error prompt message is generated.

[0020] Optionally, the above method further includes:

[0021] When the request time is later than the first time, and it is determined based on the second capacity threshold of the thread pool that there are no free thread resources in the thread pool, the query task is added to the waiting queue.

[0022] Optionally, after executing the query task, the following steps are performed:

[0023] Remove the query task from the work queue and release the target thread resources occupied by the query task;

[0024] Based on the request time of the task to be executed, select the target task to be executed from the waiting queue;

[0025] Add the target task to be executed to the work queue.

[0026] Optionally, the above method further includes:

[0027] For each task to be executed in the waiting queue, a minimum unit thread is allocated.

[0028] According to another aspect of an embodiment of the present application, a query task processing device is provided, the device comprising:

[0029] A receiving module, used for receiving a query task to be processed sent by the second processing node; the query task is generated by the second processing node based on the query request;

[0030] A determination module, used to determine the timestamp of the query task; the timestamp includes the request time of the query request;

[0031] The calling module is used to call the thread resources in the thread pool of the first processing node based on the request time, execute the query task, and generate the query result.

[0032] Optionally, the above calling module is used to:

[0033] When the request time is earlier than the first time, and based on the first capacity threshold of the thread pool, it is determined that there are free thread resources in the thread pool; or when the request time is later than the first time, and based on the second capacity threshold of the thread pool, it is determined that there are free thread resources in the thread pool;

[0034] Add the query task to the work queue of the first processing node, and execute the query task based on the free thread resources; wherein the first capacity threshold is greater than the second capacity threshold; wherein the first time is the minimum time value of the request time corresponding to all query tasks in the waiting queue and the work queue of the first processing node.

[0035] Optionally, the above device further includes a first resource monitoring module, configured to:

[0036] Determine a first difference between a first capacity threshold and a number of thread occupancy of a work queue of a first processing node;

[0037] If the first difference is not less than the number of threads required by the query task, it is determined that there are free thread resources in the thread pool based on the first capacity threshold of the thread pool.

[0038] Optionally, the above device further includes a second resource monitoring module, configured to:

[0039] Determine a second difference between a second capacity threshold and the number of threads occupied by the work queue of the first processing node;

[0040] When the second difference is not less than the required number of threads for the query task, it is determined that there are free thread resources in the thread pool based on the second capacity threshold of the thread pool.

[0041] Optionally, the above device further includes a prompt module, which is used to:

[0042] When the request time is earlier than the first time, and it is determined based on the first capacity threshold of the thread pool that there are no free thread resources in the thread pool, the query task is not executed and an error prompt message is generated.

[0043] Optionally, the above device further includes an adding module for:

[0044] When the request time is later than the first time, and it is determined based on the second capacity threshold of the thread pool that there are no free thread resources in the thread pool, the query task is added to the waiting queue.

[0045] Optionally, the above device further includes a releasing module, which is used to:

[0046] Remove the query task from the work queue and release the target thread resources occupied by the query task;

[0047] Based on the request time of the task to be executed, select the target task to be executed from the waiting queue;

[0048] Add the target task to be executed to the work queue.

[0049] Optionally, the above device further includes a distribution module, which is used to:

[0050] For each task to be executed in the waiting queue, a minimum unit thread is allocated.

[0051] According to another aspect of an embodiment of the present application, an electronic device is provided, which includes: a memory, a processor, and a computer program stored in the memory, and the processor executes the computer program to implement the steps of the method shown in the first aspect of the embodiment of the present application.

[0052] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method shown in the first aspect of the embodiments of the present application are implemented.

[0053] According to one aspect of an embodiment of the present application, a computer program product is provided, which includes a computer program, and when the computer program is executed by a processor, the steps of the method shown in the first aspect of the embodiment of the present application are implemented.

[0054] The beneficial effects of the technical solution provided by the embodiment of the present application are:

[0055] The first processing node in the embodiment of the present application receives the query task sent by the second processing node, and determines the timestamp of the query task, which timestamp includes the request time of the query request. The first processing node calls the thread resources in the thread pool based on the request time, executes the query task and generates the query result. The present application implements task scheduling based on timestamps and makes rational use of thread resources. Compared with the prior art of scheduling tasks based on the second processing node and then executing the query task based on the first processing node, the present application schedules and executes tasks based on the first processing node, and there is no need to introduce a resource synchronization mechanism, which reduces the time consumption of generating query results and improves query efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in describing the embodiments of the present application are briefly introduced below.

[0057] Figure 1 A schematic diagram of an application scenario of a query task processing method provided in an embodiment of the present application;

[0058] Figure 2 A flowchart of a method for processing a query task provided in an embodiment of the present application;

[0059] Figure 3 A schematic diagram of a process for executing a query task in a query task processing method provided in an embodiment of the present application;

[0060] Figure 4 A schematic diagram of a process for determining whether to execute a query task in a query task processing method provided in an embodiment of the present application;

[0061] Figure 5 A flowchart of a method for processing a query task provided as an example in an embodiment of the present application;

[0062] Figure 6 A schematic diagram of the structure of a query task processing device provided in an embodiment of the present application;

[0063] Figure 7 A schematic diagram of the structure of an electronic device for processing a query task provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] The embodiments of the present application are described below in conjunction with the drawings in the present application. It should be understood that the implementation methods described below in conjunction with the drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0065] It will be understood by those skilled in the art that, unless specifically stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the terms "including" and "comprising" used in the embodiments of the present application refer to that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and / or components, but do not exclude the implementation as other features, information, data, steps, operations, elements, components and / or combinations thereof supported by the technical field. It should be understood that when we say that an element is "connected" or "coupled" to another element, the one element may be directly connected or coupled to the other element, or it may refer to that the one element and the other element establish a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein indicates at least one of the items defined by the term, for example, "A and / or B" may be implemented as "A", or as "B", or as "A and B".

[0066] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0067] The MPP system is composed of many loosely coupled processing units. The CPU in each unit has its own private resources, such as bus, memory, hard disk, etc. The biggest feature of this structure is that resources are not shared between processing units.

[0068] A database that uses the MPP architecture is called an MPP database, which has strong parallel data computing capabilities and massive data storage capabilities. Traditional databases cannot support large-scale clusters, and the performance and scalability of a single machine are limited and costly, while MPP databases can meet the analysis needs of massive data; at the same time, complex queries often use multi-table joins, full table scans, etc., and the amount of data involved is often very large; MPP databases can support complex SQL (Structured Query Language) queries and large data scales.

[0069] In the prior art, the MPP database runs on multiple machines, and the machines exchange data through network connections. These machines perform two roles, coordinators and workers. Each role can be configured multiple times to improve the scalability of the system. There are network connections between workers and between workers and coordinators, but there is no network connection between coordinators. The coordinator will receive query statements from multiple clients. When there are multiple coordinators, multiple coordinators will send query tasks to multiple workers concurrently. When the coordinator globally controls and schedules these concurrent queries, it is necessary to introduce an additional resource synchronization mechanism, and the cost of the resource synchronization mechanism increases with the expansion of the processing unit cluster, increasing the time consumption of task processing.

[0070] The inventors found that these tasks can be scheduled on workers, but the following three problems need to be solved:

[0071] 1. Scheduling these tasks needs to ensure that they do not exceed the hardware resource limits of the machine to avoid downtime;

[0072] 2. Whether scheduling these tasks on the workers can achieve the effect of global scheduling, ensuring that incoming requests can be executed and completed one by one without deadlock;

[0073] 3. Can the worker schedule these multiple SQL tasks in a fair manner?

[0074] The query task processing method, device, electronic device and computer-readable storage medium provided in this application are intended to solve the above technical problems in the prior art.

[0075] The embodiment of the present application provides a method for processing a query task, which is applied to a server cluster in an MPP database, and the server cluster may include a first processing node and a second processing node. The first processing node in the embodiment of the present application receives the query task to be processed sent by the second processing node, determines the timestamp of the query task, and then calls the thread resources in the thread pool of the first processing node based on the request time, executes the query task, and generates a query result; the embodiment of the present application implements task scheduling based on timestamps, reasonably utilizes thread resources, and improves query efficiency.

[0076] The following describes several exemplary embodiments to illustrate the technical solutions of the embodiments of the present application and the technical effects produced by the technical solutions of the present application. It should be noted that the following embodiments can refer to, draw on or combine with each other, and the same terms, similar features and similar implementation steps in different embodiments will not be described repeatedly.

[0077] like Figure 1 As shown, the query task processing method of the present application can be applied to Figure 1 In the scenario shown, specifically, the first server 103 receives a query task to be processed sent by the second server 102, wherein the query task is generated by the second server 102 based on the query request of the client 101, and the first server 103 determines the timestamp of the query task. Since the timestamp may include the request time of the query request, the thread resources in the thread pool of the first server are called based on the request time to execute the query task and generate the query result; then the first server 103 sends the query result to the second server 102 so that the second server 102 forwards the query result to the client 101.

[0078] Figure 1 In the scenario shown, the query task processing method can be executed in the server, and in other scenarios, it can also be executed in the terminal.

[0079] Technicians in this technical field can understand that the "terminal" used here can be a mobile phone, a tablet computer, a PDA (Personal Digital Assistant), a MID (Mobile Internet Device), etc.; the "server" can be implemented by an independent server or a server cluster consisting of multiple servers.

[0080] The present application provides a method for processing a query task, such as Figure 2 As shown, it can be applied to a first processing node, and the method includes:

[0081] S201, receiving a query task to be processed sent by a second processing node;

[0082] The query task is generated by the second processing node based on the query request.

[0083] Specifically, the second processing node receives the query request, generates a query task in response to the query request, and sends the query task to the first processing node.

[0084] Among them, the first processing node or the second processing node can be a node device in the MPP database system, for example, a computer, a node device in the network, such as a base station, a controller device, etc., and no specific limitation is made in the embodiments of the present application.

[0085] Specifically, the first processing node may receive the query task sent by the second processing node through the application layer; the query task is generated after the second processing node compiles and instantiates the query request. The query request may be a SQL query statement.

[0086] In an embodiment of the present application, after receiving a query request, i.e., an SQL query statement, sent by a user, the second processing node can compile the SQL query statement to obtain a distributed execution plan, and then instantiate the distributed execution plan into multiple query tasks, and then send the multiple query tasks to the first processing node.

[0087] S202, determining the timestamp of the query task.

[0088] The timestamp may include the request time of the query request.

[0089] Specifically, the first processing node may allocate a globally unique timestamp to the query task according to the system time.

[0090] In some implementations, the first processing node may parse the query task and directly obtain a timestamp corresponding to the query task.

[0091] In other implementations, the timestamp may further include an identifier of a query request corresponding to the query task, and the first processing node may further obtain the identifier based on query task analysis, and determine the timestamp corresponding to the query task based on the identifier.

[0092] S203: Calling thread resources in the thread pool of the first processing node based on the request time, executing the query task, and generating query results.

[0093] Each query request corresponds to at least two query tasks.

[0094] Specifically, the first processing node can determine whether at least two query tasks corresponding to the query request enter the working queue or the waiting queue based on the request time. When the query task enters the working queue, the thread resources in the thread pool are called to execute the query tasks in sequence, and the query result of the query request is determined based on the execution results of all query tasks. When the query task enters the waiting queue, the thread resources are called after it enters the working queue to execute the query task and generate the query result.

[0095] In an embodiment of the present application, when the query request is to query the target data by combining the information in Table A and Table B, after the second processing node receives the above query request, it compiles and instantiates the above query request to obtain four query tasks, namely scanning, connection, projection and aggregation; when the first processing node determines that the above query task enters the work queue based on the request time, it calls the thread resources to execute the above four query tasks in sequence to obtain the corresponding query results, namely the target data.

[0096] The first processing node in the embodiment of the present application receives the query task sent by the second processing node, and determines the timestamp of the query task, which timestamp includes the request time of the query request. The first processing node calls the thread resources in the thread pool based on the request time, executes the query task and generates the query result. The present application implements task scheduling based on timestamps and makes rational use of thread resources. Compared with the prior art of scheduling tasks based on the second processing node and then executing the query task based on the first processing node, the present application schedules and executes tasks based on the first processing node, and there is no need to introduce a resource synchronization mechanism, which reduces the time consumption of generating query results and improves query efficiency.

[0097] A possible implementation method is provided in the embodiment of the present application, such as Figure 3 As shown, in the above step S203, calling the thread resources in the thread pool of the first processing node based on the request time to execute the query task may include:

[0098] When the request time is earlier than the first time, and it is determined based on the first capacity threshold of the thread pool that there are free thread resources in the thread pool;

[0099] or,

[0100] When the request time is later than the first time, and it is determined based on the second capacity threshold of the thread pool that there are free thread resources in the thread pool;

[0101] Add the query task to the work queue of the first processing node, and execute the query task based on the free thread resources; wherein the first capacity threshold is greater than the second capacity threshold; wherein the first time is the minimum time value of the request time corresponding to all query tasks in the waiting queue and the work queue of the first processing node.

[0102] Specifically, the first processing node can determine the free thread resources of the thread pool based on the request time of the query request, and add the query task to the work queue of the first processing node, and execute the query task based on the free thread resources. When the query task is added to the work queue, the first time can be updated in real time according to the update of the work queue.

[0103] The capacity of the thread pool can be dynamically adjusted based on the amount of tasks, the first capacity threshold is the maximum number of threads that the first processing node can apply for, and the second capacity threshold is the safety value of the number of threads that the first processing node can apply for. The specific steps for determining the free thread resources of the thread pool are as follows:

[0104] In the embodiment of the present application, the first processing node may determine the first time T based on the minimum time value corresponding to all query tasks in the working queue and the waiting queue. mi0 , the request time T of the current query request is earlier than T min , then based on the first capacity threshold, i.e., thread hard limit, it is determined whether there are free thread resources in the thread pool; the request time T of the current query request is later than T min , it is determined whether there are free thread resources in the thread pool based on the second capacity threshold, namely, thread soft limit.

[0105] A possible implementation method is provided in the embodiment of the present application, such as Figure 4 As shown, the above method may further include:

[0106] When the request time is earlier than the first time, and it is determined based on the first capacity threshold of the thread pool that there are no free thread resources in the thread pool, the query task is not executed and an error prompt message is generated.

[0107] In the embodiment of the present application, the first processing node may determine the first time T based on the minimum time value corresponding to all query tasks in the working queue and the waiting queue. min , when the query request time T is earlier than T min , based on the first capacity threshold, it is determined whether there are free thread resources in the thread pool. When there are no free thread resources, it means that the query task cannot be executed immediately, that is, it cannot be added to the work queue. The first processing node can generate an error prompt message to remind the user and cancel all query tasks corresponding to the query request.

[0108] A possible implementation method is provided in an embodiment of the present application. The method further includes:

[0109] When the request time is later than the first time, and it is determined based on the second capacity threshold of the thread pool that there are no free thread resources in the thread pool, the query task is added to the waiting queue.

[0110] In the embodiment of the present application, the first processing node may determine the first time T based on the minimum time value corresponding to all query tasks in the working queue and the waiting queue. min , the request time T of the current query request is later than T min , it is determined whether there are free thread resources in the thread pool based on the second capacity threshold, namely, thread soft limit. When there are no free thread resources, it means that the query task cannot be executed immediately. The first processing node can directly add the query task to the waiting queue to facilitate reasonable scheduling of subsequent thread resources.

[0111] The embodiment of the present application controls the scheduling and execution of query tasks based on the first capacity threshold and the second capacity threshold in the thread pool in the first processing node, which can ensure that the thread resources in the thread pool will not exceed the hardware resource limit of the first processing node, thereby avoiding the first processing node from crashing. At the same time, the capacity of the thread pool is dynamically adjustable within a certain range, further improving the adaptability of the first processing node.

[0112] The above describes the process of determining whether to execute or not execute a query task based on the request time and the capacity threshold of the thread pool. The following describes in detail the process of determining free thread resources based on different capacity thresholds.

[0113] A possible implementation method is provided in an embodiment of the present application. The method further includes:

[0114] Determine a first difference between a first capacity threshold and the number of threads occupied by a work queue of a first processing node; if the first difference is not less than the number of threads required by the query task, determine that there are free thread resources in the thread pool based on the first capacity threshold of the thread pool.

[0115] In an embodiment of the present application, taking the first capacity threshold of 10,000 threads as an example, the number of free threads based on the maximum capacity can be determined based on the difference between the first capacity threshold and the current number of threads occupied by the first processing node, such as 6,000 threads, that is, the first difference is 4,000 threads. When the number of threads required for all query tasks corresponding to the query request is 1,500 threads, it means that the current free thread resources can meet the thread resource requirements of all query tasks in the query request, that is, there are corresponding free thread resources in the thread pool.

[0116] A possible implementation method is provided in an embodiment of the present application. The method further includes:

[0117] Determine a second difference between the second capacity threshold and the number of threads occupied by the work queue of the first processing node; when the second difference is not less than the number of threads required by the query task, determine that there are free thread resources in the thread pool based on the second capacity threshold of the thread pool.

[0118] In an embodiment of the present application, taking the second capacity threshold of 8000 threads as an example, the number of free threads based on the maximum capacity can be determined based on the difference between the second capacity threshold and the current number of threads occupied by the first processing node, such as 6000 threads, that is, the second difference is 2000 threads. When the number of threads required for all query tasks corresponding to the query request is 1500 threads, it means that the current free thread resources can meet the thread resource requirements of all query tasks in the query request, that is, there are corresponding free thread resources in the thread pool.

[0119] The embodiment of the present application controls the capacity of the thread pool through a first capacity threshold and a second capacity threshold, wherein the first capacity threshold is the upper limit value of the number of threads that the first processing node can apply for, and the second capacity threshold is a safety value of the number of threads that the first processing node can apply for. The above-mentioned first capacity threshold and second capacity threshold can ensure that the capacity of the thread pool is within a reasonable tolerance range, and can reasonably and fully utilize the thread resources in the thread pool while avoiding the downtime of the first processing node due to excessive capacity of the thread pool, thereby further improving the query efficiency.

[0120] A possible implementation method is provided in an embodiment of the present application. After the query task is executed, the following steps are included:

[0121] The query task is removed from the work queue to release the target thread resources occupied by the query task; based on the request time of the task to be executed, the target task to be executed is selected from the waiting queue; and the target task to be executed is added to the work queue.

[0122] Specifically, the first processing node may select a target task to be executed from the waiting queue based on resource data required for the task to be executed and current free thread resources of the first processing node; wherein the current free thread resources may be determined according to the first capacity threshold or the second capacity threshold.

[0123] The to-be-executed task may be at least one query task set, and each query task set corresponds to a query request. The to-be-executed tasks in the waiting queue are arranged in ascending order based on the request time.

[0124] In an embodiment of the present application, when the target thread resources released by the first processing node are 1,500 threads, the first three query task sets in the waiting queue can be obtained as set 1, set 2 and set 3; and the thread resources required for set 1 are 500, the thread resources required for set 2 are 600, and the thread resources required for set 3 are 700, then set 1 and set 2 can be first taken as target tasks to be executed and added to the work queue.

[0125] The embodiment of the present application uses query requests as units to determine the tasks to be executed in the waiting queue. When a query request is executed, all corresponding query tasks can be executed, which can ensure the integrity of all query tasks when processing a single query request. When the second processing node receives a query request, the first processing node schedules the task for it. The single query request is either fully executed or not executed, which avoids thread deadlock due to resource competition among multiple query tasks in a single query request, thereby ensuring query efficiency.

[0126] A possible implementation method is provided in an embodiment of the present application. The method further includes:

[0127] For each task to be executed in the waiting queue, a minimum unit thread is allocated.

[0128] In an embodiment of the present application, the first processing node can allocate a thread to each pending task in the waiting queue so that the pending task can be started; when the pending task is a short query, the short query can be executed immediately and then exit the waiting queue; when the pending task is a long query, the long query can be started in the waiting queue first, and when the long query is transferred from the waiting queue to the working queue, the first processing node can continue to allocate the required thread resources to it so that the long query can be executed.

[0129] This application allocates the minimum unit thread to the waiting queue so that short queries in the tasks to be executed can be completed as soon as possible without waiting for long queries. This can avoid short queries being blocked by long queries, causing thread starvation, and achieve fair scheduling of query tasks.

[0130] In order to better understand the processing method of the above query task, the following Figure 5 An example of a method for processing a query task of the present application is described in detail, which is applied to a server cluster in an MPP database. The server cluster may include a first processing node and a second processing node. The method includes the following steps:

[0131] S501: After compiling and instantiating the received query request, the second processing node generates a query task and sends the query task to the first processing node.

[0132] The first processing node or the second processing node may be a node device in an MPP database system, for example, a computer, a node device in a network, such as a base station, a controller device, etc., which is not specifically limited in the embodiments of the present application. The received query request may be compiled and instantiated using methods known in the art, which is also not specifically limited in the embodiments of the present application.

[0133] Specifically, the first processing node may receive the query task sent by the second processing node through the application layer.

[0134] S502, the first processing node determines a timestamp of the query task; the timestamp includes a request time of the query request.

[0135] The timestamp may be a globally unique timestamp assigned to the query task by the first processing node according to the system time.

[0136] In some implementations, the first processing node may parse the query task and directly obtain a timestamp corresponding to the query task.

[0137] In other implementations, the timestamp may further include an identifier of a query request corresponding to the query task, and the first processing node may further obtain the identifier based on query task analysis, and determine the timestamp corresponding to the query task based on the identifier.

[0138] S503, when the request time is earlier than the first time, and the first processing node determines that there are free thread resources in the thread pool based on the first capacity threshold of the thread pool;

[0139] or,

[0140] When the request time is later than the first time, and it is determined based on the second capacity threshold of the thread pool that there are free thread resources in the thread pool;

[0141] Add the query task to the work queue of the first processing node, and execute the query task based on the free thread resources; wherein the first capacity threshold is greater than the second capacity threshold; wherein the first time is the minimum time value of the request time corresponding to all query tasks in the waiting queue and the work queue of the first processing node.

[0142] Specifically, the first processing node may determine free thread resources of the thread pool based on the request time of the query request, and execute the query task based on the free thread resources.

[0143] The capacity of the thread pool can be dynamically adjusted based on the amount of tasks, the first capacity threshold is the maximum number of threads that the first processing node can apply for, and the second capacity threshold is the safety value of the number of threads that the first processing node can apply for. The specific steps for determining the free thread resources of the thread pool are as follows:

[0144] In the embodiment of the present application, the first processing node may determine the first time T based on the minimum time value corresponding to all query tasks in the working queue and the waiting queue. min , the request time T of the current query request is earlier than T min , then based on the first capacity threshold, i.e., thread hard limit, it is determined whether there are free thread resources in the thread pool; the request time T of the current query request is later than T min , it is determined whether there are free thread resources in the thread pool based on the second capacity threshold, namely, thread soft limit.

[0145] S504: When the request time is earlier than the first time, and the first processing node determines that there are no free thread resources in the thread pool based on the first capacity threshold of the thread pool, the query task is not executed and an error prompt message is generated.

[0146] In the embodiment of the present application, the first processing node may determine the first time T based on the minimum time value corresponding to all query tasks in the working queue and the waiting queue. min , when the query request time T is earlier than T min , then based on the first capacity threshold, it is determined whether there are free thread resources in the thread pool. When there are no free thread resources, it means that the query task cannot be executed immediately, that is, it cannot be added to the work queue. The first processing node can generate an error prompt message to remind the user and cancel all query tasks corresponding to the query request.

[0147] S505: When the request time is later than the first time, and the first processing node determines that there are no free thread resources in the thread pool based on the second capacity threshold of the thread pool, the query task is added to the waiting queue.

[0148] In the embodiment of the present application, the first processing node may determine the first time T based on the minimum time value corresponding to all query tasks in the working queue and the waiting queue. min , the request time T of the current query request is later than T min, it is determined whether there are free thread resources in the thread pool based on the second capacity threshold, namely, thread soft limit. When there are no free thread resources, it means that the query task cannot be executed immediately. The first processing node can directly add the query task to the waiting queue to facilitate reasonable scheduling of subsequent thread resources.

[0149] S506, after executing the query task, the first processing node removes the query task from the work queue, releases the target thread resources occupied by the query task; and based on the request time of the task to be executed, selects a target task to be executed from the waiting queue; and adds the target task to be executed to the work queue.

[0150] The to-be-executed task may be at least one query task set, and each query task set corresponds to a query request. The to-be-executed tasks in the waiting queue are arranged in ascending order based on the request time.

[0151] In an embodiment of the present application, when the target thread resources released by the first processing node are 1,500 threads, the first three query task sets in the waiting queue can be obtained as set 1, set 2 and set 3; and the thread resources required for set 1 are 500, the thread resources required for set 2 are 600, and the thread resources required for set 3 are 700, then set 1 and set 2 can be first taken as target tasks to be executed and added to the work queue.

[0152] The first processing node in the embodiment of the present application receives the query task sent by the second processing node, and determines the timestamp of the query task, which timestamp includes the request time of the query request. The first processing node calls the thread resources in the thread pool based on the request time, executes the query task and generates the query result. The present application implements task scheduling based on timestamps and makes rational use of thread resources. Compared with the prior art of scheduling tasks based on the second processing node and then executing the query task based on the first processing node, the present application schedules and executes tasks based on the first processing node, and there is no need to introduce a resource synchronization mechanism, which reduces the time consumption of generating query results and improves query efficiency.

[0153] The present application embodiment provides a query task processing device, such as Figure 6 As shown, the query task processing device 60 may include: a receiving module 601, a determining module 602 and a calling module 603;

[0154] The receiving module 601 is used to receive a query task to be processed sent by the second processing node; the query task is generated by the second processing node based on the query request;

[0155] A determination module 602 is used to determine a timestamp of the query task; the timestamp includes a request time of the query request;

[0156] The calling module 603 is used to call the thread resources in the thread pool of the first processing node based on the request time, execute the query task, and generate the query result.

[0157] The present application embodiment provides a possible implementation method, wherein the calling module 603 is used to:

[0158] When the request time is earlier than the first time, and based on the first capacity threshold of the thread pool, it is determined that there are free thread resources in the thread pool; or when the request time is later than the first time, and based on the second capacity threshold of the thread pool, it is determined that there are free thread resources in the thread pool;

[0159] Add the query task to the work queue of the first processing node, and execute the query task based on the free thread resources; wherein the first capacity threshold is greater than the second capacity threshold; wherein the first time is the minimum time value of the request time corresponding to all query tasks in the waiting queue and the work queue of the first processing node.

[0160] In an embodiment of the present application, a possible implementation manner is provided, wherein the above-mentioned device further includes a first resource monitoring module, which is used to:

[0161] Determine a first difference between a first capacity threshold and a number of thread occupancy of a work queue of a first processing node;

[0162] If the first difference is not less than the number of threads required by the query task, it is determined that there are free thread resources in the thread pool based on the first capacity threshold of the thread pool.

[0163] In an embodiment of the present application, a possible implementation is provided, wherein the above-mentioned device further includes a second resource monitoring module, which is used to:

[0164] Determine a second difference between a second capacity threshold and the number of threads occupied by the work queue of the first processing node;

[0165] When the second difference is not less than the required number of threads for the query task, it is determined that there are free thread resources in the thread pool based on the second capacity threshold of the thread pool.

[0166] A possible implementation method is provided in an embodiment of the present application. The above-mentioned device further includes a prompt module, which is used to:

[0167] When the request time is earlier than the first time, and it is determined based on the first capacity threshold of the thread pool that there are no free thread resources in the thread pool, the query task is not executed and an error prompt message is generated.

[0168] A possible implementation method is provided in an embodiment of the present application. The above-mentioned device further includes an adding module for:

[0169] When the request time is later than the first time, and it is determined based on the second capacity threshold of the thread pool that there are no free thread resources in the thread pool, the query task is added to the waiting queue.

[0170] A possible implementation method is provided in an embodiment of the present application. The above-mentioned device further includes a release module, which is used to:

[0171] Remove the query task from the work queue and release the target thread resources occupied by the query task;

[0172] Based on the request time of the task to be executed, select the target task to be executed from the waiting queue;

[0173] Add the target task to be executed to the work queue.

[0174] A possible implementation method is provided in an embodiment of the present application. The above-mentioned device further includes an allocation module, which is used to:

[0175] For each task to be executed in the waiting queue, a minimum unit thread is allocated.

[0176] The device of the embodiments of the present application can execute the method provided by the embodiments of the present application, and the implementation principles are similar. The actions performed by each module in the device of each embodiment of the present application correspond to the steps in the method of each embodiment of the present application. For the detailed functional description of each module of the device, please refer to the description in the corresponding method shown in the previous text, which will not be repeated here.

[0177] The first processing node in the embodiment of the present application receives the query task sent by the second processing node, and determines the timestamp of the query task, which timestamp includes the request time of the query request. The first processing node calls the thread resources in the thread pool based on the request time, executes the query task and generates the query result. The present application implements task scheduling based on timestamps and makes rational use of thread resources. Compared with the prior art of scheduling tasks based on the second processing node and then executing the query task based on the first processing node, the present application schedules and executes tasks based on the first processing node, and there is no need to introduce a resource synchronization mechanism, which reduces the time consumption of generating query results and improves query efficiency.

[0178] An electronic device is provided in an embodiment of the present application, including a memory, a processor and a computer program stored in the memory, and the processor executes the above-mentioned computer program to implement the steps of the query task processing method. Compared with the related art, it can be achieved: the first processing node in the embodiment of the present application receives the query task sent by the second processing node, and determines the timestamp of the query task, and the timestamp includes the request time of the query request. The first processing node calls the thread resources in the thread pool based on the request time, executes the query task and generates the query result. The present application implements task scheduling based on timestamps and makes rational use of thread resources. Compared with the prior art of scheduling tasks based on the second processing node and then executing the query task based on the first processing node, the present application schedules and executes tasks based on the first processing node, and there is no need to introduce a resource synchronization mechanism, which reduces the time consumption of generating query results and improves query efficiency.

[0179] In an alternative embodiment, an electronic device is provided, such as Figure 7 As shown, Figure 7 The electronic device 700 shown includes: a processor 701 and a memory 703. The processor 701 and the memory 703 are connected, such as through a bus 702. Optionally, the electronic device 700 may also include a transceiver 704, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 704 is not limited to one, and the structure of the electronic device 700 does not constitute a limitation on the embodiments of the present application.

[0180] Processor 701 may be a CPU (Central Processing Unit), a general purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 701 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0181] The bus 702 may include a path to transmit information between the above components. The bus 702 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 702 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0182] The memory 703 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), magnetic disk storage medium, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation herein.

[0183] The memory 703 is used to store the computer program for executing the embodiment of the present application, and the execution is controlled by the processor 701. The processor 701 is used to execute the computer program stored in the memory 703 to implement the steps shown in the above method embodiment.

[0184] The electronic devices include, but are not limited to, mobile terminals such as mobile phones, notebook computers, PADs, etc. and fixed terminals such as digital TVs, desktop computers, etc.

[0185] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps and corresponding contents of the aforementioned method embodiment can be implemented.

[0186] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that when the computer device executes the computer instructions, the following conditions are achieved:

[0187] receiving a query task to be processed sent by the second processing node; the query task is generated by the second processing node based on the query request;

[0188] Determine the timestamp of the query task; the timestamp includes the request time of the query request;

[0189] The thread resources in the thread pool of the first processing node are called based on the request time to execute the query task and generate the query result.

[0190] The terms "first", "second", "third", "fourth", "1", "2", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that shown or described in the drawings.

[0191] It should be understood that, although each operation step is indicated by arrows in the flowchart of the embodiment of the present application, the implementation order of these steps is not limited to the order indicated by the arrows. Unless clearly stated herein, in some implementation scenarios of the embodiment of the present application, the implementation steps in each flowchart can be performed in other orders according to demand. In addition, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on actual implementation scenarios. Some or all of these sub-steps or stages may be executed at the same time, and each sub-step or stage in these sub-steps or stages may also be executed at different times respectively. In different scenarios of execution time, the execution order of these sub-steps or stages may be flexibly configured according to demand, and the embodiment of the present application does not limit this.

[0192] The above is only an optional implementation method for some implementation scenarios of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the solution of the present application, other similar implementation methods based on the technical ideas of the present application are also within the protection scope of the embodiments of the present application.

Claims

1. A method for processing a query task, applied to a first processing node, characterized in that: include: receiving a query task to be processed sent by a second processing node; The query task is generated by the second processing node based on the query request; Determine a timestamp of the query task; the timestamp includes a request time of the query request; Calling thread resources in the thread pool of the first processing node based on the request time, executing the query task, and generating query results; The calling of thread resources in the thread pool of the first processing node based on the request time to execute the query task includes: When the request time is earlier than the first time, and based on the first capacity threshold of the thread pool, it is determined that there are free thread resources in the thread pool; or when the request time is later than the first time, and based on the second capacity threshold of the thread pool, it is determined that there are free thread resources in the thread pool; The query task is added to the work queue of the first processing node, and the query task is executed based on the free thread resources; wherein the first capacity threshold is greater than the second capacity threshold; wherein the first time is the minimum time value of the request time corresponding to all query tasks in the waiting queue and the work queue of the first processing node.

2. The method according to claim 1, characterized in that The method further comprises: Determine a first difference between the first capacity threshold and the number of threads occupied by the work queue of the first processing node; If the first difference is not less than the required number of threads of the query task, it is determined that there are free thread resources in the thread pool based on the first capacity threshold of the thread pool.

3. The method according to claim 1, characterized in that The method further comprises: Determine a second difference between the second capacity threshold and the number of threads occupied by the work queue of the first processing node; When the second difference is not less than the required number of threads of the query task, it is determined that there are free thread resources in the thread pool based on the second capacity threshold of the thread pool.

4. The method according to claim 1, characterized in that: The method further comprises: When the request time is earlier than the first time, and it is determined based on the first capacity threshold of the thread pool that there are no free thread resources in the thread pool, the query task is not executed and an error prompt message is generated.

5. The method according to claim 1, characterized in that The method further comprises: When the request time is later than the first time, and it is determined based on the second capacity threshold of the thread pool that there are no free thread resources in the thread pool, the query task is added to the waiting queue.

6. The method according to claim 1, characterized in that After executing the query task, the method further comprises: Removing the query task from the work queue to release the target thread resources occupied by the query task; Based on the request time of each to-be-executed task in the waiting queue, selecting a target to-be-executed task from the waiting queue; Add the target task to be executed to the work queue.

7. The method according to claim 1, characterized in that The method further comprises: A minimum unit thread is allocated to each task to be executed in the waiting queue.

8. A query task processing device, characterized in that: The device is applied to a first processing node, comprising: A receiving module, configured to receive a query task to be processed sent by a second processing node; the query task is generated by the second processing node based on a query request; A determination module, configured to determine a timestamp of the query task; the timestamp includes a request time of the query request; Calling module for Calling thread resources in the thread pool of the first processing node based on the request time, executing the query task, and generating query results; When the calling module is used to call the thread resources in the thread pool of the first processing node based on the request time, execute the query task, and generate the query result, it is specifically used to: When the request time is earlier than the first time, and based on the first capacity threshold of the thread pool, it is determined that there are free thread resources in the thread pool; or when the request time is later than the first time, and based on the second capacity threshold of the thread pool, it is determined that there are free thread resources in the thread pool; The query task is added to the work queue of the first processing node, and the query task is executed based on the free thread resources; wherein the first capacity threshold is greater than the second capacity threshold; wherein the first time is the minimum time value of the request time corresponding to all query tasks in the waiting queue and the work queue of the first processing node.

9. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images