Method and apparatus for processing high concurrency query requests

[0069] Hereinafter, exemplary embodiments of the present invention will be described in more detail with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0070] The embodiment of the present invention provides a method for processing high concurrent query requests, such as figure 1 As shown, the method is to determine the initial query request and the same query request corresponding to the initial query request contained in multiple concurrent query requests received within a period of time, and execute the initial query request, and compare the initial query request to The result is multiplexed to the same query request corresponding to the initial query request to reduce query workload and improve the work efficiency of query data. For this, the embodiment of the present invention provides the following specific steps:

[0071] 101. Receive multiple concurrent query requests.

[0072] Wherein, the concurrent query request includes the initial query request and the same query request corresponding to the initial query request. The same query request refers to the data received by the server during the period from when the server receives the initial query request to the completion of the query operation of the initial query request. The initial query request is the same concurrent query request.

[0073] In the embodiment of the present invention, when concurrent query requests are received, the structured query language (Structured Query Language, SQL) statement corresponding to each query request can be extracted, and the query conditions contained in the SQL statement and The query keywords are analyzed to realize whether there are multiple identical query requests in the concurrent query requests received. For the embodiment of the present invention, the method for determining whether the same query request exists in concurrent query requests is not limited.

[0074] When processing data through Online Analytical Processing (OLAP), the usual approach is to pre-calculate the indicator set according to the combination of dimensions for a time range of data. This pre-calculation is called pre-aggregation. Then save the pre-aggregation result. According to the above-mentioned pre-aggregation result, the user can apply OLAP to analyze the data more specifically in multiple dimensions. Based on the above-mentioned pre-aggregated results for query analysis, it can be found that the query request is idempotent, that is, the query results corresponding to the same query request executed multiple times are the same. According to this, when the query request status is high, the query There may be many identical concurrent query requests in the request.

[0075] For the embodiment of the present invention, when a high concurrent query request is initiated for the pre-aggregated result in the OLAP data processing process, there will be many identical concurrent query requests in the query request. For these same concurrent query requests, according to the query instructions received by the server The chronological order of, the query request corresponding to the query command received by the server for the first time can be determined as the initial query request, and the query request corresponding to the query command received later by the server can be determined as the same query request corresponding to the initial query request.

[0076] For example, such as figure 2 As shown, a high concurrent query request is initiated for the pre-aggregated result in the OLAP data processing process. Within a certain time range, such as the time range from t1 to t4, the server sequentially receives the same concurrent query requests q1 and q3 in chronological order , Q2, q4, q5, where the server receives the query request q1 at time t1, and the execution is completed at time t2, and within the time range from t1 to t2, the server receives the query request q2 and q3 successively, then it can be determined from t1 to In the time range of t2, q1 is the initial query request and q2 and q3 are the same query requests corresponding to q1. Similarly, in the time range from t3 to t4, q4 is the initial query request, and q5 is the same query request corresponding to q4.

[0077] 102. Extract the initial query request.

[0078] For the embodiment of the present invention, when a high-concurrency query request is initiated for the pre-aggregated result in the OLAP data processing process, the server will receive multiple concurrent query requests, and determine the initial query request included in the multiple concurrent query requests according to step 101 , And extract the initial query request to separate the initial query request from the same query request. For example, according to the example in step 101 above, determine the initial query request q1 and the same query corresponding to q1 within the time range from t1 to t4 The same query q5 corresponding to q2 and q3, the initial query request q4 and q4, and the initial query request q1 and q4 are extracted.

[0079] 103. Block the same query request.

[0080] For the embodiment of the present invention, after determining the initial query request and the same query request contained in multiple concurrent query requests, according to the initial query request, create a synchronization waiting structure corresponding to the initial query request to store the same query request and block The query operation requested by the same query. For example, according to the example in step 101 above, q1 is the initial query request, q2 and q3 are the same query requests corresponding to q1, then a synchronous waiting structure corresponding to q1 is created to store q2 and q3, and block q2 and q3 Query operation.

[0081] 104. Determine the query result corresponding to the initial query request.

[0082] After determining the initial query request contained in the multiple concurrent query requests, the server executes the query instruction of the initial query request and saves the query result in the server cache.

[0083] 105. Determine the query result corresponding to the initial query request as the query result corresponding to the same query request and return it to the corresponding query user.

[0084] After the server finishes executing the query instruction of the initial query request and determines the query result, the server calling thread wakes up the same query request blocked in the synchronous waiting structure, and the same query request can obtain the query result corresponding to the initial query request through the query server cache. That is, the query result corresponding to the same query request can be determined, and the query result will be returned to each query user according to concurrent query requests initiated by different query users.

[0085] When the same query request blocked in the synchronization waiting structure obtains the corresponding query result, the server will cancel the synchronization waiting structure. According to this, the synchronization waiting structure created in the embodiment of the present invention has a certain life cycle, that is, according to the corresponding The initial query request of is created, and after the same query request reuses the query result of the initial query request, it is cancelled by the server. Through the above-mentioned method of dynamically establishing and deregistering the synchronization waiting structure, server resources can not be occupied too much and the processing performance of the server can be improved.

[0086] The present invention provides a method and device for processing high-concurrency query requests. When a high-concurrency query request is initiated for the pre-aggregated results in the OLAP data processing process, the embodiment of the present invention does not require multiple concurrent query requests received. When performing a query operation, by determining the initial query request and the same query request corresponding to the initial query request contained in multiple concurrent query requests received within a period of time, only the initial query request is executed, and the same query request is blocked. By performing an initial query operation, the query result is reused to the same query request corresponding to the initial query request, that is, by performing a query operation, the query result of the initial query request and the same query request corresponding to the initial query request can be obtained , Which greatly reduces the query workload corresponding to high concurrent query requests, saves a lot of processing resources, and improves the efficiency of querying data.

[0087] Based on the specific implementation of the steps in the above embodiment, the embodiment of the present invention provides another method for processing high concurrent query requests, such as image 3 As shown, this method uses an OLAP query system based on a distributed framework, blocks the same query request on the local node, performs the query operation of the initial query request on the remote node, and stores the query result in the distributed cache of the remote node. The distributed cache of each node has the consistency of read and write data, the same query request can reuse the result of the initial query request, and the corresponding result can be obtained in the local distributed cache, which reduces the workload of query and improves the efficiency of query data For this, the embodiment of the present invention provides the following specific steps:

[0088] 201. Receive multiple concurrent query requests.

[0089] Wherein, the concurrent query request includes the initial query request and the same query request corresponding to the initial query request. For a specific description, refer to step 101, which will not be repeated here.

[0090] It should be noted that for the embodiment of the present invention, a distributed framework is adopted to construct data storage, data analysis, and data calculation on a cluster composed of multiple nodes. When the pre-aggregation result in the OLAP processing data process is initiated high concurrency When querying a request, the embodiment of the present invention uses the aforementioned distributed framework to process high-concurrency query requests.

[0091] For example, such as Figure 4 The OLAP query system based on the Akka framework is shown. The entire query system consists of two clusters, including: Node1, Node2 and Node3 in the figure are 3 nodes in the distributed framework, composed of these 3 peers A peer-to-peer Akka cluster; another cluster is composed of distributed caches on 3 nodes. In a distributed cache cluster, when data is written in the distributed cache of any node, in the distributed cache of other nodes The data can be read to ensure the consistency of reading and writing. Among them, the nodes Node1, Node2, and Node3 have the same structure and function, that is, you can choose to receive concurrent query requests at any of Node1, Node2, and Node3, and send the initial query request to any of the other nodes Perform OLAP query, so nodes Node1, Node2, and Node3 are equal. For the distributed framework adopted in the embodiment of the present invention, there may be multiple nodes, and each node is equivalent. In the embodiment of the present invention, only the nodes Node1, Node2, and Node3 are exemplified.

[0092] For the examples of the embodiments of the present invention, it is not limited to the OLAP query system based on the Akka framework, but because based on the Akka framework and the Scala programming language, the Akka framework does not need application layer code to pay attention to data transmission when implementing basic message communication. And Akka's Actor-based programming model makes it easier to write scalable distributed applications. Therefore, the embodiments of the present invention can be explained more clearly by using examples of the OLAP query system implemented based on the Akka framework. As for the OLAP query system implemented by other distributed frameworks, the method of implementing concurrent query requests can be equivalent to the OLAP query system implemented based on the Akka framework in the embodiment of the present invention, and the embodiment of the present invention does not need to give an example.

[0093] 202. Query whether the query result corresponding to the concurrent query request exists in the distributed cache of the local node.

[0094] In the distributed framework, for multiple concurrent query requests received by a local node, firstly, each concurrent query request is judged in turn, and whether there is a query result corresponding to the concurrent query request in the distributed cache of the local node. For a cluster composed of the distributed cache of each node, when data is written in the distributed cache of one node, the data can be read by the distributed cache of other nodes. Therefore, by querying the distributed cache of the local node, it can be determined whether there is a query result corresponding to the concurrent query request in the distributed framework. For the embodiment of the present invention, the query result corresponding to the concurrent query request can be obtained by querying the distributed cache of the local node, which can improve query efficiency.

[0095] 203a. If there is a query result corresponding to the concurrent query request, send the query result to the querying user.

[0096] Through the judgment in step 202 above, the query result corresponding to the concurrent query request exists in the distributed cache of the local node. For the embodiment of the present invention, it is also necessary to determine whether the query result is valid according to the preset cache time validity period.

[0097] Because the data processed by OLAP is continuously increasing in batches, if the cache time validity period is not set, the processing pressure on the server will increase, and as the data is updated, the corresponding query results are also time-sensitive, so if the query results do not exceed the cache Time validity period, the query result is valid data, if the query result exceeds the cache time validity period, the query result is invalid data, and the invalid query result is cleared.

[0098] If the query result corresponding to the concurrent query request is valid in the distributed cache of the local node, you can query the distributed cache of the local node to obtain the query result corresponding to the concurrent query request, and send the query result to the corresponding Query users.

[0099] 203b. Determine whether the concurrent query request is an initial query request.

[0100] Through the judgment of step 202 above, if there is no query result corresponding to the concurrent query request in the distributed cache of the local node, it is necessary to perform subsequent processing on the concurrent query request received by the local node, and first determine whether the concurrent query request is the initial query request , The specific description includes: query the local node whether there is a synchronization waiting structure corresponding to the concurrent query request, if it exists, it means that the local node has received the same concurrent query request before receiving the concurrent query request, and according to the initial The query request establishes a corresponding synchronous waiting structure, then the concurrent query request is not the initial query request but the same query request. If it does not exist, it means that the local node has not received the same concurrent query request, and the concurrent query request is the initial query request.

[0101] 204a. If the concurrent query request is an initial query request, extract the initial query request, and create a synchronous waiting structure according to the initial query request.

[0102] If the concurrent query request is an initial query request, then extract the initial query request, refer to the example in step 102, which will not be repeated here. According to the initial query request, the synchronization waiting structure corresponding to the initial query request is created, which is used to store the same query request in the synchronization waiting structure when the concurrent query request received by the local server is the same as the initial query request. Block the query operation of the same query request.

[0103] For example, such as Figure 4 As shown, the local server Node1 receives multiple concurrent query requests, and by judging the concurrent query request currently received, if it is an initial query request, it creates a synchronization waiting structure corresponding to the initial query request in the concurrency control blocking waiting module in Node1. Later, when the concurrent query request received by the local server Node1 is the same as the initial query request, the same query request is stored in the synchronous waiting structure, so as to block the query operation of the same query request.

[0104] For Figure 4 The illustrated example of the embodiment of the present invention, further, an example of the programming code of the synchronous waiting structure based on the Akka framework and the Scala programming language is as follows:

[0105]

[0106]

[0107] The programming codes listed above are explained one by one according to the annotations in the programming code, as follows:

[0108] (1) Declare the lock and condition used in the synchronization waiting structure;

[0109] (2) Define the variable closed to determine whether the initial query request has been executed and returned;

[0110] (3) When the local node receives the same query request, it calls reEntranceWait to block the same query request;

[0111] (4) Block the same query request on the condition variable condition;

[0112] (5) Return the response set by the destroy method. After the initial query is executed, a query completion message will be sent, which includes the query exception or query success;

[0113] (6) The remote node sends a query completion message to the local node. When the initial query request returns, the thread is obtained from the thread pool and the destroy method is called to wake up the same query request in the synchronization waiting structure;

[0114] (7) Set the query result result of the initial query request to response, and the query result can be reused for the same query request;

[0115] (8) Set the closed flag to true, indicating that the initial query request corresponding to the synchronous waiting structure has returned, so that the thread blocking the same query request exits from the while of the reEntranceWait method, and no longer stores the same query request to the synchronous waiting structure;

[0116] (9) Through condition.signalAll() to wake up the thread blocking the same query request on the condition variable condition.

[0117] Specifically, the above-exemplified programming code is described. The synchronization waiting structure corresponding to the initial query request is realized by QueryStatus. It is a dynamic runtime structure. When the local node receives the initial query request, the synchronization waiting structure is created, and the synchronization waiting structure is created in a remote location. After the node executes the initial query request, the same query request reuses the query result of the initial query request and is cancelled. The destroy method is called when the remote node performs the initial query request and returns. When the closed flag is set to true, pass the condition .signalAll() wakes up the blocked thread of the same query request, the reEntranceWait() method is called by the same query request, and awakens when the same query request is blocked by condition.await() waiting for the initial query request to return.

[0118] 204b. If the concurrent query request is not the initial query request, determine the concurrent query request as the same query request, and add the same query request to the corresponding synchronization waiting structure to block the same query request.

[0119] If the concurrent query request is not the initial query request, it can be determined that before the local server receives the concurrent query request, it has first received the same query request as the concurrent query request, that is, the initial query request corresponding to the concurrent query request, then the concurrent query request The query request is the same query request corresponding to the initial query request, and according to the above step 203c, it can be determined that the concurrent control blocking waiting module in the local server already has a synchronization waiting structure corresponding to the initial query request, so the concurrent query request is added In the corresponding synchronous waiting structure, the query operation of the concurrent query request is blocked.

[0120] For Figure 4 The illustrated example of the embodiment of the present invention, further, based on the Akka framework and the Scala programming language, realizes the above-mentioned programming code for adding concurrent query requests to the corresponding synchronous waiting structure as follows:

[0121] private val queryInProcess=new ConcurrentHashMap[OlapQuery,QueryStatus]asScala

[0122] Each concurrent query request received by the local node is represented by OlapQuery, and the corresponding relationship with QueryStatus is stored in the concurrent hash Map queryInProcess, OlapQuery is the key, QueryStatus is the value, and is written to queryInProcess through the method putIfAbsent. Among them, OlapQuery corresponds to the concurrent query request received by the local node, and QueryStatus corresponds to the synchronization waiting structure of the local node. Through the putIfAbsent method, the concurrent query request hash is added to the processing queue Map queryInProcess in the corresponding synchronization waiting structure. The specific code is not described here.

[0123] 205. Determine the query result corresponding to the initial query request.

[0124] Based on the initial query request extracted in step 204c, the specific description of this step includes:

[0125] First, extract the initial query request at the local node and send it to the remote node in the distributed cluster. The remote node executes the query operation. The query result corresponding to the initial query request is obtained at the remote node, and the query result is written to the remote node Distributed cache.

[0126] Then, the local node receives the query completion message returned by the remote node, and the query completion message indicates whether the initial query result is successful or abnormal.

[0127] Finally, since the read and write data of the distributed cache of each node in the distributed cluster can be kept consistent, the query result of the initial query request written on the remote node can be obtained by reading the distributed cache of the local node.

[0128] For example, such as Figure 4 As shown in the OLAP query system based on the Akka framework, when the local node Node1 is receiving multiple concurrent query requests, the initial query request is extracted through the above step 204c, and the initial query request is sent to the distributed cluster in the distributed framework The remote node Node2 executes the initial query request on the remote node Node2 and obtains the query result. After the query result is written into the distributed cache of the remote node Node2, the remote node Node2 sends a query completion message to the local node Node1, where the query completion message includes If the query is successful and the query is abnormal, the local node Node1 receives the query completion message, and reads the query result written in the distributed cache of the remote node Node2 in the distributed cache of the local node, that is, the initial query request can be obtained on the local node Node1 The corresponding query result. Such as Figure 4 As shown, the remote node Node3 is equivalent to the remote node Node2, and the local node Node1 can also send the extracted initial query request to the remote node Node3, which will not be described in detail here.

[0129] 206. Determine the query result corresponding to the initial query request as the query result corresponding to the same query request and send it to the corresponding query user.

[0130] For the embodiment of the present invention, the detailed description of this step includes: when the local node receives the query completion message of the initial query request sent by the remote node, wake up the same query request that is blocked by the synchronous waiting structure, and read from the distributed cache of the local node The obtained query result of the initial query request is determined to be the query result corresponding to the same query request, and the synchronization waiting structure corresponding to the initial query request of the local node is cancelled.

[0131] For example, such as Figure 4 The shown OLAP query system based on the Akka framework, when the local node Node1 receives the query completion message for the initial query request, wake up the synchronization waiting structure corresponding to the initial query request in the concurrency control blocking waiting module in the local node Node1, The same query request corresponding to the initial query request stored in the synchronous waiting structure By querying the query result in the distributed cache of the local node Node1, the query result corresponding to the initial query request written in the distributed cache of the remote node Node2 can be obtained, that is, the same The query request can obtain the corresponding query result, and the local node Node1 cancels the synchronization waiting structure corresponding to the initial query request.

[0132] For such as Figure 4 The illustrated example of the embodiment of the present invention, further, based on the Akka framework and the Scala programming language, in the above steps 205 and 206, the implementation of the initial query request, query scheduling in the distributed framework and cancellation of the synchronization waiting structure programming code as follows:

[0133]

[0134]

[0135] The programming codes listed above are explained one by one according to the annotations in the programming code, as follows:

[0136] (1) createQueryActor implements the scheduling of initial query requests within the distributed cluster through the Actor model;

[0137] (2) After the initial query request is executed, the remote node will send a query completion message corresponding to the initial query request;

[0138] (3) The query completion message indicates that the initial query request is successful, and the local node fetches the query result written by the distributed cache of the remote node from the distributed cache;

[0139] (4) Return the query result in (3) to the corresponding query user;

[0140] (5) Call the destroy method on QueryStatus. When the closed flag is set to true, it indicates that the initial query request has awakened the same query request in QueryStatus;

[0141] (6) The corresponding relationship between query query and QueryStatus(qs) is deleted from queryInProcess, that is, the synchronization waiting structure corresponding to the initial query request is cancelled.

[0142] The time window between (5) and (6) in the above code will not affect the normal processing logic of QueryStatus. If (5) has not been executed (6), the local node receives the same concurrent For the query request, since the same query request in the synchronous waiting structure has been awakened, the same query request will not be added to the synchronous waiting structure, but the query result corresponding to the initial query request is directly obtained. If after executing (5) and (6), the local node receives the same concurrent query request, because the synchronous waiting structure has been cancelled, assuming that there is no query cache result here, then the same concurrent query request is New initial query request.

[0143] It should be noted that, for example, for the OLAP query system implemented based on the Akka framework in the embodiment of the present invention, only one thread pool is used to process the received concurrent query request and after the query result corresponding to the initial query request is determined, the query result is determined The logical process of requesting corresponding query results for the same query. Assuming that the thread pool can only start up to 3 threads, at this time, when the multiple concurrent query requests received by the local node include 4 identical query requests, then in addition to sending the initial query request to the remote node, there are 3 remaining The same query request will be blocked in the synchronous waiting structure and occupy all 3 threads in the thread pool. When the remote node executes the initial query request, in the logical process of determining the query result as the query result corresponding to the same query request, it is necessary to obtain threads in the thread pool to perform corresponding processing, but at this time, the remaining 3 are the same The query request occupies all 3 threads in the thread pool, and there are no idle threads in the thread pool, so a deadlock is formed.

[0144] Based on the above-mentioned deadlock problem, for the embodiment of the present invention, it should be noted that two thread pools are created in the embodiment of the present invention to prevent the pre-aggregation results in the OLAP data processing process from initiating high concurrent query requests. When the local node receives a concurrent query request, the thread created in the first thread pool is used to wake up when the remote node sends an initial query request query completion message to the local node. The specific description includes: The same query request in the synchronous waiting structure of the local node uses the threads created in the second thread pool, and the first thread pool and the second thread pool are thread pools obtained by dividing corresponding processing resources according to preset rules, where the preset rules are Refers to the rule of dividing the first thread pool and the second thread pool based on the limited processing resources of the server. Because the number of threads required when the local node receives multiple concurrent query requests is much larger than the threads required for the same query in the synchronous waiting structure to wake up the local node Therefore, when dividing the limited processing resources of the server, more processing resources should be allocated to the first thread pool, and less processing resources should be allocated to the second thread pool. According to such preset rules, reasonable allocation can improve the utilization of threads. Rate, thereby improving the processing performance of the server.

[0145] Specifically, for Figure 4 The illustrated example of the embodiment of the present invention, further, based on the Akka framework and the Scala programming language, the programming code for creating two thread pools is as follows:

[0146] private val senderExecutor=new ThreadPoolExecutor(10,30,0L,

[0147] TimeUnit.MILLISECONDS,new LinkedBlockingQueue[Runnable](100),newThreadPoolExecutor.CallerRunsPolicy());

[0148] private val receiverExecutor=newThreadPoolExecutor(10,30,0L,

[0149] TimeUnit.MILLISECONDS,new LinkedBlockingQueue[Runnable](100),newThreadPoolExecutor.CallerRunsPolicy());

[0150] private val senderEC=ExecutionContext.fromExecutorService(senderExecutor) private val receiverEC=

[0151] ExecutionContext.fromExecutorService(receiverExecutor)

[0152] Two thread pools are defined here, including: the first thread pool senderExecutor and the second thread pool receiverExecutor. The senderExecutor is used to process the local node to receive multiple concurrent requests, and the receiverExecutor is used to send the initial query request query completion message to the local node on the remote node. When a node is awakened, the same query request in the synchronization waiting structure of the local node is used. Two thread pools are used to prevent possible deadlock problems.

[0153] Among them, the first thread pool is used to process multiple concurrent query requests received by the local node. Specifically, the programming code based on the Akka framework and the Scala programming language is as follows:

[0154]

[0155]

[0156] For the above code, receive is the message receiver of the Akka framework, and each case processes a received message type. When an OlapQuery message is received, that is, when the local node receives a concurrent query request, the future method of Futures is used to execute the query request asynchronously, and the specific query process is encapsulated in the query method. It should be noted here that the execution thread of the future method comes from the thread created by the first thread pool.

[0157] Based on another method for processing high concurrent query requests provided by the above embodiment of the present invention, the query process of the above method in the embodiment of the present invention is as follows: Figure 5 As shown, the specific description is as follows:

[0158] First, when the local node receives multiple concurrent query requests, it queries the distributed cache of the local node to see if the query result corresponding to the concurrent query request exists. If so, it needs to determine whether the query result is valid data according to the validity period of the cache time. If it is valid data, the query result is returned to the corresponding query user; if it is invalid data, the invalid data is cleared in the cache.

[0159] Second, if you query the distributed cache of the local node and there is no query result corresponding to the concurrent query request, you need to determine whether the concurrent query request is the initial query request, and you can query the local node's concurrency control blocking waiting module to see if the concurrent query request exists If the corresponding synchronous waiting structure exists, it is determined that the concurrent query request is the same query request, that is, the local node has previously received the same concurrent query request, and if it does not exist, the concurrent query request is determined to be the initial query request;

[0160] Third, if the concurrent query request is an initial query request, a synchronization waiting structure corresponding to the initial query request is created in the concurrency control blocking waiting module of the local node to store the same query request subsequently received by the local node. The query operation of the query request, extract the initial query request and send it to the remote node to execute the query request; if the concurrent query request is the same query request, store the same query request in the synchronous waiting structure, blocking the query of the same query request operating;

[0161] Fourth, when the remote node has executed the initial query request, the query result is stored in the distributed cache of the remote node, and the query completion message is sent to the local node to wake up the same query request in the synchronization waiting structure of the local node. The distributed cache of each node in the cluster maintains data consistency, so the query result corresponding to the initial query request can be obtained by querying the distributed cache of the local node, that is, the initial query request and the same query request can obtain the corresponding query result After that, the synchronization waiting structure in the concurrency control blocking module of the local node is cancelled, and the processing resources are released.

[0162] Specifically, for Figure 5 The query flowchart shown, further, based on the Akka framework and the Scala programming language, the programming code to implement the above query process is as follows:

[0163]

[0164] The programming codes listed above are explained one by one according to the annotations in the programming code, as follows:

[0165] (1) Declare the default query response result;

[0166] (2) Find the cache data corresponding to OlapQuery, that is, according to the concurrent query request received by the local node, find whether there is a corresponding query result in the distributed cache cluster;

[0167] (3) If there is no query result corresponding to the concurrent query request in the distributed cache of the local node;

[0168] (4) Create a new QueryStatus structure, that is, create a synchronous waiting structure corresponding to the initial query request;

[0169] (5) Determine whether the received concurrent query request is an initial query request by querying whether the local node has a synchronous query request corresponding to the received concurrent query request. The specific code is to call putIfAbsent to write OlapQuery and qs into the concurrent hash MapqueryInProcess. If there is already an item corresponding to OlapQuery, putIfAbsent returns the old QueryStatus value, that is, the concurrent query request is not the initial query request, that is (6), otherwise the OlapQuery is the initial request, putIfAbsent can successfully write the newly created QueryStatus into queryInProcess , That is, according to the initial query request, establish a synchronization waiting structure, and the execution flow proceeds to (9);

[0170] (6) The same concurrent query request has been received before the concurrent query request received locally, that is, it is determined that the concurrent query request is the same query request;

[0171] (7) Call reEntranceWait to block waiting on the QueryStatus structure oldValue established by the initial query request, that is, to block the query operation of the same query request by storing the same query request in the synchronous waiting structure;

[0172] (8) When the remote node sends the initial query request query completion message to the local node, the processing thread corresponding to the same query request blocked in the synchronous waiting structure is awakened. After the thread is awakened, the query result res corresponding to the initial query request is returned to the caller replyTo, which is returned to the corresponding query user;

[0173] (9) DirectQuery is scheduled by the system, that is, the initial query request is sent to the remote node, and the query operation is performed on the remote node;

[0174] (10) After the local node receives the concurrent query request, it can first query whether the corresponding query result exists in the distributed cache. If it exists, call isValid4CacheResult to determine whether the query result is still valid. If it is valid, return the query result to the caller replyTo, Among them, the isValid4CacheResult method here is a Lazy processing method, that is, when the query result is obtained, it is judged whether the query result is expired;

[0175] (11) If there is a query result in the distributed cache, and the cached data is no longer valid (for example, new data causes the previously queried cache data to become invalid), clear the cache in (12), and call the query method recursively in (13) Perform the above query process once.

[0176] In order to achieve the above-mentioned object, according to another aspect of the present invention, an embodiment of the present invention also provides a storage medium, the storage medium includes a stored program, wherein, when the program runs, the device where the storage medium is located is controlled to execute The above-mentioned method for processing high concurrent query requests.

[0177] In order to achieve the above objective, according to another aspect of the present invention, an embodiment of the present invention also provides a processor, the processor is configured to run a program, wherein the program executes the above-mentioned processing high concurrent query request when the program is running. Methods.

[0178] Further, as for the above figure 1 , image 3 For the realization of the method shown, another embodiment of the present invention also provides an apparatus for processing high-concurrency query requests. This device embodiment corresponds to the foregoing method embodiment. For ease of reading, this device embodiment will not repeat the details of the foregoing method embodiment one by one, but it should be clear that the device in this embodiment can correspondingly implement the foregoing method implementation Everything in the example. The device is used to improve the query processing efficiency in response to high concurrent query requests, such as Image 6 As shown, the device includes:

[0179] The receiving unit 301 is configured to receive multiple concurrent query requests, where the concurrent query requests include an initial query request and the same query request corresponding to the initial query request;

[0180] The extracting unit 302 is configured to extract the initial query request received by the receiving unit 301;

[0181] The blocking unit 303 is configured to block the same query request received by the receiving unit 301;

[0182] The determining unit 304 is configured to determine the query result corresponding to the initial query request extracted by the extracting unit 302;

[0183] The determining unit 304 is further configured to determine that the query result is the query result corresponding to the same query request blocked by the blocking unit 303 and return to the corresponding query user.

[0184] Further, such as Figure 7 As shown, the determining unit 304 includes:

[0185] The sending module 3041 is configured to send the initial query request extracted by the extraction unit 302 to the remote node in the distributed cluster, and the remote node performs the query operation, obtains the query result corresponding to the initial query request, and transfers the The query result is written into the distributed cache of the remote node;

[0186] The receiving module 3042 is configured to receive the query completion message returned by the remote node;

[0187] The obtaining module 3043 is configured to obtain the query result in the distributed cache of the local node according to the query completion message, where each node included in the distributed cluster has a distributed cache, and when the query is in a node When data is written in the distributed cache, the data can be read by the distributed cache of other nodes.

[0188] Further, such as Figure 7 As shown, the device further includes:

[0189] The determining unit 305 is configured to determine whether the concurrent query request received by the receiving unit 301 is an initial query request;

[0190] The extracting unit 302 is further configured to extract the initial query request when the determining unit 305 determines that the concurrent query request is an initial query request;

[0191] The first creation unit 306 is configured to create a synchronization waiting structure according to the initial query request extracted by the extraction unit 302, where the synchronization waiting structure is used to block the same query request;

[0192] The determining unit 304 is further configured to determine that the concurrent query request is the same query request when the determining unit 305 determines that the concurrent query request is not an initial query request;

[0193] The adding unit 307 is configured to add the same query request determined by the determining unit 304 to the corresponding synchronization waiting structure, so that the blocking unit 303 blocks the same query request.

[0194] Further, such as Figure 7 As shown, the determining unit 304 includes:

[0195] The wake-up module 3044 is configured to wake up the same query request blocked by the synchronous waiting structure when the receiving module 3042 receives the query completion message;

[0196] The determining module 3045 is configured to determine the query result obtained by the obtaining module 3043 in the distributed cache of the local node as the query result corresponding to the same query request;

[0197] The logout module 3046 is used to logout the synchronization waiting structure corresponding to the initial query request.

[0198] Further, such as Figure 7 As shown, the device further includes:

[0199] The query unit 308 is configured to query whether the query result corresponding to the concurrent query request received by the receiving unit 301 exists in the distributed cache of the local node;

[0200] The sending unit 309 is configured to send the query result of the query by the query unit 308 to the query user if there is a query result corresponding to the concurrent query request.

[0201] Further, such as Figure 7 As shown, the device further includes:

[0202] The second creating unit 310 is configured to create a processing thread corresponding to the concurrent query request in the first thread pool according to the concurrent query request received by the receiving unit 301;

[0203] The third creating unit 311 is configured to create a processing thread for obtaining the query result in the second thread pool when the determining unit 304 determines the query result, where the first thread pool and the query result The second thread pool is a thread pool obtained by dividing corresponding processing resources according to preset rules.

[0204] In summary, a method and device for processing high-concurrency query requests adopted in the embodiments of the present invention. When a high-concurrency query request is initiated for the pre-aggregated results in the OLAP data processing process, the embodiment of the present invention adopts distributed The OLAP query system implemented by the framework, when the local node uses the thread created in the first thread pool to receive multiple concurrent query requests, determine the initial query request and the same query request included in the concurrent query request, and send the initial query request to a remote location Node processing, while blocking the same query request in the synchronization waiting structure of the local node. When the remote node has completed the initial query request, the thread created in the second thread pool is used to wake up the same query request in the synchronization waiting structure. The same query request can be Reuse the query result of the initial query request, that is, implement the initial query request once to obtain the initial query request and the query result of the same query request corresponding to the initial query request, greatly reducing the workload of querying data, saving processing resources, and In the distributed framework, there are multiple nodes to obtain more processing resources. Therefore, the processing performance of the processor can be improved by the above-mentioned method of using processing resources, thereby improving the efficiency of querying data. In addition, the aforementioned use of threads in two different thread pools to process data can also prevent the deadlock from being available when no idle thread is available when the same query request in the synchronization waiting structure of the local node is awakened. In addition to the above statement, after the local node receives multiple concurrent query requests, it can first query the distributed cache of the distributed cluster. Since the data stored in the distributed cache of each node can maintain consistency, by querying the distributed cache of the local node , It can be determined whether the query result corresponding to the concurrent query request exists in the entire distributed cluster, and if it exists, the result can be directly returned to the query user, which further improves the efficiency of querying data.

[0205] The device for processing high concurrent query requests includes a processor and a memory. The receiving unit, extracting unit, blocking unit, and determining unit are all stored as program units in the memory, and the above program units stored in the memory are executed by the processor. Realize the corresponding function.

[0206] The processor contains the kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the working efficiency of data can be queried by adjusting kernel parameters.

[0207] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and/or non-volatile memory, such as read-only memory (ROM) or flash memory (flash RAM), and the memory includes at least one Memory chip.

[0208] The embodiment of the present invention provides a storage medium on which a program is stored. When the program is executed by a processor, the method for processing high-concurrency query requests is realized.

[0209] The embodiment of the present invention provides a processor, the processor is used to run a program, wherein the method for processing a high concurrent query request is executed when the program is running.

[0210] The embodiment of the present invention provides a device. The device includes a processor, a memory, and a program stored on the memory and running on the processor, and the processor implements the following steps when the program is executed:

[0211] A method for processing high concurrent query requests, the method comprising: receiving multiple concurrent query requests, the concurrent query requests including an initial query request and the same query request corresponding to the initial query request; extracting the initial query Request; block the same query request; determine the query result corresponding to the initial query request; determine the query result as the query result corresponding to the same query request and return it to the corresponding query user.

[0212] Further, the determining the query result corresponding to the initial query request includes: sending the initial query request to a remote node in a distributed cluster, and the remote node performs a query operation to obtain the query result corresponding to the initial query request. Query result, write the query result into the distributed cache of the remote node; receive the query completion message returned by the remote node; obtain the query result from the distributed cache of the local node according to the query completion message , Wherein each node included in the distributed cluster has a distributed cache, and when data is written in the distributed cache of one node, the data can be read by the distributed cache of other nodes.

[0213] Further, after the receiving multiple concurrent query requests, the method further includes: judging whether the concurrent query request is an initial query request; if so, extracting the initial query request and creating it according to the initial query request The synchronization waiting structure is used to block the same query request; if not, the concurrent query request is determined to be the same query request, and the same query request is added to the corresponding synchronization waiting structure to block The same query request.

[0214] Further, determining that the query result is the query result corresponding to the same query request to the corresponding query user includes: waking up the same query request blocked by the synchronization waiting structure when the query completion message is received; The query result obtained in the distributed cache of the local node is determined to be the query result corresponding to the same query request; the synchronization waiting structure corresponding to the initial query request is cancelled.

[0215] Further, before judging whether the concurrent query request is an initial query request, the method further includes: querying whether the query result corresponding to the concurrent query request exists in the distributed cache of the local node; The query result is sent to the query user.

[0216] Further, the method further includes: according to the received concurrent query request, creating a processing thread corresponding to the concurrent query request in a first thread pool; when determining the query result, creating a thread for processing in the second thread pool The processing thread for obtaining the query result, wherein the first thread pool and the second thread pool are thread pools obtained by dividing corresponding processing resources according to preset rules.

[0217] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0218] This application also provides a computer program product, which when executed on a data processing device, is suitable for executing program code that initializes the following method steps: receiving multiple concurrent query requests, the concurrent query requests including an initial query request and The same query request corresponding to the initial query request; extract the initial query request; block the same query request; determine the query result corresponding to the initial query request; determine the query result to correspond to the same query request The result of the query is sent to the corresponding query user.

[0219] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0220] This application is described with reference to flowcharts and/or block diagrams of methods, equipment (systems), and computer program products according to the embodiments of this application. It should be understood that each process and/or block in the flowchart and/or block diagram, and the combination of processes and/or blocks in the flowchart and/or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing equipment to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing equipment are generated In the process Figure one Process or multiple processes and/or boxes Figure one A device with functions specified in a block or multiple blocks.

[0221] These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing equipment to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction device. The device is implemented in the process Figure one Process or multiple processes and/or boxes Figure one Function specified in a box or multiple boxes.

[0222] These computer program instructions can also be loaded on a computer or other programmable data processing equipment, so that a series of operation steps are executed on the computer or other programmable equipment to produce computer-implemented processing, so as to execute on the computer or other programmable equipment. Instructions are provided to implement the process Figure one Process or multiple processes and/or boxes Figure one Steps of functions specified in a box or multiple boxes.

[0223] In a typical configuration, the computing device includes one or more processors (CPU), input/output interfaces, network interfaces, and memory.

[0224] The memory may include non-permanent memory in a computer readable medium, random access memory (RAM) and/or non-volatile memory, such as read only memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.

[0225] Computer-readable media includes permanent and non-permanent, removable and non-removable media, and information storage can be realized by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disc (DVD) or other optical storage, Magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media can be used to store information that can be accessed by computing devices. According to the definition in this article, computer-readable media does not include transitory media, such as modulated data signals and carrier waves.

[0226] It should also be noted that the terms "include", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, product or equipment including a series of elements not only includes those elements, but also includes Other elements that are not explicitly listed, or include elements inherent to the process, method, commodity, or equipment. If there are no more restrictions, the element defined by the sentence "including a..." does not exclude the existence of other identical elements in the process, method, commodity or equipment that includes the element.

[0227] Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, this application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0228] The above are only examples of the application, and are not used to limit the application. For those skilled in the art, this application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included in the scope of the claims of this application.