Request processing method, device, electronic device and computer readable storage medium

Abstract

The present application provides a request processing method, device, electronic device and computer-readable storage medium, which relate to the field of data processing technology. The method includes: obtaining n requests; wherein each request corresponds to a coroutine on a thread, n is an integer greater than 0 and less than or equal to a preset threshold; according to a preset competition rule, a dominant coroutine is determined among the n coroutines, and the remaining n-1 coroutines are all follower coroutines; the n requests are merged by the dominant coroutine to obtain the merged request data; and the merged request data is submitted to the computing device. In an embodiment of the present application, when processing multiple requests input by a user using the request processing method, by introducing a coroutine as an execution unit, the n requests obtained are respectively corresponded to each coroutine, and the dominant coroutine is determined according to the preset competition rule to merge the n requests, thereby reducing the server resource consumption caused by cross-thread competition, so that the computing performance of the server can be fully utilized.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a request processing method, device, electronic device and computer-readable storage medium. Background Art

[0002] With the emergence of massive data, people tend to use machine learning technology to mine value from data. Machine learning can continuously improve the performance of the system itself by simulating or realizing human learning behavior, acquiring new knowledge or skills, and reorganizing existing knowledge structures. It is an inevitable product of the development of artificial intelligence research to a certain stage, and its applications are spread across all fields of artificial intelligence.

[0003] Since the use of machine learning is inseparable from the necessary machine learning framework, the explosive growth of machine learning has also promoted the development of many open source tools, making it easier for developers to learn its technology. For example, in the field of deep learning in machine learning research, TensorFlow Serving launched by Google is one of the typical machine learning frameworks with relatively excellent performance. In order to improve data processing performance, multi-threading technology is used in existing machine learning frameworks to process multiple concurrent requests, that is, when the batch processing strategy is enabled in the machine learning framework and a batch of concurrent requests are processed, a batch of requests are processed simultaneously based on multiple threads.

[0004] However, when the above-mentioned existing machine learning framework processes multiple concurrent requests through multi-threading technology, a batch of requests will occupy multiple threads at the same time, and there is cross-core scheduling and cross-thread competition between different threads, which consumes a large amount of server resources, resulting in the server's computing performance cannot be fully utilized. Summary of the invention

[0005] In view of this, the purpose of the embodiments of the present application is to provide a request processing method, device, electronic device and computer-readable storage medium, which can reduce the server resource consumption caused by cross-thread contention when processing requests, thereby better exerting the computing performance of the server.

[0006] In a first aspect, an embodiment of the present application provides a request processing method, including:

[0007] Obtain n requests, wherein each request corresponds to a coroutine on a thread, a thread contains one or more coroutines, and n is an integer greater than 0 and less than or equal to a preset threshold; according to a preset competition rule, determine a dominant coroutine among the n coroutines, and the remaining n-1 coroutines are all follower coroutines; merge the n requests through the dominant coroutine to obtain merged request data; and submit the merged request data to the computing device.

[0008] Optionally, according to a preset competition rule, a leading coroutine is determined among the n coroutines, and the remaining n-1 coroutines are all follower coroutines, including:

[0009] According to the order in which n requests are written into the queue, the coroutine corresponding to the first request in the queue is determined as the leading coroutine, and the coroutines corresponding to other requests are all follower coroutines.

[0010] Optionally, after submitting the merge request data to the computing device, the method further includes:

[0011] Receive the calculation results returned by the computing device; split the calculation results and obtain the response results corresponding to each request.

[0012] Optionally, the leading coroutine is in a running state and the following coroutine is in a waiting state; after splitting the calculation results and obtaining the response result corresponding to each request, it also includes:

[0013] The leading coroutine wakes up the following coroutine corresponding to each request and feeds back the corresponding response result to the following coroutine.

[0014] Optionally, the leading coroutine wakes up the following coroutine corresponding to each request, and feeds back the corresponding response result to the following coroutine, including:

[0015] According to the order in which the n requests are written into the queue, the leading coroutine wakes up the follower coroutine corresponding to each request in turn, and feeds back the corresponding response result to the follower coroutine.

[0016] Optionally, the leading coroutine wakes up the following coroutine corresponding to each request, and feeds back the corresponding response result to the following coroutine, including:

[0017] Determine the wake-up order of the follower coroutines in each thread, wake up other follower coroutines through the leading coroutine according to the wake-up order, and feed back corresponding response results to other follower coroutines.

[0018] Optionally, get n requests, including:

[0019] If the number of currently unprocessed requests is greater than or equal to a preset threshold, the preset threshold number of requests are obtained; or, if the number of currently unprocessed requests is less than the preset threshold, all currently unprocessed requests are obtained.

[0020] Optionally, the leading coroutine is a coroutine that acquires the lock; before submitting the merge request data to the computing device, the method further includes:

[0021] Release the lock by the leading coroutine.

[0022] Optionally, after splitting the calculation results and obtaining the response results corresponding to each request, the following is also included:

[0023] Acquire the lock through the leading coroutine.

[0024] Optionally, the computing device is a machine learning processor MLU.

[0025] In a second aspect, an embodiment of the present application further provides a request processing device, comprising: an acquisition module, a first processing module, a merging module and a submission module.

[0026] An acquisition module is used to acquire n requests, where each request corresponds to a coroutine on a thread, a thread contains one or more coroutines, and n is an integer greater than 0 and less than or equal to a preset threshold; a first processing module is used to determine a dominant coroutine among n coroutines according to preset competition rules, and the remaining n-1 coroutines are all follower coroutines; a merging module is used to merge n requests through the dominant coroutine to obtain merged request data; a submission module is used to submit the merged request data to a computing device.

[0027] Optionally, the first processing module is specifically used to determine, according to the order in which the n requests are written into the queue, that the coroutine corresponding to the first request in the queue is the leading coroutine, and the coroutines corresponding to other requests are all follower coroutines.

[0028] Optionally, the apparatus further includes: a receiving module and a second processing module; the receiving module is used to receive the calculation results returned by the computing device; the second processing module is used to split the calculation results and obtain the response results corresponding to each request.

[0029] Optionally, the leading coroutine is in a running state and the following coroutine is in a waiting state; the second processing module is also used to wake up the following coroutine corresponding to each request through the leading coroutine after splitting the calculation results and obtaining the response results corresponding to each request, and feed back the corresponding response results to the following coroutine.

[0030] Optionally, the second processing module is specifically used to wake up the follower coroutine corresponding to each request in turn through the leading coroutine according to the order in which the n requests are written into the queue, and feed back the corresponding response result to the follower coroutine.

[0031] Optionally, the second processing module is specifically used to determine the wake-up order of the following coroutines in each thread, wake up other following coroutines through the leading coroutine according to the wake-up order, and feed back corresponding response results to other following coroutines.

[0032] Optionally, the acquisition module is specifically used to acquire a preset threshold number of requests if the number of currently unprocessed requests is greater than or equal to a preset threshold; or, if the number of currently unprocessed requests is less than the preset threshold, acquire all currently unprocessed requests.

[0033] Optionally, the dominant coroutine is the coroutine that acquires the lock; the first processing module is further used to release the lock through the dominant coroutine before the submission module submits the merge request data to the computing device.

[0034] Optionally, the second processing module is further used to acquire the lock through the leading coroutine after splitting the calculation results and obtaining the response results corresponding to each request.

[0035] Optionally, the computing device is a machine learning processor MLU.

[0036] In a third aspect, an embodiment of the present application further provides an electronic device, including: a processor, a storage medium and a bus.

[0037] The storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the storage medium through a bus, and the processor executes the machine-readable instructions to perform the steps of the request processing method provided in the first aspect above.

[0038] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the request processing method provided in the first aspect are executed.

[0039] Based on any of the above aspects, the beneficial effects of this application are:

[0040] In an embodiment of the present application, when processing multiple requests, n requests are obtained from the unprocessed requests each time, and by introducing a coroutine as an execution unit, the n requests obtained are respectively corresponded to each coroutine, and according to a preset competition rule, a dominant coroutine is determined among the n coroutines corresponding to the n requests. The n requests are merged through the dominant coroutine, and the obtained merged request data is submitted to the computing device for calculation, thereby avoiding competition between multiple threads and reducing the server resource consumption caused by cross-thread competition, so as to better exert the computing performance of the server. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 A schematic diagram of a process flow of a request processing method provided in an embodiment of the present application;

[0043] Figure 2Another flowchart of the request processing method provided in the embodiment of the present application;

[0044] Figure 3 Another flowchart of the request processing method provided in the embodiment of the present application;

[0045] Figure 4 Another flowchart of the request processing method provided in the embodiment of the present application;

[0046] Figure 5 A schematic diagram of a structure of a request processing device provided in an embodiment of the present application;

[0047] Figure 6 Another structural diagram of a request processing device provided in an embodiment of the present application;

[0048] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0050] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0051] In order to make the contents of the following embodiments of the present application easier to understand, some necessary terms are explained as follows:

[0052] Thread: Sometimes called a lightweight process, it is the smallest unit of program execution flow. A standard thread consists of a thread identification (ID), a current instruction pointer, a register set, and a stack. In addition, a thread is an entity in a process and is a basic unit that is independently scheduled and dispatched by the system. A thread does not own system resources, but only has a few resources that are essential for operation. However, it can share all the resources owned by the process with other threads belonging to the same process. A thread can create and cancel another thread, and multiple threads in the same process can be executed concurrently. Due to the mutual constraints between threads, threads appear intermittent during operation. Threads also have three basic states: ready, blocked, and running. The ready state means that the thread has all the conditions for operation, can logically run, and is waiting for the processor; the running state means that the thread occupies the processor and is running; the blocked state means that the thread is waiting for an event (such as a semaphore) and cannot be executed logically. Every program has at least one thread. If a program has only one thread, it is the program itself. A thread is a single sequential control flow in a program. There is a relatively independent and schedulable execution unit in the process, which is the basic unit of the system to independently schedule and dispatch the instructions of the central processing unit (CPU) when the program is running. Running multiple threads simultaneously in a single program to complete different tasks is called multithreading.

[0053] Concept of coroutine: Coroutine refers to a user-mode execution unit implemented on the basis of operating system threads, which can include symmetric coroutines (stackful coroutines) and asymmetric coroutines (stackless coroutines).

[0054] Coroutine switching principle: Taking stackful coroutine as an example, each coroutine has its own stack in the stackful coroutine design, and the running stack space and code segment switching of different coroutines is achieved by switching the instruction pointer (IP) register and the stack pointer (SP) register. When switching, all registers of the original coroutine are first saved to the specified location in the stack, and then the registers of the target coroutine are restored. Each physical thread can only execute one coroutine at the same time. When the coroutine exits or encounters a blocking operation and hangs, the scheduling engine will switch the current coroutine to other coroutines in an executable state, otherwise it will be in a waiting wake-up state.

[0055] The embodiment of the present application takes a server that provides data processing to the outside as an example to illustrate the specific application and beneficial effects of the request processing method provided by the present application. When the server receives multiple requests input by the user, the server processes the multiple requests according to the request processing method, and then returns the processing results to the user. Of course, the execution subject of the embodiment of the present application can also be other devices with request processing functions, which are not limited here.

[0056] Optionally, in one embodiment of the present application, the server and the user end are implemented based on an I / O (Input / Output) framework, and the user can input request data to the server based on the I / O framework, such as text, audio, video, images, etc. The server can batch process the request data input by the user according to the steps of the following request processing method, and return the processing results to the user.

[0057] like Figure 1 As shown, the request processing method provided in the embodiment of the present application includes the following steps:

[0058] S101. Obtain n requests.

[0059] Each request corresponds to a coroutine on a thread, a thread includes one or more coroutines, and n is an integer greater than 0 and less than or equal to a preset threshold.

[0060] When a user inputs multiple requests to the server, the server can batch process the multiple requests according to the request processing method, using the coroutine as the execution unit.

[0061] First, the server obtains n requests from the received multiple requests, and the obtained n requests correspond to a coroutine on the thread respectively. Optionally, the server has multiple threads, each thread may include one or more coroutines, and the obtained n requests may correspond to coroutines in different threads respectively.

[0062] In addition, the preset threshold may refer to the total number of coroutines that can be processed at one time in all threads in the server, and its specific value may be configured according to different types of servers. For example, when the server uses a quad-core, quad-thread processor, the preset threshold may be set to any value among 2, 4, 6, or 8; or, when the server uses a quad-core, eight-thread processor, n may be set to a larger value such as 2, 4, 6, 8, or 16. There may be multiple ways to set the preset threshold, and this application does not limit this.

[0063] As described above, in the request processing method provided in the embodiment of the present application, when the server batch processes the requests input by the user, it can determine the number of requests obtained based on a preset threshold and the number of unprocessed requests, so as to achieve self-adaptation of the number of requests processed each time, without the server having to do other work or adjustments.

[0064] In one embodiment, the above-mentioned coroutines can be symmetric coroutines, that is, each coroutine performs the same function. Compared with other coroutines, symmetric coroutines have their own independent stacks. When switching, there is no need to serialize objects on the stack to the heap or other locations. The switching operation can be completed directly by switching registers, so it has better switching performance and is convenient for diagnostic tools to climb the stack. For example, a stackful symmetric coroutine can be used in this embodiment.

[0065] In another embodiment, the above-mentioned coroutine may also be an asymmetric coroutine, that is, the functions performed by each coroutine may be different, and the present application does not make any special limitation here.

[0066] Optionally, in some implementations, the server may implement the acquisition request through an input / output (I / O) framework. The I / O framework is capable of receiving external input request data, for example, the request data may include different types of data such as graphic data, text data, audio data, etc. After receiving the external input request data, the I / O framework uses the request data as a request and waits for acquisition.

[0067] S102. According to a preset competition rule, a leading coroutine is determined among the n coroutines, and the remaining n-1 coroutines are all follower coroutines.

[0068] Optionally, the preset competition rules can be pre-configured in the server as a basis for the server to determine the leader and follower. When the server obtains n requests, it can determine a leader from the n coroutines corresponding to the n requests according to the preset competition rules, and the remaining n-1 coroutines will be followers.

[0069] S103. Merge n requests through the leading coroutine to obtain merged request data.

[0070] Specifically, when the server determines to obtain a leading coroutine, the server merges the obtained n requests through the leading coroutine to obtain merged request data, wherein the merged request data includes all data information of the n requests.

[0071] Optionally, the leading coroutine is in a running state, while other following coroutines are in a waiting state.

[0072] S104. Submit the merge request data to the computing device.

[0073] After the server merges n requests to obtain merged request data, it can submit the obtained merged request data to the computing device. The computing device can calculate the received merged request data, obtain the calculation result, and return the calculation result.

[0074] Optionally, the computing device may be a central processing unit (CPU), a graphics processing unit (GPU), or a machine learning unit (MLU), etc.

[0075] As described above, in the request processing method provided in the embodiment of the present application, when processing multiple requests, n requests are obtained from the unprocessed requests each time, and by introducing a coroutine as an execution unit, the obtained n requests are respectively corresponded to each coroutine, and according to the preset competition rules, a dominant coroutine is determined among the n coroutines corresponding to the n requests, and the n requests are merged through the dominant coroutine, and the merged request data is submitted to the computing device for calculation, thereby avoiding competition between multiple threads and reducing the server resource consumption caused by cross-thread competition, so as to better exert the computing performance of the server.

[0076] Optionally, the above-mentioned obtaining of n requests may specifically be: if the number of currently unprocessed requests is greater than or equal to a preset threshold, obtaining the preset threshold number of requests; or, if the number of currently unprocessed requests is less than the preset threshold, obtaining all currently unprocessed requests.

[0077] For example, when the number of unprocessed requests is greater than or equal to a preset threshold, the server can obtain the preset threshold number of requests at a time for merging, that is, n is equal to the preset threshold, and the remaining unprocessed requests can be processed in the subsequent batch processing process. When the number of unprocessed requests is less than the preset threshold, the server can obtain all the unprocessed requests at a time for merging, that is, n is equal to the total number of unprocessed requests.

[0078] Optionally, in one implementation, the above-mentioned method of determining a leading coroutine among n coroutines and the remaining n-1 coroutines as follower coroutines according to a preset competition rule includes:

[0079] According to the order in which n requests are written into the queue, the coroutine corresponding to the first request in the queue is determined as the leading coroutine, and the coroutines corresponding to other requests are all follower coroutines.

[0080] It should be noted that in this embodiment, after the server obtains n requests, the n requests will be written into the queue in order. The preset competition rule can be: the server determines the coroutine of a certain sequence as the leading coroutine according to the order in which the n requests are written into the queue. For example, the coroutine corresponding to the first request in the queue can be determined as the leading coroutine, and the coroutines corresponding to other requests are follower coroutines.

[0081] In the above implementation, since the server can determine the leading coroutine and the following coroutine according to the order in which n requests are written into the queue, there is less competition among all coroutines when determining the leading coroutine, thereby reducing the server resources occupied by the coroutine competition, further enabling the computing performance of the server to be better utilized and reducing the response time of the server when receiving a processor request.

[0082] like Figure 2 As shown, the request processing method provided in the embodiment of the present application may further include, after submitting the merge request data to the computing device:

[0083] S201. Receive calculation results returned by a computing device.

[0084] Optionally, the computing device may calculate the merge request data submitted by the server, obtain a calculation result, and return the obtained calculation result. The server can receive the calculation result returned by the computing device.

[0085] S202: Split the calculation results and obtain the response results corresponding to each request.

[0086] Optionally, the above calculation result may be a set of response results corresponding to n requests. After receiving the calculation result returned by the computing device, the server may split the calculation result to obtain the response result corresponding to each request.

[0087] For example, the server may refer to the combined request data of n requests, split the calculation results, and obtain a response result corresponding to each request.

[0088] As described above, in the request processing method provided by the embodiment of the present application, the leading coroutine merges n requests to obtain the merged request data for submission, wherein the leading coroutine is in a running state and the other following coroutines are in a waiting state. Then, in the process of feeding back the response result, it is necessary to wake up the corresponding coroutine to feed back the response result.

[0089] Optionally, the request processing method provided in the embodiment of the present application, after splitting the calculation results and obtaining the response results corresponding to each request, may further include:

[0090] The leading coroutine wakes up the follower coroutine corresponding to each request, and feeds back the corresponding response result to the follower coroutine. After the server splits the calculation result to obtain the response result corresponding to each request, the leading coroutine wakes up the follower coroutine corresponding to each request, so that the follower coroutine corresponding to each request changes from the waiting state to the running state, and feeds back the response result corresponding to each request obtained by splitting to the awakened follower coroutine.

[0091] Optionally, in one implementation, waking up the follower coroutine corresponding to each request by the leading coroutine and feeding back the corresponding response result to the follower coroutine may include:

[0092] According to the order in which the n requests are written into the queue, the leading coroutine wakes up the follower coroutine corresponding to each request in turn, and feeds back the corresponding response result to the follower coroutine.

[0093] In this embodiment, since the leading coroutine wakes up the follower coroutine corresponding to each request in sequence according to the order in which n requests are written to the queue, the object to be awakened is the coroutine rather than the thread, and there is no competition between the coroutines in the thread where the leading coroutine is located and the coroutines in other threads, thereby reducing the server resource consumption problem caused by competition during the awakening process, and can also further enhance the computing performance of the server.

[0094] For example, suppose the server receives 8 requests from the user, and the preset threshold is 6, that is, at most 6 requests are processed as a batch each time, then 6 requests can be written into the queue for processing each time. The 6 requests correspond to 6 coroutines (w1, w2, w3, w4, w5 and w6), and suppose the order in which the 6 requests are written into the queue is w1, w2, w3, w4, w5 and w6, then in this implementation, the corresponding coroutines are woken up in the order of w1, w2, w3, w4, w5 and w6 and the response results are fed back.

[0095] Optionally, in another implementation manner, waking up the follower coroutine corresponding to each request by the leading coroutine and feeding back the corresponding response result to the follower coroutine may also include:

[0096] Determine the wake-up order of the follower coroutines in each thread, wake up other follower coroutines through the leading coroutine according to the wake-up order, and feed back corresponding response results to other follower coroutines.

[0097] In this embodiment, the server first determines the wake-up order of the follower coroutines in each thread, and then wakes up other follower coroutines through the leading coroutine according to the determined wake-up order. For example, the priorities of other follower coroutines can be pre-set in the server. When the server wakes up other follower coroutines through the leading coroutine, the wake-up order can be determined by referring to the pre-set priorities of other follower coroutines, so as to wake up in sequence according to the wake-up order. Alternatively, the wake-up order can also be determined by letting other follower coroutines compete, etc., which is not limited by this application.

[0098] For example, taking the above-mentioned w1, w2, w3, w4, w5 and w6 coroutines as an example, assuming that w1, w2, w3 belong to the first thread, w4, w5, w6 belong to the second thread, and w1 is the leading coroutine, starting from the first thread, first determine the wake-up order of w2 and w3. For example, the wake-up order of the following coroutines in each thread can be determined according to the order of writing to the queue, that is, first wake up w2 and then wake up w3. Similarly, determine the wake-up order of w4, w5, w6 in the second thread. Assuming that the wake-up order is also determined as w4, w5, w6 according to the order of writing to the queue, then wake up w3 and then wake up w4, w5, w6 in turn. Of course, when determining the wake-up order of the following coroutines in each thread, it can also be determined according to some preset competition rules, which is not limited in the embodiments of the present application.

[0099] Optionally, in the request processing method provided by the present application, the leading coroutine may be a coroutine that acquires the lock; before submitting the merge request data to the computing device, the method may further include:

[0100] Release the lock by the leading coroutine.

[0101] It should be noted that a lock is a mechanism used in multi-threaded programming to prevent two threads from reading and writing the same common resource (such as a global variable) at the same time.

[0102] When the leading coroutine obtains the lock, it merges the n requests obtained by the leading coroutine to obtain the merged request data. In the process, the lock obtained by the leading coroutine can protect the data of the above n requests, prevent other request data from affecting the merging process, and avoid request data conflicts.

[0103] Accordingly, before submitting the merge request data to the computing device, the server can release the acquired lock through the leading coroutine and submit the merge request data to the computing device. After the lock is released through the leading coroutine, the coroutines corresponding to other unprocessed requests can compete to acquire the above lock.

[0104] Optionally, the request processing method provided by the present application, after splitting the calculation results and obtaining the response results corresponding to each request, may further include:

[0105] Acquire the lock through the leading coroutine.

[0106] As described above, after obtaining the response result corresponding to each request, the lock can be obtained again by the leading coroutine, and then the following coroutine corresponding to each request is awakened by the leading coroutine, and the corresponding response result is fed back to the following coroutine.

[0107] Among them, after the leading coroutine acquires the above lock again, it can wake up the follower coroutine corresponding to each request through the leading coroutine, and in the process of feeding back the corresponding response result to the follower coroutine, the lock can prevent the required resources from being occupied by other threads, which will not be repeated here.

[0108] Please refer to Figure 3 and Figure 4 , this application hereby takes the machine learning framework as an example to explain in detail the request processing method described in the aforementioned method embodiment. Assume that the server includes a first thread and a second thread. Among them, the first thread includes: a first coroutine w1, a second coroutine w2 and a third coroutine w3; the second thread includes: a fourth coroutine w4, a fifth coroutine w5 and a sixth coroutine w6. In this request processing method, the server uses the aforementioned 6 coroutines as execution units to process multiple requests input by the user, that is, it is assumed that the preset threshold described in the aforementioned method embodiment is equal to 6.

[0109] like Figure 3 As shown, the specific steps of using this request processing method to process the request are as follows:

[0110] S301. Obtain n requests.

[0111] S302. According to the order in which the n requests are written into the queue, determine that the coroutine corresponding to the first request in the queue is the leading coroutine, and the coroutines corresponding to the other requests are all follower coroutines.

[0112] S303. Acquire the lock through the leading coroutine.

[0113] S304. Merge n requests through the leading coroutine to obtain merged request data.

[0114] S305. Release the lock through the leading coroutine.

[0115] S306. Submit the merge request data to the computing device.

[0116] S307: Receive the calculation result returned by the calculation device.

[0117] S308. Split the calculation results and obtain the response results corresponding to each request.

[0118] S309. Acquire the lock through the leading coroutine.

[0119] S310. The leading coroutine wakes up the following coroutine corresponding to each request, and feeds back the corresponding response result to the following coroutine.

[0120] As described above, in step S301, if the number of currently unprocessed requests is greater than or equal to 6, then n is equal to 6, that is, 6 requests are obtained; if the number of currently unprocessed requests is less than 6, then n is equal to the total number of currently unprocessed requests, that is, all unprocessed requests are obtained.

[0121] Furthermore, the server will use the coroutine corresponding to the first request written into the queue as the leading coroutine, and the coroutines corresponding to the remaining requests in the queue as the following coroutines according to the order in which the acquired requests are written into the queue.

[0122] For example, if the number of requests n obtained is 6, the first coroutine at the top of the queue is used as the leading coroutine, and the other second coroutines, third coroutines, fourth coroutines, fifth coroutines and sixth coroutines are follower coroutines. Then, the lock is acquired by the first coroutine as the leading coroutine. Then, the requests corresponding to the above 6 coroutines are merged through the first coroutine to obtain the merged request data, and submit it to the computing device for processing. After processing the merged request data, the computing device returns the calculation result. The server splits the returned calculation result to obtain the response result corresponding to each request, and after acquiring the lock again through the first coroutine, the other 5 follower coroutines are awakened by the first coroutine in the order in which the above 6 requests are written to the queue, and the corresponding response result is sent to each coroutine, and finally the request processing result is fed back to the user.

[0123] It should be noted that in the above request processing method, if the number of currently unprocessed requests is greater than 6, the server obtains the first batch of 6 requests and merges them, obtains the merged request processing data, submits it to the computing device, and continues to obtain the next batch of requests from the remaining unprocessed requests for processing. The request processing process is looped until all requests are processed, and then the processing process is terminated.

[0124] The beneficial effects of the request processing method described in this application example that are the same as those described in the aforementioned method embodiment of this application will not be repeated here.

[0125] In addition, in the current machine learning field (such as deep learning), most machine learning development frameworks focus on cloud computing and online prediction, while there is still a lot of room for improvement in scenarios where local predictions are made on the end-side device. Since the request processing method provided by this application can improve the computing performance of the server, reduce the response time of the server when processing requests, and improve efficiency, from the perspective of user-side speed experience, offline prediction solution requirements, and full utilization of end-side device computing resources, the request processing method can also help the prediction part of machine learning (or deep learning) to be implemented cross-platform and lightweight, and will greatly improve the productization efficiency of the prediction part.

[0126] In order to make the technical solution of the request processing method described in the above embodiments of the present application clearer, the present application also uses the following application examples to illustrate:

[0127] Assume that the server receives 8 requests from the user, and the preset threshold is 6, that is, a maximum of 6 requests are obtained each time as a batch for processing, then 6 requests can be written into the queue for processing each time. The first batch of 6 requests obtained by the server correspond to 6 coroutines (w1, w2, w3, w4, w5 and w6), respectively, and the order in which the 6 requests are written into the queue corresponds to the above 6 coroutines. According to the order in which the 6 requests are written into the queue, the server can determine w1 as the leading coroutine, and w2 to w5 are all follower coroutines. After determining that w1 is the leading coroutine, the lock is obtained through w1, and then w1 merges the above 6 requests to obtain the merged request data, releases the acquired lock, and submits it to the computing device for calculation.

[0128] Furthermore, after the computing device returns the calculation result, w1 will split the calculation result, obtain the response results corresponding to the above 6 requests respectively, and wake up the other w2 to w5 in the waiting state to send the corresponding response results.

[0129] In the process of processing the first batch of 6 requests, after w1 releases the lock, the coroutines corresponding to the remaining 2 unprocessed requests (w7 and w8) can continue to repeat the above w1-w6 process, and process the remaining 2 requests as the second batch. For example, in the process of processing the second batch of requests, w7 and w8 are written into the queue in sequence, and the server can determine w7 as the leading coroutine.

[0130] like Figure 5 As shown, an embodiment of the present application further provides a request processing device, including: an acquisition module 501, a first processing module 502, a merging module 503 and a submitting module 504.

[0131] The acquisition module 501 is used to acquire n requests, wherein each request corresponds to a coroutine on a thread, a thread includes one or more coroutines, and n is an integer greater than 0 and less than or equal to a preset threshold.

[0132] The first processing module 502 is used to determine a leading coroutine among n coroutines according to a preset competition rule, and the remaining n-1 coroutines are all follower coroutines.

[0133] The merging module 503 is used to merge n requests through the leading coroutine to obtain the merged request data.

[0134] The submission module 504 is used to submit the merge request data to the computing device.

[0135] Optionally, the first processing module 502 is specifically configured to determine, according to the order in which the n requests are written into the queue, that the coroutine corresponding to the first request in the queue is the leading coroutine, and the coroutines corresponding to the other requests are all follower coroutines.

[0136] Alternatively, if Figure 6 As shown, the request processing device further includes: a receiving module 505 and a second processing module 506 .

[0137] The receiving module 505 is used to receive the calculation results returned by the calculation device.

[0138] The second processing module 506 is used to split the calculation results and obtain the response results corresponding to each request.

[0139] Optionally, the leading coroutine is in a running state and the following coroutine is in a waiting state. The second processing module 506 is also used to wake up the following coroutine corresponding to each request through the leading coroutine after splitting the calculation results and obtaining the response result corresponding to each request, and feed back the corresponding response result to the following coroutine.

[0140] Optionally, the second processing module 506 is specifically configured to wake up the follower coroutine corresponding to each request in turn through the leading coroutine according to the order in which the n requests are written into the queue, and feed back the corresponding response result to the follower coroutine.

[0141] Optionally, the second processing module 506 is specifically used to determine the wake-up order of the following coroutines in each thread, wake up other following coroutines through the leading coroutine according to the wake-up order, and feed back corresponding response results to other following coroutines.

[0142] Optionally, the acquisition module 501 is specifically configured to acquire a preset threshold number of requests if the number of currently unprocessed requests is greater than or equal to a preset threshold; or to acquire all currently unprocessed requests if the number of currently unprocessed requests is less than the preset threshold.

[0143] Optionally, the dominant coroutine is the coroutine that acquires the lock; the first processing module 502 is further configured to release the lock through the dominant coroutine before the submission module 504 submits the merge request data to the computing device.

[0144] Optionally, the second processing module 506 is further configured to acquire the lock through the leading coroutine after splitting the calculation result and acquiring the response result corresponding to each request.

[0145] Optionally, the computing device is an MLU, but is not limited thereto and may also be a CPU, a GPU, etc.

[0146] As described above, the request processing device provided in the embodiment of the present application executes the request processing method described in the aforementioned method embodiment, and therefore has all the beneficial effects described in the aforementioned method embodiment, which will not be described one by one in this application.

[0147] In addition, the above modules can be connected or communicated with each other via wired connection or wireless connection. The wired connection can include metal cables, optical cables, hybrid cables, etc., or any combination thereof. The wireless connection can include connections in the form of LAN, WAN, Bluetooth, ZigBee, or NFC, or any combination thereof. Two or more modules can be combined into a single module, and any one module can be divided into two or more units.

[0148] It should be noted that the device embodiments described above are only illustrative. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be through some communication interface, and the indirect coupling or communication connection of the device or module may be electrical, mechanical or other forms.

[0149] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0150] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0151] like Figure 7As shown, an embodiment of the present application also provides an electronic device, including: a processor 601, a storage medium 602 and a bus; the storage medium 602 stores machine-readable instructions executable by the processor 601. When the electronic device is running, the processor 601 communicates with the storage medium 602 through the bus, and the processor 601 executes the machine-readable instructions to perform the steps of the request processing method provided in the aforementioned method embodiment.

[0152] Specifically, the machine-readable instructions stored in the storage medium 602 are the execution steps of the request processing method described in the aforementioned embodiment of the present application. The processor 601 can execute the request processing method to process the request. Therefore, the electronic device also has all the beneficial effects described in the aforementioned method embodiment, and this application will not repeat the description.

[0153] It should be noted that the electronic device can be a general-purpose computer or a special-purpose computer, as well as other servers for processing data, etc., all of which can be used to implement the request processing method of this application. Although this application only describes the request processing method through computers and servers, for convenience, the functions described in this application can also be implemented in a distributed manner on multiple similar platforms to balance the processing load.

[0154] For example, the electronic device may include one or more processors, a communication bus, and different forms of storage media, such as a disk, ROM, or RAM, or any combination thereof, for executing program instructions. Exemplarily, the computer platform may also include program instructions stored in ROM, RAM, or other types of non-transitory storage media, or any combination thereof. The method of the present application may be implemented according to these program instructions.

[0155] For ease of description, only one processor is described in the electronic device. However, it should be noted that the electronic device in the present application may also include multiple processors, so the steps performed by one processor described in the present application may also be performed jointly or individually by multiple processors.

[0156] In addition, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the request processing method described in the aforementioned method embodiment are executed.

[0157] Specifically, if the functions described in the aforementioned embodiments of the present application are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0158] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A request processing method, characterized in that: The method comprises: Obtain n requests, where each request corresponds to a coroutine on a thread, a thread includes one or more coroutines, and n is an integer greater than 0 and less than or equal to a preset threshold; According to the preset competition rules, a leading coroutine is determined among the n coroutines, and the remaining n-1 coroutines are all follower coroutines; Merge n requests through the leading coroutine to obtain merged request data; submitting the merge request data to a computing device; According to the preset competition rules, a leading coroutine is determined among the n coroutines, and the remaining n-1 coroutines are all follower coroutines, including: According to the order in which the n requests are written into the queue, the coroutine corresponding to the first request in the queue is determined as the leading coroutine, and the coroutines corresponding to the other requests are all follower coroutines. After submitting the merge request data to the computing device, the method further includes: Receiving a calculation result returned by the computing device, wherein the calculation result includes a response result corresponding to each of the n requests; The leading coroutine wakes up the following coroutine corresponding to each request, and feeds back the corresponding response result to the following coroutine.

2. The method according to claim 1, characterized in that: After submitting the merge request data to the computing device, the method further includes: The calculation result is split, and a response result corresponding to each of the requests is obtained.

3. The method according to claim 2, characterized in that The leading coroutine is in a running state, and the following coroutine is in a waiting state.

4. The method according to claim 1, characterized in that The waking up the follower coroutine corresponding to each of the requests by the leading coroutine, and feeding back the corresponding response result to the follower coroutine, includes: According to the order in which the n requests are written into the queue, the leading coroutine wakes up the following coroutine corresponding to each request in turn, and feeds back the corresponding response result to the following coroutine.

5. The method according to claim 1, characterized in that The waking up the follower coroutine corresponding to each of the requests by the leading coroutine, and feeding back the corresponding response result to the follower coroutine, includes: Determine the wake-up order of the following coroutines in each of the threads, wake up other following coroutines through the leading coroutine according to the wake-up order, and feed back the corresponding response results to other following coroutines.

6. The method according to claim 1, characterized in that The obtaining of n requests includes: If the number of currently unprocessed requests is greater than or equal to the preset threshold, obtaining the preset threshold number of requests; or, If the number of currently unprocessed requests is less than the preset threshold, all currently unprocessed requests are obtained.

7. The method according to claim 1, characterized in that The dominant coroutine is the coroutine that acquires the lock; Before submitting the merge request data to the computing device, the method further includes: The lock is released by the leading coroutine.

8. The method according to claim 2, characterized in that: After splitting the calculation results and obtaining the response results corresponding to each of the requests, the method further includes: The lock is acquired by the leading coroutine.

9. The method according to any one of claims 1 to 8, characterized in that: The computing device is a machine learning processor MLU.

10. A request processing device, characterized in that: include: An acquisition module, a first processing module, a merging module and a submitting module; The acquisition module is used to acquire n requests, wherein each request corresponds to a coroutine on a thread, a thread includes one or more coroutines, and n is an integer greater than 0 and less than or equal to a preset threshold; The first processing module is used to determine a leading coroutine among the n coroutines according to a preset competition rule, and the remaining n-1 coroutines are all follower coroutines; The merging module is used to merge the n requests through the leading coroutine to obtain merged request data; The submitting module is used to submit the merge request data to the computing device, The first processing module is further used to determine, according to the order in which the n requests are written into the queue, that the coroutine corresponding to the first request in the queue is the leading coroutine, and the coroutines corresponding to the other requests are all follower coroutines; The device also includes a receiving module and a second processing module. The receiving module is used to receive the calculation result returned by the computing device, wherein the calculation result includes the response result corresponding to each request in the n requests, The second processing module is also used to wake up the following coroutine corresponding to each request through the leading coroutine after splitting the calculation result and obtaining the response result corresponding to each request, and feed back the corresponding response result to the following coroutine.

11. The device according to claim 10, characterized in that The second processing module is used to split the calculation result and obtain a response result corresponding to each of the n requests.

12. The device according to claim 11, characterized in that The leading coroutine is in a running state, and the following coroutine is in a waiting state.

13. The device according to claim 12, characterized in that The second processing module is specifically used to wake up the follower coroutine corresponding to each request in turn through the leading coroutine according to the order in which the n requests are written into the queue, and feed back the corresponding response result to the follower coroutine.

14. The device according to claim 12, characterized in that The second processing module is specifically used to determine the wake-up order of the following coroutines in each of the threads, wake up other following coroutines through the leading coroutine according to the wake-up order, and feed back the corresponding response results to other following coroutines.

15. The device according to claim 10, characterized in that The acquisition module is specifically configured to acquire a preset threshold number of the requests if the number of currently unprocessed requests is greater than or equal to the preset threshold; or If the number of currently unprocessed requests is less than the preset threshold, all currently unprocessed requests are obtained.

16. The device according to claim 10, characterized in that The dominant coroutine is the coroutine that acquires the lock; The first processing module is further configured to release the lock through the leading coroutine before the submitting module submits the merge request data to the computing device.

17. The device according to claim 12, characterized in that The second processing module is also used to acquire the lock through the leading coroutine after splitting the calculation result and obtaining the response result corresponding to each request.

18. The device according to any one of claims 10 to 17, characterized in that: The computing device is a machine learning processor MLU.

19. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the request processing method as described in any one of claims 1-9.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the request processing method according to any one of claims 1 to 9 is executed.

Images