Request processing method and apparatus, electronic device, and storage medium

Abstract

The present disclosure provides a request processing method and device, electronic equipment and storage medium, relates to the technical field of computers, in particular to the technical field of artificial intelligence such as deep learning and intelligent search, and the technical field of chips. The specific implementation scheme is: in the process of controlling the first component to process the first request for the i-th time, controlling the second component to process the second request for the k-th time; i and k are integers greater than 1; in the case where it is determined that the first component processes the first request for the i-th time to obtain the i-th intermediate data of the first request, controlling the second component to process the first request for the i+1-th time according to the i-th intermediate data, and obtaining the i+1-th intermediate data of the first request; obtaining the processing result of the first request according to the i+1-th intermediate data of the first request, and obtaining the processing result of the second request according to the k-th intermediate data of the second request obtained by the second component processing the second request for the k-th time. The embodiment of the present disclosure can improve the request processing efficiency.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, in particular to the technical field of artificial intelligence such as deep learning and intelligent search, and the technical field of chip. BACKGROUND

[0002] With the development of computer technology, more and more data need to be processed by computers, and the processing difficulty is getting greater and greater. In order to meet the growing use demand of users, the request processing technology is also facing new challenges. Therefore, how to more efficiently process the received request and obtain the corresponding processing result has become a problem to be solved. SUMMARY

[0003] The present disclosure provides a request processing method and device, electronic equipment and storage medium.

[0004] According to a first aspect of the present disclosure, a request processing method is provided, comprising:

[0005] In the process of controlling the first component to process the first request for the i-th time, the second component is controlled to process the second request for the k-th time; i and k are integers greater than 1;

[0006] In the case where it is determined that the first component processes the first request for the i-th time to obtain the i-th intermediate data of the first request, the second component is controlled to process the first request for the i+1-th time according to the i-th intermediate data, to obtain the i+1-th intermediate data of the first request;

[0007] According to the i+1-th intermediate data of the first request, the processing result of the first request is obtained, and according to the k-th intermediate data of the second request obtained by the second component processing the second request for the k-th time, the processing result of the second request is obtained.

[0008] According to a second aspect of the present disclosure, a request processing device is provided, comprising:

[0009] The first processing module is configured to control the second component to process the second request for the k-th time in the process of controlling the first component to process the first request for the i-th time; i and k are integers greater than 1;

[0010] The second processing module is configured to control the second component to process the first request for the i+1-th time according to the i-th intermediate data in the case where it is determined that the first component processes the first request for the i-th time to obtain the i-th intermediate data of the first request, to obtain the i+1-th intermediate data of the first request;

[0011] The first processing result module is configured to obtain the processing result of the first request according to the i+1-th intermediate data of the first request;

[0012] The second processing result module is configured to obtain a processing result of the second request according to the kth intermediate data of the second request obtained by the second component processing the second request for the kth time.

[0013] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0014] at least one processor; and

[0015] a memory in communication with the at least one processor; wherein

[0016] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the information display method of the first aspect.

[0017] According to a fourth aspect of the present disclosure, a non-transitory computer readable storage medium storing computer instructions is provided, the computer instructions being used to cause a computer to perform the foregoing method.

[0018] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the foregoing method.

[0019] It should be understood that the contents described in this part are not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.

[0020] The scheme provided by the embodiment can synchronize the kth processing of the second request by the second component in the process of controlling the first component to process the intermediate data of the first request, so that different requests can be input to the first component or the second component in turn, and the first component and the second component can be controlled to perform request processing operations in parallel, thereby improving the request processing efficiency and the utilization rate of processing resources. BRIEF DESCRIPTION OF DRAWINGS

[0021] The accompanying drawings serve to better understand the scheme and do not constitute limitations on the present disclosure. Among them:

[0022] Figure 1 is a flowchart of a request processing method according to an embodiment of the present disclosure;

[0023] Figure 2 is a flowchart of a request processing method according to another embodiment of the present disclosure;

[0024] Figure 3 is another flowchart of a request processing method according to another embodiment of the present disclosure;

[0025] Figure 4This is a schematic diagram of a target model based on an example of this disclosure;

[0026] Figure 5 This is a single-card execution diagram based on an example of this disclosure;

[0027] Figure 6 This is a schematic diagram of two pipeline stages according to an example of this disclosure;

[0028] Figure 7A , 7B This is a schematic diagram comparing the efficiency of sequential execution and execution using the method provided in the embodiments of this disclosure;

[0029] Figure 8 This is a schematic diagram of the composition structure of a request processing apparatus according to an embodiment of the present disclosure;

[0030] Figure 9 This is a schematic diagram of another component structure of a request processing apparatus according to an embodiment of the present disclosure;

[0031] Figure 10 This is a schematic diagram of the composition of a request processing apparatus according to another embodiment of the present disclosure;

[0032] Figure 11 This is a schematic diagram of another component structure of a request processing apparatus according to another embodiment of the present disclosure;

[0033] Figure 12 This is a schematic diagram of the composition structure of a request processing apparatus according to yet another embodiment of the present disclosure;

[0034] Figure 13 This is a schematic diagram of another component structure of a request processing apparatus according to yet another embodiment of the present disclosure;

[0035] Figure 14 This is a block diagram of an electronic device used to implement the request processing method of the embodiments of this disclosure. Detailed Implementation

[0036] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0037] The first aspect of this disclosure provides a request processing method, such as... Figure 1 As shown, it includes:

[0038] Step S101: During the process of controlling the first component to process the first request for the i-th time, control the second component to process the second request for the k-th time; i and k are integers greater than 1;

[0039] Step S102: If it is determined that the first component processes the first request for the i-th time and obtains the i-th intermediate data of the first request, the second component is controlled to process the first request for the (i+1)-th time based on the i-th intermediate data to obtain the (i+1)-th intermediate data of the first request.

[0040] Step S103: Based on the (i+1)th intermediate data of the first request, obtain the processing result of the first request, and based on the kth intermediate data of the second request obtained by the second component in the kth processing of the second request, obtain the processing result of the second request.

[0041] The solution provided in this embodiment can be applied to electronic devices, especially terminal devices, such as personal computers, tablets, mobile phones, etc.

[0042] The aforementioned first component can be a resource for processing data, which may include software resources, such as operators; it may also include multiple resources, such as operators and their corresponding runtime spaces. The aforementioned runtime space can be provided by a processor, such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a MCU (Microcontroller Unit).

[0043] The first component mentioned above can also be hardware, such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a MCU (Microcontroller Unit).

[0044] In all embodiments of this disclosure, the processing procedure for the first request can be the same as the processing procedure for the second request.

[0045] The second component can be any other processing component of the same type and structure as the first component; for example, both the first and second components could be CPUs. Each time the first request is processed by one component, an intermediate result for the first request is generated. Upon receiving an initially unprocessed first request, the first component can be controlled to perform the first processing of the first request, obtaining the first intermediate data of the first request.

[0046] In another possible implementation, controlling the first component to process the first request for the i-th time can be either controlling the first component to process the first request for the first time in the first instance. Controlling the first component to process the first request for the i-th time can also involve controlling the first component to process the first request based on intermediate data generated from the processing of the first request by other components.

[0047] In one implementation, where only the first component and the second component exist, the first request and the second request need to be processed sequentially by the first component and the second component, respectively. The first component can be controlled to process the first request according to the first stage steps of the first request processing flow, obtaining the first intermediate data of the first request. The second component can then be controlled to process the first request a second time based on the first intermediate data of the first request, according to the second stage steps of the first request processing flow, obtaining the second intermediate data of the first request.

[0048] In one implementation, the operation performed by the first component on the first request and the operation performed by the second component on the first request can be two time-adjacent operations in the process of processing the first request.

[0049] In other implementations, more components can exist. Similarly, other components can be made to process multiple requests synchronously with the first and second components. For example, within the same processing time period, the first component can be controlled to process the first request, the second component can be controlled to process the second request, and the third component can be controlled to process the third request.

[0050] In other implementations, if there are more components, the first component does not necessarily refer to the component corresponding to the operation in the first stage of timing, and the second component does not necessarily refer to the component corresponding to the operation in the second stage of timing.

[0051] For example, the first component and the second component can be controlled to execute operations of two sequentially adjacent processing stages during the processing of the first request. For instance, from the first request to its processing result, stages A, B, C, and D are required. The first component can be controlled to execute stage A, and the second component can be controlled to execute stage B. Alternatively, the first component can be controlled to execute stage B, and the second component can be controlled to execute stage C. Or, the first component can be controlled to execute stage D, and the second component can be controlled to execute stage C.

[0052] In the case of controlling multiple component processing requests in embodiments of this disclosure, any two components used to process adjacent timing stages can, according to... Figure 1 The first and second components shown in the diagram process the request and obtain the corresponding intermediate data.

[0053] In one possible implementation, the processing stages of two adjacent time sequences executed by the first component and the second component can be obtained by dividing the processing of the first request or the second request.

[0054] In one possible implementation, since the first component and the second component are respectively executed in two adjacent time-series processing stages, the processing of the first request by the first component may depend on the intermediate result of the previous processing of the first request by the second component, or the processing of the first request by the second component may depend on the intermediate result of the previous processing of the first request by the first component.

[0055] In another possible implementation, if the processing of the first request or the second request is divided into two sub-processes, each sub-process corresponds to an operation. The first component can be used to process the operation corresponding to one of the sub-processes, and the second component can be used to process the operation corresponding to the other sub-process.

[0056] For example, during the process of controlling the first component to process the first request for the i-th time, controlling the second component to process the second request for the k-th time can mean that the process of controlling the first component to process the first request for the i-th time and the process of controlling the second component to process the second request for the k-th time are at least partially parallel, and the time ratio of the parallel part exceeds a set threshold. For example, if the threshold is set to x% (x can be greater than 0), and the time for the first component to process the first request for the i-th time and obtain the i-th intermediate data of the first request is X1, and the time for the second component to process the second request for the k-th time and obtain the k-th intermediate data of the second request is X2, then the smaller value of X1 and X2 can be selected and multiplied by x% to obtain the specific duration for the process of the first component processing the first request and the process of the second component processing the second request to be synchronized.

[0057] For example, the first and second components can be controlled to execute the steps of a submodule of a request processing module. The aforementioned request processing module may be, for example, a model or a function. The aforementioned request processing module may have a fixed input submodule for receiving input data. The first request and the second request may be data input to the input submodule sequentially. The request processing module can execute the steps of the corresponding submodule according to the first request and the second request respectively to obtain the corresponding request processing result.

[0058] In this embodiment, the second component can be controlled to execute the processing stage adjacent to the request processing stage corresponding to the first component. In this case, the second component can be triggered when the first component finishes processing the first request for the i-th time and obtains the i-th intermediate data of the first request, so that the second component can be controlled to process the first request for the (i+1)-th time. For example, when the i-th intermediate data of the first request is generated, the i-th intermediate data of the first request can be stored in a specified storage address. When new data is stored in this storage address, information for triggering the second component can be generated, so that the second component can be controlled to process the request immediately after the current task ends based on the newly stored data.

[0059] In one possible implementation, information for triggering the second component can be generated when the i-th intermediate data of the first request is generated, so that the second component can be controlled to process the first request immediately after the current task ends based on the i-th intermediate data of the first request.

[0060] In another specific implementation, the processing result of the first request is obtained based on the (i+1)th intermediate data of the first request. This can be achieved by performing similar operations as steps S101-S103 on the (i+1)th intermediate data of the first request again, and obtaining the processing result of the first request after at least one loop.

[0061] In another specific implementation, the processing result of the first request is obtained based on the (i+1)th intermediate data of the first request. This can be done by directly using the (i+1)th intermediate data of the first request as the processing result of the first request, or by using it as part of the processing result of the first request.

[0062] In another specific implementation, while controlling the second component to process the first request for the (i+1)th time based on the i-th intermediate data, the first component can be controlled to process the third request for the nth time. Alternatively, while controlling the second component to process the first request for the (i+1)th time based on the i-th intermediate data, the first component can be controlled to process the second request for the (k+1)th time.

[0063] The method provided in this disclosure can be applied to deep learning models, machine learning models, and other request processing modules with data calculation, transformation, and parsing functions. The aforementioned deep learning models and machine learning models can be prediction models such as weather forecasting models, biological growth forecasting models, traffic flow forecasting models, and user preference forecasting models; they can also be analysis models such as text processing models and numerical processing models; they can be audio and video processing models; or other models that generate output data based on input data.

[0064] When the method of this disclosure is applied to a model, the first component and the second component can be controlled to execute model stages in adjacent time sequences. These model stages can be generated based on the layers within the model, the relationships between layers, and the operators or functions included in the layers. The processing result of the first request can be the result output by the model based on the first request.

[0065] As can be seen, the above scheme can simultaneously control the second component to process the second request k times while the first component processes the intermediate data of the first request. This allows different requests to be input into the first component or the second component in sequence, and controls the first component and the second component to execute request processing operations in parallel, thereby improving the efficiency of request processing and the utilization rate of request processing resources such as components per unit time. When the number of requests to be processed is large, it has a more significant efficiency improvement effect.

[0066] In one implementation, obtaining the processing result of the first request based on the (i+1)th intermediate data of the first request includes:

[0067] Based on the (i+1)th intermediate data of the first request, determine the number of loop processing times for the first request;

[0068] If the number of loop processing times reaches a preset loop threshold, the (i+1)th intermediate data of the first request is taken as the processing result of the first request.

[0069] As can be seen, the above embodiments can apply parallel processing to the request loop processing process, so that other requests can be processed in parallel during the loop processing of one request, thereby further improving the request processing efficiency.

[0070] In one implementation, the request processing method further includes:

[0071] If the number of loop processing times does not reach the preset loop threshold, control the first component and the second component to update the number of loop processing times according to the (i+1)th intermediate data of the first request.

[0072] After updating the loop count, loop processing can continue for the first request. The loop processing after this update can still be performed concurrently with... Figure 1 The process shown is similar to, or includes, that of other processes. Figure 1 The steps are shown.

[0073] As can be seen, in this embodiment, parallel processing can be applied to the request loop processing process, so that other requests can be processed in parallel during the loop processing of one request, thereby further improving the request processing efficiency.

[0074] In one implementation, such as Figure 2As shown, the request processing method also includes:

[0075] Step S201: Obtain the first number of original words;

[0076] Step S202: Based on the first number of original words, obtain the first request; the processing result of the first request is used to obtain the second number of predicted words; the second number is equal to the loop threshold.

[0077] In the embodiments of this disclosure, the processing and generation processes of the second request can be similar to or the same as those of the first request.

[0078] When the second quantity equals the loop threshold, a prediction word can be generated after the control unit executes the last request processing stage during each loop process; thus, after the number of loops reaches the loop threshold, a number of prediction words equal to the loop threshold can be generated.

[0079] As can be seen, the above embodiments enable the generation of a number of predicted words from a certain number of original words, thereby achieving the goal of generating predicted words based on existing words during text processing. Through this embodiment, raw data generated from different original words can be processed simultaneously during text processing, improving the efficiency of text processing.

[0080] In one implementation, controlling the first component to process the first request for the i-th time includes controlling the first component to process the first request for the i-th time through a first model layer; the first model layer is one of at least one model layer in the target model.

[0081] Controlling the second component to process the second request for the kth time includes: controlling the second component to process the second request for the kth time through the second model layer; the second model layer is one of at least one model layer in the target model, and the second model layer is different from the first model layer.

[0082] The target model mentioned above can be a generative model, that is, a model used to generate the output results required by the user based on the input raw data, such as various prediction models, image processing models, text processing models, and other deep learning or machine learning models.

[0083] For example, the target model can be divided into multiple model layers, with each model layer deployed on a component. Alternatively, a portion of the target model can be divided into multiple model layers, with each model layer deployed on a component.

[0084] For example, at least one model layer may have M model layers.

[0085] For example, the target model is used to perform loop processing operations on the requests that need to be processed, or a part of the target model is used to perform loop processing operations on the requests that need to be processed.

[0086] This embodiment enables at least some layers in the target model used for performing loop processing operations on requests to process intermediate data of different requests in parallel, thereby improving component utilization and the efficiency of the target model in processing requests. Furthermore, this embodiment can divide the target model into multiple model layers, which, even as the target model size increases, can improve model processing efficiency while reducing model deployment difficulty.

[0087] In one implementation, such as Figure 3 As shown, the request processing method also includes:

[0088] Step S301: Based on the request processing operators included in the original model and the relationships between different request processing operators, obtain the operator graph of the original model;

[0089] Step S302: Based on the operator graph of the original model, delete redundant operators in the request processing operators, or based on the operator graph of the original model, merge duplicate operators in the request processing operators to obtain the processed operator graph.

[0090] Step S303: Obtain the target model based on the processed operator graph.

[0091] In this embodiment, the request processing operator can be a function within the model or other sub-networks used to process data.

[0092] This embodiment enables the optimization of the target model, improving the efficiency of processing the input data of the target model from multiple aspects.

[0093] In the field of natural language processing, generative models are a common type of prediction model. These models generate Y words from X input words. During prediction inference, only one word is generated per computation, and the goal of generating Y words is achieved by looping Y times; that is, the model includes a loop body (while block). When the model is deployed on a single GPU for prediction, the presence of the loop body does not cause problems. However, when applying generative models to distributed prediction pipelines in parallel scenarios, the presence of the while block poses functional and performance challenges to distributed prediction.

[0094] First, for pipelined parallel scenarios, the model can be deployed across multiple GPUs (equivalent to the components in the aforementioned embodiment). Therefore, during pipelined prediction, the loop termination condition needs to be propagated across all GPUs to ensure each GPU correctly executes the loop processing operation. Variable values ​​changed during the loop also need to be synchronized across all GPUs. Furthermore, the pipelined parallel prediction mode feeds data in the first pipeline stage and fetches results in the last pipeline stage, which contradicts the prediction mechanism's logic of fetching data from the GPU on which it was fed.

[0095] Secondly, when using pipelined parallelism for distributed inference prediction, the lack of a robust safeguard mechanism can lead to low overall hardware utilization. In a scenario with a pipelined parallel prediction pipeline degree of 2, running the model on two GPUs (GPU1 and GPU0), if a traditional sequential executor is used, when GPU0 finishes its current loop's prediction calculation and waits for the next loop, while GPU1 is performing its current loop's prediction calculation, GPU0 can only wait for GPU1 to finish its calculation without performing any effective computation. This results in only one GPU performing effective prediction calculations, wasting hardware resources. This waste becomes more pronounced with increased pipelined parallelism granularity. When the pipeline degree is N, only one pipeline stage is performing predictions at any given time, resulting in an overall machine utilization of 1 / N.

[0096] Furthermore, as the model size increases, how to perform efficient distributed large model inference is also a very challenging problem.

[0097] This disclosed embodiment can be applied to generative models. When a generative model is deployed on multiple processors, the model can be segmented, with each segmented model layer deployed on a separate processor. The final processing result is synchronously sent to the first component that receives the raw data, thus achieving output from the same input location. The loop termination condition can be set in the first and last processing units. Each processing unit executes the loop steps sequentially, with the last processing unit updating the loop condition. Except for the stages of processing the initial input request and generating the final result of the request, the processing efficiency of the remaining stages can reach close to 100%. Furthermore, as the model size for processing requests increases, the model deployment difficulty is reduced, further improving the efficiency of distributed model inference.

[0098] In one embodiment, when the first component receives the first request, the request processing method further includes:

[0099] The processing result of the first request is sent to the first component.

[0100] In this embodiment, sending the processing result of the first request to the first component may include sending the processing result of the first request to a first storage unit of the first component. The first storage unit may be a storage unit that the first component has access and read permissions for, and can be represented by a first storage address. The second component may correspond to a second storage unit, which may be a storage unit that the second component has access and read permissions for, and can be represented by a second storage address.

[0101] In this embodiment, intermediate data required by the second component to process the first request or the second request can also be stored in the second storage unit. And intermediate data required by the first component to process the first request or the second request can also be stored in the first storage unit.

[0102] It can be seen that when the first and second components process raw or intermediate data, they can obtain it from their respective storage units with access and read permissions, ensuring the smooth progress of the parallel processing of multiple raw data.

[0103] In text prediction scenarios, products that perform cyclic operations in the prediction process in parallel include Deep Speed ​​and Fast Transformer. Deep Speed ​​uses tensor parallelism (the first and second layers of the large model, as well as other related layers, are all placed on a single GPU) for distributed large model prediction.

[0104] Fast Transformer implements tensor parallelism and pipelined parallelism in large model inference, but due to the presence of loops, it cannot efficiently utilize hardware.

[0105] In terms of performance optimization, both Deep Speed ​​and Fast Transformer employ schemes such as fuse kernel (operator fusion), fp16 (half-precision floating-point 16, 16-bit half-precision floating-point data) calculation, and int8 (initial 8, 8-bit integer) quantization to improve prediction speed.

[0106] Deep Speed ​​does not support pipelined parallelism for distributed inference, which increases communication overhead when the model is large enough to require inter-machine communication. Fast Transformer, lacking support for loops, wastes hardware resources during prediction and inference due to the sequential executor, failing to achieve optimal hardware resource utilization. The optimization techniques in DeepSpeed ​​and Fast Transformer do not effectively reuse existing optimizations from training, resulting in secondary development costs. In view of the shortcomings of tools like Deep Speed ​​and Fast Transformer, this disclosure presents an example that supports pipelined parallel inference using a while-block inference network (i.e., a model network containing only forward computation). Based on this, this example divides the model using a conventional pipelined parallelism method. For pipeline checkpoints (input / output points, i.e., nodes between different processing stages) variables that already need to be transmitted between pipeline stages in the original pipelined parallelism, the upstream pipeline stage still calculates and sends them to the downstream pipeline stage. This step is the same as the conventional pipelined parallelism logic.

[0107] In this example, the loop control variable of the while block (a counter in the generative model) is updated by the last pipeline stage, and the value of the counter is synchronized to the processor (e.g., GPU) of each pipeline stage after the update.

[0108] In this disclosed example, furthermore, the updated variable values ​​in the while block are calculated and updated by the last pipeline stage, and then the updated values ​​are synchronized to the pipeline stages that need them by sending intermediate data to the corresponding storage address of the corresponding component. Due to the parallel inference characteristics of the pipeline, all variables that need to be updated in the while block will receive a final update in the last pipeline stage.

[0109] For example, for the final output of the program, the last pipeline stage adds a send operation to synchronize the final result to the first pipeline stage (i.e., send it to the first component). This achieves the function of fetching data on the same GPU that was fed data.

[0110] Based on the message-driven and asynchronous execution characteristics of asynchronous pipelined executors, the above adaptation was made for pipelined parallel predictive inference with while blocks, which improves the overall hardware (GPU) utilization of inference networks with while blocks in pipelined parallel mode.

[0111] Taking a two-card pipelined parallelism as an example, the following steps are used for pipelined parallel prediction to improve GPU utilization. At each time step (T), the model and the operations corresponding to each processing unit of the model include:

[0112] T0: The model receives two requests, query 0 and query 1 (equivalent to the first and second requests in the previous embodiment), both of which require the generation of Y words (performing Y loops);

[0113] T1: GPU0 (equivalent to the first component of the first model layer) performs the first loop calculation on query0 and saves the intermediate results to scope0 (equivalent to the first storage unit in the aforementioned embodiment);

[0114] T2: GPU1 (equivalent to the second component deploying the second part of the model layer) performs the first loop calculation on query0 and saves the intermediate results to scope0. At the same time, GPU0 performs the first loop calculation on query1 and saves the intermediate results to scope1 (equivalent to the second storage unit in the aforementioned embodiment).

[0115] T3: GPU0 performs a second loop calculation on query0 based on the saved results in scope0, and updates the intermediate results in scope0. Simultaneously, GPU1 performs a first loop calculation on query1, and saves the intermediate results in scope1.

[0116] T4: GPU1 performs a second loop calculation on query0 based on the saved results in scope0, and updates the intermediate results in scope0. Simultaneously, GPU1 performs a second loop calculation on query1 based on the results in scope1, and updates the intermediate results in scope1.

[0117] And so on,

[0118] T(2Y-1): GPU0 performs the Y-th loop on query0, and GPU1 performs the Y-1-th loop on query1;

[0119] T(2Y): GPU0 performs the Y-th loop on query1, GPU1 performs the Y-th loop on query0, and the result of query0 has been produced;

[0120] T(2Y+1): GPU1 performs the Y-th iteration on query1, and the result of query1 has been generated;

[0121] From an overall process perspective, except for the initial T0 and the final TN+2 times when only one GPU is performing effective computation (overall utilization of 50%), both GPUs are performing effective computation at other times (overall utilization of 100%). Compared to a pure sequential executor, the pipeline parallelism time can be reduced from the original 4Y to (2Y+1). Furthermore, when the scale of pipeline parallelism is further increased to N pipeline stages, the overall GPU utilization of the traditional sequential executor is only (1 / N). However, through the strategy of this solution, the overall GPU utilization can be maintained at 100% after the pipeline warmup phase (i.e., the first component processes the initial input request for the first time), significantly improving the overall hardware resource utilization efficiency.

[0122] The feasibility of this scheme is achieved through the message-driven and asynchronous execution characteristics of the asynchronous pipelined executor. After GPU0 completes its computation, it triggers GPU1's computation via a message. When GPU1 finishes its previous round of computation, it triggers GPU0's next round of computation by sending a message. This mechanism ensures that after GPU0 finishes its computation on query0, it can proceed with the computation on query1 even without receiving a message from GPU1 regarding intermediate data for query0. Unlike sequential executors, this avoids the dependency relationship of send / recv (receive / send) to control the operation of GPU0 / GPU1 (data input to GPU0 can be sent from GPU0; GPU1 can be triggered by messages, and GPU0 can be triggered by data), which would cause GPU0 to hang after its computation is complete at the point of receiving the result from GPU1, preventing further effective computation.

[0123] In one possible implementation, to reduce secondary development costs, all optimization techniques from the training process are reused during prediction. Performance is improved through PaaS (Platform as Service) integration. Furthermore, after the system loads the model network, graph optimization can be used. First, the network (model or model components) that the user needs to infer is transformed into a graph, and then the overall network is optimized through graph compilation and graph optimization. These two methods work together to improve the efficiency of inference and prediction.

[0124] In a specific example, such as Figure 4As shown, the target model can include components such as the Main Block 401, the loop body 402, and the Beam Search 403. For a query with X words, the generation network first encodes the query through the main block. Then, in each while block and the corresponding Beam Search, the next word is predicted. After Y while loops, the final result is output, thus realizing the function of predicting Y words based on the query (raw data) generated from X words.

[0125] In specific examples, the loop body can be split into separate segments. Generally, if the loop body is not split, it can be deployed on a single processor for single-card operation, such as... Figure 5 As shown:

[0126] Step 500: Set the counter value for the number of loop iterations to 0;

[0127] Step 501: Determine if the counter value is less than the set threshold (condition); if the counter value is less than the set threshold, proceed to step 502; otherwise, if the counter value is equal to the set threshold, output the final result (final result).

[0128] Step 502: Obtain the (i-1)th intermediate data for processing the first request;

[0129] Step 503: Process the (i-1)th intermediate data of the first request to obtain the ith intermediate data of the first request and update it (update val 1).

[0130] Step 504: Update the counter. Specifically, after the current loop ends (and val 1 is obtained), update the counter (updatecounter), and then return to step 501 for judgment.

[0131] In a specific example, after dividing the loop body, two processing units are obtained, namely the first component and the second component. The pipeline operation process includes, as follows: Figure 6 In the pipeline stages 0 and 1 shown, each processing unit counts the number of loops. In the initial stage, the first counter value of the first component is set to 0, and the second counter value of the second component is set to 0. In pipeline stage 0, after initializing the first counter value, a judgment step 600 is executed to determine whether the first counter value is less than the loop threshold. If it is less, step 601 is executed; otherwise, pipeline stage 0 is determined to end.

[0132] Step 601: Send the (i-1)th intermediate data of the first request to the second component corresponding to the pipeline stage 1.

[0133] Specifically, it can be done as follows: obtain the (i-1)th intermediate data of the first request (update val 0), that is, obtain val 0. Then simultaneously send the (i-1)th intermediate data of the first request (update val 0) and null data to indicate that the second component corresponding to pipeline stage 1 needs to update the null data as the i-th intermediate data of the first request (val 1), that is, send val0 and val 1.

[0134] In pipeline stage 1, step 610 is executed to determine whether the value of the second counter is less than the loop threshold. If it is less, step 611 is executed; otherwise, pipeline stage 1 is determined to end and the final result is generated.

[0135] Step 611: Receive the (i-1)th intermediate data of the first request sent by the first component.

[0136] That is, the second component receives val 0 (the (i-1)th intermediate data of the first request) and val 1 (the i-th intermediate data of the first request), at which time val 1 is an empty value.

[0137] Step 612: Obtain the i-th intermediate data of the first request.

[0138] Specifically, the second component processes the first request for the (i-1)th time based on the (i+1)th intermediate data to obtain the i-th intermediate data of the first request. That is, it obtains the updated i-th intermediate data of the first request (updating val1), and obtains the updated val0 and val1.

[0139] This step may also include updating the second counter value of pipeline stage 1.

[0140] Step 613: Send the (i-1)th intermediate data of the first request and the ith intermediate data of the first request to the first component.

[0141] Specifically, this can be done by controlling the second component to send val 0, val 1, and the second counter value to the first component of the pipeline stage 0 and returning to the judgment step 610.

[0142] The first component continues to execute step 602 in the flow stage 0, receiving the (i-1)th intermediate data of the first request and the i-th intermediate data of the first request.

[0143] Specifically, the first component can receive val0, val1, and the second counter value, update the first counter value accordingly, and return to the judgment step 600. If the judgment condition is not met during the execution of judgment step 600 in pipeline stage 0, pipeline stage 0 ends. If the judgment condition is not met during the judgment step 610 in pipeline stage 1, the final result is generated. Finally, the final result is synchronized to the first component corresponding to pipeline stage 0.

[0144] Generally, if sequential execution is used, then... Figure 7A As shown, when the first component corresponding to pipeline stage 0 processes query0 or its intermediate data, the second component corresponding to pipeline stage 1 is idle. Then, when the second component corresponding to pipeline stage 1 processes the intermediate data of query0, the first component corresponding to pipeline stage 0 is idle. However, using the method provided in this disclosure example, as follows... Figure 7B As shown, when the first component corresponding to pipeline stage 0 processes query0 or its intermediate data, the second component corresponding to pipeline stage 1 can process the intermediate data of query1. When the second component corresponding to pipeline stage 1 processes the intermediate data of query0, the first component corresponding to pipeline stage 0 processes query1 or its intermediate data, thereby significantly improving the utilization rate of each processing unit.

[0145] This disclosure enables pipelined parallel prediction with while blocks in distributed inference. Compared to Deep Speed, it significantly reduces the cost of inter-machine communication. Furthermore, through special handling of while blocks in pipelined parallelism, combined with the message-driven and asynchronous execution characteristics of asynchronous pipeline executors, the utilization of GPUs or other processors during prediction is greatly improved. In addition, by integrating an optimized PaaS for training and graph compilation optimization in the prediction phase, the efficiency of distributed large-scale model prediction is further improved.

[0146] This disclosure also provides a request processing apparatus, such as... Figure 8 As shown, it includes:

[0147] The first processing module 801 is used to control the second component to process the second request for the kth time during the process of controlling the first component to process the first request for the i-th time; i and k are integers greater than 1.

[0148] The second processing module 802 is used to control the second component to process the first request for the (i+1)th time based on the i-th intermediate data of the first request, when it is determined that the first component processes the first request for the i-th time and obtains the i-th intermediate data of the first request.

[0149] The first processing result module 803 is used to obtain the processing result of the first request based on the (i+1)th intermediate data of the first request;

[0150] The second processing result module 804 is used to obtain the processing result of the second request based on the kth intermediate data of the second request obtained by the second component in the kth processing of the second request.

[0151] In one implementation, such as Figure 9 As shown, the first processing result module includes:

[0152] The loop processing count submodule 901 is used to determine the loop processing count of the first request based on the (i+1)th intermediate data of the first request;

[0153] The result determination submodule 902 is used to take the (i+1)th intermediate data of the first request as the processing result of the first request when the number of loop processing times reaches a preset loop threshold.

[0154] In one implementation, such as Figure 10 As shown, the first processing result module also includes:

[0155] The loop control submodule 1001 is used to control the first component and the second component to update the loop processing count according to the (i+1)th intermediate data of the first request when the number of loop processing counts has not reached the preset loop threshold.

[0156] In one implementation, such as Figure 11 As shown, the request processing device further includes:

[0157] The original word module 1101 is used to obtain the first number of original words;

[0158] The first request generation module 1102 is used to obtain a first request based on a first number of original words; the processing result of the first request is used to obtain a second number of predicted words; the second number is equal to the loop threshold.

[0159] In one implementation, such as Figure 12 As shown, the first processing module includes:

[0160] The first model layer calls submodule 1201 to control the first component to process the first request for the i-th time through the first model layer; the first model layer is one of at least one model layer in the target model.

[0161] The second model layer calls submodule 1202 to control the second component to process the second request for the kth time through the second model layer; the second model layer is one of at least one model layer in the target model, and the second model layer is different from the first model layer.

[0162] In one implementation, such as Figure 13 As shown, the request processing device further includes:

[0163] The operator graph module 1301 is used to obtain the operator graph of the original model based on the request processing operators included in the original model and the relationships between different request processing operators.

[0164] The optimization module 1302 is used to delete redundant operators in the request processing operators based on the operator graph of the original model, or to merge duplicate operators in the request processing operators based on the operator graph of the original model, so as to obtain the processed operator graph.

[0165] The target model generation module 1303 is used to obtain the target model based on the processed operator graph.

[0166] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0167] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0168] Figure 14 A schematic block diagram of an example electronic device 1400 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0169] like Figure 14 As shown, the electronic device 1400 includes a computing unit 1401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1402 or a computer program loaded from a storage unit 1408 into a random access memory (RAM) 1403. The RAM 1403 may also store various programs and data required for the operation of the electronic device 1400. The computing unit 1401, ROM 1402, and RAM 1403 are interconnected via a bus 1404. An input / output (I / O) interface 1405 is also connected to the bus 1404.

[0170] Multiple components in electronic device 1400 are connected to I / O interface 1405, including: input unit 1406, such as keyboard, mouse, etc.; output unit 1407, such as various types of monitors, speakers, etc.; storage unit 1408, such as disk, optical disk, etc.; and communication unit 1409, such as network card, modem, wireless transceiver, etc. Communication unit 1409 allows electronic device 1400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0171] The computing unit 1401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1401 executes the request processing method described above. For example, in some embodiments, the request processing method described above can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1408. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 1400 via ROM 1402 and / or communication unit 1409. When the computer program is loaded into RAM 1403 and executed by the computing unit 1401, one or more steps of the request processing method described above can be performed. Alternatively, in other embodiments, the computing unit 1401 can be configured to execute the request processing method described above by any other suitable means (e.g., by means of firmware).

[0172] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0173] Program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable request processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0174] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0175] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0176] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0177] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0178] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0179] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for processing a request, comprising: controlling a first component to process a first request for an i th time and a second component to process a second request for a k th time, wherein i and k are integers greater than 1; in a case where it is determined that the first component processes the first request for the i th time to obtain i th intermediate data of the first request, controlling the second component to process the first request for an i + 1 th time according to the i th intermediate data to obtain i + 1 th intermediate data of the first request; obtaining a processing result of the first request according to the i + 1 th intermediate data of the first request and a processing result of the second request according to k th intermediate data of the second request obtained by the second component processing the second request for the k th time; the method further comprises, in a process of controlling the second component to process the first request for the i + 1 th time according to the i th intermediate data, controlling the first component to process the second request for a k + 1 th time; the method further comprises: obtaining a first number of original words; and obtaining the first request according to the first number of original words; the processing result of the first request is used to obtain a second number of predicted words, and the processing process and the generation process of the second request are the same as those of the first request. the method further comprises:

2. The method of claim 1, wherein, in a case where the i + 1 th intermediate data of the first request is determined as the processing result of the first request when the number of cycles reaches a preset cycle threshold. 3.The method of claim 2, further comprising: in a case where the number of cycles does not reach the preset cycle threshold, controlling the first component and the second component to update the number of cycles according to the i + 1 th intermediate data of the first request. the second number is equal to the cycle threshold. the controlling the first component to process the first request for the i th time comprises: controlling the first component to process the first request for the i th time through a first model layer; and the first model layer is one of at least one model layer in a target model.

4. The method of claim 2 or 3, wherein, the controlling the second component to process the second request for the k th time comprises: controlling the second component to process the second request for the k th time through a second model layer; and the second model layer is one of at least one model layer in the target model, and the second model layer is different from the first model layer.

5. The method of claim 1, wherein, 6.The method of claim 5, further comprising: obtaining an operator graph of an original model according to request processing operators included in the original model and relationships between different request processing operators; deleting redundant operators in the request processing operators according to the operator graph of the original model, or merging duplicate operators in the request processing operators according to the operator graph of the original model to obtain a processed operator graph; obtaining the target model according to the processed operator graph. 7.A device for processing a request, comprising: ​ ​ The first processing module is configured to control the second component to process the second request for the kth time during the process of controlling the first component to process the first request for the ith time. i and k are integers greater than 1; The second processing module is configured to control the second component to process the first request for the ith+1 time according to the ith intermediate data of the first request, and obtain the ith+1 intermediate data of the first request, in the case that the first component processes the first request for the ith time and obtains the ith intermediate data of the first request. The first processing result module is configured to obtain the processing result of the first request according to the ith+1 intermediate data of the first request. The second processing result module is configured to obtain the processing result of the second request according to the kth intermediate data of the second request obtained by the second component processing the second request for the kth time. The second processing module is configured to control the first component to process the second request for the kth+1 time during the process of controlling the second component to process the first request for the ith+1 time according to the ith intermediate data. The device further comprises: an original word module configured to obtain a first number of original words; and a first request generation module configured to obtain the first request according to the first number of original words; the processing result of the first request is configured to obtain a second number of predicted words, and the processing process and the generation process of the second request are the same as those of the first request.

8. The apparatus of claim 7, wherein, The first processing result module comprises: A loop processing number submodule configured to determine the loop processing number of the first request according to the ith+1 intermediate data of the first request. A result determination submodule configured to take the ith+1 intermediate data of the first request as the processing result of the first request in the case that the loop processing number reaches a preset loop threshold.

9. The apparatus of claim 8, wherein, The first processing result module further comprises: A loop control submodule configured to control the first component and the second component to update the loop processing number according to the ith+1 intermediate data of the first request in the case that the loop processing number does not reach the preset loop threshold.

10. The apparatus of claim 8 or 9, wherein, The second number is equal to the loop threshold.

11. The apparatus of claim 7, wherein, The first processing module comprises: A first model layer calling submodule configured to control the first component to process the first request for the ith time through a first model layer; the first model layer is one of at least one model layer in a target model; A second model layer calling submodule configured to control the second component to process the second request for the kth time through a second model layer; the second model layer is one of at least one model layer in the target model, and the second model layer is different from the first model layer.

12. The device of claim 11, further comprising: An operator graph module configured to obtain an operator graph of an original model according to request processing operators included in the original model and relationships between different request processing operators; An optimization module configured to delete redundant operators in the request processing operators according to the operator graph of the original model, or to combine repeated operators in the request processing operators according to the operator graph of the original model, and obtain a processed operator graph. A target model generation module configured to obtain the target model according to the processed operator graph. 13.An electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

14. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1-6. 15.A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-6.

Images