Inference method and device, equipment and medium

Abstract

The invention provides a reasoning method and device, equipment and a medium. The method comprises the steps that when a reasoning request is received, model parameters of a target model deployed on each GPU are obtained, the reasoning request comprises information of a reasoning task, and the information of the reasoning task comprises input data; then, according to the structure information of the target model, segmenting the reasoning task into a plurality of groups of subtasks corresponding to the GPU according to a combination mode of a calculation task and a communication task; segmenting the input data into input sub-data corresponding to each group of sub-tasks; afterwards, generating a CUDA flow corresponding to the GPU according to a sequence dependency relationship between the calculation task and the communication task in each group of subtasks, the CUDA flow being used for realizing parallel execution of multiple groups of subtasks; then, a CUDA kernel on the GPU is scheduled according to the CUDA flow, so that the GPU executes multiple groups of subtasks in parallel according to model parameters and input subdata, and a reasoning result corresponding to the reasoning request is obtained. According to the invention, the reasoning time delay can be reduced.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and specifically to a reasoning method, device, equipment and medium. Background Art

[0002] With the rise of LLM (Large Language Model) represented by the Transformer architecture, the scale of deep learning models continues to expand, and the number of parameters has reached hundreds of billions and trillions, which has promoted the rapid development of the field of artificial intelligence. However, corresponding challenges have also arisen: the traditional single-machine single-card mode can no longer meet the requirements for reasoning of large models.

[0003] At present, the relevant technology has designed a method of submitting computing tasks on the CUDA core of a CUDA (Compute Unified Device Architecture) stream allocated on a single GPU (Graphic Processing Unit), and submitting communication tasks on the CUDA core of another CUDA stream. However, there is the following problem: submitting the communication task and computing task on a single GPU to the two CUDA cores respectively causes a large gap when the CPU schedules tasks on two different CUDA cores, resulting in a large inference delay, which has become a problem that needs to be solved urgently. Summary of the invention

[0004] In view of this, the embodiments of the present application are committed to providing an inference method, apparatus, device and medium to reduce inference delay.

[0005] In a first aspect, an inference method is provided, the method comprising: in response to receiving an inference request for a target model, obtaining model parameters of the target model deployed on each graphics processor, wherein the inference request includes information about the inference task, and the information about the inference task includes input data; according to the structural information of the target model, dividing the inference task into a plurality of groups of subtasks corresponding to the graphics processor in a combination of computing tasks and communication tasks; wherein each group of subtasks includes a communication task and a computing task; and, dividing the input data into input sub-data corresponding to each group of subtasks; according to the sequential dependency between the computing tasks and the communication tasks in each group of subtasks, generating a CUDA stream corresponding to the graphics processor; wherein the CUDA stream is used to implement parallel execution of the plurality of groups of subtasks; and scheduling the CUDA kernel on the graphics processor according to the CUDA stream so that the graphics processor executes the plurality of groups of subtasks in parallel according to the model parameters and the input sub-data, and obtains an inference result corresponding to the inference request.

[0006] In a second aspect, an inference device is provided, which includes: an acquisition module, which is used to obtain model parameters of the target model deployed on each graphics processor in response to receiving an inference request for the target model, wherein the inference request includes information about the inference task, and the information about the inference task includes input data; a segmentation module, which is used to segment the inference task into multiple groups of subtasks corresponding to the graphics processor in a combination of computing tasks and communication tasks according to the structural information of the target model; wherein each group of subtasks includes a communication task and a computing task; and, segment the input data into input sub-data corresponding to each group of subtasks; a generation module, which is used to generate a CUDA stream corresponding to the graphics processor according to the sequential dependency between the computing tasks and the communication tasks in each group of subtasks; wherein the CUDA stream is used to implement parallel execution of multiple groups of subtasks; and an inference module, which is used to schedule the CUDA kernel on the graphics processor according to the CUDA stream, so that the graphics processor can execute multiple groups of subtasks in parallel based on the model parameters and the input sub-data to obtain an inference result corresponding to the inference request.

[0007] According to a third aspect, an electronic device is provided, comprising: one or more processors; and a memory associated with the one or more processors, the memory being used to store program instructions, which, when read and executed by the one or more processors, execute the method described in the first aspect above.

[0008] According to a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method described in the first aspect is implemented.

[0009] According to a fifth aspect, a program product is provided, comprising computer program instructions, which, when executed by a processor, enable the processor to implement the method described in the first aspect.

[0010] In this embodiment, first, when receiving an inference request, according to the structural information of the target model, the inference task is divided into a plurality of groups of subtasks corresponding to the graphics processor in a combination of computing tasks and communication tasks; wherein each group of subtasks includes a communication task and a computing task; and, the input data is divided into input sub-data corresponding to each group of subtasks; and the input data is divided to obtain input sub-data corresponding to each group of subtasks, wherein each group of subtasks includes a computing task and a communication task; then, according to the sequential dependency between the computing tasks and the communication tasks in each group of subtasks, a CUDA stream for the graphics processor is generated, and the CUDA stream is used to implement parallel execution of multiple groups of subtasks; then, according to the CUDA stream, the CUDA kernel on the graphics processor is scheduled so that the graphics processor can execute multiple groups of subtasks in parallel according to the input sub-data and the model parameters deployed on the graphics processor to obtain the inference result corresponding to the inference request. This embodiment can solve the problem of a large gap between the CUDA kernel for executing computing tasks and the CUDA kernel for executing communication tasks on the scheduling GPU in the related technology, and thus can solve the problem of inference delay caused by the gap, thereby improving the inference efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a schematic diagram of the architecture of the application scenario of the reasoning method provided in the embodiment of the present application;

[0012] Figure 2 is a flow chart of the reasoning method provided in the embodiment of the present application;

[0013] Figure 3 It is a category table of communication tasks and computing tasks;

[0014] Figure 4 It is a schematic diagram of dividing the computing tasks and communication tasks;

[0015] Figure 5 It is a schematic diagram of the split full connection operation;

[0016] Figure 6 It is a schematic diagram of the split-all-reduce operation;

[0017] Figure 7 It is a schematic diagram of the split tensor operation (+);

[0018] Figure 8 It is a schematic diagram of the tasks executed in parallel in each reasoning round;

[0019] Fig. 9 It is a schematic diagram of the two-round communication algorithm;

[0020] Fig.10 It is a schematic diagram of the ring communication algorithm;

[0021] Fig.11 is a flow chart of the reasoning method provided in the embodiment of the present application;

[0022] Fig.12 is a schematic diagram of an inference device provided in an embodiment of the present application;

[0023] Fig.13 It is a structural diagram of the CPU host side provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0025] In this embodiment, the inference method may include: the CPU host side generates a CUDA stream corresponding to the GPU device side; the CPU host side schedules the CUDA kernel on the GPU device side for parallel calculation according to the CUDA stream to obtain the inference result, which can solve the problem of large gaps in the related technology in scheduling different CUDA kernels on the GPU device side to respectively execute computing tasks and communication tasks.

[0026] For ease of understanding, this article first introduces the application scenarios to which the reasoning method provided by this application is applicable. Figure 1 , the CPU host end 110 is connected to the GPU device end 120 for communication, and the GPUs in the GPU device end 120 are connected to each other for communication. For example, PCIe (Peripheral Component Interconnect Express) is used for communication between the CPU host end 110 and the GPU device end 120; NVLink (connection) is used for communication between the GPUs in the GPU device end 120; wherein NVLink is a bus communication protocol developed and launched by NVIDIA. Among them, the GPU device end 120 is used to execute tasks in parallel; the CPU host end 110 is used to receive inference requests for the target model, and for the model parameters of the target model deployed on each graphics processor, the CPU host end 110 performs the following operations:

[0027] (1) in response to receiving an inference request for a target model, obtaining model parameters of the target model deployed on each graphics processor, wherein the inference request includes information of an inference task, and the information of the inference task includes input data;

[0028] (2) dividing the inference task into a plurality of groups of subtasks corresponding to the graphics processor according to the combination of computing tasks and communication tasks according to the structure information of the target model; wherein each group of subtasks includes a communication task and a computing task; and dividing the input data into input sub-data corresponding to each group of subtasks;

[0029] (3) generating a CUDA stream corresponding to the graphics processor according to the sequential dependency between the computing tasks and the communication tasks in each group of subtasks; wherein the CUDA stream is used to implement parallel execution of multiple groups of subtasks;

[0030] (4) Scheduling the CUDA kernel on the graphics processor according to the CUDA stream so that the graphics processor executes multiple groups of subtasks in parallel according to the model parameters and the input sub-data to obtain an inference result corresponding to the inference request.

[0031] The CPU host end 110 may be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, and portable wearable devices. The GPU device end 120 may be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, and portable wearable devices.

[0032] Figure 2 A flowchart of the reasoning method provided in the embodiment of the present application, in which the scheme is applied to Figure 1 The CPU host in the example is used to illustrate. Figure 2 As shown in , the method may include the following steps:

[0033] Step S201: in response to receiving an inference request for a target model, obtaining model parameters of the target model deployed on each graphics processor, wherein the inference request includes information of an inference task, and the information of the inference task includes input data.

[0034] In this embodiment, on the CPU host side (eg Figure 1 When the CPU host side 110 shown in FIG. 1 receives an inference request for the target model, the GPU device side (eg Figure 1 The GPU device end 120 shown in the figure determines the GPU used to deploy the target model, and obtains the model parameters of the target model deployed on each GPU. Among them, the inference request is used to transmit the information of the inference task to the CPU host end, and the information of the inference task includes input data, which is the input data for inference.

[0035] Here, the setting of model parameters is crucial to ensure that the model can correctly and efficiently process input data and produce accurate output. These parameters usually involve multiple aspects such as model input, processing, and output. Prior to this, the model parameters of the target model mainly include the weight matrix and bias vector of each layer in the target model.

[0036] In one example, after receiving the reasoning request, the reasoning method further includes: generating a reasoning task according to the reasoning request.

[0037] In this embodiment, the inference request may be a combination of operators, for example, a communication operator and a fully connected operator, an activation operator, a reduction operator, and a tensor operator.

[0038] It should be noted that an operator is a special operation that maps or transforms functions or data spaces. For example, the full connection operator, activation operator, reduction operator, tensor operator, etc. are all operators that perform specific operations on functions.

[0039] Here, the tensor operations (i.e., tensor operators) included in the computational subtasks do not have the concept of layers, because tensor operations (+, -, *, ÷, Pow, Sqrt) themselves do not constitute an independent layer structure. However, tensor operations are widely used in different layers of the target model and can affect or constitute the computational process of certain layers. Therefore, the parameters of tensor operations can be determined based on the weight matrices and bias vectors corresponding to certain layers.

[0040] For example, the convolution layer extracts features by performing multiplication and accumulation operations on the convolution kernel and the local area of ​​the input tensor, which can be regarded as a special tensor multiplication operation. The pooling layer reduces the spatial dimension of the data by downsampling the local area of ​​the input tensor (such as maximum pooling, average pooling, etc.), which can also be regarded as a special tensor operation.

[0041] In one example, the parameters of the target model deployed on each graphics processor may include: determining the scale information of the graphics processor based on the tensor parallel information of the target model, the scale information including the number of graphics processors, and the number of graphics processors is multiple; then, based on the structural information of the target model, splitting the model weight file of the target model into multiple copies, and loading them on multiple graphics processors to obtain the model parameters deployed on each graphics processor.

[0042] Here, the inference request may further include a model name, which may be the name of a target model, and the structural information of the target model is obtained according to the model name.

[0043] Among them, TP (Tensor Parallelism) is a type of model parallelism and a technology that accelerates the inference process. Tensor parallelism is to split the tensors in the model (such as weight matrices) along specific dimensions and distribute these split tensors to different GPUs for parallel computing to accelerate the inference process.

[0044] Here, the tensor parallel information may include a tensor parallel mode and / or a tensor parallel quantity. The tensor parallel mode includes row parallelism, column parallelism, one-dimensional tensor parallelism, and multi-dimensional tensor parallelism.

[0045] In one example, obtaining model parameters of a target model deployed on each graphics processor may include: obtaining model parameters of the target model deployed on each GPU via PCIe (Peripheral Component Interconnect Express) or NVLink (connection).

[0046] In one example, the inference request may be an instruction issued by the application layer for inference on the target model. The application layer may be used to complete the processing of the client's inference request and response reception. It should be noted that the client may be a separate physical device that is connected to the GPU host by communication, such as a wired or wireless connection; the client may also be a separate application or software on the GPU host.

[0047] Step S202: according to the structural information of the target model, the inference task is divided into a plurality of groups of subtasks corresponding to the graphics processor in a combination of computing tasks and communication tasks; each group of subtasks includes a communication task and a computing task; and the input data is divided into input sub-data corresponding to each group of subtasks.

[0048] In this embodiment, when an inference request for a target model is received in step S201, the inference task is divided into a plurality of groups of subtasks corresponding to the GPU in a combination of computing tasks and communication tasks according to the structural information of the target model, and the input data is divided to obtain input sub-data corresponding to each group of subtasks. The input sub-data may be the input data of the first task in each group of subtasks.

[0049] Here, the combination of computing tasks and communication tasks is used to divide the inference task into subtasks, each of which includes computing tasks and communication tasks. Each subtask group includes computing tasks and communication tasks; computing tasks are used to improve the computing efficiency of tensor parallel computing; communication tasks are used to transmit data between different subtask groups to ensure that they can work together and complete the entire inference task.

[0050] In this embodiment, the structural information of the target model can be used to reflect the model architecture of the target model, such as the composition of each layer of the target model, etc. There is a sequential dependency relationship between the layers in the target model, so the sequential dependency relationship between the operators in the reasoning task can be determined according to the structural information of the target model. For example, the fully connected operator (i.e., the fully connected operation) corresponding to the fully connected layer.

[0051] It should be noted that tensor operations do not correspond to the above layers, because tensor operations (+, -, *, ÷, Pow, Sqrt) themselves do not constitute an independent layer structure. However, tensor operations are widely used in different parts and levels of the target model, and can affect or constitute the calculation process of certain layers.

[0052] For example, the convolution layer extracts features by performing multiplication and accumulation operations on the convolution kernel and the local area of ​​the input tensor, which can be regarded as a special tensor multiplication operation. The pooling layer reduces the spatial dimension of the data by downsampling the local area of ​​the input tensor (such as maximum pooling, average pooling, etc.), which can also be regarded as a special tensor operation.

[0053] The above step S202, namely "divide the reasoning task into a plurality of groups of subtasks corresponding to the graphics processor in a combination of computing tasks and communication tasks according to the structural information of the target model" is described in detail below in conjunction with the embodiments.

[0054] In an embodiment of the present application, the sequential dependency between operators in the reasoning task is determined based on the structural information of the target model; based on the sequential dependency between operators in the reasoning task, the number of graphics processors and the type of operators in the reasoning task, the reasoning task is divided into multiple groups of subtasks corresponding to the graphics processor in a combination of computing tasks and communication tasks.

[0055] In this embodiment, the reasoning task is divided according to the sequence defined by the sequential dependency between the operators in the reasoning task, and is divided into corresponding dimensions according to the number of image processors and the type of operators to obtain multiple groups of subtasks corresponding to each GPU.

[0056] exist Figure 3In the embodiment, the tasks are divided in the order of full-connection operation, full-reduce operation, tensor operation and reduction operation (i.e., the order defined by the sequential dependency); when dividing the full-connection operation, the full-connection operation is divided into corresponding dimensions according to the number of GPUs and the type of the full-connection operation; then, when dividing the full-reduce operation, the full-reduce operation is divided into corresponding dimensions according to the division dimension of the full-connection operation and the type of the full-reduce operation; when dividing the tensor operation, the tensor operation is divided into corresponding dimensions according to the division dimension of the full-reduce operation and the type of the tensor operation; when dividing the reduction operation, the reduction operation is divided into corresponding dimensions according to the division dimension of the tensor operation and the type of the reduction operation, and finally a plurality of groups of subtasks are obtained.

[0057] In one example, after executing step S201, the type of each operator in the reasoning task can be identified so that when the reasoning task is divided, it is divided according to the division dimension corresponding to the type of each operator and the number of GPUs. For example, the matrix multiplication operator corresponds to the 2*2 division dimension.

[0058] Correspondingly, in this example, the division of the reasoning task is actually the division of each operator.

[0059] It should be noted that the type of operator determines whether the task is split horizontally or vertically; the number of GPUs determines the number of parts into which the task is split.

[0060] In one example, NCCL (NVIDIA Collective Communication Library) is a library for communication between GPUs, including aggregate communication and point-to-point communication, etc., where aggregate communication includes all-gather, all-reduce, broadcast, reduce, reduce-scatter, etc.

[0061] In this embodiment, NCCL is used to support efficient data exchange and parallel computing between GPUs to speed up the reasoning process.

[0062] exist Figure 3In the communication task, at least one of the following operators can be included: All-gather operation, All-Reduce operation, broadcast operation, Reduce operation, Reduce-scatter operation. Among them, All-Reduce aggregates the data on multiple GPUs and distributes them on each GPU so that each GPU generates the same aggregated result; All-gather collects the data of each GPU to all GPUs so that each GPU has the data of all GPUs.

[0063] exist Figure 3 In the example, a computing task includes at least one of the following operators: a fully connected (Matrix-Multiplication) operation, an activation operation, a reduction operation, and a tensor operation.

[0064] Among them, the full connection operation is a commonly used layer operation in neural networks. In a fully connected layer, all input neurons and all output neurons have an independent weight connection. Preferably, the full connection operation in large model reasoning can be implemented in the form of matrix multiplication or inner product operation.

[0065] Activation operation, a nonlinear function applied to the output of each neuron in a neural network. The purpose of the activation operation is to introduce nonlinearity so that the neural network can learn and represent complex functional relationships. Preferably, the activation operation in large model inference can include SiLU (Sigmoid Linear Unit), ReLU (Rectified Linear Unit) and GeLU (Gaussian Linear Unit).

[0066] A reduction operation aggregates the elements in a certain dimension of a tensor, usually used to transform multidimensional data into a lower dimension or scalar. The reduction operation may include LayerNorm (LayerNormalization), RmsNorm (Root Mean Square Normalization), Softmax (normalized exponential function). Preferably, a tensor can be a multidimensional data structure.

[0067] Tensor operations refer to various mathematical operations performed on multidimensional array structures. Tensors can be seen as the generalization of vectors and matrices. In an example, tensor operations cover many types of mathematical operations, including (+, -, *, ÷), square operations (Pow), square root operations (Sqrt), etc.

[0068] In an example, a fully connected operation is taken as a matrix multiplication operation, a full reduction operation is taken as All-Reduce, a tensor operation is taken as a tensor operation (+), and a reduction operation is taken as RmsNorm, and a process of dividing an inference task into four groups of subtasks is described in detail.

[0069] exist Figure 4 In the figure, the model parameters of the target model are deployed on GPU1 and GPU2 respectively. The principle of segmentation is to ensure that there is no sequential dependency between computing tasks and communication tasks among the four groups of subtasks corresponding to GPU1, and the same is true for GPU2 and GPU1. The following is the order defined by the sequential dependency between the operators in the inference task. The matrix multiplication operation, All-Reduce, tensor operation (+) and RmsNorm are segmented in turn to obtain the four groups of subtasks corresponding to GPU1 or GPU2. The specific description is as follows:

[0070] exist Figure 5 In the example, the formula corresponding to the matrix multiplication operation (i.e., the matrix multiplication operator) is Z=X*Y. The matrix Z in the inference request is divided by 2*2 (i.e., the number of GPUs (2 GPUs) and the type of matrix multiplication), and Z1=X1*Y1, Z2=X1*Y2, Z3=X2*Y1, and Z4=X2*Y2 corresponding to GPU1 or GPU2 are obtained. Among them, X is the input sub-data, and Y is the parameter. When dividing the inference task, the input data in step S201 is also divided according to the type of the first operator (i.e., the matrix multiplication operator).

[0071] exist Figure 6 In the example, the All-Reduce operation (i.e., the All-Reduce operator) is used to sum the output results of All-Reduce on GPU1 (i.e., Z1, Z2, Z3, and Z4) and the output results of All-Reduce on GPU2 (i.e., D1, D2, D3, and D4), and then propagate the summed results (i.e., E1, E2, E3, and E4) to GPU1 and GPU2 respectively, so that both GPU1 and GPU2 have E1, E2, E3, and E4. Among them, the input of All-Reduce is the matrix Z in the same group of subtasks, so All-Reduce needs to be split into 2*2 according to the same dimension as the matrix Z.

[0072] exist Figure 7In the above example, the tensor operation (+) is split. The formula of the tensor operation (+) is G=E+F. F is split according to the same dimension as E to obtain F1, F2, F3 and F4. Then G is obtained according to the formula of the tensor operation (+). G includes G1=E1+F1, G2=E2+F2, G3=E3+F3, and G4=E4+F4. RmsNorm is further split according to the type of RmsNorm and the dimension corresponding to G to obtain four groups of subtasks corresponding to GPU1 or GPU2. Among them, F is the parameter corresponding to the tensor operation (+).

[0073] It should be noted that since there is no sequential dependency between the computing tasks and the communication tasks among the four groups of subtasks, the four groups of subtasks can be executed in parallel, thereby speeding up the reasoning process and improving the efficiency of reasoning.

[0074] In this embodiment, there is no sequential dependency between computing tasks and communication tasks among the four groups of subtasks, so the four groups of subtasks can be executed in parallel, thereby speeding up the reasoning process and improving the efficiency of reasoning.

[0075] It should be noted that, before executing step S202 , the type of each operator in the inference task may be pre-identified, so that the inference task can be divided according to the number of GPUs and the type of each operator in step S202 .

[0076] Step S203: generating a CUDA stream corresponding to the graphics processor according to the sequential dependency between the computing tasks and the communication tasks in each group of subtasks; wherein the CUDA stream is used to implement parallel execution of multiple groups of subtasks.

[0077] In this embodiment, for the GPU (i.e., each GPU), a CUDA stream for implementing parallel execution of multiple groups of subtasks is generated according to the sequential dependency between the computing tasks and the communication tasks in each group of subtasks. The computing tasks and computing tasks in each group of subtasks refer to the fact that the completion of one task in the group of subtasks depends on the completion of one or more tasks in the group of subtasks, that is, the sequential dependency defines the execution order of the computing tasks and the communication tasks in the group of subtasks.

[0078] Here, a CUDA stream may be a sequence for executing multiple groups of subtasks in parallel. The CUDA stream is used to manage and organize the scheduling of CUDA kernels on a GPU so that the GPU can execute multiple groups of subtasks in parallel.

[0079] In this embodiment, a CUDA stream for executing multiple groups of subtasks in parallel is generated by computing tasks and computing tasks, so that when the CUDA stream is executed on the GPU host side, the CUDA kernel on the GPU is scheduled to execute the multiple groups of subtasks in parallel to speed up the reasoning process.

[0080] Step S204: scheduling the CUDA kernel on the graphics processor according to the CUDA stream, so that the graphics processor executes multiple groups of subtasks in parallel according to the model parameters and the input sub-data, and obtains the inference result corresponding to the inference request.

[0081] In this embodiment, after the CPU host generates a CUDA stream in step S203, the CPU host can schedule the CUDA kernel on the GPU according to the CUDA stream, so that the GPU can execute multiple groups of subtasks in parallel according to the model parameters and the input sub-data to obtain the inference result corresponding to the inference request. The CUDA kernel is defined by the parallel functions running on the GPU, and these parallel functions are marked by a specific syntax, so that when the CPU host runs the CUDA stream, the mark can be identified and the CUDA kernel on the GPU can be scheduled, so that the GPU can execute multiple groups of subtasks in parallel.

[0082] It should be noted that the inference result may include part or all of the GPU output results.

[0083] In this embodiment, the inference result is ultimately determined based on the output results of part or all of the GPUs according to the type of communication operation in the communication task.

[0084] For example, executing All-Reduce corresponding to each GPU will make the output results of All-Reduce corresponding to each GPU the same; then execute the tensor operation (+), the input of the tensor operation (+) is the output result of All-Reduce, at this time, the tensor operation (+) corresponding to each GPU has the same input, and the output of the tensor operation (+) corresponding to each GPU will also be the same; then execute RmsNorm, the input of RmsNorm is the output of the tensor operation (+), then the output of RmsNorm corresponding to each GPU is the same, at this time, the output result on one GPU can be used as the final inference result.

[0085] For example, in Reduce, the data in multiple GPUs are operated according to the specified mapping function, and the output result is saved in a specified GPU, and other GPUs will not get this output result. In this case, the inference result includes the output results of all GPUs.

[0086] In this embodiment, the communication tasks and computing tasks in multiple groups of subtasks are all executed in parallel by the CPU host side according to the CUDA stream scheduling of the same CUDA kernel on the GPU, thereby solving the problem of a large gap between the CUDA kernel used to execute computing tasks and the CUDA kernel used to execute communication tasks on the scheduling GPU in the related technology, and further solving the problem of inference delay caused by the gap, thereby improving inference efficiency.

[0087] It can be seen from the above process that, first, when an inference request is received, the inference task is divided into multiple groups of subtasks corresponding to the graphics processor according to the combination of computing tasks and communication tasks according to the structural information of the target model; wherein each group of subtasks includes a communication task and a computing task; and the input data is divided into input sub-data corresponding to each group of subtasks; then, according to the sequential dependency between the computing tasks and the communication tasks in each group of subtasks, a CUDA stream for the graphics processor is generated, and the CUDA stream is used to implement parallel execution of multiple groups of subtasks; then, the CUDA kernel on the graphics processor is scheduled according to the CUDA stream, so that the graphics processor can execute multiple groups of subtasks in parallel according to the input sub-data and the model parameters deployed on the graphics processor, so as to obtain the inference result corresponding to the inference request. This embodiment can solve the problem of a large gap between the CUDA kernel used to execute computing tasks and the CUDA kernel used to execute communication tasks on the scheduling GPU in the related technology, and then solve the problem of inference delay caused by the gap, thereby improving the inference efficiency.

[0088] First, the above step S203, namely "generating a CUDA stream corresponding to the graphics processor according to the sequential dependency relationship between the computing tasks and the communication tasks in each group of subtasks" is described in detail in conjunction with the embodiment.

[0089] In this embodiment, based on the sequential dependency between the computing tasks and the communication tasks in each group of subtasks and the performance analysis results, the tasks of the multiple groups of subtasks to be executed in parallel in each reasoning round are determined; based on the tasks of the multiple groups of subtasks to be executed in parallel in each reasoning round, a CUDA stream corresponding to the graphics processor is generated.

[0090] The following describes in detail the task of "determining the tasks of executing multiple groups of subtasks in parallel in each reasoning round according to the sequential dependency relationship between the computing tasks and the communication tasks in each group of subtasks and the performance analysis results."

[0091] In this embodiment, the multiple groups of subtasks are divided into multiple reasoning rounds through the calculation tasks and the performance analysis results in each group of subtasks, and the tasks to be executed in parallel in each reasoning round are determined. Among them, the reasoning round can be the number of times the same CUDA kernel on the GPU needs to be scheduled when executing the reasoning task. After all the reasoning rounds are executed, the multiple groups of subtasks can be executed in parallel.

[0092] In this embodiment, when analyzing the performance analysis of the CUDA kernel, it is generally necessary to coordinate the CPU host side and the GPU device side to perform performance analysis. Because performance analysis also needs to be performed on the CPU host side and the GPU device side. Among them, the CPU host side is responsible for scheduling and monitoring, while the GPU device side is responsible for executing tasks. By combining the data of the CPU host side and the GPU device side, the performance of the CUDA kernel can be more comprehensively evaluated.

[0093] In one example, the performance analysis can be performed on each CUDA kernel on the GPU by using the NVIDIA NSYS (Nsight Systems) tool to obtain a performance analysis result.

[0094] Optionally, the performance analysis result may include task execution time; wherein the task execution time may be the time taken to execute the task.

[0095] It should be noted that the CPU host can monitor and measure the performance indicators of the CUDA kernel, such as execution time and memory usage. Optionally, these performance indicators can usually be implemented using functions in the CUDA Runtime API (Application Programming Interface) to record the start and end time of communication tasks and computing tasks, thereby obtaining the task execution time.

[0096] It should be noted that CUDA kernels are executed on GPUs. Therefore, the performance of CUDA kernels will be affected by multiple factors such as GPU architecture, resource utilization, memory, bandwidth, etc. Based on this, when deploying the target model, you can choose a GPU with performance that is more consistent with the target model (for example, larger memory) for model deployment.

[0097] In addition, performance-related data (e.g., cache hit rate, instruction execution efficiency, etc.) is also generated on the GPU device side. These data may be captured and analyzed through specific tools (e.g., NVIDIA's Nsight Compute (Compute Performance Analyzer), Visual Profiler (Visual Profiler), etc.).

[0098] In this embodiment, the performance analysis of the CUDA kernel involves both task scheduling, monitoring and measurement on the CPU host side and computing task execution and performance data collection on the GPU device side. By comprehensively analyzing the data on the CPU host side and the GPU device side, the performance of the CUDA kernel can be analyzed more accurately, and a CUDA kernel or GPU with better performance can be selected to speed up the reasoning process.

[0099] In this embodiment, the NVIDIA NSYS (Nsight Systems) tool can be used to collect performance indicators on the CPU host side or the GPU device side (for example, through hardware counters, etc.), which can provide an intuitive interface to display the performance analysis results on the one hand, and optimize the reasoning process on the other hand to speed up the reasoning process.

[0100] The following describes in detail "generating a CUDA stream corresponding to the graphics processor according to the tasks of executing multiple groups of subtasks in parallel in each inference round".

[0101] In this embodiment, after determining the tasks of multiple groups of subtasks to be executed in parallel in each reasoning round, a task sequence (i.e., CUDA stream) sorted in the order of executing each reasoning round is generated according to the tasks to be executed in parallel in each reasoning round; when executing each reasoning round in sequence on the GPU host side, the same CUDA kernel on the GPU is scheduled until the tasks to be executed in parallel in all reasoning rounds are completed, so as to obtain the reasoning result by final reasoning.

[0102] The following embodiment describes in detail the above step S204, namely "scheduling the CUDA kernel on the graphics processor according to the CUDA stream, so that the graphics processor executes multiple groups of subtasks in parallel according to the model parameters and the input sub-data".

[0103] In this embodiment, the CUDA stream is used to sequentially implement the tasks executed in parallel in each inference round, so as to ultimately realize the parallel execution of multiple groups of subtasks; when the GPU host side executes the CUDA stream, it sequentially schedules the same CUDA kernel on the GPU so that the GPU can execute the tasks executed in parallel in each inference round in parallel according to the model parameters and input sub-data until the parallel execution of multiple groups of subtasks is completed.

[0104] In this embodiment, CUDA streams are used to implement tasks that are executed in parallel in each inference round, and the performance analysis results of the CUDA kernel and the computing tasks are comprehensively considered, so that the accuracy of the inference results can be ensured while also ensuring the inference speed.

[0105] The above step of "determining the tasks of multiple groups of subtasks to be executed in parallel in each reasoning round according to the sequential dependency relationship between the computing tasks and the communication tasks in each group of subtasks and the performance analysis results" is described in detail below in conjunction with the embodiments.

[0106] In this embodiment, for each group of subtasks, according to the computing tasks and the computing tasks, a first computing task whose task execution time matches the task execution time of the communication task in the group of subtasks is first determined from multiple groups of subtasks, the communication task in the group of subtasks is executed in parallel with the first computing task, and a second computing task other than the first computing task is determined from multiple groups of subtasks according to the computing tasks and the computing tasks; then, the first computing task and the second computing task are respectively used as tasks executed in parallel in their corresponding reasoning rounds.

[0107] Here, the communication task in each group of subtasks and the first computing task do not belong to the same group of subtasks, and the task execution time of the first computing task is consistent with the task execution time of the communication task in each group of subtasks. The task execution time refers to the time taken to execute the task. Optionally, the task execution time of the first computing task is less than or equal to the task execution time of the communication task in each group of subtasks.

[0108] It should be noted that when the GPU executes communication tasks and computing tasks, the resources occupied by computing tasks and communication tasks that are not in the same sub-group of tasks (for example, different hardware resources on the GPU) are separated to achieve resource isolation and reduce the possibility of resource competition, so that computing tasks and communication tasks that are not in the same sub-group of tasks can be executed in parallel. In general, the task execution time of executing a communication task is usually longer than the task execution time of a computing task. Therefore, multiple computing tasks and communication tasks that are not in the same sub-group of tasks can be executed in parallel to increase the occupancy rate of the GPU's computing resources, speed up the reasoning process, and improve the efficiency of obtaining reasoning results.

[0109] In this embodiment, as long as the inference result can be accurate while the inference speed can be guaranteed, multiple groups of subtasks can be divided into tasks executed in parallel in several inference rounds according to the requirements for the inference result and / or the inference speed. Therefore, the specific division of multiple groups of subtasks into tasks executed in parallel in several inference rounds is not the only one, and will not be repeated here.

[0110] In one example, taking the first computing task and the second computing task as tasks to be executed in parallel in their corresponding reasoning rounds may include:

[0111] In this embodiment, based on the first computing task that is executed in parallel with the communication task in each group of subtasks, and the second computing task that is not executed in parallel with the communication task in each group of subtasks, multiple groups of subtasks are divided into several parallel execution task combinations, and each parallel task combination represents all tasks executed in parallel each time; the number of reasoning rounds involved in the reasoning process is the number of reasoning rounds required to complete the reasoning process, and the tasks executed in parallel in each reasoning round are the tasks that need to be executed in parallel for each reasoning; then, several parallel execution task combinations are respectively corresponded to the tasks executed in parallel in each reasoning round, so that the tasks that need to be executed in parallel for each reasoning can be determined; next, the tasks executed in parallel in each reasoning round are executed in sequence until the final reasoning result is obtained.

[0112] In this embodiment, multiple groups of subtasks are split into a first computing task that is executed in parallel with the communication task and a second computing task that is not executed in parallel with the communication task based on sequential dependencies and task execution time. This fully takes into account that the task execution time of the communication task is usually longer than the task execution time of the computing task. Therefore, when executing the communication task, more consideration will be given to executing the first computing task in parallel with the communication task, and then the second computing task in parallel. This can shorten the time for executing multiple groups of subtasks as a whole, thereby speeding up the reasoning process and improving the reasoning speed.

[0113] It should be noted that the terms "first" and "second" in this application do not have limitations on order and quantity, but are only used to distinguish between the two computing tasks. For example, "first computing task" and "second computing task" are used to distinguish between the two computing tasks.

[0114] To facilitate the description of "determining the tasks to be executed in parallel in each inference round based on the sequential dependency between the computing tasks and the communication tasks in each group of subtasks and the performance analysis results", four groups of subtasks are taken as an example and explained in detail as follows:

[0115] The four groups of subtasks include a first group of subtasks, a second group of subtasks, a third group of subtasks, and a fourth group of subtasks, each of which includes a matrix multiplication operation, a full reduction operation, a tensor operation, and a reduction operation arranged in a sequence defined by the computing tasks;

[0116] Among them, the first computing task executed in parallel with the communication task in each group of subtasks includes at least one of the following: the matrix multiplication operation in the second group of subtasks executed in parallel with the full reduction operation in the first group of subtasks; the matrix multiplication operation in the third group of subtasks, the tensor operation in the first group of subtasks, and the reduction operation in the first group of subtasks executed in parallel with the full reduction operation in the second group of subtasks; the matrix multiplication operation in the fourth group of subtasks, the tensor operation in the second group of subtasks, and the reduction operation in the second group of subtasks executed in parallel with the full reduction operation in the third group of subtasks; the tensor operation in the third group of subtasks and the reduction operation in the third group of subtasks executed in parallel with the full reduction operation in the fourth group of subtasks;

[0117] The second computing task includes at least one of the following: a matrix multiplication operation in the first group of subtasks; a tensor operation in the fourth group of subtasks and a reduction operation in the fourth group of subtasks;

[0118] For the first computing task and the second computing task, respectively determine the tasks to be executed in parallel in the corresponding reasoning rounds, including:

[0119] The matrix multiplication operation in the first group of subtasks is used as a task to be executed in parallel in the first reasoning round; the matrix multiplication operation in the second group of subtasks executed in parallel with the full reduction operation in the first group of subtasks is used as a task to be executed in parallel in the second reasoning round; the matrix multiplication operation in the third group of subtasks, the tensor operation in the first group of subtasks and the reduction operation in the first group of subtasks executed in parallel with the full reduction operation in the second group of subtasks are used as tasks to be executed in parallel in the third reasoning round; the matrix multiplication operation in the fourth group of subtasks, the tensor operation in the second group of subtasks and the reduction operation in the second group of subtasks executed in parallel with the full reduction operation in the third group of subtasks are used as tasks to be executed in parallel in the fourth reasoning round; the tensor operation in the third group of subtasks executed in parallel with the full reduction operation in the fourth group of subtasks and the reduction operation in the third group of subtasks are used as tasks to be executed in parallel in the fifth reasoning round; the tensor operation in the fourth group of subtasks and the reduction operation in the fourth group of subtasks are used as tasks to be executed in parallel in the sixth reasoning round.

[0120] When performing inference, the CPU host side can perform the following scheduling according to the CUDA stream: scheduling the CUDA kernel on the GPU so that the GPU executes the tasks executed in parallel in the first inference round; then, scheduling the CUDA kernel on the GPU (that is, calling the same CUDA kernel in both the first and second inference rounds) so that the GPU executes the tasks executed in parallel in the second inference round; then, scheduling the CUDA kernel on the GPU so that the GPU executes the tasks executed in parallel in the third inference round; then, scheduling the CUDA kernel on the GPU so that the GPU executes the tasks executed in parallel in the fourth inference round; then, scheduling the CUDA kernel on the GPU so that the GPU executes the tasks executed in parallel in the fifth inference round; then, scheduling the CUDA kernel on the GPU so that the GPU executes the tasks executed in parallel in the sixth inference round.

[0121] exist Figure 8 In the figure, the first subgroup of tasks includes: matrix multiplication-1, All-Reduce-1, tensor operation (+)-1 and RmsNorm-1 as the first group of subtasks; the second subgroup of tasks includes: matrix multiplication-2, All-Reduce-2, tensor operation (+)-2 and RmsNorm-2 as the second group of subtasks; the third subgroup of tasks includes: matrix multiplication-3, All-Reduce-3, tensor operation (+)-3 and RmsNorm-3 as the third group of subtasks; the fourth subgroup of tasks includes: matrix multiplication-4, All-Reduce-4, tensor operation (+)-4 and RmsNorm-4 as the fourth group of subtasks.

[0122] Taking the execution of Matrix Multiplication-1 as an example, after scheduling the CUDA kernel on the GPU, the GPU performs the Matrix Multiplication-1 operation according to the input sub-data corresponding to Matrix Multiplication-1 and the parameters of Matrix Multiplication-1, and obtains the output result of Matrix Multiplication-1. Among them, the execution process of Matrix Multiplication-2, Matrix Multiplication-3 or Matrix Multiplication-4 is the same as the execution process of Matrix Multiplication-1, except that the input sub-data corresponding to Matrix Multiplication-2, Matrix Multiplication-3 or Matrix Multiplication-4 is different from that of Matrix Multiplication-1, which will not be repeated here.

[0123] Taking GPU1 executing All-Reduce-1 as an example, GPU1 executes the All-Reduce-1 operation according to the output result of Matrix Multiplication-1 and the parameters corresponding to All-Reduce-1 to obtain the output result of All-Reduce-1. Next, the output result of All-Reduce-1 on GPU1 and the output result of All-Reduce-1 on GPU2 are summed, and then the summed result is transmitted to All-Reduce-1 on GPU1 and All-Reduce-1 on GPU2 respectively. Among them, the execution process of All-Reduce-2, All-Reduce-3 or All-Reduce-4 on GPU1 is the same as the execution process of All-Reduce-1 on GPU1, which will not be repeated here.

[0124] Taking tensor operation (+)-1 as an example, the GPU executes tensor operation (+)-1 according to the output result of All-Reduce-1 and the parameters corresponding to tensor operation (+)-1 to obtain the output result of tensor operation (+)-1. The execution process of tensor operation (+)-2, tensor operation (+)-3 or tensor operation (+)-4 is the same as the execution process of tensor operation (+)-1, which will not be repeated here.

[0125] Taking RmsNorm-1 as an example, the GPU performs the RmsNorm-1 operation according to the output result of the tensor operation (+)-1 and the parameters corresponding to RmsNorm-1 to obtain the output result of RmsNorm-1. The execution process of RmsNorm-2, RmsNorm-3 or RmsNorm-4 is the same as the execution process of RmsNorm-1, which will not be repeated here.

[0126] It should be noted that executing All-Reduce corresponding to each GPU will make the output results of All-Reduce corresponding to each GPU the same; then execute the tensor operation (+), the input of the tensor operation (+) is the output result of All-Reduce, at this time, the tensor operation (+) corresponding to each GPU has the same input, and the output of the tensor operation (+) corresponding to each GPU will also be the same; then execute RmsNorm, the input of RmsNorm is the output of the tensor operation (+), then the output of RmsNorm corresponding to each GPU is the same, at this time, the output result on one GPU can be used as the final inference result.

[0127] Here, the model parameters in step S201 may include parameters corresponding to matrix multiplication, parameters corresponding to All-Reduce, parameters corresponding to tensor operation (+), and parameters corresponding to RmsNorm. The input sub-data corresponding to the matrix multiplication operation in each group of subtasks is the data obtained by segmenting the input data in step S201, so that each group of subtasks executes multiple groups of subtasks in parallel for the corresponding input sub-data to obtain the final inference result.

[0128] In this embodiment, the CUDA kernel on the GPU is scheduled according to the CUDA stream, so that the GPU sequentially executes the tasks executed in parallel in the first reasoning round, the tasks executed in parallel in the second reasoning round, the tasks executed in parallel in the third reasoning round, the tasks executed in parallel in the fourth reasoning round, the tasks executed in parallel in the fifth reasoning round, and the tasks executed in parallel in the sixth reasoning round, until multiple groups of subtasks are executed and the final reasoning result is obtained.

[0129] In combination with the embodiment below, after executing the "matrix multiplication operation (for example, matrix multiplication-1, matrix multiplication-2, matrix multiplication-3 or matrix multiplication-4)" in the above steps, the reasoning method also includes: for the full reduction operation in each group of subtasks, after executing the matrix multiplication operation in the group of subtasks, waking up the full reduction operation in the group of subtasks.

[0130] exist Figure 8 In the example, after executing matrix multiplication-1, All-Reduce-1 is awakened; after executing matrix multiplication-2, All-Reduce-2 is awakened; after executing matrix multiplication-3, All-Reduce-3 is awakened; after executing matrix multiplication-4, All-Reduce-4 is awakened.

[0131] In this embodiment, the purpose of waking up the communication task in the current group of subtasks only after the matrix multiplication operation in the current group of subtasks is executed can include the following aspects: (1) ensuring data consistency and integrity; (2) optimizing resource utilization; (3) improving task execution efficiency; (4) achieving orderly execution between tasks.

[0132] Ensuring data consistency and integrity means that computing tasks often involve processing and analyzing data, which is the basis for subsequent communication tasks. Only after the computing task is completed and the accuracy and integrity of the data are ensured, can the communication task be effectively communicated based on this data. If the computing task is not completed or the data is not ready, the communication task may not be executed correctly or may result in erroneous results.

[0133] Optimizing resource utilization means that when executing computing tasks, there is competition for GPU hardware resources between computing tasks and communication tasks in the same sub-group of tasks; if the communication tasks in the same sub-group of tasks are awakened at this time, the overall performance of the GPU will be reduced. Waking up the communication tasks in the same sub-group of tasks after the computing tasks are completed can more effectively utilize the GPU hardware resources to execute computing tasks and communication tasks that are not in the same sub-group of tasks in parallel, avoiding unnecessary waste of resources.

[0134] Improving task execution efficiency means: by separating the computing task from the communication task and waking up the communication task only after the computing task is completed, it can ensure that the communication task can immediately obtain the results of the computing task and execute quickly based on these results. This method avoids the communication task from waiting unnecessarily when the computing task is not completed, thereby improving the overall efficiency of task execution.

[0135] To achieve ordered execution between tasks means that during the reasoning process, there is a certain order dependency between the computing task and the communication task. The result of the computing task may be part of the input of the communication task. By ensuring that the computing task is completed first and then waking up the communication task, the ordered execution between tasks can be achieved, ensuring the correctness and efficiency of the entire reasoning process.

[0136] The above step, namely "awakening the full reduction operation in the group of subtasks", is described in detail below in conjunction with an embodiment.

[0137] In this embodiment, the all-reduce operation in the group of subtasks is awakened by a synchronization primitive.

[0138] In this embodiment, the all-reduce operation in the group of subtasks is awakened by a CUDA kernel synchronization primitive with lower overhead.

[0139] Among them, CUDA Kernel Synchronization Primitives refers to the tools and functions used to coordinate and synchronize thread execution in CUDA streams; here, threads can be threads that execute multiple groups of subtasks in parallel. These primitives are used to ensure that threads execute in the expected order during execution (that is, a CUDA stream can be a sequence for executing multiple groups of subtasks in parallel), avoid data competition, and ensure memory consistency.

[0140] In this embodiment, the all-reduce operation in the group of subtasks is awakened by a synchronization primitive to solve the problem in the related art that when waking up the communication task, the message passing between the computing task and the communication task needs to be performed through GPU global memory access, resulting in a large message passing overhead for waking up the communication task, because the computing task and the communication task are distributed in CUDA kernels of different CUDA streams.

[0141] In combination with the embodiment below, before executing the above step S201, the reasoning method also includes: determining the scale information of the graphics processor according to the tensor parallel information of the target model; wherein the scale information includes the number of graphics processors, and the number of graphics processors is multiple; according to the structural information of the target model, splitting the model weight file of the target model into multiple parts, and loading them on multiple graphics processors to obtain model parameters deployed on each graphics processor.

[0142] In this embodiment, the model weight file is a very important component in the field of deep learning. It contains the parameters learned by the model during the training process. These parameters mainly include the weight matrix and bias vector of each layer, which together determine the response of the model to the input data and the accuracy of the output results.

[0143] In this embodiment, before inference, the number of GPUs to be deployed is determined based on the tensor parallel information of the target model; then, the model weight file of the target model is split into multiple copies and loaded on multiple GPUs, and the model parameters are deployed on the multiple GPUs respectively, thereby realizing the deployment of the target model.

[0144] In conjunction with the embodiment below, after executing “determining the scale information of the graphics processor”, the reasoning method further includes:

[0145] When executing the communication tasks in each group of subtasks, data communication between the graphics processors is completed through the communication algorithm corresponding to the communication tasks and / or the topology graph in the group of subtasks.

[0146] In this embodiment, the scale information may include a topology diagram between multiple GPUs, and the topology diagram may intuitively display the physical connection between the multiple GPUs, and the physical connection may include connecting multiple GPUs through NVLink, PCIe or IB (Infini Band) channels.

[0147] In this embodiment, a corresponding communication algorithm may be determined according to the communication task and / or the topology graph, and data communication between the GPUs may be completed through the communication algorithm.

[0148] In one example, the communication modes corresponding to the all-reduce operation and the broadcast operation are different.

[0149] In this embodiment, data communication between GPUs is completed by determining a corresponding communication algorithm through communication tasks and / or topology graphs.

[0150] The “communication algorithm” is described in detail below in conjunction with the embodiments.

[0151] In this embodiment, any one of the following communication algorithms may be used: a one-round communication algorithm, a two-round communication algorithm, and a ring communication algorithm.

[0152] In one example, taking All-Reduce as an example, a one-shot communication algorithm refers to performing one round of communication between GPUs to aggregate data of all GPUs on each GPU.

[0153] In one example, in Fig. 9 In the example, GPU1 includes M1, M2 and M3, GPU2 includes P1, P2 and P3, and GPU3 includes Q1, Q2 and Q3. Taking All-Reduce as an example, the two-shot communication algorithm refers to two rounds of communication between each GPU (i.e., GPU1, GPU2 or GPU3), aggregating all the data of GPU1, GPU2 and GPU3 on each GPU, that is, finally GPU1, GPU2 or GPU3 includes M1+P1+Q1, M2+P2+Q2 and M3+P3+Q3.

[0154] In one example, taking All-Reduce as an example, a Ring-based communication algorithm requires 2*(the number of GPUs - 1) rounds of communication.

[0155] exist Fig.10 In the figure, GPU1 includes M1, M2 and M3, GPU2 includes P2, P2 and P3, GPU3 includes Q1, Q2 and Q3, and four rounds of communication are required between GPU1, GPU2 and GPU3 to aggregate all the data of GPU1, GPU2 and GPU3 on each GPU, that is, GPU1, GPU2 or GPU3 finally includes M1+P1+Q1, M2+P2+Q2 and M3+P3+Q3.

[0156] In this embodiment, a corresponding communication algorithm may be selected according to the demand for communication delay. For example, the communication delay may be reduced by reducing the number of communication rounds. The two-shot communication delay is smaller than the Ring-based communication delay.

[0157] The following describes in detail the "receiving an inference request for a target model" in the above step S201 in conjunction with an embodiment.

[0158] In this embodiment, an inference request for a target model is received using the OpenAI (Open AI Research Center, USA) API format.

[0159] In this embodiment, the API provided by OpenAI enables developers to access and use the models developed by OpenAI. The API provides a variety of functions, including natural language processing, text generation, image generation, etc., and supports a variety of application scenarios, such as chatbots, text analysis, automatic writing, etc. Developers can integrate OpenAI's models into their applications, services or products through the API to achieve smarter functions and user experience.

[0160] Fig.11 A flowchart of the reasoning method provided in the embodiment of the present application, in which the scheme is applied to Figure 1 The CPU host in the example is used to illustrate. Fig.11 As shown in , the method may include the following steps:

[0161] Step S1101: Determine the scale information of the graphics processor according to the tensor parallel information of the target model; wherein the scale information includes the number of graphics processors, and the number of graphics processors is multiple.

[0162] Step S1102: splitting the model weight file of the target model into multiple parts according to the structural information of the target model, and loading them on multiple graphics processors to obtain model parameters deployed on each graphics processor.

[0163] Step S1103: in response to receiving an inference request for a target model, obtaining model parameters of the target model deployed on each graphics processor.

[0164] Step S1104: Generate an inference task for the inference request; wherein the inference request includes information about the inference task.

[0165] In one example, the reasoning task may be a combination of operators, such as a communication operator and a computation operator.

[0166] Step S1105: Identify the types of communication operators and computation operators in the reasoning task.

[0167] In this embodiment, the types of communication operators and computing operators are identified so as to facilitate segmentation of the operators according to the types in the subsequent step 1106 .

[0168] Step S1106: Based on the structural information of the target model, the inference task is divided into a plurality of groups of subtasks corresponding to the graphics processor in a combination of computing tasks and communication tasks; each group of subtasks includes a communication task and a computing task; and the input data is divided into input sub-data corresponding to each group of subtasks.

[0169] Step S1107: Generate a CUDA stream corresponding to the graphics processor according to the sequential dependency between the computing tasks and the communication tasks in each group of subtasks; wherein the CUDA stream is used to implement parallel execution of multiple groups of subtasks; and schedule the CUDA kernel on the graphics processor according to the CUDA stream to obtain an inference result corresponding to the inference request.

[0170] Step S1108: Output the inference result.

[0171] Combination of the above Figure 2 , describes the method embodiment of the present application in detail, and the following is combined with Fig.12 , describes the device embodiment of the present application in detail. It should be understood that the description of the method embodiment corresponds to the description of the device embodiment, so the parts not described in detail can refer to the previous method embodiment.

[0172] Fig.12 FIG. 1 is a schematic diagram of the structure of the reasoning method provided by an embodiment of the present application. Fig.12 As shown, the reasoning method provided in the embodiment of the present application includes an acquisition module 1201, a segmentation module 1202, a generation module 1203 and a reasoning module 1204. Among them, the acquisition module 1201 is used to obtain the model parameters of the target model deployed on each graphics processor in response to receiving an inference request for the target model, wherein the inference request includes information about the inference task, and the information about the inference task includes input data; the segmentation module 1202 is used to segment the inference task into multiple groups of subtasks corresponding to the graphics processor in a combination of computing tasks and communication tasks according to the structural information of the target model; wherein each group of subtasks includes a communication task and a computing task; and, the input data is segmented into input sub-data corresponding to each group of subtasks; the generation module 1203 is used to generate a unified computing device architecture CUDA stream corresponding to the graphics processor according to the sequential dependency between the computing tasks and the communication tasks in each group of subtasks; wherein the CUDA stream is used to realize parallel execution of multiple groups of subtasks; the inference module 1204 is used to schedule the CUDA kernel on the graphics processor according to the CUDA stream, so that the graphics processor can execute multiple groups of subtasks in parallel according to the model parameters and the input sub-data, and obtain the inference result corresponding to the inference request.

[0173] In one embodiment, the generation module 1203 includes: a task determination unit (not shown in the figure), which is used to determine the tasks of the multiple groups of subtasks to be executed in parallel in each inference round according to the sequential dependency relationship between the computing tasks and the communication tasks in each group of subtasks and the performance analysis results; a CUDA stream generation unit (not shown in the figure), which is used to generate a CUDA stream corresponding to the graphics processor according to the tasks of the multiple groups of subtasks to be executed in parallel in each inference round;

[0174] The inference module 1204 is specifically used to: when executing each inference round in sequence, schedule the CUDA kernel on the graphics processor according to the CUDA stream, so that the graphics processor can perform tasks in parallel in the inference round according to the model parameters and the input sub-data.

[0175] In one embodiment, the task determination unit is specifically used to: for each group of subtasks, determine a first computing task whose task execution time matches the task execution time of the communication task in the group of subtasks from multiple groups of subtasks according to the computing tasks and the computing tasks, the communication tasks in the group of subtasks are executed in parallel with the first computing task, and determine a second computing task other than the first computing task from multiple groups of subtasks; wherein the first computing task and the communication task are not in the same group of subtasks; and respectively use the first computing task and the second computing task as tasks executed in parallel in their corresponding reasoning rounds.

[0176] In one embodiment, the plurality of groups of subtasks include a first group of subtasks, a second group of subtasks, a third group of subtasks, and a fourth group of subtasks, each group of subtasks including a matrix multiplication operation, a full reduction operation, a tensor operation, and a reduction operation arranged in a sequential dependency relationship;

[0177] Among them, the first computing task executed in parallel with the communication task in each group of subtasks includes at least one of the following: the matrix multiplication operation in the second group of subtasks executed in parallel with the full reduction operation in the first group of subtasks; the matrix multiplication operation in the third group of subtasks, the tensor operation in the first group of subtasks, and the reduction operation in the first group of subtasks executed in parallel with the full reduction operation in the second group of subtasks; the matrix multiplication operation in the fourth group of subtasks, the tensor operation in the second group of subtasks, and the reduction operation in the second group of subtasks executed in parallel with the full reduction operation in the third group of subtasks; the tensor operation in the third group of subtasks and the reduction operation in the third group of subtasks executed in parallel with the full reduction operation in the fourth group of subtasks;

[0178] The second computing task includes at least one of the following: a matrix multiplication operation in the first group of subtasks; a tensor operation in the fourth group of subtasks and a reduction operation in the fourth group of subtasks.

[0179] In one embodiment, the device further includes: a wake-up module (not shown in the figure) for waking up the full reduction operation in each group of subtasks after executing the matrix multiplication operation in the group of subtasks.

[0180] In one embodiment, the wake-up module is specifically configured to: wake up the full-reduce operation in the group of subtasks through a synchronization primitive.

[0181] In one embodiment, the device also includes: a determination module (not shown in the figure), which is used to determine the scale information of the graphics processor based on the tensor parallel information of the target model; wherein the scale information includes the number of graphics processors, and the number of graphics processors is multiple; a deployment module (not shown in the figure), which is used to split the model weight file of the target model into multiple parts according to the structural information of the target model, and load them on multiple graphics processors to obtain model parameters deployed on each graphics processor.

[0182] In one embodiment, the scale information also includes: a topological map between multiple graphics processors; the device also includes: a communication module (not shown in the figure), which is used to complete data communication between each graphics processor when executing the communication task in each group of subtasks through the communication algorithm corresponding to the communication task and / or the topological map in the group of subtasks.

[0183] In one embodiment, the communication algorithm includes at least one of the following: a one-round communication method, a two-round communication method, and a ring communication algorithm.

[0184] In one embodiment, the acquisition module 1201 is specifically used to: receive an inference request for a target model using the Open AI Research Center application programming interface format.

[0185] Below, reference Fig.13 To describe the CPU host side according to an embodiment of the present application. Fig.13 Shown is a schematic diagram of the structure of a CPU host side provided in one embodiment of the present application.

[0186] like Fig.13 As shown, the CPU host side includes one or more processors 1301 and a memory 1302 .

[0187] The processor 1301 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the CPU host side to perform desired functions.

[0188] The memory 1302 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on a computer-readable storage medium, and the processor 1301 may run the program instructions to implement the functions such as the reasoning method of each embodiment of the present application mentioned above. Various contents such as model parameters and input data may also be stored in the computer-readable storage medium.

[0189] In one example, the CPU host side may further include: an input device 1303 and an output device 1304, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0190] The input device 1303 may include, for example, a keyboard, a mouse, etc.

[0191] The output device 1304 can output various information to the outside, including inference results, etc. The output device 1304 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0192] Of course, to simplify, Fig.13 Only some of the components in the CPU host side that are related to the present application are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, the CPU host side may also include any other appropriate components according to specific application conditions.

[0193] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the reasoning method according to various embodiments of the present application described above in this specification.

[0194] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on the user CPU host, partially on the user device, as an independent software package, partially on the user CPU host and partially on the remote CPU host, or entirely on the remote CPU host or server.

[0195] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the reasoning method according to various embodiments of the present application described above in this specification.

[0196] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0197] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.

[0198] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0199] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0200] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0201] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

[0202] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method of reasoning, wherein: The method comprises: In response to receiving an inference request for a target model, obtaining model parameters of the target model deployed on each graphics processor, wherein the inference request includes information of an inference task, and the information of the inference task includes input data; According to the structural information of the target model, the inference task is divided into a plurality of groups of subtasks corresponding to the graphics processor in a combination of computing tasks and communication tasks; wherein each group of subtasks includes a communication task and a computing task; and the input data is divided into input sub-data corresponding to each group of subtasks; Generate a unified computing device architecture CUDA stream corresponding to the graphics processor according to the sequential dependency between the computing tasks and the communication tasks in each group of subtasks; wherein the CUDA stream is used to implement parallel execution of multiple groups of subtasks; The CUDA kernel on the graphics processor is scheduled according to the CUDA stream, so that the graphics processor executes multiple groups of subtasks in parallel according to the model parameters and the input sub-data to obtain an inference result corresponding to the inference request.

2. The method according to claim 1, wherein: The step of generating a CUDA stream corresponding to the graphics processor according to the sequential dependency between the computing tasks and the communication tasks in each group of subtasks includes: According to the sequential dependency relationship between the computing tasks and the communication tasks in each group of subtasks and the performance analysis results, determine the tasks of the multiple groups of subtasks to be executed in parallel in each reasoning round; Generate a CUDA stream corresponding to the graphics processor according to the tasks of executing the multiple groups of subtasks in parallel in each inference round; The step of scheduling the CUDA kernel on the graphics processor according to the CUDA stream so that the graphics processor executes multiple groups of subtasks in parallel according to the model parameters and the input sub-data includes: When each inference round is executed in sequence, the CUDA kernel on the graphics processor is scheduled according to the CUDA stream to provide the graphics processor with tasks to be executed in parallel in the inference round according to the model parameters and the input sub-data.

3. The method according to claim 2, wherein: The step of determining the tasks of executing the multiple groups of subtasks in parallel in each reasoning round according to the sequential dependency relationship between the computing tasks and the communication tasks in each group of subtasks and the performance analysis results includes: For each group of subtasks, according to the computing tasks and the computing tasks, determine from the multiple groups of subtasks a first computing task whose task execution time matches the task execution time of the communication task in the group of subtasks, the communication task in the group of subtasks is executed in parallel with the first computing task, and determine from the multiple groups of subtasks a second computing task other than the first computing task; wherein the first computing task and the communication task are not in the same group of subtasks; The first computing task and the second computing task are respectively used as tasks to be executed in parallel in their corresponding reasoning rounds.

4. The method according to claim 3, wherein: The multiple groups of subtasks include a first group of subtasks, a second group of subtasks, a third group of subtasks, and a fourth group of subtasks, each group of subtasks includes a matrix multiplication operation, a full reduction operation, a tensor operation, and a reduction operation arranged according to the sequential dependency relationship; Among them, the first computing task executed in parallel with the communication task in each group of subtasks includes at least one of the following: the matrix multiplication operation in the second group of subtasks executed in parallel with the full reduction operation in the first group of subtasks; the matrix multiplication operation in the third group of subtasks, the tensor operation in the first group of subtasks, and the reduction operation in the first group of subtasks executed in parallel with the full reduction operation in the second group of subtasks; the matrix multiplication operation in the fourth group of subtasks, the tensor operation in the second group of subtasks, and the reduction operation in the second group of subtasks executed in parallel with the full reduction operation in the third group of subtasks; the tensor operation in the third group of subtasks and the reduction operation in the third group of subtasks executed in parallel with the full reduction operation in the fourth group of subtasks; The second computing task includes at least one of the following: a matrix multiplication operation in the first group of subtasks; a tensor operation in the fourth group of subtasks and a reduction operation in the fourth group of subtasks.

5. The method according to claim 4, wherein: The method further includes: for the full reduction operation in each group of subtasks, after the matrix multiplication operation in the group of subtasks is executed, waking up the full reduction operation in the group of subtasks.

6. The method according to claim 5, wherein: The waking up the all-reduce operation in the group of subtasks includes: waking up the all-reduce operation in the group of subtasks through a synchronization primitive.

7. The method according to any one of claims 1 to 6, wherein: The method further comprises: Determining scale information of a graphics processor according to tensor parallel information of a target model; wherein the scale information includes the number of graphics processors, and the number of the graphics processors is multiple; According to the structural information of the target model, the model weight file of the target model is split into multiple parts and loaded on multiple graphics processors to obtain model parameters deployed on each graphics processor.

8. The method according to claim 7, wherein: The scale information also includes: a topology diagram between multiple graphics processors; the method also includes: When executing the communication tasks in each group of subtasks, data communication between the graphics processors is completed through the communication algorithm corresponding to the communication tasks in the group of subtasks and / or the topology graph.

9. An inference device, wherein: The device comprises: an acquisition module, configured to acquire model parameters of the target model deployed on each graphics processor in response to receiving an inference request for the target model, wherein the inference request includes information of the inference task, and the information of the inference task includes input data; A segmentation module, configured to segment the inference task into a plurality of groups of subtasks corresponding to the graphics processor in a combination of computing tasks and communication tasks according to the structural information of the target model; wherein each group of subtasks includes a communication task and a computing task; and segment the input data into input sub-data corresponding to each group of subtasks; A generation module, used for generating a unified computing device architecture CUDA stream corresponding to the graphics processor according to the sequential dependency between the computing tasks and the communication tasks in each group of subtasks; wherein the CUDA stream is used to implement parallel execution of multiple groups of subtasks; The inference module is used to schedule the CUDA kernel on the graphics processor according to the CUDA stream, so that the graphics processor executes multiple groups of subtasks in parallel according to the model parameters and the input sub-data to obtain an inference result corresponding to the inference request.

10. A computing device, characterized in that: including memory and processor; The memory is connected to the processor and is used to store programs; The processor is configured to implement the method according to any one of claims 1 to 8 by running the program in the memory.

11. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1 to 8 is implemented.

12. A computer program product, characterized in that The method comprises computer program instructions, which, when executed by a processor, cause the processor to implement the method according to any one of claims 1 to 8.

Images