Reasoning methods, devices, equipment and media

Abstract

The present application provides an inference method, apparatus, device and medium. The method includes: upon receiving an inference request, obtaining the model parameters of the target model deployed on each GPU, the inference request including the information of the inference task, and the information of the inference task including the input data; then, according to the structural information of the target model, dividing the inference task into multiple groups of subtasks corresponding to the GPU in a combination of computing tasks and communication tasks; and dividing the input data into input sub-data corresponding to each group of subtasks; then, according to the sequential dependency between the computing tasks and the communication tasks in each group of subtasks, generating a CUDA stream corresponding to the GPU, the CUDA stream is used to realize the parallel execution of multiple groups of subtasks; then, scheduling the CUDA kernel on the GPU according to the CUDA stream, so that the GPU can execute multiple groups of subtasks in parallel based on the model parameters and the input sub-data, and obtain the inference result corresponding to the inference request. The present application can reduce the inference latency.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and specifically to a reasoning method, apparatus, device, and medium. Background Art

[0002] With the rise of LLMs (Large Language Models), exemplified by the Transformer architecture, the scale of deep learning models continues to expand, reaching hundreds of billions and even trillions of parameters, driving rapid progress in the field of artificial intelligence. However, this has also brought with it corresponding challenges: the traditional single-machine, single-GPU model can no longer meet the requirements for reasoning on large models.

[0003] At present, the relevant technology has designed a method of submitting computing tasks to the CUDA core of a CUDA (Compute Unified Device Architecture) stream allocated on a single GPU (Graphic Processing Unit), and submitting communication tasks to the CUDA core of another CUDA stream. However, there is the following problem: submitting the communication task and computing task on a single GPU to the two CUDA cores respectively causes a large gap when the CPU schedules tasks on the two different CUDA cores, resulting in a large inference delay, which has become a problem that needs to be solved urgently. Summary of the Invention

[0004] In view of this, embodiments of the present application are dedicated to providing an inference method, apparatus, device and medium to reduce inference latency.

[0005] In a first aspect, an inference method is provided, the method comprising: in response to receiving an inference request for a target model, obtaining model parameters of the target model deployed on each graphics processor, wherein the inference request includes information about the inference task, and the information about the inference task includes input data; based on structural information of the target model, dividing the inference task into multiple groups of subtasks corresponding to the graphics processor in a combination of computing tasks and communication tasks; wherein each group of subtasks includes a communication task and a computing task; and dividing the input data into input sub-data corresponding to each group of subtasks; based on a sequential dependency relationship between the computing tasks and the communication tasks in each group of subtasks, generating a CUDA stream corresponding to the graphics processor; wherein the CUDA stream is used to implement parallel execution of multiple groups of subtasks; and scheduling a CUDA kernel on the graphics processor according to the CUDA stream so that the graphics processor executes the multiple groups of subtasks in parallel based on the model parameters and the input sub-data, thereby obtaining an inference result corresponding to the inference request.

[0006] In a second aspect, an inference device is provided, which includes: an acquisition module for acquiring model parameters of the target model deployed on each graphics processor in response to receiving an inference request for the target model, wherein the inference request includes information about the inference task, and the information about the inference task includes input data; a segmentation module for segmenting the inference task into multiple groups of subtasks corresponding to the graphics processor in a combination of computing tasks and communication tasks according to structural information of the target model; wherein each group of subtasks includes a communication task and a computing task; and, segmenting the input data into input sub-data corresponding to each group of subtasks; a generation module for generating a CUDA stream corresponding to the graphics processor according to a sequential dependency relationship between the computing tasks and the communication tasks in each group of subtasks; wherein the CUDA stream is used to implement parallel execution of multiple groups of subtasks; an inference module for scheduling the CUDA kernel on the graphics processor according to the CUDA stream, so that the graphics processor executes multiple groups of subtasks in parallel based on the model parameters and the input sub-data, and obtains an inference result corresponding to the inference request.

[0007] According to a third aspect, an electronic device is provided, comprising: one or more processors; and a memory associated with the one or more processors, the memory being used to store program instructions, which, when read and executed by the one or more processors, execute the method described in the first aspect above.

[0008] According to a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method described in the first aspect is implemented.

[0009] According to a fifth aspect, a program product is provided, comprising computer program instructions, which, when executed by a processor, enable the processor to implement the method described in the first aspect.

[0010] In this embodiment, upon receiving an inference request, the inference task is first divided into multiple groups of subtasks corresponding to the graphics processor based on the structural information of the target model, in a combination of computing tasks and communication tasks; each group of subtasks includes a computing task and a communication task; input data is divided into input sub-data corresponding to each group of subtasks; and the input data is divided to obtain input sub-data corresponding to each group of subtasks, each group of subtasks including a computing task and a communication task; then, based on the sequential dependencies between the computing tasks and the communication tasks in each group of subtasks, a CUDA stream is generated for the graphics processor, the CUDA stream being used to implement parallel execution of the multiple groups of subtasks; and then, based on the CUDA stream, a CUDA kernel on the graphics processor is scheduled so that the graphics processor executes the multiple groups of subtasks in parallel based on the input sub-data and model parameters deployed on the graphics processor to obtain inference results corresponding to the inference request. This embodiment can solve the problem of a large gap between the scheduling of CUDA kernels for executing computing tasks and CUDA kernels for executing communication tasks on the GPU in related technologies, thereby solving the problem of inference latency caused by the gap and improving inference efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a schematic diagram of the architecture of the application scenario of the reasoning method provided in the embodiment of the present application;

[0012] Figure 2 is a flow chart of the reasoning method provided in an embodiment of the present application;

[0013] Figure 3 It is a category table of communication tasks and computing tasks;

[0014] Figure 4 This is a schematic diagram of dividing computing tasks and communication tasks;

[0015] Figure 5 It is a schematic diagram of the split full connection operation;

[0016] Figure 6 It is a schematic diagram of the split-reduce operation;

[0017] Figure 7 This is a schematic diagram of the split tensor operation (+);

[0018] Figure 8 It is a schematic diagram of the tasks executed in parallel in each reasoning round;

[0019] Figure 9 It is a schematic diagram of the two-round communication algorithm;

[0020] Figure 10 It is a schematic diagram of the ring communication algorithm;

[0021] Figure 11 is a flow chart of the reasoning method provided in an embodiment of the present application;

[0022] Figure 12 is a schematic diagram of an inference device provided in an embodiment of the present application;

[0023] Figure 13 This is a schematic diagram of the structure of the CPU host side provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0025] In this embodiment, the inference method may include: the CPU host side generates a CUDA stream corresponding to the GPU device side; the CPU host side schedules the CUDA kernels on the GPU device side for parallel computing according to the CUDA stream to obtain inference results, which can solve the problem of large gaps in related technologies in scheduling different CUDA kernels on the GPU device side to respectively execute computing tasks and communication tasks.

[0026] For ease of understanding, this article first introduces the application scenarios to which the reasoning method provided by this application is applicable. Figure 1 The CPU host 110 is connected to the GPU device 120, and the GPUs in the GPU device 120 communicate with each other. For example, PCIe (Peripheral Component Interconnect Express) is used for communication between the CPU host 110 and the GPU device 120; NVLink is used for communication between the GPUs in the GPU device 120; NVLink is a bus communication protocol developed and launched by NVIDIA. The GPU device 120 is used to execute tasks in parallel; the CPU host 110 is used to receive inference requests for the target model and, based on the model parameters of the target model deployed on each graphics processor, the CPU host 110 performs the following operations:

[0027] (1) in response to receiving an inference request for a target model, obtaining model parameters of the target model deployed on each graphics processor, wherein the inference request includes information about an inference task, and the information about the inference task includes input data;

[0028] (2) based on the structural information of the target model, dividing the inference task into a plurality of subtasks corresponding to the graphics processor in a combination of computing tasks and communication tasks; wherein each subtask group includes a communication task and a computing task; and dividing the input data into input sub-data corresponding to each subtask group;

[0029] (3) generating a CUDA stream corresponding to the graphics processor based on the sequential dependency between the computing tasks and the communication tasks in each group of subtasks; wherein the CUDA stream is used to implement parallel execution of multiple groups of subtasks;

[0030] (4) Scheduling the CUDA kernel on the graphics processor according to the CUDA stream so that the graphics processor can execute multiple groups of subtasks in parallel according to the model parameters and the input sub-data to obtain an inference result corresponding to the inference request.

[0031] The CPU host end 110 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The GPU device end 120 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices.

[0032] Figure 2 A flowchart of the reasoning method provided in the embodiment of the present application, in which the scheme is applied to Figure 1 The CPU host side in the example is used to illustrate. Figure 2 As shown in , the method may include the following steps:

[0033] Step S201: in response to receiving an inference request for a target model, obtaining model parameters of the target model deployed on each graphics processor, wherein the inference request includes information of an inference task, and the information of the inference task includes input data.

[0034] In this embodiment, on the CPU host side (eg Figure 1 When the CPU host side 110 shown in FIG. 1 receives an inference request for the target model, the GPU device side (eg Figure 1 The GPU device 120 shown in FIG. 1 determines the GPU for deploying the target model and obtains the model parameters of the target model deployed on each GPU. The inference request is used to transmit inference task information to the CPU host. The inference task information includes input data, which is the input data for inference.

[0035] Setting model parameters is crucial to ensuring the model can correctly and efficiently process input data and produce accurate output. These parameters typically involve multiple aspects of the model, including input, processing, and output. Prioritizing the target model, the model parameters primarily include the weight matrices and bias vectors of each layer in the target model.

[0036] In one example, after receiving the reasoning request, the reasoning method further includes: generating a reasoning task according to the reasoning request.

[0037] In this embodiment, the inference request may be a combination of operators, for example, a communication operator and a fully connected operator, an activation operator, a reduction operator, and a tensor operator.

[0038] It should be noted that an operator is a special type of operation that maps or transforms functions or data spaces. For example, the fully connected operator, activation operator, reduction operator, and tensor operator are all operators that perform specific operations on functions.

[0039] Here, the tensor operations (i.e., tensor operators) included in the computational subtasks do not have the concept of layers, because tensor operations (+, -, *, ÷, Pow, Sqrt) themselves do not constitute an independent layer structure. However, tensor operations are widely used in different layers of the target model and can influence or constitute the computation process of certain layers. Therefore, the parameters of tensor operations can be determined based on the weight matrices and bias vectors corresponding to certain layers.

[0040] For example, the convolution layer extracts features by performing multiplication and accumulation operations on the convolution kernel and the local region of the input tensor, which can be seen as a special tensor multiplication operation. The pooling layer reduces the spatial dimension of the data by downsampling the local region of the input tensor (such as maximum pooling, average pooling, etc.), which can also be seen as a special tensor operation.

[0041] In one example, the parameters of the target model deployed on each graphics processor may include: determining the scale information of the graphics processor based on the tensor parallel information of the target model, the scale information including the number of graphics processors, and the number of graphics processors is multiple; then, based on the structural information of the target model, splitting the model weight file of the target model into multiple parts, and loading them on multiple graphics processors to obtain the model parameters deployed on each graphics processor.

[0042] Here, the inference request may further include a model name, which may be the name of a target model, and the structural information of the target model is obtained according to the model name.

[0043] Tensor Parallelism (TP) is a type of model parallelism, a technology that accelerates the inference process. It involves splitting model tensors (such as weight matrices) along specific dimensions and distributing these split tensors to different GPUs for parallel computing, accelerating the inference process.

[0044] Here, the tensor parallel information may include a tensor parallel mode and / or a quantity of tensor parallelism, wherein the tensor parallel mode includes row parallelism, column parallelism, one-dimensional tensor parallelism, and multi-dimensional tensor parallelism.

[0045] In one example, obtaining model parameters of a target model deployed on each graphics processor may include: obtaining model parameters of the target model deployed on each GPU via PCIe (Peripheral Component Interconnect Express) or NVLink (connection).

[0046] In one example, an inference request can be an instruction issued by the application layer for performing inference on a target model. The application layer can process the client's inference request and receive a response. It should be noted that the client can be a separate physical device that is connected to the GPU host via a wired or wireless connection. Alternatively, the client can be a separate application or software on the GPU host.

[0047] Step S202: Based on the structural information of the target model, the inference task is divided into multiple groups of subtasks corresponding to the graphics processor in a combination of computing tasks and communication tasks; each group of subtasks includes a communication task and a computing task; and the input data is divided into input sub-data corresponding to each group of subtasks.

[0048] In this embodiment, upon receiving an inference request for a target model in step S201, the inference task is divided into multiple groups of subtasks corresponding to the GPU based on the combination of computing tasks and communication tasks according to the structural information of the target model, and the input data is divided to obtain input sub-data corresponding to each group of subtasks. The input sub-data may be the input data of the first task in each group of subtasks.

[0049] Here, the combination of compute and communication tasks is used to split the inference task into subtasks, each of which includes both compute and communication tasks. Each subtask group includes both compute and communication tasks. Computational tasks improve the efficiency of tensor parallel computing, while communication tasks transfer data between subtasks to ensure they can work together and complete the entire inference task.

[0050] In this embodiment, the structural information of the target model can be used to reflect the model architecture of the target model, such as the components of each layer of the target model. Sequential dependencies exist between the layers in the target model, and thus the sequential dependencies between operators in the inference task can be determined based on the structural information of the target model. For example, the fully connected operator (i.e., the fully connected operation) corresponding to the fully connected layer.

[0051] It should be noted that tensor operations do not correspond to the above layers, because tensor operations (+, -, *, ÷, Pow, Sqrt) themselves do not constitute an independent layer structure. However, tensor operations are widely used in different parts and layers of the target model and can affect or constitute the calculation process of certain layers.

[0052] For example, the convolution layer extracts features by performing multiplication and accumulation operations on the convolution kernel and the local region of the input tensor, which can be seen as a special tensor multiplication operation. The pooling layer reduces the spatial dimension of the data by downsampling the local region of the input tensor (such as maximum pooling, average pooling, etc.), which can also be seen as a special tensor operation.

[0053] The following describes in detail step S202 , “divide the inference task into a plurality of subtasks corresponding to the graphics processor in a combination of computing tasks and communication tasks according to the structural information of the target model”, in conjunction with an embodiment.

[0054] In an embodiment of the present application, the sequential dependency between operators in an inference task is determined based on the structural information of a target model; based on the sequential dependency between operators in the inference task, the number of graphics processors, and the type of operators in the inference task, the inference task is divided into multiple groups of subtasks corresponding to the graphics processor in a combination of computing tasks and communication tasks.

[0055] In this embodiment, the inference task is divided according to the sequence defined by the sequential dependency between the operators in the inference task, and is divided into corresponding dimensions according to the number of image processors and the type of operators to obtain multiple groups of subtasks corresponding to each GPU.

[0056] exist Figure 3In the process, the tasks are split in the order of full-connection operation, full-reduce operation, tensor operation, and reduction operation (i.e., the order defined by the sequential dependency); when splitting the full-connection operation, the full-connection operation is split into corresponding dimensions according to the number of GPUs and the type of the full-connection operation; then, when splitting the full-reduce operation, the full-reduce operation is split into corresponding dimensions according to the splitting dimension of the full-connection operation and the type of the full-reduce operation; when splitting the tensor operation, the tensor operation is split into corresponding dimensions according to the splitting dimension of the full-reduce operation and the type of the tensor operation; when splitting the reduction operation, the reduction operation is split into corresponding dimensions according to the splitting dimension of the tensor operation and the type of the reduction operation, and finally multiple groups of subtasks are obtained.

[0057] In one example, after executing step S201, the type of each operator in the inference task can be identified so that when splitting the inference task, the task can be split according to the splitting dimensions corresponding to the type of each operator and the number of GPUs. For example, the matrix multiplication operator corresponds to a 2*2 splitting dimension.

[0058] Correspondingly, in this example, dividing the inference task actually means dividing each operator.

[0059] It should be noted that the operator type determines whether the task is split horizontally or vertically; the number of GPUs determines the number of parts the task is split into.

[0060] In one example, NCCL (NVIDIA Collective Communication Library) is a library for communication between GPUs, including aggregate communication and point-to-point communication, where aggregate communication includes all-gather, all-reduce, broadcast, reduce, and reduce-scatter.

[0061] In this embodiment, NCCL is used to support efficient data exchange and parallel computing between GPUs to speed up the inference process.

[0062] exist Figure 3In [1], a communication task can include at least one of the following operators: All-gather, All-Reduce, Broadcast, Reduce, and Reduce-scatter. All-Reduce aggregates data from multiple GPUs and distributes it across all GPUs so that each GPU generates the same aggregated result. All-gather gathers data from all GPUs so that each GPU has access to data from all GPUs.

[0063] exist Figure 3 In the , computing task includes at least one of the following operators: full connection (Matrix-Multiplication) operation, activation operation, reduction operation, tensor operation.

[0064] Fully connected operations are a common layer operation in neural networks. In a fully connected layer, all input neurons have an independent weighted connection with all output neurons. Preferably, fully connected operations in large model inference can be implemented as matrix multiplications or inner product operations.

[0065] Activation is a nonlinear function applied to the output of each neuron in a neural network. The purpose of activation is to introduce nonlinearity, enabling the neural network to learn and represent complex functional relationships. Preferred activations for large model inference can include SiLU (Sigmoid Linear Unit), ReLU (Rectified Linear Unit), and GeLU (Gaussian Linear Unit).

[0066] Reduction operations aggregate the elements along a dimension of a tensor. They are typically used to transform multidimensional data into a lower dimension or scalar. Reduction operations can include LayerNorm (LayerNormalization), RmsNorm (Root Mean Square Normalization), and Softmax (normalized exponential function). Preferably, a tensor can be a multidimensional data structure.

[0067] Tensor operations refer to various mathematical operations performed on multidimensional array structures. Tensors can be seen as a generalization of vectors and matrices. In one example, tensor operations cover a variety of mathematical operations, including (+, -, *, ÷), square operations (Pow), square root operations (Sqrt), and so on.

[0068] In an example, the fully connected operation is taken as a matrix multiplication operation, the full reduction operation is taken as All-Reduce, the tensor operation is taken as a tensor operation (+), and the reduction operation is taken as RmsNorm, and the process of dividing the reasoning task into four groups of subtasks is described in detail.

[0069] exist Figure 4 In the example, the model parameters of the target model are deployed on GPU1 and GPU2 respectively. The partitioning principle is to ensure that there is no sequential dependency between the computing tasks and the communication tasks among the four groups of subtasks corresponding to GPU1. The same applies to GPU2 and GPU1. The matrix multiplication operation, All-Reduce, tensor operation (+), and RmsNorm are partitioned in the order defined by the sequential dependency between the operators in the inference task to obtain the four groups of subtasks corresponding to GPU1 or GPU2. The details are as follows:

[0070] exist Figure 5 In the example, the matrix multiplication operation (i.e., the matrix multiplication operator) corresponds to the formula Z = X * Y. The matrix Z in the inference request is split into 2 * 2 (i.e., corresponding to the number of GPUs (2 GPUs) and the type of matrix multiplication), resulting in Z1 = X1 * Y1, Z2 = X1 * Y2, Z3 = X2 * Y1, and Z4 = X2 * Y2, corresponding to GPU1 or GPU2. X represents the input sub-data, and Y represents the parameter. When splitting the inference task, the input data in step S201 is also split according to the type of the first operator (i.e., the matrix multiplication operator).

[0071] exist Figure 6 In [1], the All-Reduce operation (i.e., the All-Reduce operator) is used to sum the All-Reduce output results on GPU1 (i.e., Z1, Z2, Z3, and Z4) with the All-Reduce output results on GPU2 (i.e., D1, D2, D3, and D4). The summed results (i.e., E1, E2, E3, and E4) are then propagated to GPU1 and GPU2, respectively, so that both GPU1 and GPU2 have E1, E2, E3, and E4. The input of All-Reduce is the matrix Z from the same group of subtasks, so All-Reduce needs to be split into 2*2 parts with the same dimensions as matrix Z.

[0072] exist Figure 7In the example, the tensor operation (+) is split. The formula for the tensor operation (+) is G = E + F. F is split according to the same dimension as E to obtain F1, F2, F3, and F4. G is then obtained according to the formula for the tensor operation (+). G includes G1 = E1 + F1, G2 = E2 + F2, G3 = E3 + F3, and G4 = E4 + F4. RmsNorm is further split according to its type and the dimension corresponding to G to obtain four groups of subtasks corresponding to GPU1 or GPU2. F is the parameter corresponding to the tensor operation (+).

[0073] It should be noted that since there is no sequential dependency between the computing tasks and the communication tasks among the four groups of subtasks, the four groups of subtasks can be executed in parallel, thereby speeding up the reasoning process and improving the efficiency of reasoning.

[0074] In this embodiment, there is no sequential dependency between the computing tasks and the communication tasks among the four groups of subtasks, so the four groups of subtasks can be executed in parallel, thereby speeding up the reasoning process and improving the efficiency of reasoning.

[0075] It should be noted that before executing step S202 , the type of each operator in the inference task may be pre-identified so that the inference task can be divided according to the number of GPUs and the type of each operator in step S202 .

[0076] Step S203: generating a CUDA stream corresponding to the graphics processor according to the sequential dependency between the computing tasks and the communication tasks in each group of subtasks; wherein the CUDA stream is used to implement parallel execution of multiple groups of subtasks.

[0077] In this embodiment, for the GPU (i.e., each GPU), a CUDA stream is generated for implementing parallel execution of multiple groups of subtasks based on the sequential dependencies between the computational tasks and the communication tasks in each group of subtasks. The computational tasks in each group of subtasks refer to the fact that the completion of one task in the group of subtasks depends on the completion of one or more other tasks in the group of subtasks. In other words, the sequential dependencies define the order in which the computational tasks and the communication tasks in the group of subtasks are executed.

[0078] Here, a CUDA stream can be a sequence for executing multiple groups of subtasks in parallel. CUDA streams are used to manage and organize the scheduling of CUDA kernels on a GPU so that the GPU can execute multiple groups of subtasks in parallel.

[0079] In this embodiment, a CUDA stream for executing multiple groups of subtasks in parallel is generated by computing tasks and computing tasks, so that when the CUDA stream is executed on the GPU host side, the CUDA kernel on the GPU is scheduled to execute multiple groups of subtasks in parallel, thereby speeding up the inference process.

[0080] Step S204: scheduling the CUDA kernel on the graphics processor according to the CUDA stream, so that the graphics processor executes multiple groups of subtasks in parallel according to the model parameters and the input sub-data, and obtains an inference result corresponding to the inference request.

[0081] In this embodiment, after the CPU host generates a CUDA stream in step S203, the CPU host can schedule CUDA kernels on the GPU based on the CUDA stream, so that the GPU can execute multiple groups of subtasks in parallel based on the model parameters and input sub-data to obtain inference results corresponding to the inference request. CUDA kernels are defined by parallel functions running on the GPU, and these parallel functions are marked using a specific syntax. This allows the CPU host to recognize the markings when running the CUDA stream and schedule the CUDA kernels on the GPU to enable the GPU to execute multiple groups of subtasks in parallel.

[0082] It should be noted that the inference result may include part or all of the GPU output results.

[0083] In this embodiment, the inference result is ultimately determined based on the output results of part or all of the GPUs according to the type of the communication operation in the communication task.

[0084] For example, executing All-Reduce corresponding to each GPU will make the All-Reduce output results corresponding to each GPU the same; then execute the tensor operation (+), the input of the tensor operation (+) is the output result of All-Reduce, at this time the tensor operation (+) corresponding to each GPU has the same input, and the output of the tensor operation (+) corresponding to each GPU will also be the same; then execute RmsNorm, the input of RmsNorm is the output of the tensor operation (+), then the output of RmsNorm corresponding to each GPU is the same, at this time the output result on one GPU can be used as the final inference result.

[0085] For example, a reduction operation applies a specified mapping function to data from multiple GPUs, obtaining an output that is stored on a specific GPU. This output is not available on other GPUs. In this case, the inference result includes the output from all GPUs.

[0086] In this embodiment, the communication tasks and computing tasks in multiple groups of subtasks are all executed in parallel by the CPU host side using the same CUDA kernel on the GPU according to the CUDA stream scheduling, thereby solving the problem of a large gap between the CUDA kernel used to execute computing tasks and the CUDA kernel used to execute communication tasks on the scheduling GPU in related technologies, thereby solving the problem of inference delay caused by the gap and improving inference efficiency.

[0087] As can be seen from the above process, first, when an inference request is received, the inference task is divided into multiple groups of subtasks corresponding to the graphics processor in the combination of computing tasks and communication tasks according to the structural information of the target model; wherein each group of subtasks includes a communication task and a computing task; and the input data is divided into input sub-data corresponding to each group of subtasks; then, according to the sequential dependency between the computing tasks and the communication tasks in each group of subtasks, a CUDA stream is generated for the graphics processor, and the CUDA stream is used to realize the parallel execution of multiple groups of subtasks; then, according to the CUDA stream, the CUDA kernel on the graphics processor is scheduled so that the graphics processor can execute multiple groups of subtasks in parallel according to the input sub-data and the model parameters deployed on the graphics processor to obtain the inference result corresponding to the inference request. This embodiment can solve the problem of a large gap between the CUDA kernel used to execute computing tasks and the CUDA kernel used to execute communication tasks on the scheduling GPU in the related technology, and thus can solve the problem of inference delay caused by the gap, thereby improving the inference efficiency.

[0088] First, the above step S203, namely "generating a CUDA stream corresponding to the graphics processor according to the sequential dependency relationship between the computing tasks and the communication tasks in each group of subtasks", is described in detail with reference to the embodiment.

[0089] In this embodiment, based on the sequential dependency relationship between the computing tasks and the communication tasks in each group of subtasks and the performance analysis results, the tasks of multiple groups of subtasks to be executed in parallel in each inference round are determined; based on the tasks of multiple groups of subtasks to be executed in parallel in each inference round, a CUDA stream corresponding to the graphics processor is generated.

[0090] The following describes in detail the task of “determining the tasks of executing multiple groups of subtasks in parallel in each inference round based on the sequential dependency relationship between the computing tasks and the communication tasks in each group of subtasks and the performance analysis results.”

[0091] In this embodiment, the computational tasks and performance analysis results within each subtask group are used to split the subtask groups into multiple inference rounds, and the tasks to be executed in parallel within each inference round are determined. An inference round can be the number of times the same CUDA kernel on the GPU is scheduled to execute the inference task. Parallel execution of the multiple subtask groups can only be completed after all inference rounds have been completed.

[0092] In this embodiment, when analyzing the performance of a CUDA kernel, it is generally necessary to coordinate the performance analysis between the CPU host and the GPU device. This is because performance analysis also needs to be performed on both the CPU host and the GPU device. The CPU host is responsible for scheduling and monitoring, while the GPU device is responsible for executing tasks. By combining data from the CPU host and the GPU device, a more comprehensive performance assessment of the CUDA kernel can be achieved.

[0093] In one example, the performance analysis can be performed on each CUDA kernel on the GPU using the NVIDIA NSYS (Nsight Systems) tool to obtain a performance analysis result.

[0094] Optionally, the performance analysis result may include task execution time; wherein the task execution time may be the time taken to execute the task.

[0095] It should be noted that the CPU host can monitor and measure performance indicators such as the execution time and memory usage of CUDA kernels. Optionally, these performance indicators can usually be implemented using functions in the CUDA Runtime API (Application Programming Interface) to record the start and end times of communication and computation tasks, thereby obtaining task execution time.

[0096] It's important to note that CUDA kernels execute on the GPU. Therefore, CUDA kernel performance is affected by various factors, including GPU architecture, resource utilization, memory, and bandwidth. Therefore, when deploying a target model, choose a GPU with performance that better matches the target model (for example, one with more memory).

[0097] In addition, performance-related data (such as cache hit rate, instruction execution efficiency, etc.) is also generated on the GPU device side. This data may be captured and analyzed through specific tools (such as NVIDIA's Nsight Compute (Compute Performance Analyzer), Visual Profiler (Visual Profiler), etc.).

[0098] In this embodiment, CUDA kernel performance analysis involves both task scheduling, monitoring, and measurement on the CPU host side, as well as computing task execution and performance data collection on the GPU device side. By comprehensively analyzing data from the CPU host and GPU device sides, CUDA kernel performance can be more accurately analyzed, and the optimal CUDA kernel or GPU can be selected to accelerate the inference process.

[0099] In this embodiment, the NVIDIA NSYS (Nsight Systems) tool can be used to collect performance indicators on the CPU host side or the GPU device side (for example, through hardware counters, etc.). On the one hand, it can provide an intuitive interface to display the performance analysis results, and on the other hand, it can optimize the inference process to speed up the inference.

[0100] The following describes in detail "generating a CUDA stream corresponding to the graphics processor according to the tasks of executing multiple groups of subtasks in parallel in each inference round."

[0101] In this embodiment, after determining the tasks of multiple groups of subtasks to be executed in parallel in each inference round, a task sequence (i.e., CUDA stream) is generated according to the tasks executed in parallel in each inference round. When each inference round is executed in sequence on the GPU host side, the same CUDA kernel on the GPU is scheduled until all the tasks executed in parallel in all inference rounds are executed, so as to obtain the inference result through final inference.

[0102] The following embodiment describes in detail the above step S204, namely "scheduling the CUDA kernel on the graphics processor according to the CUDA stream, so that the graphics processor executes multiple groups of subtasks in parallel according to the model parameters and the input sub-data."

[0103] In this embodiment, the CUDA stream is used to sequentially implement the tasks executed in parallel in each inference round, so as to ultimately implement the parallel execution of multiple groups of subtasks; when the GPU host side executes the CUDA stream, it sequentially schedules the same CUDA kernel on the GPU so that the GPU can execute the tasks executed in parallel in each inference round in parallel according to the model parameters and input sub-data until the parallel execution of multiple groups of subtasks is completed.

[0104] In this embodiment, CUDA streams are used to implement tasks executed in parallel in each inference round, taking into account the performance analysis results of the CUDA kernel and the computing tasks, thereby ensuring both the accuracy of the inference results and the speed of inference.

[0105] The above step of "determining the tasks of executing multiple groups of subtasks in parallel in each reasoning round based on the sequential dependency relationship between the computing tasks and the communication tasks in each group of subtasks and the performance analysis results" is described in detail below in conjunction with an embodiment.

[0106] In this embodiment, for each group of subtasks, based on the computing tasks and the computing tasks, a first computing task whose task execution time is consistent with the task execution time of the communication task in the group of subtasks is first determined from multiple groups of subtasks, the communication task in the group of subtasks is executed in parallel with the first computing task, and a second computing task other than the first computing task is determined from multiple groups of subtasks based on the computing tasks and the computing tasks; then, the first computing task and the second computing task are respectively used as tasks executed in parallel in their corresponding reasoning rounds.

[0107] Here, the communication task and the first computing task in each subtask group do not belong to the same subtask group, and the task execution time of the first computing task matches the task execution time of the communication task in each subtask group. The task execution time refers to the time taken to execute the task. Optionally, the task execution time of the first computing task is less than or equal to the task execution time of the communication task in each subtask group.

[0108] It should be noted that when the GPU executes communication and computation tasks, the resources occupied by computation tasks and communication tasks that are not in the same sub-group (for example, different hardware resources on the GPU) are separated to achieve resource isolation and reduce the possibility of resource competition. This allows computation tasks and communication tasks that are not in the same sub-group to be executed in parallel. Generally, the execution time of a communication task is usually longer than that of a computation task. Therefore, multiple computation tasks and communication tasks that are not in the same sub-group can be executed in parallel to increase the utilization rate of the GPU's computation resources, speed up the inference process, and improve the efficiency of obtaining inference results.

[0109] In this embodiment, as long as the accuracy of the inference results and the speed of inference can be guaranteed, multiple groups of subtasks can be divided into tasks that are executed in parallel across several inference rounds, depending on the requirements for the inference results and / or inference speed. Therefore, the specific method of dividing multiple groups of subtasks into tasks that are executed in parallel across several inference rounds is not unique and will not be elaborated on here.

[0110] In one example, taking the first computing task and the second computing task as tasks to be executed in parallel in their corresponding inference rounds may include:

[0111] In this embodiment, multiple groups of subtasks are divided into several parallel execution task combinations based on the first computing task that is executed in parallel with the communication task in each group of subtasks, and the second computing task that is not executed in parallel with the communication task in each group of subtasks. Each parallel task combination represents all tasks executed in parallel each time. The number of reasoning rounds involved in the reasoning process is the number of reasoning rounds required to complete the reasoning process, and the tasks executed in parallel in each reasoning round are the tasks that need to be executed in parallel for each reasoning. Then, several parallel execution task combinations are respectively corresponded to the tasks executed in parallel in each reasoning round, so that the tasks that need to be executed in parallel for each reasoning can be determined. Next, the tasks executed in parallel in each reasoning round are executed in sequence until the final reasoning result is obtained.

[0112] In this embodiment, based on the sequential dependency and task execution time, multiple groups of subtasks are split into a first computing task that is executed in parallel with the communication task and a second computing task that is not executed in parallel with the communication task. This fully takes into account that the task execution time of the communication task is usually longer than the task execution time of the computing task. Therefore, when executing the communication task, more consideration will be given to executing the first computing task in parallel with the communication task, and then the second computing task in parallel. This can shorten the time for executing multiple groups of subtasks as a whole, thereby speeding up the reasoning process and improving the reasoning speed.

[0113] It should be noted that the terms "first" and "second" in this application do not restrict the order or quantity of tasks, but are merely used to distinguish between them. For example, "first computing task" and "second computing task" are used to distinguish between two computing tasks.

[0114] To facilitate the description of "determining the tasks for executing multiple groups of subtasks in parallel in each inference round based on the sequential dependencies between the computational tasks and communication tasks in each group of subtasks and the performance analysis results," the following uses four groups of subtasks as an example and provides specific explanations:

[0115] The four groups of subtasks include the first group of subtasks, the second group of subtasks, the third group of subtasks, and the fourth group of subtasks. Each of the four groups of subtasks includes a matrix multiplication operation, a full reduction operation, a tensor operation, and a reduction operation arranged in the order defined by the computing tasks.

[0116] Wherein, the first computing task executed in parallel with the communication task in each group of subtasks includes at least one of the following: a matrix multiplication operation in the second group of subtasks executed in parallel with the full reduction operation in the first group of subtasks; a matrix multiplication operation in the third group of subtasks, a tensor operation in the first group of subtasks, and a reduction operation in the first group of subtasks executed in parallel with the full reduction operation in the second group of subtasks; a matrix multiplication operation in the fourth group of subtasks, a tensor operation in the second group of subtasks, and a reduction operation in the second group of subtasks executed in parallel with the full reduction operation in the third group of subtasks; and a tensor operation in the third group of subtasks and a reduction operation in the third group of subtasks executed in parallel with the full reduction operation in the fourth group of subtasks.

[0117] The second computing task includes at least one of the following: a matrix multiplication operation in the first group of subtasks; a tensor operation in the fourth group of subtasks; and a reduction operation in the fourth group of subtasks;

[0118] For the first computing task and the second computing task, respectively, tasks to be executed in parallel in their corresponding inference rounds are determined, including:

[0119] The matrix multiplication operations in the first group of subtasks are performed as tasks to be executed in parallel in the first reasoning round; the matrix multiplication operations in the second group of subtasks that are performed in parallel with the full reduction operations in the first group of subtasks are performed as tasks to be executed in parallel in the second reasoning round; the matrix multiplication operations in the third group of subtasks that are performed in parallel with the full reduction operations in the second group of subtasks, the tensor operations in the first group of subtasks, and the reduction operations in the first group of subtasks are performed as tasks to be executed in parallel in the third reasoning round; the matrix multiplication operations in the fourth group of subtasks that are performed in parallel with the full reduction operations in the third group of subtasks, the tensor operations in the second group of subtasks, and the reduction operations in the second group of subtasks are performed as tasks to be executed in parallel in the fourth reasoning round; the tensor operations in the third group of subtasks that are performed in parallel with the full reduction operations in the fourth group of subtasks, and the reduction operations in the third group of subtasks are performed as tasks to be executed in parallel in the fifth reasoning round; the tensor operations in the fourth group of subtasks and the reduction operations in the fourth group of subtasks are performed as tasks to be executed in parallel in the sixth reasoning round.

[0120] When executing inference, the CPU host side can perform the following scheduling according to the CUDA stream: scheduling the CUDA kernel on the GPU for the GPU to execute the tasks executed in parallel in the first inference round; then, scheduling the CUDA kernel on the GPU (that is, calling the same CUDA kernel in both the first and second inference rounds) for the GPU to execute the tasks executed in parallel in the second inference round; then, scheduling the CUDA kernel on the GPU for the GPU to execute the tasks executed in parallel in the third inference round; then, scheduling the CUDA kernel on the GPU for the GPU to execute the tasks executed in parallel in the fourth inference round; then, scheduling the CUDA kernel on the GPU for the GPU to execute the tasks executed in parallel in the fifth inference round; then, scheduling the CUDA kernel on the GPU for the GPU to execute the tasks executed in parallel in the sixth inference round.

[0121] exist Figure 8 In the example, the first subgroup of tasks includes: matrix multiplication-1, All-Reduce-1, tensor operation (+)-1 and RmsNorm-1 as the first group of subtasks; the second subgroup of tasks includes: matrix multiplication-2, All-Reduce-2, tensor operation (+)-2 and RmsNorm-2 as the second group of subtasks; the third subgroup of tasks includes: matrix multiplication-3, All-Reduce-3, tensor operation (+)-3 and RmsNorm-3 as the third group of subtasks; the fourth subgroup of tasks includes: matrix multiplication-4, All-Reduce-4, tensor operation (+)-4 and RmsNorm-4 as the fourth group of subtasks.

[0122] Taking the execution of Matrix Multiply-1 as an example, after scheduling the CUDA kernel on the GPU, the GPU executes the Matrix Multiply-1 operation based on the input sub-data corresponding to Matrix Multiply-1 and the parameters of Matrix Multiply-1, obtaining the output result of Matrix Multiply-1. The execution process of Matrix Multiply-2, Matrix Multiply-3, or Matrix Multiply-4 is the same as that of Matrix Multiply-1, except that the input sub-data corresponding to Matrix Multiply-2, Matrix Multiply-3, or Matrix Multiply-4 is different from that of Matrix Multiply-1, and will not be further described here.

[0123] Taking GPU1 executing All-Reduce-1 as an example, GPU1 executes the All-Reduce-1 operation based on the output result of Matrix Multiplication-1 and the corresponding parameters of All-Reduce-1, obtaining the output result of All-Reduce-1. Next, the output result of All-Reduce-1 on GPU1 and the output result of All-Reduce-1 on GPU2 are summed, and then the summed result is transmitted to All-Reduce-1 on GPU1 and All-Reduce-1 on GPU2, respectively. The execution process of All-Reduce-2, All-Reduce-3, or All-Reduce-4 on GPU1 is the same as the execution process of All-Reduce-1 on GPU1 and will not be repeated here.

[0124] Taking tensor operation (+)-1 as an example, the GPU executes tensor operation (+)-1 based on the output of All-Reduce-1 and the parameters corresponding to tensor operation (+)-1 to obtain the output of tensor operation (+)-1. The execution process of tensor operation (+)-2, tensor operation (+)-3, or tensor operation (+)-4 is the same as the execution process of tensor operation (+)-1 and is not repeated here.

[0125] Taking RmsNorm-1 as an example, the GPU performs the RmsNorm-1 operation based on the output of the tensor operation (+)-1 and the corresponding parameters of RmsNorm-1 to obtain the output of RmsNorm-1. The execution process of RmsNorm-2, RmsNorm-3, or RmsNorm-4 is the same as that of RmsNorm-1 and is not repeated here.

[0126] It should be noted that executing All-Reduce corresponding to each GPU will make the All-Reduce output results corresponding to each GPU the same; then execute the tensor operation (+), and the input of the tensor operation (+) is the output result of All-Reduce. At this time, the tensor operation (+) corresponding to each GPU has the same input, and the output of the tensor operation (+) corresponding to each GPU will also be the same; then execute RmsNorm, and the input of RmsNorm is the output of the tensor operation (+), then the output of RmsNorm corresponding to each GPU is the same. At this time, the output result on one GPU can be used as the final inference result.

[0127] Here, the model parameters in step S201 may include parameters corresponding to matrix multiplication, parameters corresponding to All-Reduce, parameters corresponding to the tensor operation (+), and parameters corresponding to RmsNorm. The input sub-data corresponding to the matrix multiplication operation in each group of subtasks is the data obtained by segmenting the input data in step S201. Each group of subtasks executes multiple groups of subtasks in parallel on the corresponding input sub-data to obtain the final inference result.

[0128] In this embodiment, the CUDA kernel on the GPU is scheduled according to the CUDA stream, so that the GPU sequentially executes the tasks executed in parallel in the first inference round, the tasks executed in parallel in the second inference round, the tasks executed in parallel in the third inference round, the tasks executed in parallel in the fourth inference round, the tasks executed in parallel in the fifth inference round, and the tasks executed in parallel in the sixth inference round, until multiple groups of subtasks are executed and the final inference result is obtained.

[0129] In conjunction with the embodiment below, after executing the "matrix multiplication operation (for example, matrix multiplication -1, matrix multiplication -2, matrix multiplication -3 or matrix multiplication -4)" in the above steps, the reasoning method further includes: for the full reduction operation in each group of subtasks, after executing the matrix multiplication operation in the group of subtasks, waking up the full reduction operation in the group of subtasks.

[0130] exist Figure 8 In the example, after executing matrix multiplication-1, All-Reduce-1 is awakened; after executing matrix multiplication-2, All-Reduce-2 is awakened; after executing matrix multiplication-3, All-Reduce-3 is awakened; after executing matrix multiplication-4, All-Reduce-4 is awakened.

[0131] In this embodiment, the purpose of waking up the communication task in the current group of subtasks after executing the matrix multiplication operation in the current group of subtasks can include the following aspects: (1) ensuring data consistency and integrity; (2) optimizing resource utilization; (3) improving task execution efficiency; and (4) achieving orderly execution between tasks.

[0132] Ensuring data consistency and integrity means that computing tasks often involve processing and analyzing data, which serves as the foundation for subsequent communication tasks. Only after the computing task is completed and the data's accuracy and integrity are ensured can communication tasks effectively communicate based on this data. If the computing task is not completed or the data is not prepared, the communication task may not execute correctly or may result in erroneous results.

[0133] Optimizing resource utilization means that when executing a compute task, there is competition for GPU hardware resources between the compute task and the communication task in the same task subgroup. Waking up the communication task in the same task subgroup at the same time will reduce overall GPU performance. Waking up the communication task in the same task subgroup after the compute task is completed can more effectively utilize GPU hardware resources to execute compute tasks and communication tasks in parallel, avoiding unnecessary resource waste.

[0134] Improving task execution efficiency means separating the computing and communication tasks, and waking up the communication task only after the computing task is complete. This ensures that the communication task can immediately obtain the results of the computing task and quickly execute based on these results. This approach avoids unnecessary waiting for the communication task while the computing task is incomplete, thereby improving overall task execution efficiency.

[0135] Enabling ordered execution between tasks means that during the inference process, there is a certain sequential dependency between the computation task and the communication task. The result of the computation task may be part of the input of the communication task. By ensuring that the computation task completes before waking up the communication task, ordered execution between tasks is achieved, ensuring the correctness and efficiency of the entire inference process.

[0136] The above step, namely “awakening the full-reduce operation in the group of subtasks”, is described in detail below with reference to an embodiment.

[0137] In this embodiment, the all-reduce operation in the group of subtasks is awakened by a synchronization primitive.

[0138] In this embodiment, the all-reduce operation in the group of subtasks is awakened by a CUDA kernel synchronization primitive with lower overhead.

[0139] CUDA Kernel Synchronization Primitives are tools and functions used to coordinate and synchronize the execution of threads within a CUDA stream. In this context, threads can be threads that execute multiple groups of subtasks in parallel. These primitives ensure that threads execute in the expected order (i.e., a CUDA stream can be a sequence for executing multiple groups of subtasks in parallel), avoid data races, and ensure memory consistency.

[0140] In this embodiment, the full-reduce operation in the group of subtasks is awakened by a synchronization primitive to solve the problem in the related art that when waking up the communication task, the message passing between the computing task and the communication task needs to be performed through GPU global memory access, resulting in a large message passing overhead for waking up the communication task because the computing task and the communication task are distributed in CUDA kernels of different CUDA streams.

[0141] In conjunction with the embodiment below, before executing the above step S201, the inference method also includes: determining the scale information of the graphics processor based on the tensor parallel information of the target model; wherein the scale information includes the number of graphics processors, and the number of graphics processors is multiple; based on the structural information of the target model, splitting the model weight file of the target model into multiple parts, and loading them on multiple graphics processors to obtain model parameters deployed on each graphics processor.

[0142] In this example, the model weight file is a crucial component in deep learning. It contains the parameters learned by the model during training. These parameters primarily include the weight matrices and bias vectors for each layer, which together determine how the model responds to input data and the accuracy of its output.

[0143] In this embodiment, before inference, the number of GPUs to be deployed is determined based on the tensor parallel information of the target model. Subsequently, the target model's model weight file is split into multiple copies and loaded onto multiple GPUs, with the model parameters deployed on each GPU. This achieves deployment of the target model.

[0144] In conjunction with the embodiment below, after executing “determining the scale information of the graphics processor”, the reasoning method further includes:

[0145] When executing the communication tasks in each group of subtasks, data communication between the graphics processors is completed through the communication algorithm corresponding to the communication tasks and / or the topology graph in the group of subtasks.

[0146] In this embodiment, the scale information may include a topology diagram between multiple GPUs, which may intuitively display the physical connections between the multiple GPUs. The physical connections may include connecting multiple GPUs via NVLink, PCIe, or IB (Infini Band) channels.

[0147] In this embodiment, a corresponding communication algorithm may be determined according to the communication task and / or the topology graph, and data communication between the GPUs may be completed through the communication algorithm.

[0148] In one example, the communication modes corresponding to the all-reduce operation and the broadcast operation are different.

[0149] In this embodiment, data communication between GPUs is accomplished by determining corresponding communication algorithms through communication tasks and / or topology graphs.

[0150] The “communication algorithm” is described in detail below in conjunction with the embodiments.

[0151] In this embodiment, any one of the following communication algorithms may be used: a one-round communication algorithm, a two-round communication algorithm, or a ring communication algorithm.

[0152] In one example, taking All-Reduce as an example, a one-shot communication algorithm refers to performing one round of communication between GPUs to aggregate data of all GPUs on each GPU.

[0153] In one example, Figure 9 In the example, GPU1 includes M1, M2, and M3, GPU2 includes P1, P2, and P3, and GPU3 includes Q1, Q2, and Q3. Taking All-Reduce as an example, the two-shot communication algorithm refers to two rounds of communication between each GPU (i.e., GPU1, GPU2, or GPU3), aggregating all the data of GPU1, GPU2, and GPU3 on each GPU. That is, GPU1, GPU2, or GPU3 finally includes M1+P1+Q1, M2+P2+Q2, and M3+P3+Q3.

[0154] In one example, taking All-Reduce as an example, a Ring-based communication algorithm requires 2*(number of GPUs - 1) rounds of communication.

[0155] exist Figure 10 In the example, GPU1 includes M1, M2, and M3; GPU2 includes P2, P2, and P3; GPU3 includes Q1, Q2, and Q3. Four rounds of communication are required between GPU1, GPU2, and GPU3 to aggregate all the data of GPU1, GPU2, and GPU3 on each GPU. That is, GPU1, GPU2, or GPU3 ultimately includes M1+P1+Q1, M2+P2+Q2, and M3+P3+Q3.

[0156] In this embodiment, a corresponding communication algorithm can be selected according to the requirements for communication delay. For example, the communication delay can be reduced by reducing the number of communication rounds. The two-shot communication delay is smaller than the ring-based communication delay.

[0157] The following describes in detail the “receiving an inference request for a target model” in the above step S201 in conjunction with an embodiment.

[0158] In this embodiment, an inference request for a target model is received using the OpenAI (Open AI Research Center, USA) API format.

[0159] In this embodiment, the API provided by OpenAI enables developers to access and use models developed by OpenAI. This API provides a variety of functions, including natural language processing, text generation, image generation, etc., supporting various application scenarios such as chatbots, text analysis, and automatic writing. Developers can use this API to integrate OpenAI's models into their applications, services, or products to achieve more intelligent functions and user experience.

[0160] Figure 11 A flowchart of the reasoning method provided in the embodiment of the present application, in which the scheme is applied to Figure 1 The CPU host side in the example is used to illustrate. Figure 11 As shown in , the method may include the following steps:

[0161] Step S1101: Determine scale information of a graphics processor according to tensor parallel information of a target model; wherein the scale information includes the number of graphics processors, and the number of graphics processors is multiple.

[0162] Step S1102: splitting the model weight file of the target model into multiple parts according to the structural information of the target model, and loading them onto multiple graphics processors to obtain model parameters deployed on each graphics processor.

[0163] Step S1103: In response to receiving an inference request for a target model, obtaining model parameters of the target model deployed on each graphics processor.

[0164] Step S1104: Generate an inference task for the inference request; wherein the inference request includes information about the inference task.

[0165] In one example, the reasoning task may be a combination of operators, such as a communication operator and a computation operator.

[0166] Step S1105: Identify the types of communication operators and computation operators in the inference task.

[0167] In this embodiment, the types of communication operators and calculation operators are identified so as to facilitate segmentation of the operators according to type in the subsequent step 1106 .

[0168] Step S1106: Based on the structural information of the target model, the inference task is divided into multiple groups of subtasks corresponding to the graphics processor in a combination of computing tasks and communication tasks; each group of subtasks includes a communication task and a computing task; and the input data is divided into input sub-data corresponding to each group of subtasks.

[0169] Step S1107: Based on the sequential dependency between the computing tasks and the communication tasks in each group of subtasks, a CUDA stream corresponding to the graphics processor is generated; wherein the CUDA stream is used to implement parallel execution of multiple groups of subtasks; and the CUDA kernel on the graphics processor is scheduled according to the CUDA stream to obtain an inference result corresponding to the inference request.

[0170] Step S1108: Output the inference result.

[0171] Combined with the above Figure 2 , describes the method embodiment of the present application in detail, and the following is combined with Figure 12 , the device embodiment of the present application is described in detail. It should be understood that the description of the method embodiment corresponds to the description of the device embodiment, so for parts not described in detail, reference can be made to the previous method embodiment.

[0172] Figure 12 The figure shows a schematic diagram of the structure of the reasoning method provided by an embodiment of the present application. Figure 12 As shown, the reasoning method provided in the embodiment of the present application includes an acquisition module 1201, a segmentation module 1202, a generation module 1203 and a reasoning module 1204. Among them, the acquisition module 1201 is used to obtain the model parameters of the target model deployed on each graphics processor in response to receiving an inference request for the target model, wherein the inference request includes information about the inference task, and the information about the inference task includes input data; the segmentation module 1202 is used to segment the inference task into multiple groups of subtasks corresponding to the graphics processor according to the combination of computing tasks and communication tasks based on the structural information of the target model; wherein each group of subtasks includes a communication task and a computing task; and, the input data is segmented into input sub-data corresponding to each group of subtasks; the generation module 1203 is used to generate a unified computing device architecture CUDA stream corresponding to the graphics processor based on the sequential dependency between the computing tasks and the communication tasks in each group of subtasks; wherein the CUDA stream is used to realize parallel execution of multiple groups of subtasks; the inference module 1204 is used to schedule the CUDA kernel on the graphics processor according to the CUDA stream, so that the graphics processor can execute multiple groups of subtasks in parallel based on the model parameters and the input sub-data, and obtain the inference result corresponding to the inference request.

[0173] In one embodiment, the generation module 1203 includes: a task determination unit (not shown in the figure), configured to determine tasks to be executed in parallel in each inference round based on the sequential dependency between the computing tasks and the communication tasks in each group of subtasks and the performance analysis results; a CUDA stream generation unit (not shown in the figure), configured to generate a CUDA stream corresponding to the graphics processor based on the tasks to be executed in parallel in each inference round based on the multiple groups of subtasks;

[0174] The inference module 1204 is specifically configured to schedule the CUDA kernel on the graphics processor according to the CUDA stream when executing each inference round in sequence, so that the graphics processor can perform tasks in parallel in the inference round according to the model parameters and input sub-data.

[0175] In one embodiment, the task determination unit is specifically used to: for each group of subtasks, determine, based on the computing tasks and the computing tasks, a first computing task whose task execution time is consistent with the task execution time of the communication task in the group of subtasks from multiple groups of subtasks, the communication task in the group of subtasks is executed in parallel with the first computing task, and determine a second computing task other than the first computing task from multiple groups of subtasks; wherein the first computing task and the communication task are not in the same group of subtasks; and the first computing task and the second computing task are respectively used as tasks executed in parallel in their corresponding reasoning rounds.

[0176] In one embodiment, the plurality of groups of subtasks include a first group of subtasks, a second group of subtasks, a third group of subtasks, and a fourth group of subtasks, each group of subtasks including a matrix multiplication operation, a full reduction operation, a tensor operation, and a reduction operation arranged in a sequential dependency relationship;

[0177] Wherein, the first computing task executed in parallel with the communication task in each group of subtasks includes at least one of the following: a matrix multiplication operation in the second group of subtasks executed in parallel with the full reduction operation in the first group of subtasks; a matrix multiplication operation in the third group of subtasks, a tensor operation in the first group of subtasks, and a reduction operation in the first group of subtasks executed in parallel with the full reduction operation in the second group of subtasks; a matrix multiplication operation in the fourth group of subtasks, a tensor operation in the second group of subtasks, and a reduction operation in the second group of subtasks executed in parallel with the full reduction operation in the third group of subtasks; and a tensor operation in the third group of subtasks and a reduction operation in the third group of subtasks executed in parallel with the full reduction operation in the fourth group of subtasks.

[0178] The second computing task includes at least one of the following: the matrix multiplication operation in the first group of subtasks; the tensor operation in the fourth group of subtasks and the reduction operation in the fourth group of subtasks.

[0179] In one embodiment, the apparatus further includes: a wake-up module (not shown in the figure) for waking up the full-reduction operation in each group of subtasks after executing the matrix multiplication operation in the group of subtasks.

[0180] In one embodiment, the wake-up module is specifically configured to: wake up the full-reduce operation in the group of subtasks through a synchronization primitive.

[0181] In one embodiment, the device also includes: a determination module (not shown in the figure) for determining the scale information of the graphics processor based on the tensor parallel information of the target model; wherein the scale information includes the number of graphics processors, and the number of graphics processors is multiple; a deployment module (not shown in the figure) for splitting the model weight file of the target model into multiple parts according to the structural information of the target model, and loading them on multiple graphics processors to obtain model parameters deployed on each graphics processor.

[0182] In one embodiment, the scale information also includes: a topology map between multiple graphics processors; the device also includes: a communication module (not shown in the figure) for completing data communication between each graphics processor when executing the communication task in each group of subtasks through the communication algorithm corresponding to the communication task and / or the topology map in the group of subtasks.

[0183] In one embodiment, the communication algorithm includes at least one of the following: a one-round communication method, a two-round communication method, and a ring communication algorithm.

[0184] In one embodiment, the acquisition module 1201 is specifically configured to receive an inference request for a target model using the Open AI Research Center application programming interface format.

[0185] Below, reference Figure 13 To describe the CPU host side according to an embodiment of the present application. Figure 13 Shown is a structural diagram of the CPU host side provided in one embodiment of the present application.

[0186] like Figure 13 As shown, the CPU host side includes one or more processors 1301 and a memory 1302 .

[0187] The processor 1301 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the CPU host side to perform desired functions.

[0188] The memory 1302 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc. One or more computer program instructions may be stored on a computer-readable storage medium, and the processor 1301 may execute the program instructions to implement functions such as the reasoning method of each embodiment of the present application mentioned above. Various contents such as model parameters and input data may also be stored in the computer-readable storage medium.

[0189] In one example, the CPU host side may further include: an input device 1303 and an output device 1304 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0190] The input device 1303 may include, for example, a keyboard, a mouse, and the like.

[0191] The output device 1304 can output various information to the outside, including inference results, etc. The output device 1304 can include, for example, a display, a speaker, a printer, a communication network and its connected remote output devices, etc.

[0192] Of course, to simplify, Figure 13 Only some of the components in the CPU host side that are relevant to the present application are shown, omitting components such as a bus, an input / output interface, etc. In addition, the CPU host side may further include any other appropriate components depending on the specific application.

[0193] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions. When the computer program instructions are executed by a processor, the processor executes the steps of the reasoning method according to various embodiments of the present application described above in this specification.

[0194] The computer program product may be written in any combination of one or more programming languages ​​to implement the program code of the embodiments of the present application, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's CPU host, partially on the user's device, as a standalone software package, partially on the user's CPU host and partially on a remote CPU host, or entirely on a remote CPU host or server.

[0195] In addition, an embodiment of the present application may also be a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the processor executes the steps of the reasoning method according to various embodiments of the present application described above in this specification.

[0196] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0197] The basic principles of the present application have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this application are merely illustrative and not restrictive, and it should not be assumed that these advantages, strengths, and effects are required of each embodiment of this application. In addition, the specific details disclosed above are merely illustrative and facilitating understanding, and are not restrictive. The above details do not limit this application to necessarily being implemented using the above specific details.

[0198] The block diagrams of the devices, devices, equipment, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "include," "comprise," "have," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0199] It should also be noted that in the apparatus, device, and method of the present application, each component or each step can be decomposed and / or recombined, and such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0200] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0201] The above description has been provided for the purpose of illustration and description. Furthermore, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

[0202] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method of reasoning, wherein The method comprises: In response to receiving an inference request for a target model, obtaining model parameters of the target model deployed on each graphics processor, wherein the inference request includes information about an inference task, and the information about the inference task includes input data; Based on the structural information of the target model, the inference task is divided into a plurality of subtasks corresponding to the graphics processor in a combination of computing tasks and communication tasks; each subtask group includes a communication task and a computing task; and the input data is divided into input sub-data corresponding to each subtask group; Generating a Compute Unified Device Architecture (CUDA) stream corresponding to the graphics processor based on the sequential dependencies between the computing tasks and the communication tasks in each group of subtasks; wherein the CUDA stream is used to implement parallel execution of the multiple groups of subtasks; The CUDA kernel on the graphics processor is scheduled according to the CUDA stream, so that the graphics processor executes multiple groups of subtasks in parallel according to the model parameters and the input sub-data, and obtains an inference result corresponding to the inference request.

2. The method according to claim 1, wherein Generating a CUDA stream corresponding to the graphics processor according to the sequential dependency between the computing tasks and the communication tasks in each group of subtasks includes: Determine the tasks that multiple groups of subtasks are to be executed in parallel in each inference round based on the sequential dependencies between the computational tasks and the communication tasks in each group of subtasks and the performance analysis results; Generate a CUDA stream corresponding to the graphics processor based on the tasks of multiple groups of subtasks executed in parallel in each inference round; Scheduling the CUDA kernel on the graphics processor according to the CUDA stream so that the graphics processor executes the multiple groups of subtasks in parallel according to the model parameters and the input sub-data includes: When each inference round is executed in sequence, the CUDA kernel on the graphics processor is scheduled according to the CUDA stream, so that the graphics processor can perform tasks in parallel in the inference round according to the model parameters and the input sub-data.

3. The method according to claim 2, wherein: The step of determining the tasks of executing the multiple groups of subtasks in parallel in each reasoning round based on the sequential dependency relationship between the computing tasks and the communication tasks in each group of subtasks and the performance analysis results includes: For each group of subtasks, determining, based on the computing tasks and the computing tasks, a first computing task from the multiple groups of subtasks whose task execution time matches the task execution time of the communication task in the group of subtasks, the communication task in the group of subtasks being executed in parallel with the first computing task, and determining a second computing task other than the first computing task from the multiple groups of subtasks; wherein the first computing task and the communication task are not in the same group of subtasks; The first computing task and the second computing task are respectively used as tasks executed in parallel in their corresponding inference rounds.

4. The method according to claim 3, wherein: The multiple groups of subtasks include a first group of subtasks, a second group of subtasks, a third group of subtasks, and a fourth group of subtasks, each group of subtasks including a matrix multiplication operation, a full reduction operation, a tensor operation, and a reduction operation arranged in the order of dependency; Wherein, the first computing task executed in parallel with the communication task in each group of subtasks includes at least one of the following: a matrix multiplication operation in the second group of subtasks executed in parallel with the full reduction operation in the first group of subtasks; a matrix multiplication operation in the third group of subtasks, a tensor operation in the first group of subtasks, and a reduction operation in the first group of subtasks executed in parallel with the full reduction operation in the second group of subtasks; a matrix multiplication operation in the fourth group of subtasks, a tensor operation in the second group of subtasks, and a reduction operation in the second group of subtasks executed in parallel with the full reduction operation in the third group of subtasks; and a tensor operation in the third group of subtasks and a reduction operation in the third group of subtasks executed in parallel with the full reduction operation in the fourth group of subtasks. The second computing task includes at least one of the following: a matrix multiplication operation in the first group of subtasks; a tensor operation in the fourth group of subtasks; and a reduction operation in the fourth group of subtasks.

5. The method according to claim 4, wherein The method further includes: for the full reduction operation in each group of subtasks, after the matrix multiplication operation in the group of subtasks is executed, waking up the full reduction operation in the group of subtasks.

6. The method according to claim 5, wherein: The waking up the all-reduce operation in the group of subtasks includes: waking up the all-reduce operation in the group of subtasks through a synchronization primitive.

7. The method according to any one of claims 1 to 6, wherein: The method further comprises: Determining scale information of a graphics processor according to tensor parallel information of a target model; wherein the scale information includes the number of graphics processors, and the number of the graphics processors is multiple; According to the structural information of the target model, the model weight file of the target model is split into multiple parts and loaded on multiple graphics processors to obtain model parameters deployed on each graphics processor.

8. The method according to claim 7, wherein: The scale information further includes: a topology diagram between the plurality of graphics processors; the method further includes: When executing the communication tasks in each group of subtasks, data communication between the graphics processors is completed through the communication algorithm corresponding to the communication tasks in the group of subtasks and / or the topology graph.

9. An inference device, wherein: The device comprises: an acquisition module, configured to acquire model parameters of the target model deployed on each graphics processor in response to receiving an inference request for the target model, wherein the inference request includes information of an inference task, and the information of the inference task includes input data; a segmentation module, configured to segment the inference task into a plurality of subtasks corresponding to the graphics processor according to a combination of computing tasks and communication tasks based on the structural information of the target model; wherein each subtask group includes a communication task and a computing task; and segment the input data into input sub-data corresponding to each subtask group; A generation module, configured to generate a Compute Unified Device Architecture (CUDA) stream corresponding to the graphics processor based on a sequential dependency relationship between computing tasks and communication tasks in each group of subtasks; wherein the CUDA stream is used to implement parallel execution of multiple groups of subtasks; An inference module is used to schedule a CUDA kernel on the graphics processor according to the CUDA stream, so that the graphics processor can execute multiple groups of subtasks in parallel according to the model parameters and the input sub-data to obtain an inference result corresponding to the inference request.

10. A computing device, characterized in that including memory and processor; The memory is connected to the processor and is used to store programs; The processor is configured to implement the method according to any one of claims 1 to 8 by running the program in the memory.

11. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

12. A computer program product, characterized in that The method comprises computer program instructions, which, when executed by a processor, cause the processor to implement the method according to any one of claims 1 to 8.

Images