Model inference method and model inference system

Abstract

Disclosed are a model inference method and a model inference system, relating to the technical field of AI. A processing node in the model inference system distinguishes a plurality of acceleration nodes, so that acceleration nodes having different resources are used for executing different tasks in model instances, thereby maximizing the utilization of resources of different acceleration nodes, and improving the processing efficiency of the model inference system for a plurality of inference requests. In addition, when the model inference system processes a plurality of inference requests in batches, different acceleration nodes run model instances on the basis of resources of the acceleration nodes, and the model instances process the plurality of inference requests in parallel, thereby avoiding the problem that after completing an inference task for one inference request, an inference card waits for an inference task for a next inference request to finish, and thus reducing inference delay of the model inference system for the plurality of inference requests and improving the inference efficiency of the model instances.

Description

Model inference method and model inference system

[0001] The present application claims priority to the Chinese patent application No. 202411322141.5, filed on September 20, 2024, and entitled "Model Inference Method and Model Inference System", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence (AI) technology, and in particular to a model inference method and a model inference system. BACKGROUND

[0003] With the development of the Internet and big data technology, AI models have made progress in information processing and knowledge extraction. The AI model (hereinafter referred to as "model" in the language of the following part) can include but is not limited to a large language model (LLM) or a model that can be used to implement a human-computer interaction function, etc. The current large language model uses multi-path inference technology to predict batch requests of users. The multi-path inference technology refers to that the model simultaneously infers multiple requests in the same time period. The model includes a prefill stage and a decode stage. In the prefill stage, different words in the request are processed to obtain key-value cache data. In the decode stage, the key-value cache data is taken as input to generate a prediction result corresponding to the request. The current multi-path inference technology deploys the model in an inference card and sequentially processes batch requests by the inference card. In the model inference process of the inference card, the prefill stage and the decode stage are processed in series. In the scenario of predicting batch requests of the model, the prefill stage and the decode stage wait for each other. After the model completes a prediction task of one inference request, it waits for the end of the prediction task of the next inference request. Therefore, the inference time delay of the inference card for batch requests is unstable, and the inference efficiency of the model is affected. SUMMARY

[0004] The present application provides a model inference method and a model inference system, which solve the problem of unstable inference time delay of batch requests in the inference card and improve the inference efficiency of the model.

[0005] The present application adopts the following technical solution.

[0006] In a first aspect, the present application provides a model inference method. The model inference method is applied to a model inference system, and the model inference system comprises a processing node and a plurality of acceleration nodes. The plurality of acceleration nodes comprises: a first group of acceleration nodes matched with resources required by a preprocessing task in a first model instance, and a second group of acceleration nodes matched with resources required by a decoding task in the first model instance. The resources comprise computing resources, transmission resources and storage resources. The model inference method provided in the first aspect of the present application comprises: the processing node acquires a plurality of inference requests, wherein each inference request carries input information. The processing node sends first input information in a first inference request to the first group of acceleration nodes to obtain first cache data. The first inference request is an inference request matched with the first model instance in the plurality of inference requests, and the first cache data comprises an intermediate vector corresponding to the first input information. In addition, the processing node sends a first inference instruction to the second group of acceleration nodes. The first inference instruction is used to instruct the second group of acceleration nodes to take the first cache data as input of the decoding task in the first model instance to obtain an inference result of the first inference request.

[0007] In a first optional example, the acceleration node is an acceleration chip.

[0008] In a second optional example, the acceleration node is an inference card.

[0009] In a third optional example, the acceleration node is an acceleration device.

[0010] The model instance deployed in the acceleration node refers to an operable object obtained by instantiating the model. For example, an object that can operate on actual data is created according to the definition and parameters of the model.

[0011] In an optional implementation of the model inference method provided in the first aspect, the first input information is multi-modal data.

[0012] In the first aspect of the present application, the processing node in the model inference system distinguishes the plurality of acceleration nodes, so that the acceleration nodes with different resources are used to perform different tasks in the model instance, which is conducive to maximizing the resources possessed by different acceleration nodes and improving the processing efficiency of the model inference system for a plurality of inference requests. Moreover, in the case of batch processing of a plurality of inference requests by the model inference system, different acceleration nodes run the model instance according to the resources possessed by the acceleration nodes, and the model instance processes a plurality of inference requests in parallel, which is conducive to avoiding the problem that the inference card waits for the end of the inference task of the next inference request after the completion of the inference task of one inference request, reducing the inference delay of the model inference system for a plurality of inference requests, and improving the inference efficiency of the model instance.

[0013] In an optional implementation of the model inference method provided in the first aspect, the first group of acceleration nodes has more computing resources than the second group of acceleration nodes, and the second group of acceleration nodes has more transmission resources than the first group of acceleration nodes.

[0014] In an optional implementation of the model inference method provided in the first aspect, the first group of acceleration nodes includes one or more first acceleration nodes in the plurality of acceleration nodes that support using computing resources greater than or equal to a first threshold. The second group of acceleration nodes includes one or more second acceleration nodes in the plurality of acceleration nodes that support using transmission resources greater than or equal to a second threshold.

[0015] In the first aspect of the present application, the model inference system distinguishes the plurality of acceleration nodes, and uses acceleration nodes with different resources to perform different tasks in the model instance, such as using the first group of acceleration nodes with more computing resources to perform the preprocessing task in the model instance, and using the second group of acceleration nodes with more transmission resources to perform the decoding task in the model instance, which is beneficial to maximize the use of resources of different acceleration nodes and improve the processing efficiency of the model inference system for multiple inference requests.

[0016] In an optional implementation of the model inference method provided in the first aspect, the plurality of acceleration nodes further includes a third group of acceleration nodes matched with resources required for the preprocessing task in the second model instance and a fourth group of acceleration nodes matched with resources required for the decoding task in the second model instance, and the second model instance is different from the first model instance. After the processing node obtains the plurality of inference requests, the model inference method provided in the first aspect includes: the processing node sends second input information in a second inference request to the third group of acceleration nodes to obtain second cache data; wherein the second inference request is an inference request matched with the second model instance in the plurality of inference requests, and the second cache data includes an intermediate vector corresponding to the second input information. And the processing node sends a second inference instruction to the fourth group of acceleration nodes, the second inference instruction being used to instruct the fourth group of acceleration nodes to use the second cache data as an input of the decoding task in the second model instance to obtain an inference result of the second inference request.

[0017] In the first aspect of the present application, multiple different model instances can be deployed in the model inference system, and after the model inference system receives multiple inference requests, acceleration nodes with different resources are matched for different model instances, so that in the case of batch processing requests in the model inference system, multi-node scheduling can be implemented in the model inference system to run the model instance, thereby reducing the waiting time of the model inference system for processing batch inference requests and improving the efficiency of model inference.

[0018] In an optional implementation of the model inference method provided in the first aspect, the network layer corresponding to the preprocessing task comprises one or a combination of the following: an attention layer, a residual connection layer, a normalization operation layer, and a feedforward layer. The first input information comprises a plurality of words. The first set of cache data comprises a key-value vector corresponding to the plurality of words.

[0019] In an optional implementation of the model inference method provided in the first aspect, the second set of acceleration nodes further deploy an incremental prediction model. The second set of acceleration nodes takes the first set of cache data as input of the decoding task in the first model instance, to obtain the inference result of the first inference request, comprising: the second set of acceleration nodes inputs the first word in the first input information and the first set of cache data into the network layer corresponding to the decoding task in the first model instance, to output an inference value of the first word; the first input information comprises a plurality of words, and the first word is the first word or an intermediate word in the plurality of words. The second set of acceleration nodes inputs the inference value of the first word and the first set of cache data into the incremental prediction model, to obtain candidate values of N words in the first input information; wherein the N words are words adjacent to and after the first word in the first input information, and N≥2. The second set of acceleration nodes inputs the N words and the first set of cache data into the network layer corresponding to the decoding task in the first model instance, to output prediction values of the N words; the second set of acceleration nodes takes a value consistent between the candidate values of the N words and the prediction values of the N words as an effective inference value of the N words. After the second set of acceleration nodes determines the inference values of all the words in the plurality of words, the inference values of all the words are weighted to obtain the inference result of the first inference request.

[0020] In the first aspect of the present application, in the decoding task of processing batch inference requests, an incremental prediction model is used to preliminarily predict a plurality of words (such as N words) in the inference requests, and a decoding task in a model instance compares the preliminary prediction result. This comparison process can compare multiple words at a time, solving the problem of high inference delay caused by the fact that the decoding task in the model instance can only infer one word at a time in the prior art, and improving the efficiency of model inference.

[0021] In an optional implementation of the model inference method provided in the first aspect, the first model instance is a language model, and the second model instance is an image model.

[0022] In an optional implementation of the model inference method provided in the first aspect, the first model instance is an image model, and the second model instance is a language model.

[0023] In an optional implementation of the model inference method provided in the first aspect, the first model instance includes a first attention layer and a plurality of first expert processing layers located after the first attention layer. The second group of acceleration nodes obtains the inference result of the first inference request by taking the first group of cached data as input of a decoding task in the first model instance, including: the second group of acceleration nodes obtains a set of expert weights corresponding to the first inference request, the set of expert weights including weights of each first expert processing layer in the plurality of first expert processing layers. And the second group of acceleration nodes inputs the first group of cached data into the first attention layer to obtain a first attention. The second group of acceleration nodes takes the first attention as input of the plurality of first expert processing layers, and processes the first attention in combination with the weights in the set of expert weights to obtain the inference result of the first inference request.

[0024] In an optional implementation of the model inference method provided in the first aspect, the second group of acceleration nodes includes a plurality of second acceleration nodes in the plurality of acceleration nodes, which support a transmission resource greater than or equal to a second threshold. The plurality of second acceleration nodes are deployed with a plurality of attention layers and a plurality of expert processing layers of a plurality of model instances. The attention instances corresponding to the plurality of attention layers are connected to the expert processing instances corresponding to the plurality of expert processing layers in a mesh structure. The plurality of model instances include the first model instance.

[0025] In a second aspect, the present application provides a processing node. The processing node includes a communication interface and a processor. The communication interface is configured to obtain a plurality of inference requests. The processor and the communication interface are configured to cooperatively perform the operation steps implemented by the processing node in the method provided in the first aspect or any optional implementation of the first aspect.

[0026] In a third aspect, the present application provides a model inference system, including a processing node and a plurality of acceleration nodes. The plurality of acceleration nodes include a first group of acceleration nodes matched with resources required by a preprocessing task in a first model instance and a second group of acceleration nodes matched with resources required by a decoding task in the first model instance. The resources include computing resources, transmission resources, and storage resources. The processing node is configured to obtain a plurality of inference requests, wherein each inference request carries an input information. The first group of acceleration nodes is configured to take first input information in a first inference request as input of the preprocessing task in the first model instance to obtain first key-value data. The first inference request is an inference request matched with the first model instance in the plurality of inference requests, and the first key-value data includes a key-value vector corresponding to the first input information. The second group of acceleration nodes is configured to take the first key-value data as input of the decoding task in the first model instance to obtain an inference result of the first inference request.

[0027] In a fourth aspect, the present application provides a computer program product. The computer program product comprises computer programs or instructions, which, when executed by an electronic device, implement the operation steps of the method provided in the first aspect or any of the optional implementation manners of the first aspect.

[0028] In a fifth aspect, the present application provides a computer readable storage medium. The computer readable storage medium comprises computer instructions. When the computer instructions are run in an electronic device, the electronic device implements the operation steps of the method provided in the first aspect or any of the optional implementation manners of the first aspect.

[0029] The beneficial effects of the second aspect to the fifth aspect can refer to the description of the first aspect or any of the optional implementation manners of the first aspect, which will not be repeated here. On the basis of the implementation manners provided in the above aspects, the present application can be further combined to provide more implementation manners. BRIEF DESCRIPTION OF DRAWINGS

[0030] FIG. 1 is a structural schematic diagram of a cluster provided by the present application.

[0031] FIG. 2 is a schematic diagram of distributed inference of a model provided by the present application.

[0032] FIG. 3 is a flow schematic diagram of a model inference method provided by the present application.

[0033] FIG. 4 is a flow schematic diagram of a model inference method provided by the present application.

[0034] FIG. 5 is a flow schematic diagram of a model inference method provided by the present application.

[0035] FIG. 6 is a flow schematic diagram of a model inference method provided by the present application.

[0036] FIG. 7 is a structural schematic diagram of a processing node provided by the present application. DETAILED DESCRIPTION

[0037] The technical scheme provided by the embodiments of the present application comprises: the processing node in the model inference system distinguishes the plurality of acceleration nodes, so that the acceleration nodes with different resources are used to perform different tasks in the model instance, which is conducive to maximizing the resources possessed by different acceleration nodes and improving the processing efficiency of the model inference system on a plurality of inference requests. Moreover, in the case that the model inference system processes a plurality of inference requests in batches, different acceleration nodes run model instances according to the resources possessed by the acceleration nodes, and the model instances process a plurality of inference requests in parallel, which is conducive to avoiding the problem that the inference card waits for the end of the inference task of the next inference request after the completion of the inference task of one inference request, thereby reducing the inference delay of the model inference system on a plurality of inference requests and improving the inference efficiency of the model instance.

[0038] The technical solutions provided in the present application can not only be applied to current model inference or distributed inference scenarios, but also can be applied to recommendation scenarios involving model inference, and can be applied to future model inference or model recommendation technologies. The terms used in the embodiments of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application. Some application scenarios and system architectures that can be involved in the present application will be exemplarily introduced below in conjunction with the accompanying drawings.

[0039] FIG. 1 is a structural schematic diagram of a cluster provided in the present application. As shown in FIG. 1, the cluster 100 includes a processing node 110 and a plurality of acceleration nodes, such as acceleration node i (i = 1, 2, 3,...), i.e., acceleration node 1 to acceleration node 12 shown in FIG. 1.

[0040] The hardware implementation of the processing node 110 and each acceleration node will be exemplarily described below in conjunction with the accompanying drawings.

[0041] In an optional example, the processing node 110 is a separate physical device. For example, the processing node 110 can be a terminal such as a computer, a mobile phone terminal, a tablet computer, a notebook computer, a virtual reality (VR) device, an augmented reality (AR) device, a mixed reality (MR) device, an extended reality (ER) device, a camera, or a vehicle-mounted computer, etc. computing device, and can also be an edge device (for example, a box with a processing-capable chip) and the like. In the present application, the processing node 110 can be a computing device connected with a base station, or a computing device deployed in a base station, such as a server or a cloud device, etc.

[0042] In another optional example, the processing node 110 is a control chip disposed on the physical device, which can be integrated on the physical device or installed on the physical device in a pluggable manner. For example, the processing node 110 can be a computing chip supporting integer calculation or floating point calculation, such as a central processing unit (CPU), a graphics processing unit (GPU) chip, a neural network processing unit (NPU) chip, a tensor processing unit (TPU) chip, a micro processing chip, a digital signal processor (DSP), an application-specific integrated circuit (ASIC) chip, or one or more integrated circuit chips for controlling the computer program product provided in the scheme.

[0043] The above two optional examples are only optional implementation manners of the processing node 110 provided by the embodiments of the present application, and should not be construed as a limitation of the present application. In FIG. 1, different acceleration nodes respectively communicate with the processing node 110, which can be connected or communicated by using, but not limited to, a leaf spine network, a single / multi-rail interconnection, and the like.

[0044] In an optional case, different acceleration nodes in the plurality of acceleration nodes have the same resources, i.e., the hardware configurations of the acceleration nodes are consistent. For example, the resources include, but are not limited to, computing resources, storage resources, and transmission resources. The computing resources are used to indicate the computing capability of the acceleration node, such as the number of integer operations or the number of floating point operations that can be executed per unit time. The storage resources are used to indicate the storage capacity and data read / write speed of the acceleration node, wherein the storage capacity refers to the amount of data that can be stored by the acceleration node, and the data read / write speed refers to the amount of data that can be read / written by the acceleration node per unit time. The transmission resources are used to indicate the amount of data that can be transmitted by the acceleration node per unit time.

[0045] In another optional case, different acceleration nodes in the plurality of acceleration nodes have different resources, i.e., the hardware configurations of the acceleration nodes are inconsistent. For example, the plurality of acceleration nodes can include, but are not limited to, inference cards or acceleration chips of multiple models.

[0046] The hardware implementation of each acceleration node shown in FIG. 1 is introduced as follows.

[0047] Take the acceleration node 1 as an example, the acceleration node 1 can be an acceleration device, an acceleration card or an acceleration chip.

[0048] In a first feasible example, the acceleration node 1 is an acceleration device. For example, the acceleration device can include but is not limited to a host, a server, a mobile terminal, a tablet computer, a notebook computer, a VR device, an AR device, an MR device, an ER device, a camera or a vehicle-mounted computer and the like.

[0049] In a second feasible example, the acceleration node 1 is an acceleration card. For example, the acceleration card can include but is not limited to a training card, an inference card, an adaptive accelerator card or other acceleration cards packaged with one or more chips. For example, the acceleration card can be integrated on a rack or a cabinet where the cluster 100 is located, or the acceleration card can be plugged into the rack or the cabinet where the cluster 100 is located, which is not limited in the present application.

[0050] In a third feasible example, the acceleration node 1 is an acceleration chip. For example, the acceleration chip can include but is not limited to a CPU, a data network processing unit (DPU) chip, a GPU chip, an NPU chip, a TPU chip, a micro processing chip, an ASIC chip, a DSP chip or other integrated circuit chips, etc. In some optional manners, a single acceleration chip can include one or more dies, which can also be referred to as a die, a crystal grain or other names. For example, the die can include but is not limited to a GPU in a GPU chip, an NPU in an NPU chip or a TPU in a TPU chip, etc. For example, if the die is a GPU in a GPU chip, the die can be used to perform mathematical and geometric calculations to achieve image rendering and the like.

[0051] Taking an acceleration node as an acceleration chip as an example, different acceleration nodes can communicate with each other through an inter-chip communication link, that is, chip to chip. For example, different acceleration chips are connected in one or more of the following ways: a high-speed custom communication system (HCCS) interface, a high-speed GPU interconnection bandwidth interface, an inter-integrated circuit (I2C) interface, a controller area network (CAN) bus, a serial peripheral interface (SPI), a queued serial peripheral interface (QSPI), a full-duplex asynchronous serial interface, a half-duplex differential serial interface, and the like. Among them, the HCCS interface is a high-speed connection channel between dies and dies, which is used to push data and computing acceleration to obtain executable results; for example, in the cluster 100, different acceleration chips are connected in pairs using the HCCS technology. The high-speed GPU interconnection bandwidth interface is a high-speed interconnection technology between GPUs, which is usually implemented through multiple pairs of wires printed on a computer board, and the two ends of a pair of wires are connected to different GPUs. The I2C bus is a source-synchronous serial bus used for short-distance communication between different integrated circuits; I2C uses two lines for data transmission: a serial data line (SDL) and a serial clock line (SCL). The CAN bus is a serial communication protocol bus for real-time applications, which can use twisted pair wires to transmit signals. The SPI bus is a 3-wire synchronous serial full-duplex communication interface, which has the advantages of simple circuit, high speed, reliable communication, and the like. The QSPI bus is based on the SPI and adds a queue transmission mechanism, and the QSPI uses a dedicated communication interface to connect single, double, or four data lines. The full-duplex asynchronous serial interface is also called a universal asynchronous receiver / transmitter (UART) interface, which is a universal serial data bus used for asynchronous communication; the UART bus can be a bidirectional communication bus, and the UART interface converts the data to be transmitted between serial communication and parallel communication, such as the RS-232 interface. The half-duplex differential serial interface is a serial communication bus interface, which adopts two-wire system, differential transmission and half-duplex mode, such as the RS-485 interface.

[0052] For example, each acceleration chip can be connected through a full mesh mode, i.e., different acceleration chips are directly connected. In some possible examples, each acceleration chip can also communicate through a newly built link, which is not limited in the present application.

[0053] The above description of the hardware implementation of the processing node 110 and each acceleration node is only an example provided by the embodiments of the present application and should not be construed as a limitation of the present application. In some cases, the processing node 110 and each acceleration node are deployed on different physical devices. In other cases, the processing node 110 and each acceleration node can be deployed on the same physical device, such as each acceleration node being set on a cabinet or rack with the processing node 110 through integration or pluggable manner.

[0054] As an optional implementation, the cluster 100 is configured to train the neural network using training data until the loss function in the neural network converges and the loss function value is less than a certain threshold, so that the neural network training is completed, and the neural network reaches a certain accuracy. Further, the cluster 100 configures the trained neural network to the cluster 100 or other distributed system. The cluster 100 or other distributed system is configured to implement training, inference or calculation operations on a request containing input information according to the trained neural network.

[0055] In the present embodiment, the above neural network can be referred to as an AI model or a model, which can refer to a large language model or other models, etc.

[0056] In an optional example, the neural network refers to a large language model (LLM). The large language model can effectively process massive data and provide efficient and accurate information processing and analysis services for users by virtue of its powerful computing power and complex algorithms. The large language model not only performs outstandingly in understanding and generating human language, but also exhibits great potential in solving complex problems and tasks. For example, the large language model is widely used in automated question and answer systems, text summary generation, machine translation, and language generation, greatly improving efficiency and accuracy. In particular, when processing large-scale data sets, the large language model can mine deep patterns and correlations from them to support decision-making. In addition, the self-learning ability of the large language model enables the large language model to continuously evolve, and the performance and intelligence of the large language model are continuously improved through continuous learning of new data. Commonly used large language models include the bidirectional encoder representations from transformers (BERT) model, which can be pre-trained on deep bidirectional representations using masked language modeling on unlabelled text. In some optional cases, the large language model can also be referred to as a large model. The large model provided by the embodiments of the present application can not only refer to a large language model, but also refer to a model with a certain number of model parameters. For example, depending on the field to which the large model is applied, the large model can also refer to a model that includes various functions such as image processing, human-computer interaction, semantic search, semantic query, or dialogue. The present application does not limit the field to which the large model can be applied or the specific name. In this paper, in order to facilitate description, the large model is named, but this should not be understood as a limitation of the present application, and will not be described in detail hereinafter. Because the large language model contains a large number of processing layers, it takes a long time to train the large language model using a single device, so the distributed training method is used to train the large language model. The learning framework applied by the LLM can include but is not limited to one or more of the following: Tensorflow, PyTorch, Mindspore, PaddlePaddle, Caffe, Theano. For the introduction of each learning framework, please refer to the content of the related technology, which will not be described here.

[0057] In an optional implementation, the neural network refers to other types of networks, such as convolutional neural networks (CNN), recurrent neural networks (RNN), or graph neural networks (GNN). For more implementations of CNN, RNN, or GNN, please refer to the description of the general technology, which will not be described here.

[0058] The scenarios to which the embodiments of the present application can be applied include, but are not limited to, model training, model inference, model recommendation, or other scenarios, etc. For example, the cluster 100 implements the target function of the neural network or model completed by training, such as identification or document verification in a business district, a school, a park, a sports venue, etc. in a city, or other functions. For another example, the neural network can implement some functions that can be achieved by a large language model (LLM), etc.

[0059] FIG. 1 is only a schematic diagram of a system architecture provided by the embodiments of the present application, and the positional relationship between the devices, components, modules, etc. shown in FIG. 1 does not constitute any limitation. According to the user's demand for model training, the cluster 100 can include more or fewer hardware components, which is not limited by the present application.

[0060] In the following, the inference process of the model is described by taking the model inference scenario as an example, and one or more processing layers of the model are deployed on each acceleration node in the cluster 100. For example, part or all of the model instances are deployed on each acceleration node. The model instance deployed on the acceleration node refers to an operable object obtained by instantiating the model. For example, an object that can operate on actual data is created according to the definition and parameters of the model.

[0061] It is assumed that the cluster 100 of the embodiments of the present application is used to infer an AI model based on a transformer architecture. In the following, a possible example is shown in FIG. 2, which is a schematic diagram of distributed inference of a model provided by the present application. Please refer to FIG. 2. The model includes multiple transformer modules, such as transformer module 1, transformer module 2, and transformer module n, etc., where n is a positive integer.

[0062] Different transformer modules rely on the cluster 100 shown in FIG. 1 or FIG. 2 to be implemented, and the attention layer corresponding to each transformer module can be inferred by one or more acceleration nodes, thereby improving the inference efficiency of the model. For example, different transformer modules in the model are respectively configured to different devices for inference. For example, the processing node 110 configures the transformer module 1 to the acceleration node 1, the transformer module 2 to the acceleration node 2, and the transformer module n to the acceleration node 5. Other transformer modules can also be configured to other acceleration nodes in the cluster 100, which is not described herein.

[0063] During the inference process of the model, the processing node 110 acquires an inference request, and in response to the inference request, establishes a link for data communication between different acceleration nodes. After the processing node 110 establishes the link for each acceleration node, the processing node 110 inputs the input information corresponding to the inference request to the model deployed on the different acceleration nodes, and processes the input information by the processing layer of the model deployed locally on each acceleration node to perform model inference.

[0064] In some optional implementations, the input information is a sequence, and the different acceleration nodes are used to infer the transformer module according to a part of the words in the sequence, and output the query result corresponding to the part of the words. It can be understood that different transformer modules in the model are inferred by different acceleration nodes or acceleration chips, which realizes the parallel inference process of the model.

[0065] The sequence is a kind of ordered data set, and each element in the sequence has a unique position. In some optional examples, the sequence is variable, and part or all of the elements can be modified; and the sequence can be traversed by iterating the elements in the sequence, and the device can also obtain a sub-sequence in the sequence by slicing. During the use of the sequence, one or more operations can also be performed on the sequence as needed, such as adding elements, deleting elements, sorting, searching, etc. For example, the elements in the sequence can include elements of any type, such as numbers, strings, objects, arrays, tuples, linked lists (or lists), tokens, stacks, and queues, etc. The string is a sequence composed of characters, which is commonly used in text processing, cryptography, image processing, etc. The array is a sequence composed of elements of the same type, which is commonly used to store a large amount of data and perform numerical calculations. The linked list is a sequence composed of nodes, which is commonly used to implement dynamic data structures and efficient insertion and deletion operations. The tuple is an immutable sequence, which is commonly used to package multiple values into a whole. The list is a variable sequence, which is commonly used to store and operate data. The stack and the queue are special sequences, which are commonly used to implement data structures and algorithms.

[0066] For example, the sequence can be a query sentence for the processing node 110 to generate semantic information according to a sequence of text or audio information, etc. The text or audio information can be input by a user, transmitted by another device, or generated by an application according to a locally configured inference data generation program of the cluster 100. Referring to FIG. 2, the query sentence includes a plurality of tokens, such as token 1 to token m corresponding to the black patterns. In this document, a token can include but is not limited to a word, a character, a punctuation mark, a special symbol (such as a calculation symbol, etc.), etc. In some optional cases, a word can also be referred to as a token, which is not limited in this application.

[0067] As the user's demand for the processing capability of the model gradually increases, the number of tokens in the query sentence increases, and the processing efficiency of the acceleration node is low when deploying the model in a single acceleration node to process a long sequence (i.e., a query sentence containing a large number of tokens). In this example, the inference process of the model is briefly described below taking the acceleration node 1 as an example. The processing node 110 randomly or sequentially allocates the query (Q) vector, the key (K) vector, and the value (V) vector corresponding to different tokens in the query sentence to different acceleration nodes. For example, the processing node 110 configures the Q, K, and V corresponding to token 1 in the query sentence to the acceleration node 1, and the acceleration node 1 performs the inference process of the transformer module 1 in the model according to the query vector stored locally in the acceleration node 1, the key vector and the value vector stored locally in the acceleration node 1, and the key vector and the value vector transmitted by other acceleration nodes or acceleration chips. For example, the acceleration node 1 obtains a first result after calculating the query vector stored locally in the acceleration node 1 with all key-value vectors associated with the query vector in the cluster 100, determines the loss function between the first result and the label of the token corresponding to the query vector stored locally in the acceleration node 1, and finally optimizes and updates the model parameters of the attention layer 1 deployed in the acceleration node 1 according to the loss function.

[0068] For possible implementation modes of the query vector (Q), the key vector (K), and the value vector (V), the following takes a token (such as token 1) in the query sentence as an example for description: the processing node 110 converts the token 1 into a vector to obtain the Q, K, and V corresponding to the token 1. For example, the process of converting the token 1 into a vector satisfies the following formulas (1) to (3). Q = KW Q Formula (1) K = XW K Formula (2) V = XW V Formula (3)

[0069] wherein X is an embedding representation corresponding to token 1, i.e., an abstract representation of mapping token 1 to a low-dimensional space; Q is a query vector, W Q is a weight matrix corresponding to the query vector, K is a key vector, W K is a weight matrix corresponding to the key vector, V is a value vector, W V is a weight matrix corresponding to the value vector.

[0070] In this document, for the sake of simplicity, the key vector and the value vector can be referred to as a key-value vector.

[0071] In some other optional implementations, the input information in FIG. 2 can also be other types of data, such as images, voices, or multi-modal data, etc. Multi-modal data refers to different types of data including text, images, videos, audio, etc. The research on multi-modal data is a fusion problem of processing and associating these data, aiming to provide more information for model decision-making and improve the accuracy of model processing. In addition, multi-modal data is unstructured and heterogeneous data. In the case where the input information is multi-modal data, the model inference method provided by the embodiments of the present application can be applied in the model running process in a complex scenario to improve the effect and efficiency of model interaction with the user.

[0072] Please continue to refer to FIG. 2. Different acceleration nodes are used for different transformer modules in the inference model, and the input data used for optimizing and updating the model parameters of different transformer modules are not completely the same, such as the query vectors and key-value vectors stored in different acceleration nodes. That is, multiple acceleration nodes are used to simultaneously infer different data, realizing the data parallel inference process of the model. Since the input data of the model is a query statement, and the query statement is a sequence, the data parallelism can also be referred to as sequence parallelism. Sequence parallelism refers to: dividing the input data of the model into multiple sub-data, and distributing different sub-data to different devices for calculation, so as to reduce the amount of calculation to be performed by a single device and improve the calculation efficiency of these devices on the input data of the model.

[0073] For example, the transformer module 1 includes an attention layer 1 and a multilayer perceptron (MLP) 1. The attention layer 1 is configured to calculate a query vector and a key value vector corresponding to input data 1 (e.g., tokens 1 and 2 in a query sentence), and finally output a result of the attention layer 1. The MLP 1 includes two linear layers, which are configured to perform linear processing on the result output by the attention layer 1 to obtain an output result (output data 1) of the transformer module 1, and update model parameters of the transformer module 1 based on the output result. The other transformer modules shown in FIG. 2 include the same contents and inference manners as the transformer module 1. For the specific network structure of the transformer module deployed in the acceleration node, the following is an exemplary description in combination with FIG. 2.

[0074] Referring to FIG. 2, the transformer module includes a prefill task and a decode task.

[0075] The preprocessing task can also be referred to as an encode task, and the prefill is used to process the input data and convert it into an internal representation (such as an encoding matrix) that the model can understand. The network layer corresponding to the preprocessing task is stacked by multiple identical layers, and the network structure of each layer includes two main substructures: a multi-head self-attention mechanism and a feed-forward neural network. As shown in FIG. 2: the network layer corresponding to the preprocessing task includes a multi-head self-attention layer 1, an Add&Norm (1) operation layer, a feed-forward layer (1), and an Add&Norm (2). Among them, the multi-head self-attention layer 1 is a network processing layer determined based on a multi-head self-attention mechanism, which is used to calculate the dot product between the Q corresponding to the token and the other K and V associated with the Q, to obtain the correlation score between each token and other tokens. The feed-forward layer (1) is a network processing layer determined based on a feed-forward neural network, which is used to perform forward feedback on the model parameters. Add&Norm (1) and Add&Norm (2) are used to prevent the network layer of the preprocessing task from degenerating and to normalize the activation values of different network layers, such as weighting and summing the correlation scores determined by the multi-head self-attention layer 1 to obtain a new representation of each token, such as an encoding matrix. In some feasible cases, a residual connection can also be provided before Add&Norm (1), and other types of processing layers can also be provided in the preprocessing task, which are not limited in the present application.

[0076] The decoding task generates output data (e.g., output data 1) according to the encoding matrix generated by the preprocessing task. The network layer corresponding to the decoding task includes a mask multi-head self-attention layer, Add&Norm (3), a multi-head self-attention layer 2, Add&Norm (4), a forward feedback layer (2), Add&Norm (5), a linear processing layer, and a normalization processing layer. The functions of Add&Norm (3) to Add&Norm (5) can be referred to the description of the preprocessing task, and will not be described here. The difference between the mask multi-head self-attention layer and the multi-head self-attention layer 2 is that the mask multi-head self-attention layer adopts a masked operation, the KV of the multi-head self-attention layer 2 uses the encoding matrix of the preprocessing task for calculation, and the Q uses the output of the previous decoding block for calculation. The previous decoding block of the multi-head self-attention layer 2 refers to the decoding task corresponding to another word before predicting the token 1. After the linear processing layer and the normalization processing layer perform linear processing and normalization operations, the predicted result corresponding to the token 1 is output, such as output data 1.

[0077] The structure of the transformer model shown in FIG. 2 enables the model to process sequence-to-sequence (seq2seq) tasks, such as machine translation.

[0078] On the basis of the cluster 100 shown in the foregoing embodiments, the model inference method provided by the embodiments of the present application is introduced. As shown in FIG. 3, FIG. 3 is a flowchart of a model inference method provided by the present application. The model inference method can be applied to a model inference system, such as the cluster 100 in the foregoing embodiments. Each acceleration node in the cluster 100 can include one or more processing layers of a model. The description of the model inference system (such as the cluster 100), the model, the model instance, and the processing layer can be referred to the description of the foregoing embodiments, and will not be described here. For example, the first model instance is an operable object obtained by instantiating the first model.

[0079] Referring to FIG. 3, the plurality of acceleration nodes include a first group of acceleration nodes matched with resources required by the preprocessing task in the first model instance, and a second group of acceleration nodes matched with resources required by the decoding task in the first model instance.

[0080] The first group of acceleration nodes includes one or more first acceleration nodes in the plurality of acceleration nodes of the model inference system (e.g., the cluster 100) that support a usage of a computing resource greater than or equal to a first threshold. The first threshold can be determined according to a combination of one or more of the following: a latency requirement of the pre-processing task of the first model instance, a data volume of the input information of the first model instance, or a number of acceleration nodes in the model inference system. For example, the first group of acceleration nodes includes the acceleration node 1 to the acceleration node 4.

[0081] The second group of acceleration nodes includes one or more second acceleration nodes in the plurality of acceleration nodes of the model inference system (e.g., the cluster 100) that support a usage of a transmission resource greater than or equal to a second threshold. The second threshold can be determined according to a combination of one or more of the following: a latency requirement of the decoding task of the first model instance, or a number of acceleration nodes in the model inference system. For example, the second group of acceleration nodes includes the acceleration node 9 to the acceleration node 12.

[0082] The first acceleration nodes and the second acceleration nodes can be different acceleration nodes, but in some possible implementations, some of the first acceleration nodes in the first group of acceleration nodes and some of the second acceleration nodes in the second group of acceleration nodes are the same, which is not limited in the present application.

[0083] It is worth noting that, in the first model instance, the computing resource required by the pre-processing task is greater than the computing resource required by the decoding task, and the transmission resource required by the pre-processing task is less than the transmission resource required by the decoding task, that is, the performance bottleneck of the prefill task is the computing power, and the bottleneck of the decode task is the bandwidth. Therefore, in an optional implementation, the first group of acceleration nodes has more computing resources than the second group of acceleration nodes, and the second group of acceleration nodes has more transmission resources than the first group of acceleration nodes.

[0084] In the embodiments of the present application, the model inference system distinguishes the plurality of acceleration nodes, and uses acceleration nodes with different resources to perform different tasks in the model instance, such as using the first group of acceleration nodes with more computing resources to perform the pre-processing task in the model instance, and using the second group of acceleration nodes with more transmission resources to perform the decoding task in the model instance, which is beneficial to maximize the resources of different acceleration nodes, and improve the processing efficiency of the model inference system for a plurality of inference requests.

[0085] Referring to FIG. 3, the model inference method provided by the embodiments of the present application includes the following S310 to S340.

[0086] S310, the processing node 110 acquires a plurality of inference requests.

[0087] Each inference request carries input information. As shown in FIG. 3, the plurality of inference requests include inference request 1 to inference request 4. The inference request 1 is also referred to as the first inference request, the inference request 2 is also referred to as the second inference request, the inference request 3 is also referred to as the third inference request, and the inference request 4 is also referred to as the fourth inference request. The number of inference requests that can be processed concurrently by the model inference system can be more or less, and the present application is not limited in this regard.

[0088] In some optional examples, the plurality of inference requests can be sent to the model inference system by the same user device.

[0089] In some other optional examples, the plurality of inference requests can be sent to the model inference system by a plurality of user devices.

[0090] The above two optional examples are only optional ways of obtaining inference requests provided by the embodiments of the present application, and should not be understood as a limitation of the present application. In some optional ways, the plurality of inference requests can also be inference requests to be processed in the prefill waiting queue stored in the model inference system. For example, the model inference system includes a memory, and when the model inference system receives an inference request from a user device, the inference request is stored in the prefill waiting queue in the memory according to the time when the inference request arrives at the model inference system or the priority of the inference request.

[0091] S320, the processing node 110 sends the first input information (input information 1) in the first inference request (inference request 1) to the first group of acceleration nodes to obtain the first group of cache data (cache data 1).

[0092] Corresponding to the process of S320, the first group of acceleration nodes receives the first input information sent by the processing node 110.

[0093] The first inference request (inference request 1) is an inference request in the plurality of inference requests that matches the first model instance. For example, the first inference request carries an inference identifier that matches the function that can be implemented by the first model instance.

[0094] The first group of cache data in S320 includes an intermediate vector corresponding to the first input information.

[0095] For example, if the input information included in the inference request 1 is a plurality of tokens, the first group of cache data is a key-value vector corresponding to the plurality of tokens, i.e., a KV cache. It is worth noting that the KV cache is cache data generated after the plurality of tokens input into the first model instance are preprocessed and calculated, and the KV cache will be used as input for a decoding task to participate in the calculation of an output inference result.

[0096] S330, the processing node 110 sends the first inference instruction to the second group of acceleration nodes.

[0097] The first inference instruction is used to instruct the second group of acceleration nodes to take the first group of cached data as the input of the decoding task in the first model instance, and obtain the inference result of the first inference request.

[0098] In an optional implementation, the processing node 110 directly sends the first inference instruction to the second group of acceleration nodes, and after the second group of acceleration nodes receives the first inference instruction, the first group of cached data corresponding to the inference request 1 is calculated and inferred, as S340.

[0099] In another optional implementation, the cluster 100 is provided with a collective communication component, which is used to maintain the communication link between the processing node 110 and the plurality of acceleration nodes, and transmit the instructions or data between the nodes. The collective communication component can be hardware or a software program.

[0100] In a feasible example, the collective communication component is hardware, such as an integrated circuit or a control circuit integrated on the processing node 110.

[0101] In another feasible example, the collective communication component is a software program, such as an application program, firmware or software unit deployed on the processing node 110.

[0102] The above two feasible examples are only optional ways of the collective communication component provided by the embodiments of the present application, and should not be understood as a limitation of the present application. In some optional implementations, the collective communication component can also be independently arranged on a hardware device outside the processing node 110 in the model processing system, such as a standby processing node, a management node in the model processing system or any acceleration node in the model processing system, which is not limited by the present application.

[0103] For example, the collective communication library used by the collective communication component includes but is not limited to: a collective communication library corresponding to a message passing interface (MPI), a Gloo TM , an NCCL( communications library), an HCCL( collective communications library) or other collective communication components, etc. Different collective communication libraries correspond to different adapters, such as MPI adapter, Gloo adapter, NCCL adapter and HCCL adapter.

[0104] MPI is a standard or specification representative, and the collective communication implemented by MPI has the following characteristics: ① Users exchange data between different nodes by explicitly sending or receiving messages, and each parallel node has its own independent address space, and access between different nodes cannot be directly performed, and needs to be implemented through explicit message passing; ② The model handles the parallel computing granularity of the system, which is suitable for large-scale scalable parallel algorithms.

[0105] Gloo TM is an open source library focused on collective communication, which provides algorithms including barrier, broadcast and allreduce for machine learning applications, allows the data transfer layer to flexibly switch between IP and InfiniBand, and supports GPUDirect technology to accelerate memory transfer between GPUs across nodes. The collective communication implemented by Gloo TM has the following characteristics: ① Data transfer is abstracted, whether in a pure IP network or in an InfiniBand environment, Gloo TM can provide good performance; ② GPU direct access, when using InfiniBand, GPU memory direct transfer is achieved through GPUDirect technology, reducing unnecessary data copying and improving transmission efficiency; ③ Memory buffer processing, Gloo TM supports both system memory buffers and GPU buffers, without the need for additional copy operations.

[0106] NCCL is a high-performance multi-GPU communication library that enables fast data transfer and collaborative computing between multiple GPUs. NCCL provides support for distributed training and data parallel acceleration in deep learning and high-performance computing. NCCL provides all-gather, all-reduce, broadcast, reduce, reduce-scatter, and point-to-point send and receive processes, which are optimized for PCIe and NVLink high-speed interconnects within nodes and high-bandwidth and low-latency between nodes.

[0107] HCCL is a high-performance collective communication library based on AI processors, providing single-machine multi-card active multi-card inter-card collective communication capabilities, supporting data parallelism, model parallelism, expert parallelism, pipeline parallelism, sequence parallelism and other acceleration schemes for large models. HCCL supports allreduce, broadcast, reduce-scatter, all-gather, all to all communication source domains, also supports Ring, Mesh, Halving-Doublin communication algorithms, supports collective communication based on HCCS, RoCE (allowing the use of remote direct memory access technology over Ethernet) and PCIe link / protocol implementation, and will support more links / protocols in the future.

[0108] The above two optional implementation manners are only optional manners adopted by the processing node 110 and the acceleration node in the model running process according to the embodiments of the present application, and should not be understood as a limitation of the present application.

[0109] S340, the second group of acceleration nodes take the first group of cached data as the input of the decoding task in the first model instance, and obtain the inference result of the first inference request.

[0110] As can be known from the model inference method shown in FIG. 3, in the case that the model inference system processes multiple inference requests in batches, different acceleration nodes run model instances according to the resources they have, and multiple inference requests are processed in parallel by the model instances, which is beneficial to avoid the problem that the inference card waits for the end of the inference task of the next inference request after the inference task of one inference request is completed, reduces the inference delay of the model inference system for multiple inference requests, and improves the inference efficiency of the model instance.

[0111] In order to avoid resource waste in the model inference system and improve the inference efficiency of the model inference system for multiple inference requests, the model inference method provided by the embodiments of the present application will be exemplarily described below taking inference request 2 as an example. As shown in FIG. 4, FIG. 4 is a flowchart of a model inference method provided by the present application. The hardware implementation of the model inference system shown in FIG. 4 can refer to the description of the foregoing embodiments. Different from the model inference system shown in FIG. 3, the model inference system shown in FIG. 4 further includes: a third group of acceleration nodes matched with the resources required by the preprocessing task in the second model instance, and a fourth group of acceleration nodes matched with the resources required by the decoding task in the second model instance.

[0112] For example, the third group of acceleration nodes includes acceleration node 5 to acceleration node 8, and the fourth group of acceleration nodes includes acceleration node 13 to acceleration node 16. The hardware implementation of acceleration node 13 to acceleration node 16 can refer to the description of each acceleration node in FIG. 1, which is not described here. It is worth noting that in the second model instance, the computing resources required by the preprocessing task are greater than the computing resources required by the decoding task, and the transmission resources required by the preprocessing task are less than the transmission resources required by the decoding task, that is, the performance bottleneck of the prefill task is computing power, and the bottleneck of the decode task is bandwidth. Therefore, in an optional implementation, the third group of acceleration nodes has more computing resources than the fourth group of acceleration nodes, and the fourth group of acceleration nodes has more transmission resources than the third group of acceleration nodes.

[0113] For the above-mentioned first model instance and second model instance, two possible scenarios are provided below.

[0114] In one possible scenario, the second model instance has the same function as the first model instance.

[0115] In another possible scenario, the second model instance has a different function from the first model instance.

[0116] For the specific way in which the second model instance has a different function from the first model instance, two optional examples are provided below.

[0117] In the first optional example, the first model instance is a language model, and the second model instance is an image model. For example, the first model instance is an operable object obtained after instantiating the language model, and the second model instance is an operable object obtained after instantiating the image model.

[0118] In the second optional example, the first model instance is an image model, and the second model instance is a language model. For example, the first model instance is an operable object obtained after instantiating the image model, and the second model instance is an operable object obtained after instantiating the language model.

[0119] The above two optional examples are only optional implementations of the first model instance and the second model instance provided by the embodiments of the present application, and should not be construed as a limitation of the present application. In other optional implementations, the first model instance and the second model instance can also be operable objects obtained by instantiating models with other functions, such as word processing models, object detection models, audio and video processing models, or enterprise architecture construction models, running models for generating plans, etc.

[0120] Referring to FIG. 4, after S310 described above, the model inference method provided by the embodiments of the present application further includes the following S350 to S370.

[0121] S350, the processing node 110 sends the second input information in the second inference request (inference request 2) to the third group of acceleration nodes to obtain the second group of cache data.

[0122] The second inference request is an inference request matched with the second model instance in the plurality of inference requests, and the second group of cache data includes an intermediate vector corresponding to the second input information. The second inference request (inference request 2) is an inference request matched with the second model instance in the plurality of inference requests. For example, the second inference request carries an inference identifier, and the inference identifier is matched with a function that can be implemented by the second model instance.

[0123] In an optional example, the input information included in the first inference request and the second inference request is different. For example, if the first input information includes a plurality of words, and the first group of cache data includes a key-value vector corresponding to the plurality of words, the second input information can include one or more images, and the second group of cache data can include a feature map corresponding to the one or more images.

[0124] S360, the processing node 110 sends the second inference instruction to the fourth group of acceleration nodes.

[0125] The second inference instruction is used to instruct the fourth group of acceleration nodes to take the second group of cache data as input of a decoding task in the second model instance to obtain an inference result of the second inference request.

[0126] S370, the fourth group of acceleration nodes takes the second group of cache data as input of a decoding task in the second model instance to obtain an inference result of the second inference request (inference request 2).

[0127] The specific implementation of S360 and S370 can refer to the description of S330 and S340, and will not be described here.

[0128] As can be known from the contents of S310 to S370, a plurality of different model instances can be deployed in the model inference system, different tasks are separated and arranged in different acceleration nodes, i.e., prefill / decode separation (P / D separation), and after the model inference system receives a plurality of inference requests, the acceleration nodes with different resources are matched for different model instances, so that in the case of batch processing requests in the model inference system, the model instances can be run by multi-node scheduling in the model inference system, thereby reducing the waiting time of the model inference system for processing batch inference requests and improving the efficiency of model inference.

[0129] For the specific implementation of S340 described above, two feasible specific implementation modes are provided in combination with FIG. 5 and FIG. 6.

[0130] FIG. 5 is a flowchart of a third model inference method provided by the present application. The specific implementation of each node in FIG. 5 can refer to the description of the foregoing embodiments, which will not be repeated here. The difference between the model inference system shown in FIG. 5 and the model inference system shown in FIG. 3 is that in the model inference system shown in FIG. 5, the second group of acceleration nodes further deploy an incremental prediction model. The incremental prediction model can predict multiple tokens in one inference and output the predicted multiple tokens.

[0131] In the embodiments of the present application, the incremental prediction model can include but is not limited to CNN, RNN, GNN or other types of network structures, etc. The difference between the incremental prediction model and the first model instance is that the model parameters of the incremental prediction model are smaller than the model parameters of the first model instance. Compared with the case where the first model instance is an LLM, the incremental prediction model can also be called a small model.

[0132] Referring to FIG. 5, in the model inference method provided by the embodiments of the present application, the process in which the second group of acceleration nodes obtains the inference result of the inference request 1 includes the following S510-S550.

[0133] S510, the second group of acceleration nodes inputs the first word in the first input information and the first group of cache data into the network layer corresponding to the decoding task in the first model instance, and outputs the inference value of the first word.

[0134] The specific implementation of the network layer corresponding to the decoding task in the first model instance can refer to the description of FIG. 2, which will not be repeated here.

[0135] Wherein, the first input information includes multiple words.

[0136] In the first feasible example, the first word is the first word in the multiple words. For example, the first word is token1.

[0137] In the second feasible example, the first word is an intermediate word in the multiple words. For example, the first word is any one of token2 to token(k-1), and k is the total number of words (tokens) in the first input information.

[0138] S520, the second group of acceleration nodes inputs the inference value of the first word and the first group of cache data into the incremental prediction model, and obtains the candidate value of N words in the first input information.

[0139] Wherein, the N words are the words adjacent to and after the first word in the first input information, and N≥2.

[0140] In a first possible example, if the first token is token1, the N tokens can be N tokens predicted by the incremental prediction model after token1. For example, N = 3, and the N tokens are token2, token3, and token4, which are candidate values of the N tokens predicted by the incremental prediction model.

[0141] In a second possible example, if the first token is token2, the N tokens can be N tokens predicted by the incremental prediction model after token2. For example, N = 3, and the N tokens are token3, token4, and token5, which are candidate values of the N tokens predicted by the incremental prediction model.

[0142] The above two possible examples are optional manners provided by the embodiments of the present application, and should not be construed as a limitation to the present application.

[0143] In S530, the second group of acceleration nodes input the N tokens and the first group of cache data into a network layer corresponding to the decoding task in the first model instance, and output predicted values of the N tokens.

[0144] Compared with the prior art in which one token can be predicted by one inference of the decoding task, the decoding task in the first model instance can predict N tokens at the same time, which is beneficial to improve the efficiency of model inference.

[0145] For example, N = 3, the first token is "I", and the predicted values of the three tokens predicted by the decoding task in the first model instance are "have", "a", and "peach".

[0146] In S540, the second group of acceleration nodes take values consistent between the candidate values of the N tokens and the predicted values of the N tokens as valid inference values of the N tokens.

[0147] For example, N = 3, the first token is "I", the candidate values of the N tokens are "have", "a", and "pear", and the predicted values of the N tokens are "have", "a", and "peach". The values of "have" and "a" are consistent, and thus the inference values of two tokens are "have" and "a".

[0148] S550, after determining the inference values of all the words in the plurality of words at the second group of acceleration nodes, weighting the inference values of all the words to obtain an inference result of the first inference request (inference request 1).

[0149] For example, the manner in which the second group of acceleration nodes weights the inference values of the words can be implemented by using a "concat" function in MySQL. In some optional manners, the second group of acceleration nodes can also use other manners to weight the inference values of the words, which are not limited in the present application.

[0150] As can be known from the contents of S510 to S550, based on the incremental prediction model, the plurality of words in the input information can be decoded in parallel, and the incremental prediction model is used to generate candidate words (candidate values) through multiple rounds of inference, and then the candidate words (candidate values) are verified through large model inference, so as to achieve the effect of generating multiple tokens through one inference.

[0151] FIG. 6 is a flowchart of a fourth example of a model inference method provided by the present application. The difference between the first model instance shown in FIG. 6 and the model shown in FIG. 2 is that in the first model instance shown in FIG. 6, the first model instance includes: an attention instance corresponding to a first attention layer, and a plurality of expert processing instances corresponding to a plurality of first expert processing layers located after the first attention layer.

[0152] For example, the first attention layer can be the attention layer 1 in the aforementioned transformer module 1, such as the first attention layer can include but is not limited to: a self-attention layer, a multi-head self-attention layer, a masked multi-head self-attention layer, or other attention layers (attention layers), etc., which are not limited in the present application. In FIG. 6, the attention instance corresponding to the first attention layer includes 4 attention instances (such as the white circular pattern corresponding to 1-4).

[0153] For another example, the first expert processing layer refers to a processing layer set according to the functions implemented by different model instances, such as the language model and the image model have different focuses in the model inference process, so the weights of the same expert processing layer are different. For example, the plurality of first expert processing layers correspond to 5 expert processing instances (such as the black circular pattern corresponding to 1-1 to 5-2).

[0154] For example, the first expert processing layer can refer to a processing layer obtained after adjustment or reconstruction of the aforementioned MLP 1, MLP 2, MLP n or other MLP in FIG. 2. The adjustment or reconstruction process can be implemented by a professional user, or can be obtained after processing of the MLP by the acceleration node or processing node according to statistical data, which is not limited in the present application.

[0155] In the second group of acceleration nodes, a plurality of second acceleration nodes are deployed with a plurality of attention layers and a plurality of expert processing layers of a plurality of model instances (including the first model instance), and the attention instances corresponding to the plurality of attention layers are connected to the expert processing instances corresponding to the plurality of expert processing layers in a mesh structure. For example, the connection relationship between the 4 attention instances and the 5 expert processing instances in FIG. 6 is in a mesh structure.

[0156] Referring to FIG. 6, in the model inference method provided by the embodiments of the present application, the process of obtaining the inference result of the first inference request by the second group of acceleration nodes includes the following S610 to S630.

[0157] S610, the second group of acceleration nodes obtains the expert weight set corresponding to the first inference request.

[0158] The expert weight set includes the weight of each first expert processing layer in the plurality of first expert processing layers. According to the different functional characteristics of different model instances, the weights of different expert processing layers in the expert weight set are different.

[0159] S620, the second group of acceleration nodes inputs the first cache data into the first attention layer to obtain the first attention.

[0160] S630, the second group of acceleration nodes inputs the first attention as the input of the plurality of first expert processing layers, and processes the first attention in combination with the weights in the expert weight set to obtain the inference result of the first inference request.

[0161] In the mixture of experts (MOE) scenario, after the model instance completes the attention, it will select a number of expert instances to continue inference according to the output attention. According to the distribution of a plurality of inference requests, some expert instances may actually be executed more frequently, and the execution frequency of some expert instances may be less. At this time, the model inference system can determine the number of instances of each expert instance (each expert instance is implemented by single card or multi-card tensor parallel) according to the statistical result, and connect all expert instances and all attention instances to form a mesh structure, so as to realize load balancing in the model inference system, thereby maximizing the efficiency of all acceleration nodes.

[0162] It can be understood that, in order to realize the functions in the above embodiments, the model inference system comprises a hardware structure and / or a software module corresponding to each function. Those skilled in the art should easily realize that, in combination with the units and method steps of each example described in the embodiments disclosed in the present application, the present application can be realized in the form of hardware or a combination of hardware and computer software. Whether a certain function is realized in the form of hardware or computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0163] The acceleration node and the model inference system provided by the embodiments of the present application can refer to the description of the foregoing embodiments, and will not be repeated here. Next, the processing node provided by the embodiments of the present application will be introduced in combination with the accompanying drawings. FIG. 7 is a structural schematic diagram of a processing node provided by the present application. The processing node 700 can be used to realize the functions of the processing node in the above method embodiments, and thus can also realize the beneficial effects possessed by the above method embodiments. In the present embodiment, the processing node can be the processing node 110 shown in FIG. 1, and can also be the processing node in the subsequent embodiments, or a module (such as a chip) applied to the processing node.

[0164] As shown in FIG. 7, the processing node 700 can comprise a processor 720. Optionally, the processing node 700 can further comprise a memory 730 and / or a communication interface 710. The processor 720 is coupled with the memory 730 and the communication interface 710, for example, through a communication bus. The communication bus can comprise, but is not limited to, a PCIe bus, or an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), etc.

[0165] Next, each component of the processing node 700 will be specifically introduced in combination with FIG. 7.

[0166] The processor 720 is the control center of the processing node 700, and can be one processor or a plurality of processing elements in general. For example, the processor 720 is one or more CPUs, and can also be an ASIC, or one or more integrated circuits configured to implement the embodiments of the present application, for example, one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs).

[0167] Optionally, the processor 720 can perform various functions of the processing node 700 by running or executing software programs stored in the memory 730, and invoking data stored in the memory 730. In a particular implementation, as an example, the processor 720 can include one or more CPUs.

[0168] Optionally, the processing node 700 can also include multiple processors. Each of the processors can be a single-CPU or a multi-CPU. The processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0169] The memory 730 is configured to store software programs for performing the model running chain building method in the solutions of the present application, and the processor 720 is configured to control the execution of the software programs. The specific implementation can refer to the model running chain building method in the method embodiments described above, and will not be described here. For example, the memory 730 can be a ROM or other types of static storage devices that can store static information and instructions, a RAM or other types of dynamic storage devices that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compact disc, a laser disc, an optical disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disc storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program codes in the form of instructions or data structures and can be accessed by a computer, but is not limited to this. The memory 730 can be integrated with the processor 720 or exist independently and coupled to the processor 720 through the interface circuit (not shown in FIG. 7) of the processing node 700, and the embodiments of the present application are not limited in this regard.

[0170] The communication interface 710 is configured to communicate with other devices. For example, the processing node 700 is a client or an application server, and the communication interface 710 can be configured to communicate with an acceleration chip or another processing node. For another example, the processing node 700 is a multi-core chip, and the communication interface 710 can be configured to communicate with another multi-core chip.

[0171] Optionally, the communication interface 710 can include a receiver and a transmitter (not shown separately in FIG. 7). Among them, the receiver is used to realize the receiving function, and the transmitter is used to realize the transmitting function. Optionally, the communication interface 710 can be integrated with the processor 720, or can exist independently and be coupled with the processor 720 through the interface circuit (not shown in FIG. 7) of the processing node 700. The embodiments of the present application do not make a specific limitation in this regard.

[0172] The method steps in the embodiments can be realized by hardware, or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in a RAM, a flash memory, a ROM, a PROM, an EPROM, an EEPROM, a register, a hard disk, a mobile hard disk, a CD-ROM, or any other form of storage medium well known in the art. An exemplary storage medium is coupled to a processor, so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a computing device. Of course, the processor and the storage medium can also exist as discrete components in a network device or a terminal device.

[0173] In the above embodiments, all or part of the embodiments can be realized by software, hardware, firmware, or any combination thereof. When realized by software, all or part of the embodiments can be realized in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer programs or instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments are executed. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, or other programmable devices. The computer programs or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer programs or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center through a wired or wireless manner. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that integrates one or more available media. The available media can be a magnetic medium, such as a floppy disk, a hard disk, a magnetic tape; an optical medium, such as a digital video disc (DVD); or a semiconductor medium, such as a solid state drive (SSD).

[0174] It is worth noting that if the computer program product can be provided to the user for use through a cloud service subscription mode, the user can choose different subscription levels according to needs; for example, the computer program product can also provide enterprise-level customized services with professional domain customization, interface personalization and extended functions according to the needs of users or enterprises. In addition, the computer program product provided by the present application capable of realizing the chain building method of the above model can also be made into value-added service to provide users, which is not limited by the present application. Or, the computer program product can also be embedded in eDataMate TM Or other large language model (large model) tool chain system, such as operator acceleration library, etc.

[0175] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A model inference method, comprising: The method is applied to a model inference system, the model inference system comprising a processing node and a plurality of acceleration nodes, the plurality of acceleration nodes comprising: a first group of acceleration nodes matched with resources required by a preprocessing task in a first model instance, and a second group of acceleration nodes matched with resources required by a decoding task in the first model instance, the resources comprising computing resources, transmission resources, and storage resources; The method comprises: The processing node obtains a plurality of inference requests, wherein each inference request carries input information; The processing node sends first input information in a first inference request to the first group of acceleration nodes to obtain first group of cached data; The first inference request is an inference request matched with the first model instance in the plurality of inference requests, and the first group of cached data comprises an intermediate vector corresponding to the first input information. The processing node sends a first inference instruction to the second group of acceleration nodes, the first inference instruction being used to instruct the second group of acceleration nodes to take the first group of cached data as input of the decoding task in the first model instance to obtain an inference result of the first inference request.

2. The method of claim 1, wherein, The first group of acceleration nodes has more computing resources than the second group of acceleration nodes, and the second group of acceleration nodes has more transmission resources than the first group of acceleration nodes.

3. The method according to claim 1 or 2, characterized in that, The first group of acceleration nodes comprises one or more first acceleration nodes in the plurality of acceleration nodes, which support using computing resources greater than or equal to a first threshold value. The second group of acceleration nodes comprises one or more second acceleration nodes in the plurality of acceleration nodes, which support using transmission resources greater than or equal to a second threshold value.

4. The method according to any one of claims 1-3, characterized in that, The plurality of acceleration nodes further comprises: a third group of acceleration nodes matched with resources required by a preprocessing task in a second model instance, and a fourth group of acceleration nodes matched with resources required by a decoding task in the second model instance, the second model instance being different from the first model instance in function. After obtaining the plurality of inference requests, the method comprises: The processing node sends second input information in a second inference request to the third group of acceleration nodes to obtain second group of cached data; The second inference request is an inference request matched with the second model instance in the plurality of inference requests, and the second group of cached data comprises an intermediate vector corresponding to the second input information. The processing node sends a second inference instruction to the fourth group of acceleration nodes, the second inference instruction being used to instruct the fourth group of acceleration nodes to take the second group of cached data as input of the decoding task in the second model instance to obtain an inference result of the second inference request.

5. The method according to any one of claims 1-4, characterized in that, The network layer corresponding to the preprocessing task comprises one or a combination of the following: an attention layer, a residual connection layer, a normalization operation layer, and a forward feedback layer. The first input information comprises a plurality of words; and the first group of cached data comprises a key-value vector corresponding to the plurality of words.

6. The method according to any one of claims 1-5, characterized in that, The second group of acceleration nodes further deploys an incremental prediction model; The second group of acceleration nodes input the first group of cache data as input of a decoding task in the first model instance, to obtain an inference result of the first inference request, including: The second group of acceleration nodes inputs the first word in the first input information and the first group of cache data into a network layer corresponding to the decoding task in the first model instance, to output an inference value of the first word; the first input information includes a plurality of words, and the first word is a first word or an intermediate word in the plurality of words; The second group of acceleration nodes inputs the inference value of the first word and the first group of cache data into the incremental prediction model, to obtain candidate values of N words in the first input information; wherein the N words are words adjacent to and after the first word in the first input information, and N≥2; The second group of acceleration nodes inputs the N words and the first group of cache data into the network layer corresponding to the decoding task in the first model instance, to output prediction values of the N words; The second group of acceleration nodes takes a value consistent between the candidate values of the N words and the prediction values of the N words as an effective inference value of the N words; After the second group of acceleration nodes determines the inference values of all the words in the plurality of words, the inference values of all the words are weighted to obtain the inference result of the first inference request.

7. The method of claim 4, wherein: the first model instance is a language model, and the second model instance is an image model; or the first model instance is an image model, and the second model instance is a language model. The first model instance includes a first attention layer and a plurality of first expert processing layers located after the first attention layer.

8. The method according to any one of claims 1-7, characterized in that, The second group of acceleration nodes input the first group of cache data as input of a decoding task in the first model instance, to obtain an inference result of the first inference request, including: The second group of acceleration nodes obtains a set of expert weights corresponding to the first inference request, the set of expert weights including a weight of each first expert processing layer in the plurality of first expert processing layers; The second group of acceleration nodes inputs the first group of cache data into the first attention layer, to obtain a first attention; The second group of acceleration nodes inputs the first attention as input of the plurality of first expert processing layers, and processes the first attention in combination with the weights in the set of expert weights, to obtain the inference result of the first inference request. The second group of acceleration nodes includes a plurality of second acceleration nodes in the plurality of acceleration nodes, which support a transmission resource greater than or equal to a second threshold value; 9. The method of claim 8, wherein, The plurality of second acceleration nodes are deployed with a plurality of attention layers and a plurality of expert processing layers of a plurality of model instances, the attention instances corresponding to the plurality of attention layers and the expert processing instances corresponding to the plurality of expert processing layers are connected in a mesh structure, and the plurality of model instances include the first model instance. The first input information is multi-modal data.

10. The method according to any one of claims 1-9, characterized in that, The acceleration node is an acceleration chip, an inference card, or an acceleration device.

11. The method according to any one of claims 1-10, characterized in that, including:

12. A processing node, comprising: a communication interface and a processor; ​ The communication interface is configured to obtain a plurality of inference requests, and the processor and the communication interface are configured to cooperatively perform the operation steps implemented by the processing node in the method of any one of claims 1-11.

13. A model inference system comprising: Comprising: a processing node and a plurality of acceleration nodes; The plurality of acceleration nodes comprises: a first group of acceleration nodes matched with resources required by a preprocessing task in a first model instance, and a second group of acceleration nodes matched with resources required by a decoding task in the first model instance, the resources including computing resources, transmission resources and storage resources; The processing node is configured to obtain a plurality of inference requests, wherein each inference request carries an input information; The first group of acceleration nodes is configured to take the first input information in the first inference request as an input of the preprocessing task in the first model instance to obtain a first group of key-value data; The first inference request is an inference request matched with the first model instance in the plurality of inference requests, and the first group of key-value data comprises a key-value vector corresponding to the first input information; The second group of acceleration nodes is configured to take the first group of key-value data as an input of the decoding task in the first model instance to obtain an inference result of the first inference request.

14. A computer program product, characterised in that, The computer program or instructions, when executed by an electronic device, implement the method of any one of claims 1-11.

15. A computer-readable storage medium, characterized in that, The computer readable storage medium comprises computer instructions; when the computer instructions are run in an electronic device, the electronic device performs the method of any one of claims 1-11.

Images