Model inference method, computing device, and related device

Abstract

The present application provides a model inference method, a computing device, and a related device. The method comprises the following steps: a computing device first allocates a key value cache to an inference model, wherein the key value cache comprises M blocks in a memory of a first processor and N blocks in a memory of a second processor in the computing device, each block comprises a plurality of mini blocks, the address interval between every two adjacent mini blocks is equal, and each block is configured to store, in the order of X layers comprised in the inference model, intermediate data generated by each layer. In this way, when the computing device needs to swap data of a first block among the M blocks to a second block among the N blocks, on the basis of a start address of the first block and a start address of the second block, the first processor can read out data of one block in a single operation by using an equidistant transfer interface or method, achieving high data swapping efficiency and low resource consumption, thereby improving model inference efficiency.

Description

A model inference method, computing device and related device

[0001] The present application claims priority to the Chinese patent application No. 202411081825.0, filed on August 7, 2024, with the State Intellectual Property Office of China, and entitled "A model inference method, computing device and related device", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence (AI), and in particular to a model inference method, computing device and related device. BACKGROUND

[0003] The intermediate results generated by a large language model (LLM) during inference need to be cached for quick access in subsequent inference processes, which is the key value cache (KV Cache) technology. The cache area for storing intermediate data is also called KV cache. Before model inference, the system allocates KV cache according to network layers, and applies a continuous memory for each network layer. Since the memory obtained each time is not necessarily continuous with the memory of the last application, the addresses of the continuous memories corresponding to different network layers are discontinuous and random. If a device side processes a large number of requests at the same time, a large amount of intermediate data needs to be stored in the KV cache of the device side, resulting in insufficient space on the device side. At this time, the KV cache data stored in the device side is swapped to the KV cache of the host side. When the device side needs to use the KV cache data of the host side, the KV cache data stored in the host side is swapped to the KV cache of the device side.

[0004] Generally, when swapping KV cache data between the host side and the device side, the swapping is usually performed in units of storage blocks. A block includes multiple mini blocks, each of which comes from the continuous memory of a different network layer. Since the continuous memories of different network layers are discontinuous and random, the addresses of these mini blocks in each block are also discrete and irregular. As a result, reading the data of a block requires multiple data reading operations according to the addresses of each mini block. Such multiple discrete data reading operations increase the system overhead, resulting in low data swapping efficiency and further affecting the latency of model inference. SUMMARY

[0005] The application provides a model inference method, a computing device and related devices, which are used to solve the problem of low exchange efficiency of KV cache data exchange in the model inference process, thereby affecting the model inference delay.

[0006] In a first aspect, a model inference method is provided. The method includes the following steps: a computing device allocates a key-value cache for an inference model, the key-value cache includes M storage blocks in the memory of a first processor of the computing device and N storage blocks in the memory of a second processor of the computing device, each storage block includes a plurality of sub-storage blocks, the address interval of adjacent two sub-storage blocks is equal, and each storage block is used to store intermediate data generated by each layer in the order of X layers included in the inference model; an exchange request is obtained, the exchange request is used to request exchange of data of a first storage block in the M storage blocks to a second storage block in the N storage blocks; an exchange instruction is generated according to the exchange request, and the exchange instruction is sent to the first processor, the exchange instruction includes the first address of the first storage block and the first address of the second storage block, and is used to instruct the first processor to read the intermediate data in the first storage block to the second storage block according to the first address of the first storage block, the first address of the second storage block and the address interval.

[0007] The method described in the first aspect is implemented. The computing device first allocates a key-value cache for the inference model, the key-value cache includes M storage blocks (blocks) in the memory of the first processor and N blocks in the memory of the second processor of the computing device, each block includes a plurality of sub-storage blocks (mini blocks), the address interval of adjacent two mini blocks is equal, and each block is used to store intermediate data generated by each layer in the order of X layers included in the inference model. Therefore, when the computing device obtains an exchange request for exchanging data of a first block of the M blocks to a second block of the N storage blocks, an exchange instruction including the first address of the first block and the first address of the second block can be generated according to the exchange request, and the exchange instruction is sent to the first processor, so that the first processor reads the data of the first block to the second block according to the first address of the first block, the first address of the second block and the address interval. Since the addresses of the mini blocks in each block are not discrete but equal address intervals, reading the data of a block does not require multiple addressing, and the data of a block can be read at one time based on the address interval, so that the data exchange efficiency is high, the resource consumption is low, and the efficiency of the model inference is improved.

[0008] In a possible implementation, when the key-value cache is allocated for the inference model, a first storage space with continuous addresses can be allocated in the memory of the first processor, a second storage space with continuous addresses can be allocated in the memory of the second processor, and continuous storage spaces are allocated for each layer in the first storage space and the second storage space in the order of the layers of the inference model, where each continuous storage space is used to store a K vector or a V vector of a layer, M and N are obtained, and the first addresses of the M storage blocks, the first addresses of the N storage blocks, and the address interval are determined according to M, N, and the size of the sub-storage blocks, where each storage block is used to store a K vector and a V vector of intermediate data of an X layer of the inference model.

[0009] Optionally, the K cache and the V cache of each network layer can be continuous, where each network layer includes a continuous storage space storing a K vector, referred to as a K cache, and a continuous storage space storing a V vector, referred to as a V cache. For example, a first continuous storage space stores a K vector of an L0 layer, a second continuous storage space stores a V vector of the L0 layer, a third continuous storage space stores a K vector of an L1 layer, and a fourth continuous storage space stores a V vector of the L1 layer, and so on. Alternatively, a first continuous storage space stores a V vector of an L0 layer, a second continuous storage space stores a K vector of the L0 layer, a third continuous storage space stores a V vector of an L1 layer, and a fourth continuous storage space stores a K vector of the L1 layer, and so on.

[0010] Optionally, the K cache of each network layer can be continuous, and the V cache can be continuous, for example, the first X continuous storage spaces store K vectors of X network layers, and the last X continuous storage spaces store V vectors of the X network layers. Alternatively, the first X continuous storage spaces store V vectors of X network layers, and the last X continuous storage spaces store K vectors of the X network layers.

[0011] It should be understood that, in the conventional manner, because the system applies memory in sequence according to network layers, the memory addresses between different network layers are not continuous, resulting in that multiple mini blocks in a block are discrete blocks. When data exchange is performed in the unit of a block, because the addresses of multiple mini blocks in the same block have nothing to do with each other and are completely random, the conventional manner needs to read each mini block in sequence according to the address of each mini block, and each reading needs to be readdressed, so that the data exchange efficiency is low and resource consumption is large.

[0012] The above implementation mode stores one layer of K vectors or V vectors in each continuous storage space. After allocating the continuous storage space corresponding to each network layer, the data in a block includes the K vectors and V vectors of X network layers according to the current block division mode. Since the addresses of the continuous storage spaces are also continuous, the tail address of the last mini block in each continuous storage space is connected to the head address of the first mini block in the next continuous storage space. If the multiple mini blocks in each continuous storage space are numbered in ascending order according to the address sequence, each block includes the mini blocks with the same number in each continuous storage space, for example, block0 includes the first mini block in each continuous storage space in the address sequence, block1 includes the second mini block in each continuous storage space in the address sequence, and so on, so that the address interval between the multiple mini blocks in a block is the same, that is, the length of 1 continuous storage space, that is, the length of M mini blocks. When data exchange is performed in units of blocks, the interface or method using equal-interval transfer can read the data of a block at a time, so that the data exchange efficiency is high, the resource consumption is low, and the model inference efficiency is improved.

[0013] In another possible implementation mode, when the key-value cache is allocated for the inference model, a first storage space with continuous addresses is allocated in the memory of the first processor, a second storage space with continuous addresses is allocated in the memory of the second processor, and continuous storage spaces are allocated for each layer in the first storage space and the second storage space in the order of the layers of the inference model, wherein each continuous storage space is used to store the K vectors and V vectors of one layer. M and N are obtained, and the head addresses of M storage blocks, the head addresses of N storage blocks, and the address interval are determined according to M, N, and the size of the sub-storage block. Each storage block is used to store the K vectors or V vectors of the intermediate data of X layers of the inference model.

[0014] In a specific implementation, the continuous storage space of each network layer includes the K vectors and V vectors of the network layer, which can be stored in the order of V vectors first and then K vectors, or in the order of K vectors first and then V vectors, and the application does not make a specific limitation.

[0015] It should be noted that in the industry general term of large language models, the data of one block usually includes K vectors and V vectors. Under this division manner, one block only has K vectors or V vectors, so under this division manner, the block IDs of the two blocks of the same network layer are the same. When data exchange is performed in units of blocks, the data of one block still includes K vectors and V vectors (for example, there are two block0s, the first block0 is called a K block and is used to store K vectors of X network layers, and the second block0 is called a V block and is used to store V vectors of X network layers), so that the adaptability of the technical solution of the present application is stronger and the landability is better.

[0016] In the above implementation manner, each continuous storage space stores K vectors and V vectors of one layer. After the corresponding continuous storage space of each network layer is allocated, each continuous storage space is divided into a plurality of address-continuous and same-size mini blocks according to the current block division manner, and the data in one block includes K vectors or V vectors of X network layers. Since the addresses between the continuous storage spaces are also continuous, the tail address of the last mini block of each continuous storage space is connected with the head address of the first mini block of the next continuous storage space. If the plurality of mini blocks in each continuous storage space are numbered in ascending order according to the address order, each block includes the mini blocks of the same number in each continuous storage space, so that the address interval between the plurality of mini blocks in one block is the same, that is, the length of one continuous storage space, that is, the length of 2M mini blocks. When data exchange is performed in units of blocks, the interface or method of equal-interval transfer can be used to quickly read the data of one block, so that the data exchange efficiency is high, the resource consumption is low, and the efficiency of model reasoning is improved.

[0017] It should be understood that the above exemplary manner is given for dividing the first storage space into a plurality of continuous storage spaces, and the continuous storage spaces can also be divided by other manners, which are not exemplified one by one here.

[0018] It should be noted that no matter how the first storage space is divided, the addresses between the plurality of continuous storage spaces obtained are continuous, so that the address interval between the plurality of mini blocks in each block is the same when data exchange is performed, so that the interface or method of equal-interval transfer can be used to quickly read the data of one block, so that the data exchange efficiency is high, the resource consumption is low, and the efficiency of model reasoning is improved.

[0019] In a possible implementation, when the exchange instruction is generated according to the exchange request, the identifier of the first storage block and the identifier of the second storage block can be obtained from the exchange request, and then the first address of the first storage block and the first address of the second storage block are obtained according to the identifier of the first storage block and the identifier of the second storage block, and the exchange instruction is generated.

[0020] In a specific implementation, the computing device can pre-store a first first address table and a second first address table. The first first address table stores the relationship between the first address of each block in the first processor and the identifier of the block, and the first address interval between adjacent mini blocks in each block. The second first address table stores the relationship between the second address of each block in the second processor and the identifier of the block, and the second address interval between adjacent mini blocks in each block. In this way, when the second processor obtains the exchange request, the first address of the first storage block and the first address interval can be determined according to the first first address table and the identifier of the first storage block, and the second address of the second storage block can be determined according to the second first address table and the identifier of the second storage block, so as to generate the exchange instruction.

[0021] The above implementation pre-stores the relationship between the first address of each block and the identifier of the block, and the corresponding address interval, so that when the computing device obtains the exchange request, the first address and the address interval can be determined according to the identifier of the block in the exchange request, so as to generate the exchange instruction. The first processor can transport data to the second storage block at equal intervals according to the exchange instruction, so that the exchange efficiency is high, the resource consumption is low, and the inference efficiency of the model is improved.

[0022] In a possible implementation, the length of the first storage space and the length of the second storage space are obtained according to model information and user configuration information. The model information includes the number X of network layers, and the user configuration information includes the number M of storage blocks in the memory of the first processor, the number N of storage blocks in the memory of the second processor, and the size b of the sub-storage block.

[0023] Optionally, if each continuous storage space stores K vectors or V vectors, a block includes K vectors or V vectors of X network layers, and the K cache and the V cache are continuous, the number of continuous storage spaces is 2X, the shape of the first storage space is (1, 2X, M, *v), indicating that one continuous storage space stores one kind of vector, such as a K vector or a V vector, the first storage space includes 2X continuous storage spaces, M blocks, each continuous storage space includes M miniblocks, each block includes 2X miniblocks, and each miniblock has a shape of *v. It should be understood that the user sets the shape of the miniblock as v, and when the storage block is applied, the shape of the miniblock can be adaptively adjusted, such as unfolding the dimension, so here it is represented by *v.

[0024] Optionally, if each continuous storage space stores K vectors and V vectors, a block includes K vectors or V vectors of X network layers, and the K cache and the V cache are continuous, the number of continuous storage spaces is 2X, the shape of the first storage space is (1, 2X, M, *v), indicating that one continuous storage space stores one kind of vector, such as a K vector or a V vector, the first storage space includes 2X continuous storage spaces, M blocks, each continuous storage space includes M miniblocks, each block includes 2X miniblocks, and each miniblock has a shape of *v. It should be understood that the user sets the shape of the miniblock as v, and when the storage block is applied, the shape of the miniblock can be adaptively adjusted, such as unfolding the dimension, so here it is represented by *v.

[0025] The above implementation is based on model information and user configuration to determine the length of the first storage space, and then applies the first storage space to the operating system, and then allocates the first storage space according to the network layer in sequence. Compared with the traditional technology, the traditional technology needs to apply memory for each K vector or V vector of the network layer according to the inference model, and if there are X network layers, the operating system needs to process 2X times of memory application. In the model inference method provided in the present application, the first processor determines the size and shape of the first storage space according to the user configuration information and the model information, and applies a block of continuous memory to store K vectors and V vectors of multiple network layers at one time, and the operating system only needs to process 1 time of memory application.

[0026] In a possible implementation, the first processor can determine the first address of the first storage block and the address interval according to the exchange instruction, and then read the first storage block from the first KV cache 210 using an interface or method of equal interval transfer.

[0027] If a continuous storage space stores a K vector or a V vector of a network layer, a block includes K vectors and V vectors of X network layers, and the address interval is the length Mb of M miniblocks, at this time, according to the address interval, 2X miniblocks can be read at one time to obtain the first storage block.

[0028] If a continuous storage space is used to store a K vector and a V vector of a network layer, a block includes K vectors or V vectors of X network layers, but the IDs of the blocks corresponding to the K vector and the V vector of the same network layer are the same, that is, a block includes a K block and a V block when data is exchanged, the K block includes a plurality of miniblocks storing K vectors, and the V block includes a plurality of miniblocks storing V vectors. In this case, if the address interval is 2M miniblocks, that is, 2Mb, 2 times of reading are required, the first time is to read the K block according to the first address of the K block, and the second time is to read the V block according to the first address of the V block to obtain the first storage block. If the address interval is M miniblocks, that is, Mb, 1 time of reading is required, and the data of the two blocks with the same ID, that is, the K block and the V block, is directly read to obtain the first storage block.

[0029] The above implementation manner, no matter what way is used to divide the first storage space, can read the data in the first storage block to the second storage block through 1 or 2 transfer operations according to the first address of the first storage block to be transferred and the address interval between adjacent miniblocks in the block. It should be understood that the conventional technology needs to read a plurality of discrete miniblocks and send them to the second processor. During data transmission, a plurality of messages need to be sent, and the message header of each message includes the source address and the destination address of each miniblock. Using the technical solution provided by the present application, since the addresses of the plurality of miniblocks read out are equidistant, and the destination addresses are also equidistant, the plurality of miniblocks in the first storage block can be packaged into one message, share one message header for transmission, so that the data transmission efficiency can be improved, the data exchange efficiency is high, the resource consumption is low, and the efficiency of model reasoning is improved.

[0030] In a possible implementation, the first processor is a general processor, and the second processor is an inference card, or the first processor is an inference card, and the second processor is a general processor, or the first processor and the second processor are both inference cards. For example, the first processor is an NPU, and the second processor is a CPU. When the NPU runs an inference model, the NPU stores intermediate data generated in the inference process in a KV cache of the NPU. When the NPU is insufficient in storage, the NPU can exchange data in a first storage block in the NPU to a second storage block in the CPU by using the model inference method provided in the present application.

[0031] In a second aspect, a computing device is provided. The computing device includes a first processor and a second processor. The first processor is configured to allocate a key-value cache for an inference model. The key-value cache includes M storage blocks in a memory of the first processor. Each storage block includes a plurality of sub-storage blocks. Adjacent two sub-storage blocks have equal address intervals. Each storage block is configured to store intermediate data generated by each layer in an order of X layers included in the inference model. The second processor is configured to allocate the key-value cache for the inference model. The key-value cache includes N storage blocks in a memory of the second processor. The second processor is configured to obtain an exchange request, generate an exchange instruction according to the exchange request, and send the exchange instruction to the first processor. The exchange request is configured to request to exchange data in a first storage block of the M storage blocks to a second storage block of the N storage blocks. The exchange instruction includes a first address of the first storage block and a second address of the second storage block. The first processor is configured to read the intermediate data in the first storage block to the second storage block according to the first address of the first storage block, the second address of the second storage block, and the address intervals.

[0032] The computing device implementing the second aspect is configured to assign a key-value cache to the inference model, the key-value cache including M blocks in the memory of the first processor and N blocks in the memory of the second processor of the computing device, each block including a plurality of mini blocks, adjacent two mini blocks having equal addresses and equal intervals, and each block being configured to store intermediate data generated by each layer of the X layers of the inference model in sequence, so that when the computing device obtains a swap request for swapping data of a first block of the M blocks to a second block of the N blocks, the computing device generates a swap instruction including a first address of the first block and a second address of the second block according to the swap request, and sends the swap instruction to the first processor, so that the first processor reads the data of the first block to the second block according to the first address of the first block, the second address of the second block, and the address interval, so that when the first processor and the second processor swap data in block units, the data of one block can be read at a time by using an interface or method of equal-interval transfer, so that the data swapping efficiency is high, the resource consumption is low, and the efficiency of model inference is improved.

[0033] In a possible implementation, the first processor is configured to allocate a first storage space in the memory of the first processor, the first storage space being address-continuous, and allocate a continuous storage space for each layer of the inference model in sequence in the first storage space, wherein each continuous storage space is configured to store a K vector or a V vector of one layer, obtain M, and determine a first address and an address interval of M blocks according to M and a size of a mini block, and each storage block is configured to store K vectors and V vectors of intermediate data of X layers of the inference model, the second processor is configured to allocate a second storage space in the memory of the second processor, the second storage space being address-continuous, and allocate a continuous storage space for each layer of the inference model in sequence in the second storage space, wherein each continuous storage space is configured to store a K vector or a V vector of one layer, obtain N, and determine a first address and an address interval of N blocks according to N and the size of the mini block, and each storage block is configured to store K vectors and V vectors of intermediate data of X layers of the inference model.

[0034] In another possible implementation, the first processor is configured to allocate a first storage space with a continuous address in a memory of the first processor, allocate a continuous storage space for each layer of the inference model in the first storage space according to an order of the layers, and store a K vector and a V vector of each layer in each continuous storage space, obtain M, and determine a first address and an address interval of M storage blocks according to the M and a size of a sub-storage block, wherein each storage block is used to store a K vector or a V vector of intermediate data of X layers of the inference model, and the second processor is configured to allocate a second storage space with a continuous address in a memory of the second processor, allocate a continuous storage space for each layer of the inference model in the second storage space according to the order of the layers, and store a K vector and a V vector of each layer in each continuous storage space, obtain N, and determine a first address and an address interval of N storage blocks according to the N and the size of the sub-storage block, wherein each storage block is used to store a K vector or a V vector of intermediate data of X layers of the inference model.

[0035] In a possible implementation, the first processor is a general processor, and the second processor is an inference card, or the first processor is an inference card, and the second processor is a general processor, or the first processor and the second processor are both inference cards.

[0036] In a possible implementation, the second processor is configured to obtain an identifier of the first storage block and an identifier of the second storage block from the exchange request, obtain a first address of the first storage block and a first address of the second storage block according to the identifier of the first storage block and the identifier of the second storage block, and generate the exchange instruction.

[0037] In a possible implementation, the length of the first storage space and the length of the second storage space are obtained according to model information and user configuration information, the model information includes a number X of network layers, and the user configuration information includes a number M of storage blocks in the memory of the first processor, a number N of storage blocks in the memory of the second processor, and a size b of a sub-storage block.

[0038] In a third aspect, a computing device is provided, which includes a first processor, a second processor, and a memory, the memory is configured to store instructions, and the second processor is configured to execute the instructions to enable the computing device to implement the method described in the first aspect.

[0039] In a fourth aspect, a computer readable storage medium is provided, which stores instructions, and the instructions are executed by a computing device or a computing device cluster to implement the method described in the first aspect.

[0040] In a fifth aspect, a computing device cluster is provided, the computing device cluster comprising at least one computing device, each of the at least one computing device comprising a processor and a memory, the processor of the at least one computing device configured to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to implement the method described in the first aspect.

[0041] In a sixth aspect, a computer program product comprising instructions is provided, the computer program product comprising instructions that are executable on a computing device or stored in any available medium or software or program product, when the computer program product is executed on the computing device or the computing device cluster, causing the computing device or the computing device cluster to perform the method described in the first aspect.

[0042] On the basis of the implementation manners of the above aspects, the present application can be further combined to provide more implementation manners. BRIEF DESCRIPTION OF DRAWINGS

[0043] FIG. 1 is an example diagram of storage block division of a KV cache in a conventional scheme;

[0044] FIG. 2 is an architecture diagram of an inference system provided by the present application;

[0045] FIG. 3 is an example diagram of a computing device provided by the present application deployed in a cloud environment;

[0046] FIG. 4 is an example diagram of another division manner provided by the present application;

[0047] FIG. 5 is a step flow diagram of a model inference method provided by the present application in an initialization stage;

[0048] FIG. 6 is a step flow diagram of a model inference method provided by the present application in an application stage;

[0049] FIG. 7 is a software structure diagram of a computing device provided by the present application;

[0050] FIG. 8 is a hardware structure diagram of a computing device provided by the present application;

[0051] FIG. 9 is an example diagram of a computing device cluster provided by the present application. DETAILED DESCRIPTION

[0052] First, the application scenario involved in the present application is explained.

[0053] LLM is a deep learning-based natural language processing technology that can simulate human language capabilities to generate coherent text, and is a popular research topic in the current AI field. LLM is an autoregressive model that performs multiple forward calculations during inference, generating a predicted word each time. Multiple predicted words form the output text. During each forward calculation, the first n predicted words are used as input to predict the n+1 predicted word. Therefore, the partial intermediate data from the previous forward calculation can be reused by the next forward calculation. If the intermediate data from each forward calculation is cached, the problem of repeated calculation during the next forward calculation can be avoided, thereby improving the efficiency of model inference. This technology is also known as key value cache (KV Cache) technology, and the cache area used to store intermediate data is called KV cache.

[0054] However, after using the KV cache technology to process multiple requests, the KV cache will frequently allocate and release, resulting in small, discontinuous free blocks in the KV cache, causing memory fragmentation problems, thereby limiting the allocation of subsequent large blocks of memory and affecting the memory utilization efficiency of the system. To solve this problem, the KV cache is usually divided into multiple storage blocks, and each allocation and release operation is performed in units of storage blocks. This not only avoids the generation of a large number of small fragments, but also allows the system to allocate or release storage blocks on demand based on usage, optimizing memory usage and improving the performance of large model inference.

[0055] FIG. 1 is an example diagram of storage block division of a KV cache in a conventional scheme. As shown in FIG. 1, the KV cache is divided into N storage blocks (blocks), each block including multiple small storage blocks (mini blocks), each mini block being the smallest unit of memory allocation. The mini blocks in a column belong to the same block, the mini blocks in a row belong to different blocks, and the multiple mini blocks in the same row are continuous free blocks represented by the same color, and the mini blocks in different rows are discontinuous free blocks represented by different colors.

[0056] Before model inference, the system allocates KV cache according to network layers, and applies a continuous memory for each network layer. Since the memory obtained each time is not necessarily continuous with the memory obtained last time, the KV cache is not limited, and the allocation is more flexible. Therefore, the addresses of the continuous memory corresponding to different network layers are discontinuous and random. After each network layer obtains the corresponding continuous memory, it will split the continuous memory into multiple miniblocks according to the size of the miniblock. The final KV cache can be as shown in FIG. 1. The intermediate data of each network layer includes a key (K) vector and a value (V) vector. The multiple miniblocks in the first row are used to store the K vector of the L2 layer, the multiple miniblocks in the second row are used to store the V vector of the L1 layer, and so on. Since the continuous memory corresponding to each network layer is discontinuous, the address of the last miniblock in each row in FIG. 1 is not continuous with the address of the first miniblock in the next row, and there is no any correlation. In this way, the intermediate data of each network layer can be allocated and released on demand in units of miniblocks, thereby avoiding the generation of small fragments of different granularities and improving the utilization rate of memory.

[0057] When the model starts to infer, if the system processes a large number of requests at the same time, the V vector generated by the L1 layer of each request will be stored in the multiple miniblocks in the second row, and the K vector generated by the L2 layer of each request will be stored in the multiple miniblocks in the first row. The capacity of the KV cache limits the concurrency capability of the model. In order to solve this problem, the device running the inference model will swap the data of the entire block to other devices in the case of insufficient space. Usually, we call the device running the inference model as the device side, and other devices as the host side. When the device side is insufficient in space, the data of the KV cache in the device side will be swapped to the KV cache in the host side. The second processor side also has a KV cache as shown in FIG. 1. Here, the KV cache in the host side will not be explained again.

[0058] In data exchange, in order to improve the exchange efficiency, the device side will exchange a large number of miniblocks that will not be used temporarily to the host side, in order to facilitate memory management, data exchange is usually carried out in block units, that is, a column of data shown in FIG. 1 is exchanged out. For example, block0 in FIG. 1 includes L2K, L1V, L5V,..., L(n-1)K miniblocks. It should be understood that the data exchange is carried out in column (block) units because the miniblocks in the same row store K vectors or V vectors of the same network layer of multiple requests, and if the data exchange is carried out in row units, the K and V vectors of the row of the request currently being processed will also be exchanged out, and the column exchange can avoid such problems.

[0059] However, as known from the foregoing, each block includes multiple miniblocks, and the miniblocks store K and V vectors of different network layers, that is, each block includes multiple discontinuous and discrete small storage blocks, so that the device needs to perform a large number of discrete reading operations when data exchange, and each reading of a miniblock needs to perform a data reading operation, and the reading address of each data reading operation is different, so that the discrete reading of miniblocks increases the system overhead and reduces the data exchange efficiency. Similarly, if the device side needs the host side to exchange the data of the block back to the device side, the host side also needs to read a large number of miniblocks in such a discrete manner, which increases the host system overhead and reduces the data exchange efficiency.

[0060] As known from the foregoing, in the inference process of the inference model, when the KV cache space of the device side is insufficient and the KV cache data of the device side needs to be exchanged to the host side, data exchange is usually carried out in block units, but a block includes multiple address discontinuous and discrete miniblocks, and the discrete reading of a large number of miniblocks increases the system overhead and reduces the data exchange efficiency, thereby affecting the latency of model inference.

[0061] In order to solve the problem that the exchange efficiency of the KV cache data exchange between the device side and the host side in the inference process of the inference model is low, thereby affecting the model inference delay, the present application provides a model inference method, in which the computing device first allocates a key-value cache for the inference model, the key-value cache includes M storage blocks (blocks) in the memory of the first processor and N blocks in the memory of the second processor of the computing device, each block includes a plurality of mini blocks, the addresses and intervals of adjacent two mini blocks are equal, and each block is used to store the intermediate data generated by each layer in the order of X layers included in the inference model. In this way, when the computing device obtains an exchange request for exchanging the data of a first block of the M blocks to a second block of the N storage blocks, the exchange instruction including the first address of the first block and the first address of the second block can be generated according to the exchange request, and the exchange instruction is sent to the first processor, so that the first processor reads the data of the first block to the second block according to the first address of the first block, the first address of the second block and the address interval. When the first processor and the second processor exchange data in block units, the interface or method using equal-interval transfer can read the data of one block at a time, so that the data exchange efficiency is high, the resource consumption is low, and the efficiency of model inference is improved.

[0062] FIG. 2 is an architecture diagram of an inference system provided by the present application, the inference system 1000 includes a terminal device 100 and a computing device 400, wherein the terminal device 100 and the computing device 400 establish a communication connection through a network, which can be a wired connection or a wireless connection, the network can be a public Internet, can also be an internal LAN, can also be a virtual private network (VPN), can also be a dedicated line such as a fiber line, a copper line, a satellite connection, etc., can also be a wireless network such as a wireless fidelity (Wi-Fi) network, a cellular network, etc., and the present application does not make specific limitations. The number of terminal devices 100 and computing devices 400 in the inference system 1000 can be one or more, and FIG. 2 takes four terminal devices 100 and one computing device 400 as an example for illustration, and the present application does not make specific limitations.

[0063] The terminal device 100 is used to realize human-computer interaction, and a user holding the terminal device 100 is a user who completes reasoning by using the reasoning model. The terminal device includes a personal computer, a smart phone, a wearable device, a palm-held processing device, a tablet computer, a mobile notebook, an augmented reality (AR) device, a virtual reality (VR) device, a smart conference device, and the like, which are not specifically limited herein. The description of the computing device can refer to the foregoing, which is not repeated here.

[0064] The computing device 400 can include a bare metal server (BMS), a virtual machine, a container, or a storage device. The BMS refers to a general-purpose physical server, for example, an ARM server or an X86 server; the virtual machine refers to a complete computer system that is simulated by software, runs in a completely isolated environment, and has complete hardware system functions. The work that can be completed in an entity computer can be realized in the virtual machine. When a virtual machine is created in the computing device, part of the hard disk and memory capacity of the entity machine needs to be used as the hard disk and memory capacity of the virtual machine. Each virtual machine has an independent basic input / output system (BIOS), a hard disk, and an operating system, and can be operated like an entity machine; the container is a portable software unit that can combine an application and all its dependencies into a software package, which is not limited by the underlying host operating system, so that a complex environment does not need to be constructed again, and the process of application development to deployment is simplified.

[0065] Optionally, the terminal device 100 and the computing device 400 can also be the same computing device, which is not specifically limited herein.

[0066] Further, the computing device 400 can include the first processor 200 and the second processor 300. Optionally, the first processor 200 is a general-purpose processor, and the second processor 300 is an inference card, or the first processor 200 is an inference card, and the second processor 300 is a general-purpose processor, or the first processor 200 and the second processor 300 are both inference cards. It should be noted that the number of the first processor 200 and the second processor 300 can be one or more, and FIG. 2 takes the computing device 400 including one first processor 200 and one second processor 300 as an example for description, which is not specifically limited herein.

[0067] In specific implementation, the inference card refers to a special processor of a neural network algorithm, such as a graphics processing unit (GPU), a neural network processing unit (NPU), a data processing unit (DPU), and the like, which is not limited in the present application. In some scenarios, the inference card is also referred to as an acceleration card.

[0068] In specific implementation, the general processor can be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a micro processor (MP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a complex programmable logic device (CPLD), and the like, which is not limited in the present application.

[0069] The computing device 400 can also be a computing device cluster composed of multiple computing devices, and the first processor 200 and the second processor 300 can be deployed in the same computing device or different computing devices; if deployed in different computing devices, the computing devices where the first processor 200 and the second processor 300 are located can belong to the same or different computing device clusters. FIG. 2 illustrates the case where the first processor 200 and the second processor 300 are in the same computing device 400.

[0070] Optionally, the terminal device 100 can be a terminal device controlled by a user, and a client can run on the terminal device, which can be a software or an application program, such as a personal computer (PC) client, a world wide web (web) client based on browser access, an application (APP) client running on a mobile terminal, or a console of a cloud platform, which is not limited in the present application.

[0071] Optionally, the client running on the terminal device 100 can be a client specially used to implement a machine learning inference platform based on an inference model, or a client of a model service platform, or a client running a model inference framework, which provides a series of tools and services for deployment, management and inference of machine learning models. Taking a large language model as an example, the client can be ModelArts, OpenAI, etc., which are not limited in the present application. The large language model can include but is not limited to GPT, Bing Chat, Copilot, Gemini, etc., which are not limited in the present application. The inference framework can include faster transformer, text generation inference (TGI), vectorized large language model (vLLM), etc., which are not limited in the present application.

[0072] Optionally, the client running on the terminal device 100 can also be a comprehensive client including a model inference function. Such a client not only includes a model inference function, but also can include other functions, which are determined according to the actual application scenario of the client. For example, the client of an after-sales robot can use the inference function of a large language model to obtain the output text corresponding to the after-sales consultation question input by the user, and can also provide other functions such as order processing, return, account management, etc. For example, the client of a chat robot can use the inference function of a large language model to obtain the output text corresponding to the input text input by the user, and can also provide other functions such as playing music, playing movies, recommending news, etc. It should be understood that the large language model is widely used due to its powerful natural language generation capability, and its model inference capability can be integrated into comprehensive clients with different functions. The above examples are used for illustration, and the present application is not limited in this regard.

[0073] Optionally, the client running on the terminal device 100 can also be a client of a cloud platform, which is used to purchase and rent various cloud services by users. The model inference method provided by the present application can be one of the cloud services, and the user can separately purchase the cloud service to accelerate the data exchange efficiency of model inference, thereby accelerating the model inference efficiency. Alternatively, the cloud platform provides a comprehensive service to the user, such as an inference cloud service of a large language model. After the user purchases the cloud service, the large language model runs using the model inference method provided by the present application to implement model inference.

[0074] The possible implementation manners of the terminal device 100 and the computing device 400 are described above respectively, and the implementation manners of the terminal device 100 and the computing device 400 are exemplarily illustrated in combination with specific application scenarios as follows.

[0075] For example, it is assumed that the client running on the terminal device 100 is an office assistant of an enterprise, and an enterprise employee can use the office assistant to perform real-time question answering, technical support, data analysis and the like, in this scenario, the computing device 400 is a server purchased by the enterprise, and the terminal device 100 is an office computer of the enterprise or a mobile phone of the employee, the enterprise employee can use the terminal device 100 to run the office assistant client, input a question through the office assistant client, the terminal device 100 sends the question to the computing device 400, the computing device 400 can run a large language model to obtain a reply text corresponding to the question, and then returns the reply text to the terminal device 100, and the terminal device 100 displays the reply text in the office assistant client. Wherein, when the computing device 400 runs the large language model to obtain the reply text corresponding to the question, the computing device 400 can use the model inference method provided in the present application.

[0076] In another application scenario, the computing device 400 can be deployed in a cloud environment, for example, FIG. 3 is an example diagram of a computing device deployed in a cloud environment provided by the present application, as shown in FIG. 3, a user can initiate a purchase request of a large language model inference cloud service through the terminal device 100, after the terminal device 100 sends the purchase request to the cloud platform, the cloud platform can provide the cloud service use right of the computing device 400 to the terminal device 100, so that the user can send an input text to the computing device 400 through the terminal device 100, the computing device 400 runs a large language model to infer an output text corresponding to the input text. During the model running process, if the storage space of the first processor 200 is insufficient, part of the data in the KV cache needs to be exchanged to the second processor 300, at this time, the model inference method provided in the present application can be executed to realize it.

[0077] Wherein, the cloud platform also maintains various basic resources, including computing resources, storage resources, network resources, security resources and the like, to meet the computing requirements of the computing device 400 under different scales and loads, and these computing resources can be dynamically scaled according to the use requirements of the computing device 400, to ensure the stable operation of the computing device 400.

[0078] It should be understood that the above application scenarios are used for illustration, and the computing device 400 and the terminal device 100 can be flexibly deployed according to actual business requirements, which are not illustrated one by one here.

[0079] In the embodiments of the present application, the first processor 200 is the device side in the foregoing content, and the second processor 300 is the host side in the foregoing content. The second processor 300 can receive an input text sent by a user. The second processor 300 can establish a communication connection with a plurality of first processors 200, and then select one first processor 200 from the plurality of first processors 200 to send the input text to the first processor 200 for processing. The first processor 200 can run a large language model to infer an output text corresponding to the input text. During the inference process, if the KV cache space in the first processor 200 is insufficient, the data in the KV cache can be swapped to the second processor 300. The second processor 300 swaps the KV cache data required by the first processor 200 back to the first processor 200.

[0080] In the embodiments of the present application, when the computing device allocates the KV cache for the inference model, M blocks of memory are allocated for the model in the first processor 200, and N blocks of memory are allocated for the model in the second processor. Each block includes a plurality of mini blocks, and the address interval between adjacent two mini blocks is equal. Each block is used to store the intermediate data generated by each layer in the order of X layers included in the inference model.

[0081] Specifically, a first storage space (that is, the first KV cache 210 in FIG. 2) of a continuous address is allocated in the memory of the first processor 200, and a second storage space (that is, the second KV cache 310 in FIG. 2) of a continuous address is allocated in the memory of the second processor 300. Then, a continuous storage space is allocated for each layer in the order of the layers of the inference model in the first storage space and the second storage space, respectively. Each continuous storage space is used to store a K vector or a V vector of a layer. Then, M and N are obtained, and the first address of the M storage blocks, the first address of the N storage blocks, and the address interval of adjacent mini blocks in each block are determined according to M, N, and the size of the mini block. Each block is used to store the K vector and the V vector of the intermediate data of X layers of the inference model.

[0082] Further, the first processor 200 stores a first head address table 220, and the second processor 300 stores a second head address table 320. The first head address table is used to store the head addresses of the M blocks in the first processor 200, and the second head address table 320 is used to store the head addresses of the N blocks in the second processor 300. In this way, when the first processor 200 and the second processor 300 exchange data in units of blocks, the first processor 200 can determine the head address of a first target block according to the first head address table 220, and then read the data of multiple mini blocks in the first target block into the second processor 300 at one time according to the address interval between the mini blocks. Similarly, the second processor 300 can determine the head address of a second target block according to the second head address table 320, and then read the data of multiple mini blocks in the second target block into the first processor 200 at one time according to the address interval between the mini blocks. In this way, a large number of discrete read operations are avoided, the system overhead of data exchange is reduced, and the efficiency of data exchange is improved.

[0083] In a possible implementation, each continuous storage space is used to store a K vector or a V vector of a network layer, and each block is used to store a K vector and a V vector of intermediate data of an X layer of the inference model.

[0084] Specifically, the computing device 400 can allocate a continuous first storage space in the memory of the first processor 200, and then allocate a continuous storage space for each network layer in the order of the network layers of the inference model. Each continuous storage space of each network layer is used to store a K vector or a V vector of the network layer. The computing device can determine the head addresses and the address interval of the M storage blocks according to M and the size of the mini block, and update the first head address table 220.

[0085] After the continuous storage space corresponding to each network layer is allocated, each continuous storage space is divided into a plurality of address-continuous and same-size mini blocks according to the current block division manner, and the data in one block includes K vectors and V vectors of X network layers. Since the addresses between the continuous storage spaces are also continuous, the tail address of the last mini block of each continuous storage space is connected with the head address of the first mini block of the next continuous storage space. If the plurality of mini blocks in each continuous storage space are numbered in ascending order according to the address sequence, each block includes the same-numbered mini blocks in each continuous storage space, for example, block0 includes the first mini block in each continuous storage space in the address sequence, block1 includes the second mini block in each continuous storage space in the address sequence, and so on, so that the address interval between the plurality of mini blocks in one block is the same, that is, the length of 1 continuous storage space, that is, the length of M mini blocks.

[0086] Optionally, the K vectors and the V vectors of the same network layer can be stored in two adjacent continuous storage spaces, that is, the addresses of the K cache and the V cache of the same network layer are continuous. Taking FIG. 2 as an example, the first KV cache 210 is the first storage space, which is divided into 2X continuous storage spaces according to the number X of network layers, the addresses between the continuous storage spaces are continuous, the tail address of each continuous storage space is connected with the head address of the next continuous storage space, and each continuous storage space is used for storing K vectors or V vectors of one network layer, for example, each row in FIG. 2 represents a continuous storage space, the continuous storage space Add0-Add5 corresponding to the first row is the K cache of L0 layer, which is used for storing K vectors of L0 layer, the continuous storage space Add6-Add11 corresponding to the second row is the V cache of L0 layer, which is used for storing V vectors of L0 layer, and the continuous storage space Add12-Add17 corresponding to the third row is the K cache of L1 layer, which is used for storing K vectors of L1 layer. In this way, the K cache and the V cache corresponding to each network layer are continuous, and the K cache and the V cache of adjacent network layers are also continuous.

[0087] After each continuous storage space is divided into M continuous mini blocks according to the number M of blocks, each continuous storage space includes 6 mini blocks, and the address interval of each mini block is represented by Add0-Addx. The address interval of the last mini block of each row is continuous with the address interval of the first mini block of the next row. For example, the address interval of the last mini block of the first row is Add5, and the address interval of the first mini block of the second row is Add6. The two address intervals Add5 and Add6 are continuous. In this way, a block includes a column of mini blocks, and the address intervals of the mini blocks in the column are the same. For example, in block0, the address interval of the first mini block is Add0, the address interval of the second mini block is Add6, the address interval of the third mini block is Add12, and so on. The address intervals of two adjacent mini blocks are the same, and each is the length of the address interval of 6 mini blocks, that is, the length of a continuous storage space. FIG. 2 is used for illustration, and the number of blocks is not limited in the present application.

[0088] It should be understood that in the conventional manner shown in FIG. 1, since the system applies memory in sequence according to the network layer, the memory addresses between different network layers are not continuous, resulting in that multiple mini blocks in a block are discrete blocks. When data exchange is performed in units of blocks, because the addresses of multiple mini blocks in the same block have nothing to do with each other and are completely random, the conventional manner needs to read each mini block address in sequence, and each reading needs to be readdressed, resulting in low data exchange efficiency and large resource consumption.

[0089] However, in the data exchange manner provided by the present application shown in FIG. 2, the system first applies a first storage space as a KV cache, then divides the first storage space into multiple continuous storage spaces according to the number of network layers, and the addresses between the multiple continuous storage spaces are also continuous. Then, after each continuous storage space is divided based on the size of the mini block, the address intervals of multiple mini blocks in each block are the same. In this way, when data exchange is performed in units of blocks, an interface or method using equal-interval transfer can read the data of a block at one time, resulting in high data exchange efficiency, low resource consumption, and further improving the efficiency of model reasoning.

[0090] It should be noted that in the example of FIG. 2, the K cache address of each network layer is prior to the V cache address, and the V cache address of each network layer can also be prior to the K cache address, such as the first continuous storage space being used for storing L0V, the second continuous storage space being used for storing L0K, the third continuous storage space being used for storing L1V, the fourth continuous storage space being used for storing L1K, and so on, which will not be repeated here.

[0091] Optionally, the manner shown in FIG. 2 is that the K cache and the V cache are continuous, and the K cache can also be continuous, and the V cache is continuous, that is, the addresses between the continuous storage spaces storing the K vectors are continuous, and the addresses between the continuous storage spaces storing the V vectors are continuous, such as the first X continuous storage spaces being used for storing the K vectors of X network layers, and the last X continuous storage spaces being used for storing the V vectors of X network layers, or the first X continuous storage spaces being used for storing the V vectors of X network layers, and the last X continuous storage spaces being used for storing the K vectors of X network layers.

[0092] Similarly, the second processor 300 can also split the second storage space in the manner described above, which will not be repeated here.

[0093] In another possible implementation, each continuous storage space is used for storing the K vectors and the V vectors of X network layers, and each block is used for storing the K vectors or the V vectors of X network layers.

[0094] Specifically, after the computing device allocates a continuous first storage space for the first processor 200, the computing device can allocate a continuous storage space for each network layer in the first storage space according to the order of the layers of the inference model, where the continuous storage space of each network layer is used for storing the K vectors and the V vectors of the network layer, and the computing device can determine the first address and the address interval of the M blocks according to M and the size of the mini block, and update the first first address table 220.

[0095] After the continuous storage space corresponding to each network layer is allocated, each continuous storage space is divided into a plurality of address-continuous and same-size mini blocks according to the current block division manner, and the data in one block includes K vectors or V vectors of X network layers. Since the addresses between the continuous storage spaces are also continuous, the tail address of the last mini block of each continuous storage space is connected with the head address of the first mini block of the next continuous storage space. If the plurality of mini blocks in each continuous storage space are numbered in ascending order according to the address sequence, each block includes the same-numbered mini blocks in each continuous storage space, so that the address interval between the plurality of mini blocks in one block is the same, that is, the length of 1 continuous storage space, that is, the length of 2M mini blocks.

[0096] For example, FIG. 4 is an example diagram of another division manner provided by the present application, in which each row represents a continuous storage space, and each continuous storage space includes 2M mini blocks, the first M mini blocks are used to store K vectors, and the last M mini blocks are used to store V vectors of the network layer. Each block includes X mini blocks, which are used to store K vectors or V vectors of the network layer, for example, the block0 in the first column includes X mini blocks for storing K vectors, and the block0 in the 7th column includes X mini blocks for storing V vectors.

[0097] It should be noted that in the industry general term of large language models, the data of one block usually includes K vectors and V vectors. For the example shown in FIG. 4, a column of blocks only has K vectors or V vectors, so when the KV cache is divided in the manner shown in FIG. 4, the block IDs of the two blocks of the same network layer are the same. Therefore, when the data exchange is performed in units of blocks, the data of one block still includes K vectors and V vectors (for example, block0 in FIG. 4 has two, which are in the first column and the 7th column, the first column block stores K vectors of X network layers, and the 7th column block0 stores V vectors of X network layers), so that the adaptability of the technical solution of the present application is stronger, and the good landing performance is good.

[0098] Similarly, the second processor 300 can use the same division manner to divide the second storage space, which is not repeated here.

[0099] It should be understood that the above illustrates two exemplary division manners of dividing the first storage space into a plurality of continuous storage spaces in combination with FIG. 2 and FIG. 4, and the continuous storage spaces can also be divided in other manners, which are not illustrated one by one here.

[0100] It should be noted that no matter how the first storage space is divided, the addresses between the plurality of continuous storage spaces obtained are continuous, so as to ensure that the address intervals between the plurality of mini blocks in each block are the same when data is exchanged, so that the data of one block can be read at one time when the interface or method of equal interval carrying is used when data is exchanged, so that the data exchange efficiency is high, the resource consumption is low, and the efficiency of model reasoning is improved.

[0101] It should be understood that FIG. 2 and FIG. 4 use a graphical method for ease of explanation, and one continuous storage space is represented by a row of mini blocks and one block is represented by a column of mini blocks in the KV cache. In an actual application scenario, the KV cache is a continuous physical memory, and each block and continuous storage space is not divided horizontally and vertically on the physical memory, but is virtually divided, so that the memory management is more efficient and the scheme explanation is clearer.

[0102] It should be understood that FIG. 2 and FIG. 4 simply draw the components existing in the first processor 200 and the second processor 300, and these components are involved in the model reasoning method of the application. In order to clearly explain the technical scheme of the application, some necessary components are not drawn in the figure. In an actual business scenario, the first processor 200 and the second processor 300 can further include more components, such as a communication interface, a memory, a storage medium and other components, to ensure the normal operation of the first processor 200 and the second processor 300, which are not limited in the application.

[0103] It should be noted that the first address table 220 and the second address table 320 can be stored in the first processor 200 and the second processor 300 as shown in FIG. 2. In some possible implementation manners, the first address table 220 can also be stored in the second processor 300, or in the storage of the computing device, and the second address table 320 can also be stored in the storage of the computing device, which are not limited in the application.

[0104] In summary, the model inference system provided in the application, the computing device first allocates a key-value cache for the inference model, the key-value cache includes M storage blocks (blocks) in the memory of the first processor and N blocks in the memory of the second processor of the computing device, each block includes a plurality of sub-storage blocks (mini blocks), the addresses and intervals of adjacent two miniblocks are equal, and each block is used to store the intermediate data generated by each layer in the order of X layers included in the inference model. When the computing device obtains a switching request for switching the data of the first block of the M blocks to the second storage block of the N storage blocks, the switching instruction including the first address of the first block and the first address of the second block can be generated according to the switching request, and the switching instruction is sent to the first processor, so that the first processor reads the data of the first block to the second block according to the first address of the first block, the first address of the second block and the address interval. When the first processor and the second processor exchange data in block units, the interface or method using equal-interval transportation can read the data of one block at a time, so that the data exchange efficiency is high, the resource consumption is low, and the efficiency of model inference is improved.

[0105] The model inference system provided in the application is described in detail above in combination with FIGS. 2-4, and the model inference method provided in the application is explained and described below in combination with FIGS. 5 and 6. The model inference method can include an initialization stage and an application stage, FIG. 5 illustrates the step flow in the initialization stage, and FIG. 6 illustrates the step flow in the application stage.

[0106] FIG. 5 is a step flow diagram of a model inference method provided in the application in the initialization stage, which can be applied to the first processor 200 and the second processor 300 shown in FIGS. 2-4.

[0107] S510: The second processor 300 obtains an initialization request. The initialization request is used to inform the first processor 200 and the second processor 300 to start allocating their respective KV caches.

[0108] In a specific implementation, the initialization request can be generated by the computing device when initializing the system, or can be a request sent by other devices to the computing device. The other devices can be a management node of a cluster where the computing device is located. The management node can coordinate the work between multiple computing devices and send an initialization request to the computing device 400 at a suitable time point. The application does not make specific limitations.

[0109] In a specific implementation, the initialization request can include model information and user configuration, for the first processor 200 and the second processor 300 to allocate respective KV caches. The user configuration refers to the configuration input by a training user who trains the inference model. Alternatively, the model information can include the number X of network layers of the inference model, and the user configuration information can include the total number M of blocks in the first processor, the total number N of blocks in the second processor, the shape v of the mini block, the size b of the mini block, the division manner, etc. The user configuration can be a custom configuration input by the user, or a system default configuration. The model information and the user configuration information can also include more content, which is not limited in the present application.

[0110] The shape v of the mini block is used to describe the data stored in the mini block, and the division manner refers to how the continuous storage space of each layer stores the K vector and the V vector.

[0111] For example, the first division manner can refer to the division manner corresponding to FIG. 2, each continuous storage space stores a K vector or a V vector, and a block includes K vectors and V vectors of X network layers, and the K cache and the V cache are continuous.

[0112] The second division manner can refer to the division manner corresponding to FIG. 4, each continuous storage space stores a K vector and a V vector, and a block includes K vectors or V vectors of X network layers.

[0113] The third division manner is also shown in FIG. 2, each continuous storage space stores a K vector or a V vector, and a block includes K vectors and V vectors of X network layers, but different from FIG. 2, the K cache is continuous, and the V cache is continuous, that is, the first X continuous storage spaces store K vectors of X network layers, and the last X continuous storage spaces store V vectors of X network layers.

[0114] Of course, there are other division manners, which are not exemplified one by one here.

[0115] It should be noted that some information in the user configuration can also be default, and then the initialization request can not include this part of the default information, for example, the division manner is default, and the initialization request can not include the division manner, which is not limited in the present application.

[0116] S520: The second processor 300 sends an initialization instruction to the first processor 200.

[0117] In a specific implementation, the second processor 300 can generate initialization instructions recognizable by the first processor 200 according to the initialization request, and then send the initialization instructions to the first processor 200. The initialization instructions can include the number M of blocks in the first processor 200, the shape v of the miniblock, the size b of the miniblock, the division mode, the number X of network layers, and the like. Of course, if the first processor 200 has a default setting for part or all of the information, the initialization instructions can not carry the information of the default setting, which is not limited in the present application.

[0118] It should be noted that in the example shown in FIG. 5, the first processor 200 is notified by the second processor 300 to start initializing the KV cache. In other possible implementations, the first processor 200 can also notify the second processor 300 to start initializing the KV cache, or both processors can obtain the initialization request by themselves, which is not limited in the present application.

[0119] S530A: The first processor 200 allocates a first storage space with continuous addresses. The first storage space is the first KV cache 210 shown in FIG. 2.

[0120] In a specific implementation, the KV cache includes memory, disk, hybrid storage, flash memory, cloud storage, etc. The KV cache is used to store intermediate data generated during the running of an inference model. The inference model can include a large language model. It should be understood that the large language model is a self-recurrent model. When inferring the output text according to the input text, multiple forward calculations are required. Each forward calculation can obtain a predicted word. The input data of each forward calculation includes the predicted word obtained by the last inference, until all predicted words contained in the output text are obtained, thereby obtaining the output text. Therefore, the intermediate data of the last forward calculation is cached in the KV cache, which can reduce repeated calculations for the next forward calculation, thereby improving the inference efficiency of the large language model.

[0121] Generally, the intermediate data can include a key (K) vector and a value (V) vector obtained by each network layer calculation. The K vector is obtained by performing linear transformation on the input attention layer vector of the attention layer of the large language model using a first weight matrix W k The V vector is obtained by performing linear transformation on the input attention layer vector of the attention layer using a second weight matrix W vAfter linear transformation, since the weight matrix of each network layer is unchanged in the inference stage, the input vector of each forward calculation includes the input vector of the last forward calculation, so the K vector V vector obtained by each forward calculation is cached in the KV cache, and the K vector and V vector calculated before can be directly read from the KV cache in the next forward calculation, which can reduce repeated calculation, thereby improving the calculation efficiency of each forward calculation, and further improving the inference efficiency of the large language model.

[0122] In an embodiment, the length information of the first storage space can be determined according to the size b of the mini block, the number X of network layers, and the total number M of blocks, wherein the first storage space includes a total of 2XM mini blocks, and thus the length of the first storage space is at least 2XMb. The shape of the first storage space can be determined according to the division manner and the shape v of the mini block. The possible shapes of the first storage space are described below with reference to two division manners shown in FIG. 2 and FIG. 4.

[0123] Optionally, if the first division manner shown in FIG. 2 is adopted, each continuous storage space stores a K vector or a V vector, one block includes K vectors and V vectors of X network layers, and the K cache and the V cache are continuous, in this case, the number of continuous storage spaces is 2X, and the shape of the first storage space is (1, 2X, M, *v), which means that one continuous storage space stores one kind of vector, such as a K vector or a V vector, the first storage space includes 2X continuous storage spaces, M blocks, each continuous storage space includes M mini blocks, each block includes 2X mini blocks, and the shape of each mini block is *v. It should be understood that the user sets the shape of the mini block as v, and when applying for a storage block, the shape of the mini block can be adaptively adjusted, such as unfolding the dimension, so *v is used here.

[0124] Optionally, if the second division manner shown in FIG. 4 is adopted, each continuous storage space stores a K vector and a V vector, one block includes K vectors or V vectors of X network layers, in this case, the number of continuous storage space data is X, and the shape of the first storage space is (2, X, M, *v), which means that one continuous storage space stores two kinds of vectors, that is, a K vector and a V vector, the first storage space includes X continuous storage spaces, M blocks, each continuous storage space includes 2M mini blocks, each block includes X mini blocks, and the shape of each mini block is *v.

[0125] It should be understood that the shape of the first storage space is exemplarily given above in both the two division manners of FIG. 2 and FIG. 4, and the shape of the first storage space can be determined in a similar manner in other division manners, which will not be exemplarily given here.

[0126] It should be understood that in the conventional technology, the first processor needs to apply for memory for each K vector or V vector of each network layer of the inference model according to the network layer, and if there are X network layers, the operating system needs to handle 2X times of memory application. In the model inference method provided in the present application, the first processor determines the size and shape of the first storage space according to the user configuration information and the model information, and applies for a piece of continuous memory to store the K vectors and V vectors of multiple network layers at one time, and the operating system only needs to handle 1 time of memory application.

[0127] S530B: The second processor 300 allocates a second storage space with a continuous address. The first storage space is the second KV cache 310 shown in FIG. 2.

[0128] It should be noted that the second processor 300 can also apply for the second storage space as the second KV cache 310 in the manner described in S530A. Similarly, if the total number of blocks of the second processor 300 is N, the length of the first storage space applied for by the second processor 300 is 2XNb.

[0129] In the case of the first division manner, that is, the division manner shown in FIG. 2, the number of continuous storage spaces is 2X, and the shape of the first storage space is (1, 2X, N, *v). In the case of the second division manner, that is, the division manner shown in FIG. 4, the number of continuous storage spaces is X, and the shape of the first storage space is (2, X, N, *v). For related description, reference can be made to the foregoing description, which will not be repeated here.

[0130] It should be noted that the second processor 300 and the first processor 200 can execute S530A and S530B simultaneously, or execute S530A and S530B sequentially. Specifically, the case of executing sequentially can include that the first processor 200 executes S530A first, and then the second processor 300 executes S530B; or the second processor 300 executes S530B first, and then the first processor 200 executes S530A; or the first processor 200 and the second processor 300 execute S530A and S530B sequentially, and then execute S540A and S540B sequentially, and so on; or the first processor 200 executes S530A-S560A, and then the second processor 300 executes S530B-S560B; or the second processor 300 executes S530B-S560B, and then the first processor 200 executes S530A-S560A, which is not limited in the present application.

[0131] It should be understood that the suffixes A and B of each step hereinafter represent that the two processors can execute the step simultaneously or sequentially, and the description is similar to the above S530A and S530B, which will not be described here.

[0132] S540A: The first processor 200 allocates a continuous storage space for each layer in the first storage space according to the order of the layers of the inference model.

[0133] In a specific implementation, the first processor 200 can divide the first storage space into a plurality of continuous storage spaces of the same size, and the addresses of the continuous storage spaces are continuous. If the plurality of continuous storage spaces are numbered in address order, the addresses of two continuous storage spaces with adjacent numbers are continuous, and the start address of each continuous storage space is exactly the end address of the last numbered continuous storage space. For example, the first storage space is divided into 10 continuous storage spaces numbered 0-9, and the addresses of 0 and 1 are continuous, the addresses of 1 and 2 are continuous, the addresses of 2 and 3 are continuous, and so on. The above example is used for illustration, and the present application is not limited specifically.

[0134] In a specific implementation, when the first storage space is divided into a plurality of continuous storage spaces, the number and size of the continuous storage spaces can be determined according to the division manner, and each network layer is sequentially allocated a corresponding continuous storage space according to the network layer order and the address order of the continuous storage spaces, and each continuous storage space is used to store the intermediate data of the same network layer.

[0135] For example, according to the division manner shown in FIG. 2, the number of continuous storage spaces is 2X, the length of each continuous storage space is Mb, and each continuous storage space is used to store a K vector or a V vector of a network layer. The first continuous storage space (represented by the first row of blocks corresponding to Add0-Add5 in FIG. 2) is used to store the K vector of the L0 layer, the second continuous storage space (represented by the second row of blocks corresponding to Add6-Addd11 in FIG. 2) is used to store the K vector of the L0 layer, the third continuous storage space (represented by the third row of blocks corresponding to Add12-Addd17 in FIG. 2) is used to store the K vector of the L1 layer, and so on, which will not be described here.

[0136] For example, if the division mode shown in FIG. 4 is adopted, the number of continuous storage spaces is X, the length of each continuous storage space is 2Mb, and each continuous storage space is used to store the K vector and the V vector of the same network layer. The first continuous storage space (represented by the first row of blocks corresponding to Add0-Add11 in FIG. 4) is used to store the K vector and the V vector of the L0 layer, the second continuous storage space (represented by the second row of blocks corresponding to Add12-Add23 in FIG. 4) is used to store the K vector and the V vector of the L1 layer, and so on, which will not be repeated here.

[0137] It should be understood that the above examples are divided into two modes of FIG. 2 and FIG. 4, and the technical solution of the present application can also support other division modes, such as sequentially allocating the V vector and the K vector of each row, that is, in FIG. 2, the first row is L0V, the second row is L0K, the third row is L1V, and the fourth row is L1K, and so on. The K vector of each network layer can be allocated first, and then the V vector of each network layer can be allocated, that is, in FIG. 2, the first X rows are L0K-LXK, and the X+1-2X rows are L0V-LXV. There are more division modes, which can be referred to in the foregoing content, which will not be repeated here. Of course, other division modes also have corresponding numbers and lengths, which will not be repeated here. For example, the present application is not limited.

[0138] S540B: The second processor 300 allocates continuous storage spaces for each layer in the second storage space according to the order of the layers of the inference model.

[0139] It should be noted that the second processor 300 can also divide the second storage space in the manner described in S540A and allocate continuous storage spaces for each layer. If the division mode shown in FIG. 2 is adopted, the number of continuous storage spaces is 2X, the length of each continuous storage space is Nb, and each continuous storage space is used to store the K vector or the V vector of a network layer. If the division mode shown in FIG. 4 is adopted, the number of continuous storage spaces is X, the length of each continuous storage space is 2Nb, and each continuous storage space is used to store the K vector and the V vector of the same network layer. For specific descriptions, reference can be made to the related descriptions of the first storage space in S540A, which will not be repeated here.

[0140] It should be noted that the second processor 300 and the first processor 200 can simultaneously perform S540A and S540B, or can perform S540A and S540B in sequence. For specific explanations of the sequence, reference can be made to S530A and S530B, which will not be repeated here.

[0141] S550A: The first processor 200 determines the first address and the address interval of the M blocks according to the block number M and the size of the mini block.

[0142] In a specific implementation, the first processor 200 can divide the continuous storage space corresponding to each network layer into a plurality of mini blocks with address continuity and the same size according to the size b and the shape v of the mini block. If the plurality of mini blocks are numbered in address order, the addresses of two adjacent numbered mini blocks are continuous, and the starting address of each mini block is exactly the ending address of the last numbered mini block.

[0143] In a specific implementation, if the first storage space is divided in the manner shown in FIG. 2 to obtain 2X continuous storage spaces, each of which stores a K vector or a V vector of a network layer, the first processor 200 divides each continuous storage space into M mini blocks according to the size b and the shape v of the mini block, where M is the number of blocks in the first processor configured by the user. If the first storage space is divided in the manner shown in FIG. 4 to obtain X continuous storage spaces, the first processor 200 divides each continuous storage space into 2M mini blocks according to the size b and the shape v of the mini block. Similarly, the second processor 300 can also divide the continuous storage space according to the same logic, which is not repeated here.

[0144] In a specific implementation, after the mini blocks in each continuous storage space are divided, the mini blocks included in each block can be determined, and then the first address and the address interval of each block can be determined. Each storage block is used to store the intermediate data generated by each layer in the order of the X layers included in the inference model. Each block includes a plurality of mini blocks, the plurality of mini blocks come from different continuous storage spaces, and the address interval of two adjacent mini blocks is equal, and the address interval is the length of a continuous storage space. After the mini blocks included in each block are determined, the first address of each block can be obtained according to the addresses of the plurality of mini blocks. The address interval refers to the difference between the addresses of two adjacent mini blocks in the block after the mini blocks in the block are sorted in address order.

[0145] It should be noted that the first address table can store a mapping relationship between block information and a first address of each block. The block information can include an identification (ID) of the block or an address of the block. For example, as shown in FIG. 2, the block 0 corresponds to the first address Add0. The above example is used for illustration only, and the present application is not limited in this regard. The address interval can be stored in the first address table, or can be stored in another location, and the present application is not limited in this regard.

[0146] Specifically, if the plurality of mini blocks in each continuous storage space are numbered in address order, the plurality of mini blocks included in each block are the same numbered mini blocks in different continuous storage spaces. For example, in the example of FIG. 2 or FIG. 4, each mini block in the block 0 is from a different continuous storage space, and the address order is the first mini block. Each mini block in the block 1 is from a different continuous storage space, and the address order is the second mini block. The first address of each block is the address of each mini block in the first continuous storage space. Based on this, the first address table can be updated. The first address table 220 in FIG. 2 is an example, and the present application is not limited in this regard.

[0147] Alternatively, if the first storage space is divided in the manner shown in FIG. 2 to obtain 2X continuous storage spaces, each continuous storage space includes M mini blocks. At this time, one continuous storage space is a row of mini blocks shown in FIG. 2, and one block is a column of mini blocks shown in FIG. 2. Each block includes 2X mini blocks, and the address order numbers of these blocks in the continuous storage space are the same. For example, each mini block in the block 5 is the last mini block in address order in the continuous storage space (a row), that is, the fifth column of mini blocks. The first address of each block is the address of each mini block in the first continuous storage space in address order, that is, the address of the first row of mini blocks. The first address of the block 0 is Add0, the first address of the block 1 is Add1, and so on. At this time, the address interval is the length Mb of M mini blocks.

[0148] Optionally, if the first storage space is divided in the manner shown in FIG. 4 to obtain X continuous storage spaces, each of which includes 2M mini blocks, a continuous storage space is a row of mini blocks shown in FIG. 4, and a block includes a K block for storing K vectors and a V block for storing V vectors, such as block 0 in FIG. 4, which includes X mini blocks in the first column (K block) and X mini blocks in the seventh column (V block). In the K block of the same block, the address order number of each mini block in the continuous storage space is the same, and in the V block of the same block, the address order number of each mini block in the continuous storage space is also the same. For example, in block 0 in FIG. 4, the K block for storing K vectors is the first column of data, and each mini block is the first storage block in the row (continuous storage space), and the V block for storing V vectors is the seventh column of data, and each mini block is the seventh storage block in the row (continuous storage space). The first address of the K block in block 0 is Add0, and the first address of the V block is Add6. The first address of the K block in block 1 is Add1, and the first address of the V block is Add7. This is not repeated here.

[0149] It should be noted that in the industry general term of large language models, when data is exchanged, the data of a block usually includes K vectors and V vectors. For the example shown in FIG. 4, a column of blocks has only K vectors or V vectors. Therefore, when the KV cache is divided in the manner shown in FIG. 4, the block IDs of two blocks of the same network layer are the same, and one block corresponds to two columns of mini blocks, which are referred to as K block and V block. The K block stores the K vectors of each network layer, and the B block stores the V vectors of each network layer. In this way, when data is exchanged in units of blocks, the data of a block still includes K vectors and V vectors (such as block 0 in FIG. 4, which has two blocks, the first column and the seventh column. The first column block stores K vectors of X network layers, and the seventh column block 0 stores V vectors of X network layers), so that the adaptability of the technical solution of the present application is stronger and the implementability is better.

[0150] It should be noted that for the example shown in FIG. 4, the address interval can be the length of 2M miniblocks or the length of M miniblocks. If the length is 2M miniblocks, when data exchange is performed, if data in block 0 is to be read, K blocks in block 0 are first read based on the address interval, and then V blocks are read based on the address interval. If the length is M miniblocks, K blocks and V blocks can be directly read based on the address interval. The above two methods can use an interface or method of equal interval transfer to complete data exchange, thereby improving data exchange efficiency.

[0151] It should be noted that the first address table in the first processor 200 can be stored in the first processor 200 or other memory of the computing device 400, and the present application does not make specific limitations.

[0152] It should be understood that, for the convenience of explanation, the present application proposes a numbering method for miniblocks in each continuous storage space to determine miniblocks included in each block, that is, each block includes miniblocks with the same number in each continuous storage space. This numbering is only for better explanation and illustration. In the actual processing process, each continuous storage space can also not be numbered, and each continuous storage space can be directly determined according to the address sequence, and numbering is not required, and the present application does not make specific limitations.

[0153] S550B: The second processor 300 determines the first address and the address interval of the N blocks according to the number N of blocks and the size of miniblocks.

[0154] In a specific implementation, the second processor 300 can also determine the first address of the N blocks in the manner described in S550A, update the second address table 320, and determine the address interval, which is not repeated and described herein.

[0155] It should be noted that the second processor 300 and the first processor 200 can simultaneously perform S540A and S540B, or can perform S540A and S540B in sequence. The specific explanation of performing S540A and S540B in sequence can refer to S530A and S530B, which is not repeated and described herein.

[0156] S560A: The first processor 200 stores the first address of the M blocks in the first address table.

[0157] S560B: The second processor 300 stores the first addresses of the N blocks in a second first address table.

[0158] It should be understood that the model inference method provided in the present application, in the initialization stage, applies to the operating system for a first storage space as a KV cache, and then divides the first storage space into a plurality of continuous contiguous storage spaces, and then divides each contiguous storage space into a plurality of mini blocks, so that the tail address of each contiguous storage space corresponds to a mini block 1, and the first address of the next contiguous storage space corresponds to a mini block 2, and the address therebetween is continuous. In this way, if the mini blocks in each contiguous storage space are numbered in address order, then the first address interval between the mini blocks with the same number in each contiguous storage space is the same, that is, the length of a contiguous storage space. When data exchange is performed in units of blocks, the interface or method using equal-interval transfer can read out the data of multiple mini blocks at a time, so that the data exchange efficiency is high, the resource consumption is low, and the efficiency of model inference is improved.

[0159] The above explains and describes the step flow of the model inference method provided in the present application in the initialization stage in combination with FIG. 5, and the step flow of the model inference method provided in the present application in the application stage is explained and described below in combination with FIG. 6.

[0160] FIG. 6 is a step flow diagram of a model inference method provided in the present application in the application stage, as shown in FIG. 6, the method can include the following steps:

[0161] S610: The second processor 300 acquires a swap request.

[0162] Optionally, the swap request includes information of a first storage block, wherein the first storage block is a source block that needs to be read from the first processor 200, and the information of the first storage block can include an identification (ID) of the block or an address of the block, which is uniformly exemplified below by taking the ID as an example. As known from the foregoing, the first processor 200 and the second processor 300 perform data exchange (swap) in units of blocks, so the ID of the block is included in the swap request, and the number of IDs here can be one or more. For example, assuming that the first processor 200 is as shown in FIG. 2, the first storage block includes data stored in block0~block3, and then the data exchange request includes ID information of block0~block3. The above example is used for illustration, and the present application is not limited specifically.

[0163] Optionally, the exchange request can further include information of the second storage block, where the second storage block is a target block to be stored in the second processor 300, and the information of the second storage block can include an ID or an address of the block.

[0164] In specific implementations, the exchange request can be generated by the second processor 300 of the computing device 400, or can be generated by another computing device, such as a management node according to an inference of an inference model, where the management node can be a scheduling management role in a distributed inference architecture when the inference model is distributedly deployed in multiple computing devices, and is used to schedule and manage the multiple computing devices 400.

[0165] In specific implementations, the exchange request can be generated under certain conditions, such as when the first KV cache of the first processor 200 is close to full or exceeds a preset threshold, triggering the generation of the exchange request. Alternatively, the exchange request can be generated at a fixed frequency, or can be generated at a fixed time, or can be manually initiated by a development user of the trained inference model, or can be generated by more conditions, which are not listed one by one here.

[0166] S620: The second processor 300 sends an exchange instruction to the first processor 200.

[0167] Optionally, if the second processor 300 stores the first address table, or the second address table 320 is stored in the memory of the computing device 400, the second processor 300 can determine the first address of the first storage block according to the first address table, determine the first address of the second storage block according to the second address table, obtain the address interval between the miniblocks in the first processor 200, and generate the exchange instruction, where the exchange instruction includes the first address of the first storage block, the first address of the second storage block, and the address interval.

[0168] Optionally, if the second processor 300 does not store the first address table, and the first address table is only stored in the first processor 200, the second processor 300 can also determine the first address of the second storage block according to the second address table, and then generate the exchange instruction, where the exchange instruction includes the block identifier of the first storage block and the first address of the second storage block, and the first processor 200 queries the first address table 220 according to the block identifier of the first storage block to determine the first address of the first storage block and the address interval. Both of the above two methods can be used, and the present application is not limited to specific implementations.

[0169] S630: The first processor 200 reads the first storage block to the second storage block according to the first address of the first storage block, the first address of the second storage block, and the address interval.

[0170] In a specific implementation, the first processor 200 can determine the first address and the address interval of the first storage block according to the exchange instruction, and then read the first storage block from the first KV cache 210 using the interface or method of equal interval transfer. In a specific implementation, if the exchange instruction includes the IDs of multiple blocks, the multiple blocks can be read one by one in the manner described above.

[0171] If the first KV cache 210 is divided in the manner of FIG. 2, one continuous storage space is used to store the K vectors or V vectors of one network layer, one block includes the K vectors and V vectors of X network layers, that is, one column of miniblocks in FIG. 2, and the address interval is the length Mb of M miniblocks. At this time, 2X miniblocks can be read at one time according to the address interval to obtain the first storage block.

[0172] If the first KV cache 210 is divided in the manner of FIG. 4, one continuous storage space is used to store the K vectors and V vectors of one network layer, one block includes the K vectors or V vectors of X network layers, but the IDs of the blocks corresponding to the K vectors and V vectors of the same network layer are the same, that is, one block includes a K block and a V block during data exchange, the K block includes multiple miniblocks storing K vectors, and the V block includes multiple miniblocks storing V vectors.

[0173] In this case, if the address interval is the length of 2M miniblocks, that is, 2Mb, 2 times of reading are required, the first time is to read the K block according to the first address of the K block, and the second time is to read the V block according to the first address of the V block to obtain the first storage block. If the address interval is the length of M miniblocks, that is, Mb, 1 time of reading is required, the data of the two blocks with the same ID, that is, the K block and the V block, is directly read to obtain the first storage block.

[0174] For example, taking FIG. 4 as an example, if data of block0 needs to be read, block0 includes Kblock in the first column and Vblock in the seventh column, at this time, the length of Add0-Add5 can be taken as the address interval, that is, the length of 6 miniblocks, Add0, Add6, Add12, Add18, … are read at one time at equal intervals, thereby obtaining the data of 2 block0 at one time to obtain the first storage block. Alternatively, the length of Add0-Add11 can be taken as the address interval, that is, the length of 12 miniblocks, 2M, through 2 times of equal interval reading, X miniblocks in Kblock are read at one time in the first equal interval reading, that is, Add0, Add12, …, and X miniblocks in Vblock are read at one time in the second equal interval reading, that is, Add6, Add18, …, thereby obtaining the first storage block. The above examples are used for illustration, and the present application is not limited specifically.

[0175] In a specific implementation, the interface for equal interval transfer can include an aclrtMemcpy2d interface, which is an affinity equal interval small storage block transfer interface. The interface can be called to transfer the data of an entire block at one time, thereby reducing the number of times of performing a reading operation, avoiding discrete reading of a large number of small miniblocks, and further improving the efficiency of data exchange.

[0176] Alternatively, the efficiency of data exchange can be further optimized according to actual business conditions. For example, before the interface is called, process binding can be performed, that is, the process or thread calling the data exchange interface is bound to the processor core to which the first processor 200 is affinity, so that the data transmission capability between the processor core of the second processor and the first processor can be maximized to ensure efficient transmission of the first storage block and further improve the efficiency of data exchange.

[0177] For another example, when the KV cache of the second processor 300 is initialized, the pin-memory option can be enabled to ensure that data is placed in pin-memory, and the memory pages in the pin-memory will not be swapped out to the virtual memory (disk page file) by the operating system, but will always be kept in the physical memory, thereby avoiding the performance loss caused by storing data in the memory pages and swapping the memory pages to the disk page.

[0178] It should be understood that the above examples exemplarily give some possible operation steps for optimizing the efficiency of data exchange. The model inference method of the present application can also be combined with other operation steps to further optimize the efficiency of data exchange, which will not be exemplarily illustrated here.

[0179] It should be understood that the conventional technology needs to read multiple discrete miniblocks to send to the second processor, and multiple messages need to be sent during data transmission, and the message header of each message includes the source address and the destination address of each miniblock, and using the technical solution provided in the application, since the addresses between the multiple miniblocks read out are equidistant, and the destination addresses are also equidistant, the multiple miniblocks in the first storage block can be packed into one message, and one message header is used for transmission, so that the data transmission efficiency can also be improved, the data exchange efficiency is high, the resource consumption is low, and the efficiency of model reasoning is improved.

[0180] Optionally, after the first processor 200 acquires the first storage block according to S630, the first processor 200 can also directly write the first storage block into the second storage block of the second processor 300 according to the destination address information of the first storage block, that is, the identifier of the second storage block, by using data transmission technologies such as direct memory access (Direct Memory Access, DMA) and remote direct memory access (Remote Direct Memory Access, RDMA), without intervention of the CPU of the second processor 300, thereby realizing efficient transmission of data.

[0181] It should be understood that after the first processor 200 performs S630, the space of the first storage block of the KV cache is released, and the first storage block can be used to store KV vectors in the reasoning process. After that, if the KV cache space of the first processor 200 is sufficient, the second processor 300 can perform S640 to exchange the data of the third storage block in the second processor 300 into the fourth storage block of the first processor 200.

[0182] S640: The second processor 300 reads the third storage block to the fourth storage block according to the first addresses and the address intervals of the third storage block and the fourth storage block.

[0183] In a specific implementation, the second processor 300 can also acquire an exchange request carrying representations of the third storage block and the fourth storage block, wherein the third storage block is a source block located in the second KV cache 310 of the second processor 300, and the fourth storage block is a target block located in the first KV cache 210 of the first processor 200. The second processor 300 can acquire the first address and the address interval of the third storage block from the second first address table 320 according to the ID of the third storage block, and then read multiple miniblocks in the third storage block by using the equidistant carrying interface or method, and write the miniblocks into the fourth storage block of the first processor 200. For details, reference can be made to the related description of S630, which is not repeated here.

[0184] It should be noted that S630 and S630 can be executed simultaneously, or can be executed in sequence. In the case of execution in sequence, S630 can be executed first, followed by S640. At this time, the exchange request of S610 also includes the identifiers of the third storage block and the fourth storage block, for the second processor 300 to execute S640. S630 can also be executed first, and S640 can be executed after a period of time. At this time, the second processor 300 can execute S610 again, and the exchange request in this S610 includes the identifiers of the third storage block and the fourth storage block, for the second processor 300 to execute S640.

[0185] In short, the first processor 200 and the second processor 300 can exchange memory blocks to each other's memory at the same time, or can execute S630 when the memory of the first processor 200 is insufficient, or execute S640 when the memory of the first processor is sufficient or the first processor needs to use the data of the third storage block. The present application does not make specific limitations.

[0186] In summary, the model inference method provided by the present application first allocates a key-value cache for the inference model, the key-value cache includes M storage blocks (blocks) in the memory of the first processor and N blocks in the memory of the second processor of the computing device, each block includes a plurality of mini storage blocks (mini blocks), the addresses and intervals of adjacent two miniblocks are equal, and each block is used to store intermediate data generated by each layer in the order of X layers included in the inference model. When the computing device obtains an exchange request for exchanging the data of a first block of the M blocks to a second storage block of the N storage blocks, an exchange instruction including the first address of the first block and the first address of the second block can be generated according to the exchange request, and the exchange instruction is sent to the first processor. The first processor reads the data of the first block to the second block according to the first address of the first block, the first address of the second block, and the address interval, so that when the first processor and the second processor exchange data in block units, the interface or method of equal interval transportation can read the data of one block at a time, so that the data exchange efficiency is high, the resource consumption is low, and the efficiency of model inference is improved.

[0187] The model inference method provided by the present application is described in detail above. The software structure of the computing device provided by the present application will be explained and described below in combination with FIG. 7. The computing device can be the computing device 400 shown in FIGS. 1-6.

[0188] FIG. 7 is a schematic diagram of a software structure of a computing device 400 provided by the present application. As shown in FIG. 7, the computing device 400 includes an allocation unit 410, an obtaining unit 420, a first address confirmation unit 430, and an equal-interval carrying unit 440. It should be understood that FIG. 7 is an exemplary division manner, and the computing device 400 can be divided into more or fewer unit modules, which is not specifically limited by the present application.

[0189] The allocation unit 410, the obtaining unit 420, the first address confirmation unit 430, and the equal-interval carrying unit 440 can be implemented by software or by hardware. For example, the implementation of the allocation unit 410 is described below. Similarly, the implementation of the obtaining unit 420, the first address confirmation unit 430, and the equal-interval carrying unit 440 can refer to the implementation of the allocation unit 410.

[0190] As an example of the software function unit, the allocation unit 410 can include code running on a computing instance. The computing instance can include at least one of a physical host (computing device), a virtual machine, and a container. Further, the computing instance can be one or more. For example, the allocation unit 410 can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers for running the code can be distributed in the same region, or can be distributed in different regions. Further, the multiple hosts / virtual machines / containers for running the code can be distributed in the same availability zone (AZ), or can be distributed in different AZs, each AZ including a data center or multiple data centers with similar geographical locations. Generally, one region can include multiple AZs.

[0191] Similarly, the multiple hosts / virtual machines / containers for running the code can be distributed in the same virtual private cloud (VPC), or can be distributed in multiple VPCs. Generally, one VPC is set in one region, and a communication gateway needs to be set in each VPC for cross-region communication between two VPCs in the same region and between VPCs in different regions, so as to realize the interconnection between the VPCs through the communication gateway.

[0192] As an example of the hardware function unit, the allocation unit 410 can include at least one computing device, such as a server, etc. Alternatively, the allocation unit 410 can also be a device implemented by a central processing unit (CPU), or implemented by an application-specific integrated circuit (ASIC), or implemented by a programmable logic device (PLD), etc. Among them, the PLD can be implemented by a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system on chip (SoC), an offload card, an inference card, or any combination thereof.

[0193] The plurality of computing devices included in the allocation unit 410 can be distributed in the same region, or can be distributed in different regions. The plurality of computing devices included in the allocation unit 410 can be distributed in the same AZ, or can be distributed in different AZs. Similarly, the plurality of computing devices included in the allocation unit 410 can be distributed in the same VPC, or can be distributed in multiple VPCs. Among them, the plurality of computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offload cards, inference cards, etc.

[0194] It should be noted that in other embodiments, the steps implemented by the allocation unit 410, the acquisition unit 420, the first address confirmation unit 430, and the equidistant carrying unit 440 can be specified as needed, and the entire function of the computing device 400 can be implemented by the allocation unit 410, the acquisition unit 420, the first address confirmation unit 430, and the equidistant carrying unit 440 respectively implementing different steps in the model inference method.

[0195] The functions of the allocation unit 410, the acquisition unit 420, the first address confirmation unit 430, and the equidistant carrying unit 440 will be explained and described below. Among them, the computing device 400 includes the first processor 200 and the second processor 300.

[0196] The allocation unit 410 is configured to allocate a key-value cache for the inference model, the key-value cache comprising M storage blocks in a memory of a first processor of the computing device and N storage blocks in a memory of a second processor of the computing device, each of the storage blocks comprising a plurality of sub-storage blocks, adjacent two of the sub-storage blocks having an equal address interval, and each of the storage blocks being configured to store intermediate data generated by each of X layers of the inference model in a sequence of the X layers. The allocation unit 410 can be implemented by the first processor 200 and the second processor 300, and the specific implementation can refer to S530A-S560B and optional steps thereof in the embodiment of FIG. 5.

[0197] The acquisition unit 420 is configured to acquire a swap request, generate a swap instruction according to the swap request, and send the swap instruction to the first processor. The swap request is used to request swapping data in a first storage block of the M storage blocks to a second storage block of the N storage blocks. The swap instruction comprises a first address of the first storage block and a second address of the second storage block, and is used to instruct the first processor to read the intermediate data in the first storage block to the second storage block according to the first address of the first storage block, the second address of the second storage block, and the address interval. The acquisition unit 420 can be implemented by the second processor 300, and the specific implementation can refer to S610-S620 and optional steps thereof in the embodiment of FIG. 6.

[0198] The equal-interval carrying unit 440 is configured to read the intermediate data in the first storage block to the second storage block according to the first address of the first storage block, the second address of the second storage block, and the address interval. The equal-interval carrying unit 440 can be implemented by the first processor 200 or the second processor 300, and the specific implementation can refer to S630, S640 and optional steps thereof in the embodiment of FIG. 6.

[0199] In a possible implementation, the allocation unit 410 is configured to allocate a first storage space of a continuous address in the memory of the first processor, allocate a second storage space of a continuous address in the memory of the second processor, and allocate a continuous storage space for each layer in the first storage space and the second storage space in a sequence of the layers of the inference model, respectively. Each of the continuous storage spaces is configured to store a K vector or a V vector of one layer. M and N are acquired, and the first address of the M storage blocks, the first address of the N storage blocks, and the address interval are determined according to M, N, and the size of the sub-storage blocks. In the M and N storage blocks, each of the storage blocks is configured to store the K vector and the V vector of the intermediate data of the X layers of the inference model. The specific implementation can refer to S530A-S560B and optional steps thereof in the embodiment of FIG. 5, and this implementation corresponds to the division mode shown in FIG. 2.

[0200] In another possible implementation, the allocating unit 410 is configured to allocate a first storage space with a continuous address in the memory of the first processor, allocate a second storage space with a continuous address in the memory of the second processor, and allocate a continuous storage space for each layer of the inference model in the first storage space and the second storage space respectively according to the order of the layers, where each continuous storage space is used to store a K vector and a V vector of a layer, obtain M and N, and determine the first address of the M storage blocks, the first address of the N storage blocks, and the address interval according to M, N, and the size of the sub-storage block, where each storage block in the M and N storage blocks is used to store a K vector or a V vector of intermediate data of an X layer of the inference model. The implementation is specifically used for implementing S530A-S560B and optional steps thereof in the embodiment of FIG. 5, and the implementation corresponds to the division manner shown in FIG. 4.

[0201] In a possible implementation, the first processor is a general processor, and the second processor is an inference card, or the first processor is an inference card, and the second processor is a general processor, or the first processor and the second processor are both inference cards.

[0202] In a possible implementation, the first address confirming unit 430 is configured to obtain the identification of the first storage block and the second storage block from the exchange request, obtain the first address of the first storage block and the first address of the second storage block according to the identification of the first storage block and the identification of the second storage block, and generate the exchange instruction. The first address confirming unit 430 can be implemented by the second processor 300 and is specifically used for implementing S620 and optional steps thereof.

[0203] In a possible implementation, the length of the first storage space and the length of the second storage space are obtained according to model information and user configuration information, the model information includes the number X of network layers, and the user configuration information includes the number M of storage blocks in the memory of the first processor, the number N of storage blocks in the memory of the second processor, and the size b of the sub-storage block.

[0204] In summary, the computing device provided in the present application first allocates a key-value cache for the inference model, the key-value cache includes M storage blocks (blocks) in the memory of the first processor and N blocks in the memory of the second processor of the computing device, each block includes a plurality of mini blocks, the addresses and intervals of adjacent two mini blocks are equal, and each block is used to store intermediate data generated by each layer in the order of X layers included in the inference model. When the computing device obtains a switching request for switching the data of the first block of the M blocks to the second storage block of the N storage blocks, the switching instruction including the first address of the first block and the first address of the second block can be generated according to the switching request, and the switching instruction is sent to the first processor, so that the first processor reads the data of the first block to the second block according to the first address of the first block, the first address of the second block and the address interval. When the first processor and the second processor exchange data in block units, the interface or method using equal-interval transportation can read the data of one block at a time, so that the data exchange efficiency is high, the resource consumption is low, and the efficiency of model inference is improved.

[0205] The hardware structure of the computing device provided in the present application will be explained and described below in combination with FIGS. 8-9.

[0206] FIG. 8 is a schematic diagram of the hardware structure of a computing device provided in the present application. As shown in FIG. 8, the computing device 800 includes a bus 802, a processor 804, an inference card 807, a memory 806 and a communication interface 808. The processor 804, the memory 806, the inference card 807 and the communication interface 808 communicate through the bus 802. The computing device 800 can be the first processor 200 or the second processor 300 in the foregoing. It should be understood that the number of processors, memories and inference cards in the computing device 800 is not limited in the present application.

[0207] The bus 802 can be a peripheral component interconnect Express (PCIe) bus or an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), or the like. The unified bus can also be referred to as a coherent bus. The bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one line is represented in FIG. 8, but it does not mean that there is only one bus or only one type of bus. The bus 802 can include a path for transmitting information between various components (e.g., the memory 806, the processor 804, the communication interface 808) of the computing device 800. The unified bus can also be referred to as a coherent bus.

[0208] The processor 804 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), an ASIC, a FPGA, a CPLD, an NPU, a SoC, an offload card, an inference card, or the like computing device.

[0209] The memory 806 can include a volatile memory, such as a random access memory (RAM) including a dynamic or static RAM (DRAM or SRAM). The memory 806 can also include a non-volatile memory, such as a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically EPROM (EEPROM), a flash memory, a mechanical hard disk, or a solid state hard disk. In addition, the memory 806 can also be implemented by a storage class memory (SCM), a phase change memory (PCM), or other types of storage media.

[0210] It is worth mentioning that the same type of storage medium can be configured in the same computing device to implement the function of the memory 806, or two or more types of storage media can be configured to implement the function of the memory 806, and the present application does not limit this.

[0211] The inference card 807 can be implemented by a computing device or an AI chip such as a DPU, an NPU, a GPU, an XPU, or a SoC, and the present application does not make specific limitations.

[0212] The communication interface 808 uses a transceiver module such as but not limited to a network interface card or a transceiver to implement the communication between the computing device 800 and other devices, such as the communication between the second processor 300 and the first processor 200.

[0213] The memory 806 stores executable program code, and the processor 804 executes the executable program code to implement the operation steps of the model inference method described in the above embodiments of FIGS. 5-6. The program code can include the code of one or more software units, and the one or more software units can be the allocation unit 410, the acquisition unit 420, the first address confirmation unit 430, and the equidistant carrying unit 440 shown in FIG. 7, wherein the allocation unit 410 is used to implement S510A-S560B and optional steps thereof in the embodiment of FIG. 5, the acquisition unit 420 is used to implement S610 and optional steps thereof in the embodiment of FIG. 6, the first address confirmation unit 430 is used to implement S620 and optional steps thereof in the embodiment of FIG. 6, and the equidistant carrying unit 440 is used to implement S630-S640 and optional steps thereof in the embodiment of FIG. 6. Here, the various units will not be expanded and described again.

[0214] The inference card 807 includes a processing unit and a power supply circuit, and the power supply circuit is used to supply power to the processing unit so that the processing unit executes the operation steps of the model inference method described in the above embodiments of FIGS. 5-6. For example, the processing unit can implement the functions of the above allocation unit 410 and equidistant carrying unit 440, and the function descriptions of the above multiple units can refer to the related content above, which will not be expanded and described again.

[0215] In a possible implementation, the processor 804 is the second processor 300, and the inference card 807 is the first processor 200. The processor 804 can read the program code in the memory 806 to implement the operation steps required by the second processor 300 in the above embodiments of FIGS. 5-6, and the inference card 807 can implement the operation steps required by the first processor 200 in the above embodiments of FIGS. 5-6 through the processing unit.

[0216] The application also provides a computing device cluster. As shown in FIG. 9, FIG. 9 is an example diagram of a computing device cluster provided by the application, which includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a notebook computer, or a smartphone.

[0217] Optionally, the memory 806 in one or more computing devices 800 in the computing device cluster can store instructions for implementing the operation steps required by the second processor 300 in the model inference method. The inference card 807 in each computing device 800 is used to implement the same operation steps required by the first processor 200 in the embodiments of FIGS. 5-6 described above.

[0218] Optionally, the memory 806 in one or more computing devices 800 in the computing device cluster can also store partial instructions for implementing the operation steps required by the second processor 300 in different model inference methods, respectively. In other words, the combination of one or more computing devices 800 can collectively execute instructions for implementing the operation steps required by the second processor 300 in the data exchange method. The inference card 807 in each computing device 800 is used to implement different operation steps required by the first processor 200 in the embodiments of FIGS. 5-6 described above.

[0219] The embodiments of the application also provide a computer program product containing instructions. The computer program product can be a software or program product containing instructions that can run on a computing device or be stored in any available medium. When the computer program product runs on at least one computing device, it causes the at least one computing device to perform the model inference method provided by the application.

[0220] The embodiments of the application also provide a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state disk), etc. The computer-readable storage medium includes instructions that instruct the computing device to perform the inference method based on the large language model, or instruct the computing device to perform the inference method.

[0221] Finally, it should be noted that: the above examples are used to illustrate the technical solutions of the present application, but not limited to them; although the present application is described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A model inference method, comprising: The method comprises: allocating a key-value cache for an inference model, the key-value cache comprising M storage blocks in a memory of a first processor of a computing device and N storage blocks in a memory of a second processor of the computing device, each storage block comprising a plurality of sub-storage blocks, adjacent two sub-storage blocks having an equal address interval, and each storage block being configured to store intermediate data generated by each layer in an order of X layers included in the inference model; obtaining a swap request for requesting to swap data in a first storage block of the M storage blocks to a second storage block of the N storage blocks; generating a swap instruction according to the swap request, and sending the swap instruction to the first processor, the swap instruction comprising a first address of the first storage block and a first address of the second storage block, and being configured to instruct the first processor to read the intermediate data in the first storage block to the second storage block according to the first address of the first storage block, the first address of the second storage block and the address interval.

2. The method of claim 1, wherein, The allocating the key-value cache for the inference model comprises: allocating a first storage space in the memory of the first processor and a second storage space in the memory of the second processor, the first storage space and the second storage space being address-continuous; allocating a continuous storage space for each layer in the first storage space and the second storage space in an order of layers of the inference model, respectively, wherein each continuous storage space is configured to store a K vector or a V vector of one layer; obtaining the M and the N, and determining the first address of the M storage blocks, the first address of the N storage blocks and the address interval according to the M, the N and a size of the sub-storage blocks, wherein each storage block in the M and N storage blocks is configured to store a K vector and a V vector of intermediate data of X layers of the inference model.

3. The method of claim 1, wherein, The allocating the key-value cache for the inference model comprises: allocating a first storage space in the memory of the first processor and a second storage space in the memory of the second processor, the first storage space and the second storage space being address-continuous; allocating a continuous storage space for each layer in the first storage space and the second storage space in an order of layers of the inference model, respectively, wherein each continuous storage space is configured to store a K vector and a V vector of one layer; obtaining the M and the N, and determining the first address of the M storage blocks, the first address of the N storage blocks and the address interval according to the M, the N and a size of the sub-storage blocks, wherein each storage block in the M and N storage blocks is configured to store a K vector or a V vector of intermediate data of X layers of the inference model.

4. The method according to any one of claims 1 to 3, characterized in that, The first processor is a general processor, and the second processor is an inference card, or the first processor is an inference card, and the second processor is a general processor, or the first processor and the second processor are both inference cards.

5. The method according to any one of claims 1 to 4, characterized in that, The generating the swap instruction according to the swap request comprises: obtaining an identification of the first storage block and the second storage block from the swap request; The first address of the first storage block and the first address of the second storage block are obtained according to the identification of the first storage block and the identification of the second storage block, and the exchange instruction is generated.

6. The method according to any one of claims 2 to 5, characterized in that, The length of the first storage space and the length of the second storage space are obtained according to model information and user configuration information, the model information including the network layer number X, and the user configuration information including the number M of storage blocks in the memory of the first processor, the number N of storage blocks in the memory of the second processor, and the size b of the sub-storage block.

7. A computing device, comprising: The computing device includes a first processor and a second processor, The first processor is configured to allocate a key-value cache for an inference model, the key-value cache including M storage blocks in the memory of the first processor, each storage block including a plurality of sub-storage blocks, the address interval between two adjacent sub-storage blocks being equal, and each storage block being configured to store intermediate data generated by each layer in the order of X layers included in the inference model. The second processor is configured to allocate a key-value cache for an inference model, the key-value cache including N storage blocks in the memory of the second processor. The second processor is configured to obtain an exchange request, generate an exchange instruction according to the exchange request, and send the exchange instruction to the first processor, the exchange request being used to request exchange of data in a first storage block of the M storage blocks to a second storage block of the N storage blocks, and the exchange instruction including the first address of the first storage block and the first address of the second storage block. The first processor is configured to read the intermediate data in the first storage block to the second storage block according to the first address of the first storage block, the first address of the second storage block, and the address interval.

8. The computing device of claim 7, wherein The first processor is configured to allocate a first storage space in the memory of the first processor, allocate a continuous storage space for each layer in the first storage space in the order of layers of the inference model, wherein each continuous storage space is used to store a K vector or a V vector of a layer, obtain the M, and determine the first address of the M storage blocks and the address interval according to the M and the size of the sub-storage block, and each storage block is used to store K vectors and V vectors of intermediate data of X layers of the inference model. The second processor is configured to allocate a second storage space in the memory of the second processor, allocate a continuous storage space for each layer in the second storage space in the order of layers of the inference model, wherein each continuous storage space is used to store a K vector or a V vector of a layer, obtain the N, and determine the first address of the N storage blocks and the address interval according to the N and the size of the sub-storage block, and each storage block is used to store K vectors and V vectors of intermediate data of X layers of the inference model.

9. The computing device of claim 7, wherein the first processor is configured to allocate a first storage space in the memory of the first processor, and allocate a continuous storage space for each layer in the first storage space according to the order of the layers of the inference model, wherein each continuous storage space is used to store a K vector and a V vector of a layer, obtain the M, and determine the first address of the M storage blocks and the address interval according to the M and the size of the sub-storage block, wherein each storage block is used to store a K vector or a V vector of intermediate data of an X layer of the inference model; the second processor is configured to allocate a second storage space in the memory of the second processor, and allocate a continuous storage space for each layer in the second storage space according to the order of the layers of the inference model, wherein each continuous storage space is used to store a K vector and a V vector of a layer, obtain the N, and determine the first address of the N storage blocks and the address interval according to the N and the size of the sub-storage block, wherein each storage block is used to store a K vector or a V vector of intermediate data of an X layer of the inference model.

10. The computing device of any of claims 7 to 9, wherein, The first processor is a general processor, and the second processor is an inference card, or the first processor is an inference card, and the second processor is a general processor, or the first processor and the second processor are both inference cards.

11. The computing device of any of claims 7-10, wherein the second processor is configured to obtain the first storage block and the second storage block from the exchange request; the second processor is configured to obtain the first address of the first storage block and the first address of the second storage block according to the identification of the first storage block and the identification of the second storage block, and generate the exchange instruction.

12. The computing device of any of claims 8 to 11, wherein, The length of the first storage space and the length of the second storage space are obtained according to model information and user configuration information, the model information includes the number X of network layers, and the user configuration information includes the number M of storage blocks in the memory of the first processor, the number N of storage blocks in the memory of the second processor, and the size b of the sub-storage block.

13. A computer-readable storage medium, characterized in that, The computer program instructions, when executed by a computing device, cause the computing device to perform the operation steps of the method of any of claims 1-6.

14. A computer program product comprising instructions, characterized in that, The instructions, when executed by a computing device, cause the computing device to perform the operation steps of the method of any of claims 1-6.

Images