Model reasoning method, computing device and related device

Abstract

The invention provides a model reasoning method, computing equipment and related equipment, and the method comprises the following steps: firstly, the computing equipment allocates a key value cache for a reasoning model, the key value cache comprises M storage blocks in a memory of a first processor and N storage blocks in a memory of a second processor of the computing equipment, each storage block comprises a plurality of sub-storage blocks, addresses and intervals of two adjacent sub-storage blocks are equal, and each storage block is used for storing intermediate data generated by each layer according to a sequence of X layers included in the inference model, so that when the computing equipment needs to exchange data of a first storage block of the M storage blocks to a second storage block of the N storage blocks, the data of the first storage block of the M storage blocks can be exchanged to the second storage block of the N storage blocks. According to the technical scheme, the first processor can read the data of one storage block at a time by using an equal-interval carrying interface or method according to the first address of the first storage block and the first address of the second storage block, so that the data exchange efficiency is high, the resource consumption is low, and the model reasoning efficiency is further improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence (AI), in particular to a model inference method, a computing device and related equipment. BACKGROUND

[0002] The intermediate results generated by a large language model (LLM) during inference need to be cached for quick access in subsequent inference processes. This is the key value cache (KV Cache) technology, and the cache area used to store intermediate data is also called KV cache. Before model inference, the system allocates KV cache according to network layers, and applies a continuous memory for each network layer. Since the memory obtained each time is not necessarily continuous with the memory obtained last time, the addresses of the continuous memories corresponding to different network layers are discontinuous and random. If a large number of requests are processed simultaneously on the device side, a large amount of intermediate data needs to be stored in the KV cache on the device side, resulting in insufficient space on the device side. At this time, the KV cache data stored on the device side is swapped to the KV cache on the host side. When the device side needs to use the KV cache data on the host side, the KV cache data stored on the host side is swapped to the KV cache on the device side.

[0003] Generally, when swapping KV cache data between the host side and the device side, the swapping is usually performed in units of storage blocks. A block includes multiple mini blocks, each of which comes from the continuous memory of a different network layer. Since the continuous memories of different network layers are discontinuous and random, the addresses of these mini blocks in each block are also discrete and irregular. As a result, reading the data of a block requires multiple data reading operations according to the addresses of each mini block. Such multiple discrete data reading operations increase the system overhead, resulting in low data swapping efficiency and further affecting the latency of model inference. SUMMARY

[0004] The present application provides a model inference method, a computing device and related equipment, which are used to solve the problem of low swapping efficiency of KV cache data swapping during model inference, which further affects the latency of model inference.

[0005] In a first aspect, a model inference method is provided. The method includes the following steps: a computing device assigns a key-value cache to an inference model, the key-value cache includes M blocks in a memory of a first processor of the computing device and N blocks in a memory of a second processor of the computing device, each block includes a plurality of mini blocks, adjacent two mini blocks have equal address intervals, and each block is used to store intermediate data generated by each layer in a sequence of X layers included in the inference model; an exchange request is obtained, the exchange request is used to request exchange of data in a first block of the M blocks to a second block of the N blocks; an exchange instruction is generated according to the exchange request, and the exchange instruction is sent to the first processor, the exchange instruction includes a first address of the first block and a first address of the second block, and is used to instruct the first processor to read the intermediate data in the first block to the second block according to the first address of the first block, the first address of the second block, and the address interval.

[0006] The method described in the first aspect is implemented. The computing device first assigns a key-value cache to the inference model, the key-value cache includes M blocks in a memory of a first processor and N blocks in a memory of a second processor of the computing device, each block includes a plurality of mini blocks, adjacent two mini blocks have equal address intervals, and each block is used to store intermediate data generated by each layer in a sequence of X layers included in the inference model. Thus, when the computing device obtains an exchange request for exchanging data in a first block of the M blocks to a second block of the N blocks, an exchange instruction including a first address of the first block and a first address of the second block can be generated according to the exchange request, and the exchange instruction is sent to the first processor, so that the first processor reads the data in the first block to the second block according to the first address of the first block, the first address of the second block, and the address interval. Since the addresses of the mini blocks in each block are not discrete but have equal address intervals, reading the data of a block does not require multiple addressing, and the data of a block can be read at one time based on the address interval, so that the data exchange efficiency is high, the resource consumption is low, and the efficiency of model inference is improved.

[0007] In a possible implementation, when the key-value cache is allocated for the inference model, a first storage space with continuous addresses can be allocated in the memory of the first processor, a second storage space with continuous addresses can be allocated in the memory of the second processor, and continuous storage spaces are allocated for each layer in the first storage space and the second storage space in the order of the layers of the inference model, where each continuous storage space is used to store a K vector or a V vector of a layer, M and N are obtained, and the first addresses of the M storage blocks, the first addresses of the N storage blocks, and the address interval are determined according to M, N, and the size of the sub-storage blocks, where each storage block is used to store a K vector and a V vector of intermediate data of an X layer of the inference model.

[0008] Optionally, the K cache and the V cache of each network layer can be continuous, where each network layer includes a continuous storage space storing a K vector, referred to as a K cache, and a continuous storage space storing a V vector, referred to as a V cache. For example, a first continuous storage space stores a K vector of an L0 layer, a second continuous storage space stores a V vector of the L0 layer, a third continuous storage space stores a K vector of an L1 layer, and a fourth continuous storage space stores a V vector of the L1 layer, and so on. Alternatively, a first continuous storage space stores a V vector of an L0 layer, a second continuous storage space stores a K vector of the L0 layer, a third continuous storage space stores a V vector of an L1 layer, and a fourth continuous storage space stores a K vector of the L1 layer, and so on.

[0009] Optionally, the K cache of each network layer can be continuous, and the V cache can be continuous, for example, the first X continuous storage spaces store K vectors of X network layers, and the last X continuous storage spaces store V vectors of the X network layers. Alternatively, the first X continuous storage spaces store V vectors of X network layers, and the last X continuous storage spaces store K vectors of the X network layers.

[0010] It should be understood that, in the conventional manner, because the system applies memory in sequence according to network layers, the memory addresses between different network layers are not continuous, resulting in that multiple mini blocks in a block are discrete blocks. When data exchange is performed in the unit of a block, because the addresses of multiple mini blocks in the same block have nothing to do with each other and are completely random, the conventional manner needs to read each mini block in sequence according to the address of each mini block, and each reading needs to be readdressed, so that the data exchange efficiency is low and resource consumption is large.

[0011] The above implementation mode stores one layer of K vectors or V vectors in each continuous storage space. After allocating the continuous storage space corresponding to each network layer, the data in a block includes the K vectors and V vectors of X network layers according to the current block division mode. Since the addresses of the continuous storage spaces are also continuous, the tail address of the last mini block in each continuous storage space is connected to the head address of the first mini block in the next continuous storage space. If the multiple mini blocks in each continuous storage space are numbered in ascending order according to the address sequence, each block includes the mini blocks with the same number in each continuous storage space, for example, block0 includes the first mini block in each continuous storage space in the address sequence, block1 includes the second mini block in each continuous storage space in the address sequence, and so on, so that the address interval between the multiple mini blocks in a block is the same, that is, the length of 1 continuous storage space, that is, the length of M mini blocks. When data exchange is performed in units of blocks, the interface or method using equal-interval transfer can read the data of a block at a time, so that the data exchange efficiency is high, the resource consumption is low, and the model inference efficiency is improved.

[0012] In another possible implementation mode, when the key-value cache is allocated for the inference model, a first storage space with continuous addresses is allocated in the memory of the first processor, a second storage space with continuous addresses is allocated in the memory of the second processor, and continuous storage spaces are allocated for each layer in the first storage space and the second storage space in the order of the layers of the inference model, wherein each continuous storage space is used to store the K vectors and V vectors of one layer. M and N are obtained, and the head addresses of M storage blocks, the head addresses of N storage blocks, and the address interval are determined according to M, N, and the size of the sub-storage block. Each storage block is used to store the K vectors or V vectors of the intermediate data of X layers of the inference model.

[0013] In a specific implementation, the continuous storage space of each network layer includes the K vectors and V vectors of the network layer, which can be stored in the order of V vectors first and then K vectors, or in the order of K vectors first and then V vectors, and the application does not make a specific limitation.

[0014] It should be noted that in the industry-standard terminology for large language models, the data of a block typically includes K vectors and V vectors. Under this partitioning method, a block only has K vectors or V vectors. Therefore, under this partitioning method, two blocks in the same network layer have the same block ID. Thus, when exchanging data by block, the data of a block still includes K vectors and V vectors (for example, there are two blocks 0. The first block 0 is called the K block, which is used to store the K vectors of X network layers, and the second block 0 is called the V block, which is used to store the V vectors of X network layers). This makes the technical solution of this application more adaptable and easier to implement.

[0015] In the above implementation, each contiguous storage space stores the K and V vectors of one layer. After allocating the contiguous storage space corresponding to each network layer, each contiguous storage space is divided into multiple mini-blocks of the same size with contiguous addresses according to the current block partitioning method. The data in a block includes the K or V vectors of X network layers. Since the addresses between contiguous storage spaces are also contiguous, the tail address of the last mini-block in each contiguous storage space is connected to the starting address of the first mini-block in the next contiguous storage space. If the multiple mini-blocks in each contiguous storage space are numbered in ascending order of address, and each block includes mini-blocks with the same number in each contiguous storage space, making the address intervals between multiple mini-blocks in a block the same, which is the length of one contiguous storage space, or the length of 2M mini-blocks. In this way, when exchanging data in blocks, the data of a block can be quickly read using an interface or method with equal spacing, resulting in high data exchange efficiency, low resource consumption, and thus improved model inference efficiency.

[0016] It should be understood that the above example provides a way to divide the first storage space into multiple contiguous storage spaces. This application can also divide contiguous storage spaces in other ways, which will not be illustrated here.

[0017] It should be noted that regardless of the method used to divide the first storage space, the addresses of the multiple contiguous storage spaces obtained are contiguous. This ensures that during data exchange, the address intervals between the multiple mini blocks in each block are the same. In this way, the data of a block can be quickly read using an interface or method that moves data at equal intervals, resulting in high data exchange efficiency, low resource consumption, and thus improving the efficiency of model inference.

[0018] In one possible implementation, when generating a swap instruction based on a swap request, the identifiers of the first and second storage blocks can be obtained from the swap request, and then the starting address of the first and second storage blocks can be obtained based on the identifiers of the first and second storage blocks to generate the swap instruction.

[0019] In a specific implementation, the computing device can pre-store a first address table and a second address table. The first address table stores the relationship between the starting address of each block in the first processor and the block identifier, as well as the first address interval between adjacent mini-blocks within each block. The second address table stores the relationship between the starting address of each block in the second processor and the block identifier, as well as the second address interval between adjacent mini-blocks within each block. Thus, when the second processor receives a swap request, it can determine the starting address and first address interval of the first storage block based on the first address table and the identifier of the first storage block, and determine the starting address of the second storage block based on the second address table and the identifier of the second storage block, thereby generating a swap instruction.

[0020] The above implementation, by pre-storing the relationship between the starting address of each block and the block identifier, as well as the corresponding address interval, enables the computing device to determine the starting address and address interval based on the block identifier in the exchange request when it receives an exchange request. This allows the first processor to move data from the starting address to the second storage block at equal intervals according to the address interval, based on the exchange instruction. This results in high exchange efficiency, low resource consumption, and thus improved inference efficiency of the model.

[0021] In one possible implementation, the lengths of the first and second storage spaces are obtained based on model information and user configuration information. The model information includes the number of network layers X, and the user configuration information includes the number of storage blocks M in the memory of the first processor, the number of storage blocks N in the memory of the second processor, and the size b of the sub-storage blocks.

[0022] Optionally, if each contiguous storage space stores either a K vector or a V vector, and a block includes K vectors and V vectors from X network layers, and the K and V caches are contiguous, then the number of contiguous storage spaces is 2X. The shape of the first storage space is (1, 2X, M, *v), indicating that a contiguous storage space stores one type of vector, such as a K vector or a V vector. The first storage space includes 2X contiguous storage spaces, M blocks, each contiguous storage space includes M miniblocks, each block includes 2X miniblocks, and each miniblock has a shape of *v. The first storage space includes a total of 2XM miniblocks. It should be understood that the user sets the shape of the miniblock to v, and when allocating storage blocks, the shape of the miniblock can be adaptively adjusted, such as by changing the expansion dimension; therefore, *v is used here to represent it.

[0023] Optionally, if each contiguous storage space stores K vectors and V vectors, and a block includes X K vectors or V vectors from network layers, then the data volume of the contiguous storage space is X, and the shape of the first storage space is (2, X, M, *v), indicating that a contiguous storage space stores two types of vectors, namely K vectors and V vectors. The first storage space includes X contiguous storage spaces and M blocks. Each contiguous storage space includes 2M miniblocks, and each block includes X miniblocks. The shape of each miniblock is *v.

[0024] The aforementioned implementation determines the length of the first storage space based on model information and user configuration, then requests the first storage space from the operating system, and allocates the first storage space sequentially according to network layers. Compared to traditional techniques, which require allocating memory for each network layer's K or V vector based on the inference model's network layers (if there are X network layers, the operating system needs to process 2X memory allocations), the model inference method provided in this application determines the size and shape of the first storage space based on user configuration information and model information, and allocates a contiguous block of memory at once to store the K and V vectors of multiple network layers. The operating system only needs to process one memory allocation.

[0025] In one possible implementation, the first processor can determine the starting address of the first memory block and the address interval according to the exchange instruction, and then read the first memory block from the first KV cache 210 using an interface or method for equal-interval transport.

[0026] If a contiguous storage space stores the K or V vector of a network layer, and a block includes X network layer K and V vectors, with an address interval of M miniblock lengths (Mb), then based on the address interval, 2X miniblocks can be read at once to obtain the first storage block.

[0027] If a contiguous memory space is used to store the K and V vectors of a network layer, and a block includes X K or V vectors from the same network layer, but the blocks corresponding to the K and V vectors of the same network layer have the same ID (meaning a block during data exchange includes a K block and a V block), where the K block contains multiple miniblocks storing K vectors and the V block contains multiple miniblocks storing V vectors, then: if the address interval is 2M miniblocks (2Mb), two reads are needed. The first read reads the K block based on its starting address, and the second reads the V block based on its starting address, obtaining the first memory block. If the address interval is M miniblocks (Mb), only one read is needed, directly retrieving the data from two blocks with the same ID (K block and V block), obtaining the first memory block.

[0028] Regardless of the method used to divide the first storage space, the data in the first storage block can be read into the second storage block through one or two transfer operations, based on the starting address of the first storage block to be transferred and the address intervals between adjacent mini-blocks within the block. It should be understood that traditional techniques require reading multiple discrete mini-blocks and sending them to the second processor. During data transmission, multiple messages need to be sent, each message header including the source and destination addresses of each mini-block. However, using the technical solution provided in this application, since the addresses and destination addresses of the read mini-blocks are equally spaced, multiple mini-blocks in the first storage block can be packaged into a single message and transmitted using a single message header. This improves data transmission efficiency, resulting in high data exchange efficiency, low resource consumption, and ultimately, improved model inference efficiency.

[0029] In one possible implementation, the first processor is a general-purpose processor, and the second processor is an inference card; or the first processor is an inference card, and the second processor is a general-purpose processor; or both the first and second processors are inference cards. For example, the first processor is an NPU, and the second processor is a CPU. When the NPU runs the inference model, it stores the intermediate data generated during the inference process in the NPU's KV cache. When the NPU's storage is insufficient, the model inference method provided in this application can be used to swap the data in the first storage block of the NPU to the second storage block of the CPU.

[0030] In a second aspect, a computing device is provided, comprising a first processor and a second processor. The first processor is configured to allocate a key-value cache for an inference model. The key-value cache includes M storage blocks located in the memory of the first processor. Each storage block includes multiple sub-storage blocks, and the address interval between two adjacent sub-storage blocks is equal. Each storage block is used to store intermediate data generated by each layer in the order of X layers included in the inference model. The second processor is configured to allocate a key-value cache for the inference model. The key-value cache includes N storage blocks located in the memory of the second processor. The processor acquires a swap request, generates a swap instruction based on the swap request, and sends the swap instruction to the first processor. The swap request requests the swap of data from the first storage block in the M storage blocks to the second storage block in the N storage blocks. The swap instruction includes the starting address of the first storage block and the starting address of the second storage block. The first processor is configured to read intermediate data from the first storage block to the second storage block based on the starting address of the first storage block, the starting address of the second storage block, and the address interval.

[0031] The computing device described in the second aspect first allocates a key-value cache for the inference model. This key-value cache includes M storage blocks in the memory of the first processor and N blocks in the memory of the second processor of the computing device. Each block includes multiple sub-blocks, and the addresses and intervals between adjacent mini-blocks are equal. Each block is used to store intermediate data generated by each layer in the order of X layers included in the inference model. Thus, when the computing device receives an exchange request to exchange the data of the first block of M blocks to the second storage block of N storage blocks, it can generate an exchange instruction including the starting address of the first block and the starting address of the second block according to the exchange request, and send the exchange instruction to the first processor. This allows the first processor to read the data of the first block to the second block according to the starting address of the first block, the starting address of the second block, and the address interval. When the first processor and the second processor exchange data in blocks, the data of a block can be read at once using an interface or method with equal spacing, resulting in high data exchange efficiency, low resource consumption, and thus improved model inference efficiency.

[0032] In one possible implementation, a first processor is configured to allocate a contiguous first storage space in its memory, and allocate contiguous storage space for each layer in the first storage space according to the order of the layers of the inference model. Each contiguous storage space is used to store the K vector or V vector of a layer. M is obtained, and the starting address and address interval of M storage blocks are determined according to M and the size of the sub-storage blocks. Each storage block is used to store the K vector and V vector of the intermediate data of the X layer of the inference model. A second processor is configured to allocate a contiguous second storage space in its memory, and allocate contiguous storage space for each layer in the second storage space according to the order of the layers of the inference model. Each contiguous storage space is used to store the K vector or V vector of a layer. N is obtained, and the starting address and address interval of N storage blocks are determined according to N and the size of the sub-storage blocks. Each storage block is used to store the K vector and V vector of the intermediate data of the X layer of the inference model.

[0033] In another possible implementation, a first processor is configured to allocate a contiguous first storage space in its memory, and allocate contiguous storage space for each layer in the order of the inference model, wherein each contiguous storage space is used to store the K vector and V vector of a layer. M is obtained, and the starting address and address interval of M storage blocks are determined based on M and the size of the sub-storage blocks. Each of the M storage blocks is used to store the K vector or V vector of intermediate data for layer X of the inference model. A second processor is configured to allocate a contiguous second storage space in its memory, and allocate contiguous storage space for each layer in the order of the inference model, wherein each contiguous storage space is used to store the K vector and V vector of a layer. N is obtained, and the starting address and address interval of N storage blocks are determined based on N and the size of the sub-storage blocks. Each of the N storage blocks is used to store the K vector or V vector of intermediate data for layer X of the inference model.

[0034] In one possible implementation, the first processor is a general-purpose processor and the second processor is an inference card, or the first processor is an inference card and the second processor is a general-purpose processor, or both the first processor and the second processor are inference cards.

[0035] In one possible implementation, the second processor is configured to obtain the identifiers of the first storage block and the second storage block from the exchange request, and the second processor is configured to obtain the starting address of the first storage block and the starting address of the second storage block based on the identifiers of the first storage block and the second storage block, and generate an exchange instruction.

[0036] In one possible implementation, the lengths of the first and second storage spaces are obtained based on model information and user configuration information. The model information includes the number of network layers X, and the user configuration information includes the number of storage blocks M in the memory of the first processor, the number of storage blocks N in the memory of the second processor, and the size b of the sub-storage blocks.

[0037] Thirdly, a computing device is provided, comprising a first processor, a second processor, and a memory, wherein the memory is used to store instructions, and the second processor is used to execute the instructions, such that the computing device implements the method described in the first aspect.

[0038] Fourthly, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and the instructions are executed by a computing device or a cluster of computing devices to implement the method described in the first aspect.

[0039] Fifthly, a computing device cluster is provided, the computing device cluster including at least one computing device, each computing device including a processor and a memory, the processor of the at least one computing device being configured to execute instructions stored in the memory of the at least one computing device, such that the computing device cluster implements the method described in the first aspect.

[0040] In a sixth aspect, a computer program product comprising instructions is provided, the computer program product including instructions capable of running on a computing device or stored in any available medium, and when the computer program product is run on a computing device or a cluster of computing devices, causing the computing device or cluster of computing devices to perform the method described in the first aspect.

[0041] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0042] Figure 1 This is an example diagram of storage block partitioning for KV cache in a traditional scheme;

[0043] Figure 2 This is an architecture diagram of a reasoning system provided in this application;

[0044] Figure 3 This is an example diagram of a computing device deployed in a cloud environment according to this application;

[0045] Figure 4 This is an example diagram of another division method provided in this application;

[0046] Figure 5 This is a flowchart illustrating the steps of a model reasoning method provided in this application during the initialization phase.

[0047] Figure 6 This is a flowchart illustrating the steps of a model reasoning method provided in this application during the application phase.

[0048] Figure 7 This is a schematic diagram of the software structure of a computing device provided in this application;

[0049] Figure 8 This is a schematic diagram of the hardware structure of a computing device provided in this application;

[0050] Figure 9 This is an example diagram of a computing device cluster provided in this application. Detailed Implementation

[0051] First, the application scenarios involved in this application will be explained.

[0052] LLM (Learning-Based Natural Language Processing) is a deep learning-based natural language processing technique that can simulate human language abilities to generate coherent text, and it is currently a hot research topic in the field of AI. LLM is an autoregressive model that performs multiple forward computations during inference. Each forward computation generates a predicted word, and multiple predicted words form the output text. In each forward computation, the first n predicted words are used as input to predict the (n+1)th predicted word. Therefore, some intermediate data from the previous forward computation can be reused in the next. Cached intermediate data from each forward computation avoids redundant computation in subsequent computations, thus improving the efficiency of model inference. This technique is also known as key-value caching (KV Cache), and the cache used to store intermediate data is called the KV cache.

[0053] However, after processing multiple requests using KV cache technology, the KV cache undergoes frequent allocation and release, resulting in small, non-contiguous free blocks and memory fragmentation. This limits the allocation of subsequent large memory blocks and impacts system memory utilization efficiency. To address this issue, the KV cache is typically divided into multiple storage blocks, with each allocation and release operation performed on a block-by-block basis. This not only avoids the generation of numerous small fragments but also allows the system to allocate or release storage blocks as needed, optimizing memory utilization and ultimately improving the performance of large model inference.

[0054] Figure 1 This is an example diagram of storage block partitioning for a traditional KV cache. For example... Figure 1 As shown, the KV cache is divided into N storage blocks, each block includes multiple mini blocks. Each mini block is the smallest unit of memory allocation. Mini blocks in a column belong to the same block, mini blocks in a row belong to different blocks, multiple mini blocks in the same row are contiguous free blocks and are represented by the same color, and mini blocks in different rows are non-contiguous free blocks and are represented by different colors.

[0055] Before model inference, the system allocates key-value caches according to network layers, requesting a contiguous block of memory for each layer. Since the memory allocated each time is not necessarily contiguous with the previously allocated memory, this allows for more flexible allocation without limiting the key-value cache. Therefore, the addresses of the contiguous memory allocated to different network layers are ultimately non-contiguous and random. After each network layer obtains its corresponding contiguous memory, it divides its allocated contiguous memory into multiple miniblocks according to the miniblock size. The final key-value cache can be configured as follows: Figure 1 As shown, the intermediate data of each network layer includes key (K) vectors and value (V) vectors. The multiple miniblocks in the first row are used to store the K vectors of the L2 layer, the multiple miniblocks in the second row are used to store the V vectors of the L1 layer, and so on. Examples are not provided here. Because the contiguous memory corresponding to each network layer is not contiguous, therefore... Figure 1 In this architecture, the addresses of the last miniblock in each row and the first miniblock in the next row are not contiguous and have no relation. This allows intermediate data in each network layer to be allocated and released as needed, in units of miniblocks, thus avoiding the generation of small fragments of varying granularity and improving memory utilization.

[0056] When the model begins inference, if the system processes a large number of requests simultaneously, the V vector generated by the L1 layer for each request will be stored in multiple miniblocks in the second row, and the K vector generated by the L2 layer for each request will be stored in multiple miniblocks in the first row. The capacity of the KV cache limits the model's concurrency capability. To solve this problem, the device running the inference model will swap the entire block of data to other devices when space is insufficient. Typically, we refer to the device running the inference model as the device side, and other devices as the host side. When space is insufficient on the device side, data from the KV cache on the device side will be swapped to the KV cache on the host side. The second processor side will also have similar storage. Figure 1 The KV cache shown here will not be explained further on the host-side KV cache.

[0057] During data exchange, to improve efficiency, the device side will swap a large number of miniblocks that are not currently in use to the host side. For ease of memory management, data exchange is typically performed in units of blocks. Figure 1 Swap one column of data shown. For example... Figure 1Block 0 in the code includes miniblocks such as L2K, L1V, L5V...L(n-1)K. It should be understood that data is exchanged column-by-block because miniblocks in the same row store the K or V vectors of the same network layer for multiple requests. If data were exchanged row-by-row, the K and V vectors of the currently being processed request in that row would also be swapped out. Column-by-column exchange avoids this problem.

[0058] However, as mentioned above, each block comprises multiple miniblocks, which store K and V vectors from different network layers. This means each block consists of multiple non-contiguous, discrete small storage blocks. This necessitates numerous discrete read operations during data exchange. Each read operation requires a different address, increasing system overhead and reducing data exchange efficiency. Similarly, if the device needs the host to exchange the block's data back, the host also requires discrete reads of numerous miniblocks, increasing system overhead and reducing data exchange efficiency.

[0059] In summary, during the inference process, when the KV cache space on the device side is insufficient and it is necessary to exchange the KV cache data on the device side to the host side, the data exchange is usually carried out in units of blocks. However, a block includes multiple non-contiguous and discrete miniblocks. Reading a large number of discrete miniblocks will increase the system overhead, resulting in low data exchange efficiency, which in turn affects the latency of model inference.

[0060] To address the issue of low exchange efficiency of KV cache data between the device and host sides during the inference process, which affects the inference latency of the model. This application provides a model inference method. In this method, a computing device first allocates a key-value cache for the inference model. The key-value cache includes M storage blocks in the memory of a first processor and N blocks in the memory of a second processor of the computing device. Each block includes multiple sub-blocks. The addresses and intervals of two adjacent mini-blocks are equal. Each block is used to store intermediate data generated by each layer in the order of X layers included in the inference model. Thus, when the computing device receives an exchange request to exchange the data of the first block of M blocks to the second storage block of N storage blocks, it can generate an exchange instruction including the starting address of the first block and the starting address of the second block according to the exchange request, and send the exchange instruction to the first processor. The first processor reads the data of the first block to the second block according to the starting address of the first block, the starting address of the second block, and the address interval. When the first processor and the second processor exchange data in blocks, the data of a block can be read at one time using an interface or method with equal spacing, resulting in high data exchange efficiency, low resource consumption, and thus improving the efficiency of model inference.

[0061] Figure 2 This application provides an architecture diagram of an inference system 1000, which includes a terminal device 100 and a computing device 400. The terminal device 100 and the computing device 400 establish a communication connection via a network. This communication connection can be wired or wireless. The network can be the public internet, a local area network (LAN), a virtual private network (VPN), a dedicated line such as fiber optic lines, copper wires, or satellite connections, or a wireless network such as wireless fidelity (Wi-Fi) or a cellular network. This application does not specifically limit the number of such connections. The number of terminal devices 100 and computing devices 400 in the inference system 1000 can be one or more. Figure 2 This example uses four terminal devices 100 and one computing device 400 as illustrations, but this application does not impose any specific limitations on this.

[0062] Terminal device 100 is used to realize human-computer interaction. The user holding terminal device 100 is the user who uses the inference model to complete inference. Terminal devices include personal computers, smartphones, wearable devices, handheld processing devices, tablets, mobile laptops, augmented reality (AR) devices, virtual reality (VR) devices, smart conferencing devices, etc., and are not specifically limited here. The description of computing devices can be referred to the foregoing content, and will not be repeated here.

[0063] The computing device 400 may include bare metal servers (BMS), virtual machines, containers, or storage devices. A BMS refers to a general-purpose physical server, such as an ARM server or an x86 server; a virtual machine refers to a complete computer system simulated by software, possessing full hardware system functionality and running in a completely isolated environment. Any task that can be performed on a physical computer can also be performed in a virtual machine. When creating a virtual machine on a computing device, a portion of the physical machine's hard drive and memory capacity is used as the virtual machine's hard drive and memory capacity. Each virtual machine has an independent basic input / output system (BIOS), hard drive, and operating system, and can be operated like a physical machine. A container is a portable software unit that can combine an application and all its dependencies into a single software package. This package is not limited by the underlying host operating system, thus eliminating the need to build complex environments and simplifying the application development and deployment process.

[0064] Optionally, the terminal device 100 and the computing device 400 may be the same computing device; this application does not impose any specific limitations.

[0065] Furthermore, the computing device 400 may include a first processor 200 and a second processor 300. Optionally, the first processor 200 is a general-purpose processor and the second processor 300 is an inference card; or, the first processor 200 is an inference card and the second processor 300 is a general-purpose processor; or, both the first processor 200 and the second processor 300 are inference cards. It should be noted that the number of the first processor 200 and the second processor 300 may be one or more. Figure 2 The following description uses a computing device 400 that includes a first processor 200 and a second processor 300 as an example; however, this application does not impose any specific limitations.

[0066] In specific implementations, inference cards refer to dedicated processors for neural network algorithms, such as graphics processing units (GPUs), neural network processing units (NPUs), and data processing units (DPUs), etc. This application does not impose specific limitations. In some scenarios, inference cards are also referred to as accelerator cards.

[0067] In specific implementations, the general-purpose processor can be any one or more of the following processors: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP), digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), complex programmable logic device (CPLD), etc. This application does not impose any specific limitation.

[0068] The computing device 400 can also be a cluster of computing devices composed of multiple computing devices. The first processor 200 and the second processor 300 can be deployed in the same computing device or in different computing devices. If they are deployed in different computing devices, the computing devices in which the first processor 200 and the second processor 300 are located can belong to the same or different computing device clusters. Figure 2 The following explanation will be based on the example of the first processor 200 and the second processor 300 being located within the same computing device 400.

[0069] Optionally, the terminal device 100 may be a user-controlled terminal device with a client running on it. The client may be software or an application, such as a personal computer (PC) client, a web client accessed through a browser, an application (APP) client running on a mobile terminal, or a console of a cloud platform. This application does not make any specific limitations.

[0070] Optionally, the client running on the terminal device 100 can be a client specifically designed to implement a machine learning inference platform based on an inference model, a client for a model service platform, or a client running a model inference framework. Such clients provide a series of tools and services for the deployment, management, and inference of machine learning models. Taking a large language model as an example, the client could be ModelArts, OpenAI, etc., without specific limitations in this application. The large language model may include, but is not limited to, GPT, Bing Chat, Copilot, Gemini, etc., without specific limitations in this application. The inference framework may include Faster Transformer, Text Generation Inference (TGI), Vectorized Large Language Model (vLLM), etc., without specific limitations in this application.

[0071] Optionally, the client running on the terminal device 100 can also be a comprehensive client that includes model inference functionality. Such a client may include not only model inference functionality but also other functions, the specifics of which depend on the actual application scenario. For example, a client for an after-sales robot can use the inference function of a large language model based on the user's input after-sales consultation question to obtain the corresponding output text. It can also provide other functions such as order processing, returns, and account management. Similarly, a client for a chatbot can use the inference function of a large language model based on the user's input text to obtain the corresponding output text. It can also provide other functions such as playing music, movies, and recommending news. It should be understood that large language models, due to their powerful natural language generation capabilities, have a wide range of applications, and their model inference capabilities can be integrated into comprehensive clients with different functions. The above examples are for illustrative purposes only and are not intended to limit the scope of the application.

[0072] Optionally, the client running on the terminal device 100 can also be a cloud platform client, used for users to purchase and rent various cloud services. The model inference method provided in this application can be one of these cloud services. Users can purchase this cloud service separately to accelerate the data exchange efficiency of model inference, thereby accelerating the model inference efficiency. Alternatively, the cloud platform can provide users with a comprehensive service, such as a cloud service for inference of a large language model. After the user purchases this cloud service, the large language model will use the model inference method provided in this application to perform model inference when it runs.

[0073] The above describes the possible implementation methods of terminal device 100 and computing device 400. The following will illustrate the implementation methods of terminal device 100 and computing device 400 with specific application scenarios.

[0074] For example, suppose the client running on terminal device 100 is an office assistant for a company. Employees can use this office assistant for real-time problem solving, technical support, data analysis, and other operations. In this scenario, computing device 400 is a server purchased by the company, and terminal device 100 is either the company's office computer or an employee's mobile phone. Employees can use terminal device 100 to run the office assistant client, input a question through the client, and terminal device 100 sends the question to computing device 400. Computing device 400 can run a large language model to obtain the corresponding answer text and then return the answer text to terminal device 100, which displays the answer text in the office assistant client. When running the large language model to obtain the answer text, computing device 400 can use the model inference method provided in this application.

[0075] In another application scenario, computing device 400 can be deployed in a cloud environment. For example... Figure 3 This is an example diagram of a computing device deployed in a cloud environment, as provided in this application. Figure 3 As shown, a user can initiate a purchase request for a large language model inference cloud service through terminal device 100. After terminal device 100 sends the purchase request to the cloud platform, the cloud platform can grant terminal device 100 access to the cloud service of computing device 400. This allows the user to send input text to computing device 400 through terminal device 100. Computing device 400 then runs the large language model and infers the corresponding output text. During model operation, if the storage space of the first processor 200 is insufficient, some data in the KV cache needs to be swapped to the second processor 300. In this case, the model inference method provided in this application can be executed.

[0076] The cloud platform also maintains various basic resources, including computing resources, storage resources, network resources, and security resources, to meet the computing needs of the computing device 400 under different scales and loads. Furthermore, these computing resources can be dynamically scaled according to the usage needs of the computing device 400 to ensure the stable operation of the computing device 400.

[0077] It should be understood that the above application scenarios are for illustrative purposes only. The actual deployment of computing devices 400 and terminal devices 100 can be flexibly adapted to specific business needs. These scenarios will not be illustrated here.

[0078] In this embodiment, the first processor 200 is the device side mentioned above, and the second processor 300 is the host side mentioned above. The second processor 300 can receive input text sent by the user. The second processor 300 can establish communication connections with multiple first processors 200, and then select one of them to send the input text to that first processor 200 for processing. The first processor 200 can run a large language model to infer the output text corresponding to the input text. During the inference process, if the KV cache space in the first processor 200 is insufficient, the data in the KV cache can be swapped to the second processor 300, and the second processor 300 will swap the KV cache data that the first processor 200 needs back to the first processor 200.

[0079] In this embodiment of the application, when the computing device allocates KV cache for the inference model, it can allocate M blocks in the memory of the first processor 200 and N storage blocks in the memory of the second processor. Each block includes multiple mini blocks, and the address interval between two adjacent mini blocks is equal. Each block is used to store the intermediate data generated by each layer in the order of X layers included in the inference model.

[0080] Specifically, a first memory space with contiguous addresses is allocated in the memory of the first processor 200 (that is...) Figure 2 The first KV cache 210 in the memory of the second processor 300 allocates a second memory space with contiguous addresses (that is, the first KV cache 210 in the memory of the second processor 300). Figure 2 The second KV cache 310 in the first storage space and the second storage space are used to allocate contiguous storage space for each layer in the order of each layer of the inference model. Each contiguous storage space is used to store the K vector or V vector of one layer. Then, the above M and N are obtained. Based on M, N and the size of miniblock, the starting address of M storage blocks, the starting address of N storage blocks and the address interval of adjacent mini blocks in each block are determined. Each block is used to store the K vector and V vector of the intermediate data of the X layer of the inference model.

[0081] Furthermore, the first processor 200 stores a first address table 220, and the second processor 300 stores a second address table 320. The first address table stores the starting addresses of M blocks in the first processor 200, and the second address table 320 stores the starting addresses of N blocks in the second processor 300. Thus, when the first processor 200 and the second processor 300 exchange data in blocks, the first processor 200 can determine the starting address of the first target block based on the first address table 220, and then read the data of multiple mini-blocks in the first target block into the second processor 300 at once, according to the address intervals between mini-blocks. Similarly, the second processor 300 can determine the starting address of the second target block based on the second address table 320, and then read the data of multiple mini-blocks in the second target block into the first processor 200 at once, according to the address intervals between mini-blocks. This avoids a large number of discrete read operations, reduces the system overhead of data exchange, and improves the efficiency of data exchange.

[0082] In one possible implementation, each contiguous storage space is used to store the K vector or V vector of a network layer, and each block is used to store the K vector and V vector of intermediate data of the X layer of the inference model.

[0083] Specifically, the computing device 400 can allocate a contiguous first storage space in the memory of the first processor 200, and then allocate contiguous storage space for each network layer according to the order of each network layer of the inference model. The contiguous storage space of each network layer is used to store the K vector or V vector of that network layer. The computing device can determine the starting address and address interval of the M storage blocks according to the size of M and miniblock, and update the first starting address table 220.

[0084] After allocating the contiguous storage space corresponding to each network layer, each contiguous storage space is divided into multiple mini blocks of the same size with contiguous addresses, according to the current block partitioning method. The data in a block includes the K vectors and V vectors of X network layers. Since the addresses between contiguous storage spaces are also contiguous, the tail address of the last mini block in each contiguous storage space is connected to the starting address of the first mini block in the next contiguous storage space. If the multiple mini blocks in each contiguous storage space are numbered in ascending order of address, each block includes mini blocks with the same number in each contiguous storage space. For example, block 0 includes the mini block with the first address in each contiguous storage space, block 1 includes the mini block with the second address in each contiguous storage space, and so on. This ensures that the address intervals between multiple mini blocks in a block are the same, which is the length of one contiguous storage space, or the length of M mini blocks.

[0085] Optionally, the K vector and V vector of the same network layer can be stored in two adjacent contiguous memory spaces, that is, the K cache and V cache of the same network layer have contiguous addresses. Figure 2 For example, the first KV cache 210 is the first storage space, which is divided into 2X contiguous storage spaces according to the number of network layers X. The addresses of these contiguous storage spaces are contiguous, and the end address of each contiguous storage space is connected to the beginning address of the next contiguous storage space. Each contiguous storage space is used to store the K vector or V vector of one network layer, for example... Figure 2 Each row in the diagram represents a contiguous storage space. The contiguous storage spaces Add0 to Add5 corresponding to the first row are the K-cache of the L0 layer, used to store the K vectors of the L0 layer. The contiguous storage spaces Add6 to Add11 corresponding to the second row are the V-cache of the L0 layer, used to store the V vectors of the L0 layer. The contiguous storage spaces Add12 to Add17 corresponding to the third row are the K-cache of the L1 layer, used to store the vectors of the L1 layer. In this way, the K-cache and V-cache corresponding to each network layer are contiguous, and the K-cache and V-cache of adjacent network layers are also contiguous.

[0086] After dividing each contiguous storage space into M consecutive mini-blocks according to the number of blocks M, each contiguous storage space includes 6 mini-blocks. The address range of each mini-block is represented by Add0 to Addx. The address range of the last mini-block in each row is contiguous with the address range of the first mini-block in the next row. For example, the address range of the last mini-block in the first row is Add5, and the address range of the first mini-block in the second row is Add6. The address ranges Add5 and Add6 are contiguous. In this way, a block includes a column of mini-blocks with the same address interval. For example, in block 0, the address range of the first mini-block is Add0, the address range of the second mini-block is Add6, the address range of the third mini-block is Add12, and so on. The address interval of any two adjacent mini-blocks is the same, which is the length of the address range of 6 mini-blocks, that is, the length of one contiguous storage space. Figure 2 For illustrative purposes only, this application does not limit the number of blocks.

[0087] It should be understood that, Figure 1 In the traditional approach shown, the system requests memory sequentially according to the network layer, resulting in non-contiguous memory addresses between different network layers. This causes multiple mini-blocks within a block to be discrete blocks. When exchanging data on a block-by-block basis, since the addresses of multiple mini-blocks within the same block are completely random and have no relation to each other, the traditional approach requires reading each mini-block's address sequentially. Each read requires re-addressing, resulting in low data exchange efficiency and high resource consumption.

[0088] However, in Figure 2 In the data exchange method provided by this application, the system first allocates a first storage space as a KV cache, and then divides the first storage space into multiple contiguous storage spaces according to the number of network layers. The addresses between the multiple contiguous storage spaces are also contiguous. After dividing each contiguous storage space based on the size of the miniblock, the address intervals of the multiple miniblocks in each block are the same. In this way, when exchanging data in units of blocks, the data of a block can be read at one time using an interface or method that moves data at equal intervals. This makes the data exchange efficiency high and the resource consumption low, thereby improving the efficiency of model inference.

[0089] It should be noted that,Figure 2 In the example, the K cache address of each network layer takes precedence over the V cache address, or the V cache address of each network layer takes precedence over the K cache address, for example... Figure 2 The first contiguous storage space is used to store L0V, the second contiguous storage space is used to store L0K, the third contiguous storage space is used to store L1V, the fourth contiguous storage space is used to store L1K, and so on. Examples will not be repeated here.

[0090] Optionally, Figure 2 The method shown is that the K cache and V cache are contiguous. Alternatively, the K cache and V cache can be contiguous, meaning that the addresses of the contiguous storage spaces storing K vectors are contiguous, and the addresses of the contiguous storage spaces storing V vectors are contiguous. For example, the first X contiguous storage spaces can be used to store the K vectors of X network layers, and the last X contiguous storage spaces can be used to store the V vectors of X network layers, or the first X contiguous storage spaces can be used to store the V vectors of X network layers, and the last X contiguous storage spaces can be used to store the K vectors of X network layers.

[0091] Similarly, the second processor 300 can also divide the second storage space in the manner described above, which will not be repeated here.

[0092] In another possible implementation, each contiguous storage space is used to store the K vectors and V vectors of X network layers, and each block is used to store either the K vectors or V vectors of X network layers.

[0093] Specifically, after the computing device allocates a contiguous first storage space to the first processor 200, it can allocate contiguous storage space for each network layer in the first storage space according to the order of each layer of the inference model. The contiguous storage space of each network layer is used to store the K vector and V vector of that network layer. The computing device can determine the starting address and address interval of the M blocks according to the size of M and miniblock, and update the first starting address table 220.

[0094] After allocating the contiguous storage space corresponding to each network layer, each contiguous storage space is divided into multiple mini blocks of the same size with contiguous addresses, according to the current block partitioning method. The data in a block includes K vectors or V vectors of X network layers. Since the addresses between contiguous storage spaces are also contiguous, the tail address of the last mini block in each contiguous storage space is connected to the starting address of the first mini block in the next contiguous storage space. If the multiple mini blocks in each contiguous storage space are numbered in ascending order of address, and each block includes mini blocks with the same number in each contiguous storage space, making the address intervals between multiple mini blocks in a block the same, this is the length of one contiguous storage space, or the length of 2M mini blocks.

[0095] For example, Figure 4 This is an example diagram of another division method provided in this application. Figure 4 In this diagram, each row represents a contiguous storage space, and each contiguous storage space includes 2M miniblocks. The first M miniblocks are used to store the K vector, and the last M miniblocks are used to store the V vector of the network layer. Each block includes X miniblocks, used to store either the K vector or the V vector of the network layer. For example, block 0 in the first column includes X miniblocks to store the K vector, and block 0 in the 7th column includes X miniblocks to store the V vector.

[0096] It should be noted that in the industry-standard terminology for large language models, a block of data typically includes both a K vector and a V vector. Figure 4 In the example shown, a block only has either a K vector or a V vector, so... Figure 4 When partitioning the KV cache as shown, two blocks in the same network layer have the same block ID. Therefore, when exchanging data by block, the data in a block still includes both the K vector and the V vector (e.g., ...). Figure 4 There are two blocks0 in the block, namely the first column and the seventh column. The first column of the block stores the K vectors of X network layers, and the seventh column of the block stores the V vectors of X network layers. This makes the technical solution of this application more adaptable and easier to implement.

[0097] Similarly, the second processor 300 can use the same partitioning method to partition the second storage space, which will not be elaborated here.

[0098] It should be understood that the above text, in combination with... Figure 2 and Figure 4 Two exemplary methods for dividing the first storage space into multiple contiguous storage spaces are provided. This application can also divide contiguous storage spaces in other ways, which will not be illustrated here.

[0099] It should be noted that regardless of the method used to divide the first storage space, the addresses of the multiple contiguous storage spaces obtained are contiguous. This ensures that during data exchange, the address intervals between the multiple mini blocks in each block are the same. This allows the use of an interface or method that moves data at equal intervals to read the data of a block at once, resulting in high data exchange efficiency, low resource consumption, and thus improved model inference efficiency.

[0100] It should be understood that Figure 2 and Figure 4 To facilitate explanation, a graphical method is used. In KV cache, a row of miniblocks represents a contiguous storage space, and a column of miniblocks represents a block. In actual application scenarios, KV cache is a contiguous segment of physical memory. Each block and contiguous storage space is not horizontally or vertically divided on physical memory, but rather virtually partitioned, making memory management more efficient and the solution clearer.

[0101] It should be understood that Figure 2 and Figure 4 The components present in the first processor 200 and the second processor 300 are simply drawn. These components are involved in the model inference method of this application. In order to clearly illustrate the technical solution of this application, some necessary components are not drawn in the figure. In actual business scenarios, the first processor 200 and the second processor 300 may also include more components, such as communication interfaces, memory, storage media and other components, to ensure the normal operation of the first processor 200 and the second processor 300. This application does not make specific limitations.

[0102] It should be noted that the first address table 220 and the second address table 320 can be as follows: Figure 2 As shown, the first address table 220 is stored in the first processor 200 and the second processor 300, respectively. In some possible implementations, the first address table 220 may also be stored in the second processor 300, or in the memory of the computing device. The second address table 320 may also be stored in the memory of the computing device. This application does not make any specific limitations.

[0103] In summary, the model inference system provided in this application first allocates a key-value cache for the inference model. This key-value cache includes M storage blocks in the memory of the first processor and N blocks in the memory of the second processor of the computing device. Each block includes multiple sub-blocks, and the addresses and intervals of two adjacent mini-blocks are equal. Each block is used to store the intermediate data generated by each layer according to the order of the X layers included in the inference model. Thus, when the computing device receives an exchange request to exchange the data of the first block of M blocks to the second storage block of N storage blocks, it can generate an exchange instruction including the starting address of the first block and the starting address of the second block according to the exchange request, and send the exchange instruction to the first processor. This allows the first processor to read the data of the first block to the second block according to the starting address of the first block, the starting address of the second block, and the address interval. When the first processor and the second processor exchange data in blocks, the data of a block can be read at one time using an interface or method with equal spacing, resulting in high data exchange efficiency, low resource consumption, and thus improving the efficiency of model inference.

[0104] The above text combined Figures 2-4 The model inference system provided in this application is described in detail below, in conjunction with... Figure 5 and Figure 6 The model inference method provided in this application will be explained, wherein the model inference method may include an initialization phase and an application phase. Figure 5 The steps and procedures of the initialization phase are explained below. Figure 6 The steps and procedures in the application phase are explained.

[0105] Figure 5 This is a flowchart illustrating the steps of a model inference method provided in this application during the initialization phase. This method can be applied to, for example... Figures 2-4 The first processor 200 and the second processor 300 shown.

[0106] S510: The second processor 300 obtains an initialization request. This initialization request is used to notify the first processor 200 and the second processor 300 to begin allocating their respective KV caches.

[0107] In a specific implementation, the initialization request can be generated by the computing device when initializing the system, or it can be a request sent to the computing device by other devices. These other devices can be the management node of the cluster where the computing device is located. The management node can coordinate the work between multiple computing devices and send the initialization request to the computing device 400 at an appropriate time. This application does not make any specific limitations.

[0108] In a specific implementation, the initialization request may include model information and user configuration, used to allocate KV caches to the first processor 200 and the second processor 300 respectively. The user configuration refers to the configuration of the training user input for training the inference model. Optionally, the model information may include the number of network layers X of the inference model, and the user configuration information may include the total number of blocks M in the first processor, the total number of blocks N in the second processor, the shape v of the mini-block, the size b of the mini-block, the partitioning method, etc. The user configuration can be a user-input custom configuration or a system default configuration. The model information and user configuration information may also include more content, which is not specifically limited in this application.

[0109] The shape v of the miniblock describes the data stored in the miniblock, and the partitioning method refers to how the K vector and V vector are stored in the contiguous storage space of each layer.

[0110] For example, the first division method could refer to Figure 2 The corresponding partitioning method involves storing either a K vector or a V vector in each contiguous storage space. A block includes K vectors and V vectors from X network layers, and the K cache and V cache are contiguous.

[0111] The second division method could refer to... Figure 4 The corresponding partitioning method involves storing K vectors and V vectors in each contiguous storage space, and a block includes K vectors or V vectors from X network layers.

[0112] The third division method is also as follows: Figure 2 As shown, each contiguous storage space stores either a K vector or a V vector. A block includes the K vectors and V vectors of X network layers, but... Figure 2 The difference is that the K cache is continuous and the V cache is continuous. That is, the first X consecutive memory spaces store the K vectors of X network layers, and the last X consecutive memory spaces store the V vectors of X network layers.

[0113] Of course, there are other ways to classify them, but we will not give examples of them all here.

[0114] It should be noted that some information in the user configuration can also be default, so the initialization request may not include this default information. For example, if the partitioning method is default, the initialization request may not include the partitioning method. This application does not impose any restrictions on this.

[0115] S520: The second processor 300 sends an initialization command to the first processor 200.

[0116] In a specific implementation, the second processor 300 can generate an initialization instruction that the first processor 200 can recognize based on the initialization request, and then send the initialization instruction to the first processor 200. The initialization instruction may include the number of blocks M, the shape v of the miniblock, the size b of the miniblock, the partitioning method, the number of network layers X, etc., in the first processor 200. Of course, if the first processor 200 has set some or all of this information by default, the initialization instruction may not carry the default settings; this application does not limit this.

[0117] It should be noted that, in Figure 5 In the example shown, the second processor 300 notifies the first processor 200 to start initializing the KV cache. In other possible implementations, the first processor 200 may also notify the second processor 300 to start initializing the KV cache, or both processors may obtain the initialization request themselves. This application does not make any specific limitations.

[0118] S530A: The first processor 200 is allocated a contiguous first memory space. The first memory space is... Figure 2 The first KV cache 210 is shown.

[0119] In practical implementation, the KV cache includes memory, disk, hybrid storage, flash memory, cloud storage, etc. The KV cache is used to store intermediate data generated during the inference model's execution. This inference model can include a large language model. It should be understood that a large language model is an autoregressive model. When inferring the output text from the input text, it requires multiple forward computations. Each forward computation yields a predicted word, and the input data for each forward computation includes the predicted word from the previous computation, until all predicted words are obtained in the output text. Therefore, caching the intermediate data from the previous forward computation in the KV cache reduces redundant computations in subsequent forward computations, thereby improving the inference efficiency of the large language model.

[0120] Typically, this intermediate data may include the key (K) vector and value (V) vector calculated by each network layer. The K vector is the vector from the large language model's attention layer weighted by the first weight matrix W. k The vector V, obtained after linear transformation, is the vector from the attention layer to the input attention layer vector using the second weight matrix W. vAfter linear transformation, since the weight matrix of each network layer remains unchanged during the inference phase, and the input vector for each forward calculation includes the input vector of the previous forward calculation, the K and V vectors obtained from each forward calculation are cached in the KV cache. The next forward calculation can directly read the previously calculated K and V vectors from the KV cache, which can reduce redundant calculations and improve the computational efficiency of each forward calculation, thereby improving the inference efficiency of the large language model.

[0121] In one embodiment, the length information of the first storage space can be determined based on the size b of the mini-block, the number of network layers X, and the total number of blocks M. The first storage space comprises a total of 2 × M mini-blocks, therefore the length of the first storage space is at least 2 × Mb. The shape of the first storage space can be determined by the partitioning method and the shape v of the mini-blocks, as described below. Figure 2 and Figure 4 Taking two partitioning methods as examples, we will explain the possible shapes of the first storage space.

[0122] Alternatively, if according to Figure 2 The first partitioning method shown stores either a K vector or a V vector in each contiguous storage space. A block includes K vectors and V vectors from X network layers, and the K and V caches are contiguous. In this case, the number of contiguous storage spaces is 2X. The shape of the first storage space is (1, 2X, M, *v), indicating that a contiguous storage space stores one type of vector, such as a K vector or a V vector. The first storage space includes 2X contiguous storage spaces, M blocks, each contiguous storage space includes M miniblocks, each block includes 2X miniblocks, and each miniblock has a shape of *v. The first storage space includes a total of 2XM miniblocks. It should be understood that the user sets the shape of the miniblock to v. When allocating storage blocks, the shape of the miniblock can be adaptively adjusted, such as by expanding the dimensions, hence the use of *v here.

[0123] Alternatively, if according to Figure 4The second partitioning method shown stores K vectors and V vectors in each contiguous storage space. A block includes X K vectors or V vectors from network layers. In this case, the data volume of the contiguous storage space is X. The shape of the first storage space is (2, X, M, *v), which means that a contiguous storage space stores two types of vectors, namely K vectors and V vectors. The first storage space includes X contiguous storage spaces and M blocks. Each contiguous storage space includes 2M miniblocks. Each block includes X miniblocks. The shape of each miniblock is *v.

[0124] It should be understood that the above example provides... Figure 2 and Figure 4 The shape of the first storage space is determined by the two partitioning methods. Other partitioning methods can also use a similar method to determine the shape of the first storage space, which will not be illustrated here.

[0125] It should be understood that in traditional technologies, the first processor needs to allocate memory for the K vector or V vector of each network layer according to the network layers of the inference model. If there are X network layers, the operating system needs to process 2X memory allocations. However, in the model inference method provided in this application, the first processor determines the size and shape of the first storage space based on user configuration information and model information, and allocates a contiguous block of memory at once to store the K vectors and V vectors of multiple network layers. The operating system only needs to process one memory allocation.

[0126] S530B: The second processor 300 allocates a contiguous second memory space. The first memory space is... Figure 2 The second KV cache 310 is shown.

[0127] It should be noted that the second processor 300 can also request a second storage space as a second KV cache 310 in the manner described in S530A. Similarly, if the total number of blocks of the second processor 300 is N, the length of the first storage space requested by the second processor 300 is 2XNb.

[0128] In the case of the first division method, that is Figure 2 In the partitioning method shown, the number of contiguous storage spaces is 2X, and the shape of the first storage space is (1, 2X, N, *v). In the second partitioning method, that is... Figure 4 In the partitioning method shown, the number of contiguous storage spaces is X, and the shape of the first storage space is (2, X, N, * v). For related descriptions, please refer to the previous description; they will not be repeated here.

[0129] It should be noted that the second processor 300 and the first processor 200 can execute S530A and S530B simultaneously, or they can execute S530A and S530B sequentially. Specifically, the sequential execution can include: the first processor 200 executes S530A first, and then the second processor 300 executes S530B; or, the second processor 300 executes S530B first, and then the first processor 200 executes S530A; or, the first processor 200 and the second processor 300 process S530A and S530B sequentially, and then process S540A and S540B sequentially, and so on; or, the first processor 200 executes S530A to S560A, and then the second processor 300 executes S530B to S560B; or, the second processor 300 executes S530B to S560B, and then the first processor 200 executes S530A to S560A. This application does not impose specific limitations on these cases.

[0130] It should be understood that the suffixes A and B in each step below indicate that the two processors can execute the step simultaneously or sequentially. The description is similar to that of S530A and S530B above, and will not be elaborated on here.

[0131] S540A: The first processor 200 allocates contiguous storage space for each layer in the first storage space according to the order of the layers of the inference model.

[0132] In a specific implementation, the first processor 200 can divide the first storage space into multiple contiguous storage spaces of the same size, and the addresses of these contiguous storage spaces are consecutive. If the multiple contiguous storage spaces are numbered sequentially according to their addresses, the addresses of two consecutive storage spaces with adjacent numbers are consecutive, and the starting address of each contiguous storage space is exactly the ending address of the previous consecutive storage space. For example, if the first storage space is divided into 10 contiguous storage spaces, numbered 0 to 9, then the addresses of spaces 0 and 1 are consecutive, the addresses of spaces 1 and 2 are consecutive, the addresses of spaces 2 and 3 are consecutive, and so on. The above example is for illustration only, and this application does not impose any specific limitations.

[0133] In the specific implementation, when dividing the first storage space into multiple contiguous storage spaces, the number and size of the contiguous storage spaces can be determined according to the division method. According to the network layer order and the address order of the contiguous storage spaces, the corresponding contiguous storage space is allocated to each network layer in turn. Each contiguous storage space is used to store the intermediate data of the same network layer.

[0134] For example, according to Figure 2The partitioning shown uses 2X contiguous storage spaces, each with a length of Mb, and each contiguous storage space is used to store the K vector or V vector of one network layer. The first contiguous storage space ( Figure 2 The first row of squares corresponding to Add0 to Add5 is used to store the K vector of layer L0, and the second contiguous storage space ( Figure 2 The second row of squares corresponding to Add6 to Addd11 is used to store the K vector of layer L0, and the third contiguous storage space ( Figure 2 (Using the third row of squares corresponding to Add12 to Addd17 as representations) to store the K vector of the L1 layer, and so on, without giving examples here.

[0135] For another example, if we follow Figure 4 The partitioning shown has X contiguous storage spaces, each with a length of 2Mb. Each contiguous storage space stores the K vector and V vector of the same network layer. The first contiguous storage space ( Figure 4 The first row of squares corresponding to Add0 to Add11 is used to store the K vector and V vector of layer L0, and the second contiguous storage space ( Figure 4 (Represented by the second row of squares corresponding to Add12 to Add23) are used to store the K vector and V vector of the L1 layer, and so on. Examples are not given here.

[0136] It should be understood that the above refers to Figure 2 and Figure 4 Two partitioning methods have been given as examples. The technical solution of this application can also support other partitioning methods, such as sequentially assigning the V vector and K vector in each row, that is... Figure 2 In the diagram, the first row is L0V, the second row is L0K, the third row is L1V, the fourth row is L1K, and so on. Alternatively, you can first assign the K vector to each network layer, and then assign the V vector to each network layer, which is... Figure 2 In the diagram, rows 1 to X are L0K to LXK respectively, and rows X+1 to 2X are L0V to LXV respectively. There are many other ways to divide the data, which can be found in the previous section and will not be repeated here. Of course, other ways of dividing the data also have corresponding quantities and lengths, which will not be illustrated here. For example, this application does not impose specific limitations.

[0137] S540B: ​​The second processor 300 allocates contiguous storage space for each layer in the second storage space according to the order of the layers of the inference model.

[0138] It should be noted that the second processor 300 can also divide the second storage space in the manner described in S540A, allocating contiguous storage space for each layer. If according to... Figure 2The partitioning method shown has 2X consecutive storage spaces, each with a length of Nb, and each consecutive storage space is used to store the K vector or V vector of one network layer. If we follow... Figure 4 The partitioning method shown has X consecutive storage spaces, each with a length of 2Nb. Each consecutive storage space is used to store the K vector and V vector of the same network layer. For detailed descriptions, please refer to the description of the first storage space in S540A, which will not be repeated here.

[0139] It should be noted that the second processor 300 and the first processor 200 can execute S540A and S540B simultaneously, or they can execute S540A and S540B sequentially. For a detailed explanation of the sequential execution, please refer to S530A and S530B, which will not be repeated here.

[0140] S550A: The first processor 200 determines the starting address and address interval of the M blocks based on the number of blocks M and the size of the miniblock.

[0141] In a specific implementation, the first processor 200 can divide the continuous storage space corresponding to each network layer into multiple mini blocks with consecutive addresses and the same size, according to the size b and shape v of the mini block. If the multiple mini blocks are numbered in address order, the addresses of two adjacent mini blocks are consecutive, and the starting address of each mini block is exactly the ending address of the mini block with the previous number.

[0142] In specific implementation, if according to Figure 2 The first storage space is divided into 2X contiguous storage spaces as shown. Each contiguous storage space stores the K vector or V vector of one network layer. The first processor 200 then divides each contiguous storage space into M mini-blocks according to the size b and shape v of the mini-block, where M is the number of blocks configured in the first processor by the user. If... Figure 4 The first storage space is divided into X contiguous storage spaces in the manner shown. The first processor 200 then divides each contiguous storage space into 2M miniblocks according to the size b and shape v of each miniblock. Similarly, the second processor 300 can also divide the contiguous storage space using the same logic, which will not be repeated here.

[0143] In the specific implementation, after dividing each contiguous storage space into mini blocks, the mini blocks contained in each block can be determined, thereby determining the starting address and address interval of each block. Each storage block is used to store the intermediate data generated by each layer according to the order of the X layers included in the inference model. Each block includes multiple mini blocks, which come from different contiguous storage spaces, and the address interval between any two adjacent mini blocks is equal, consisting of the length of one contiguous storage space. After determining the mini blocks included in each block, the starting address of each block can be obtained from the addresses of the multiple mini blocks. This address interval refers to the difference in address between two sequentially adjacent mini blocks after the mini blocks in the block are sorted by address order.

[0144] It should be noted that the first address table can store the mapping relationship between the block information and the starting address of each block. The block information may include the block's identifier (ID) or the block's address, for example... Figure 2 As shown, block0 corresponds to the starting address Add0. The above example is for illustration only and is not specifically limited in this application. This address interval can be stored in the first starting address table or in other locations, which is not specifically limited in this application.

[0145] Specifically, if the multiple mini blocks divided from each contiguous storage space are numbered according to address order, each block includes multiple mini blocks with the same number from different contiguous storage spaces. For example... Figure 2 or Figure 4 In the example, each mini-block in block0 comes from a different contiguous memory space, and is always the first mini-block in address order. Similarly, each mini-block in block1 comes from a different contiguous memory space, and is always the second mini-block in address order. The starting address of each block is the address of each mini-block within the first contiguous memory space. Based on this, the first address table can be updated. Figure 2 The first address table 220 in the document is an example, and this application does not impose any specific limitations.

[0146] Alternatively, if according to Figure 2The method shown divides the first storage space into 2X contiguous storage spaces, each containing M mini blocks. At this point, one contiguous storage space is... Figure 2 The image shows a row of miniblocks; one block is... Figure 2 The diagram shows a series of mini blocks, each block comprising 2X mini blocks. These blocks share the same address sequence in contiguous memory. For example, each mini block in block 5 is the last mini block in its corresponding row of memory, i.e., the fifth mini block in the column. The starting address of each block is the address of every mini block in the first contiguous memory space in terms of address sequence, i.e., the address of the mini block in the first row. The starting address of block 0 is Add0, the starting address of block 1 is Add1, and so on. The address interval is the length of M mini blocks (Mb).

[0147] Alternatively, if according to Figure 4 The method shown divides the first storage space into X contiguous storage spaces, each containing 2M mini-blocks. At this point, one contiguous storage space is... Figure 4 The image shows a row of miniblocks. A block consists of a Kblock for storing K vectors and a Vblock for storing V vectors, as shown below. Figure 4 Block 0 in the array includes X miniblocks (K blocks) in column 1 and X miniblocks (V blocks) in column 7. Within the K blocks of the same block, each miniblock has the same address sequence number in contiguous memory space. Similarly, within the V blocks of the same block, each miniblock has the same address sequence number in contiguous memory space. For example... Figure 4 In block0, the Kblock storing the K vector is the first column of data, and each miniblock is the first storage block in its row (contiguous storage space). The Vblock storing the V vector is the 7th column of data, and each miniblock is the 7th storage block in its row (contiguous storage space). The starting address of the Kblock in block0 is Add0, and the starting address of the Vblock is Add6. The starting address of the Kblock in block1 is Add1, and the starting address of the Vblock is Add7, and so on. These will not be explained in detail here.

[0148] It should be noted that in the industry-standard terminology for large language models, during data exchange, a block of data typically includes both a K vector and a V vector. Figure 4 In the example shown, a block only has either a K vector or a V vector, so... Figure 4 When partitioning the KV cache as shown, two blocks within the same network layer have the same block ID. Each block corresponds to two miniblocks, called Kblock and Vblock respectively. The Kblock stores the K vector for each network layer, and the Bblock stores the V vector for each network layer. Thus, when exchanging data in blocks, the data in a block still includes both the K vector and the V vector (e.g., ...). Figure 4 There are two blocks0 in the block, namely the first column and the seventh column. The first column of the block stores the K vectors of X network layers, and the seventh column of the block stores the V vectors of X network layers. This makes the technical solution of this application more adaptable and easier to implement.

[0149] It should be noted that, for Figure 4 In the example shown, the address interval can be either 2M miniblocks or M miniblocks. If it's 2M miniblocks, to read data from block 0 during data exchange, it's necessary to first read block K from block 0 based on the address interval, and then read block V based on the address interval. If it's M miniblocks, data from block K and block V can be read directly based on the address interval. Both methods can use interfaces or methods with equal-interval data transfer to complete the data exchange, improving data exchange efficiency.

[0150] It should be noted that the first address table in the first processor 200 can be stored in the first processor 200 or in other memory of the computing device 400; this application does not make any specific limitation.

[0151] It should be understood that, for ease of explanation, this application proposes a method of numbering the mini blocks in each contiguous memory space to determine the mini blocks included in each block; that is, a block includes mini blocks with the same number in each contiguous memory space. This numbering is only for better explanation. In actual processing, numbering may not be used. The mini blocks with the first address in each contiguous memory space belong to the same block1, and the mini blocks with the second address in each contiguous memory space belong to the same block. They can be determined directly according to the address order without numbering. This application does not limit this.

[0152] S550B: The second processor 300 determines the starting address and address interval of the N blocks based on the number of blocks N and the size of the miniblock.

[0153] In a specific implementation, the second processor 300 can also determine the starting address of N blocks, update the second starting address table 320, and determine the address interval in the manner described in S550A. This will not be elaborated on here.

[0154] It should be noted that the second processor 300 and the first processor 200 can execute S540A and S540B simultaneously, or they can execute S540A and S540B sequentially. For a detailed explanation of the sequential execution, please refer to S530A and S530B, which will not be repeated here.

[0155] It should be understood that the model inference method provided in this application, during the initialization phase, requests a first storage space from the operating system as a KV cache. Then, the first storage space is divided into multiple contiguous storage spaces, and each contiguous storage space is further divided into multiple mini-blocks. This ensures that the addresses of mini-block1 corresponding to the tail address of each contiguous storage space and mini-block2 corresponding to the first address of the next contiguous storage space are consecutive. Thus, if the mini-blocks in each contiguous storage space are numbered sequentially by address, the starting address interval between mini-blocks with the same number in each contiguous storage space is the same, which is the length of a contiguous storage space. Therefore, when exchanging data in blocks, using an interface or method with equal-interval data transfer allows multiple mini-blocks to be read at once, resulting in high data exchange efficiency, low resource consumption, and thus improved model inference efficiency.

[0156] The above text combined Figure 5 The steps and flow of the model inference method provided in this application during the initialization phase are explained below. Figure 6 The steps and procedures of the model reasoning method provided in this application during the application stage are explained.

[0157] Figure 6 This is a flowchart illustrating the steps of a model reasoning method provided in this application during the application phase, as shown below. Figure 6 As shown, the method may include the following steps:

[0158] S610: Second processor 300 receives an exchange request.

[0159] Optionally, the exchange request includes information about the first storage block, which is the source block to be read from the first processor 200. The information about the first storage block may include the block's identification (ID) or address; the following example will use ID. As mentioned above, the first processor 200 and the second processor 300 exchange data in blocks, so the exchange request includes the block's ID. The number of IDs can be one or more. For example, suppose the first processor 200... Figure 2 As shown, the first storage block includes data stored in blocks 0 to 3, so the data exchange request includes the ID information of blocks 0 to 3. The above example is for illustration only and is not intended to be specific.

[0160] Optionally, the exchange request may also include information about a second storage block, wherein the second storage block is the target block to be stored in the second processor 300, and the information about the second storage block may include the block's ID or address.

[0161] In a specific implementation, the exchange request can be generated by the second processor 300 of the computing device 400, or by other computing devices, such as the management node, based on the inference of the inference model. The management node can be the scheduling and management role in the distributed inference architecture when the inference model is distributed across multiple computing devices, and is used to schedule and manage multiple computing devices 400.

[0162] In practice, the exchange request can be generated under certain conditions, such as when the first KVcache of the first processor 200 is close to full load or exceeds a preset threshold. Alternatively, the exchange request can be generated at a fixed frequency, or it can be generated periodically. It can also be manually initiated by the development user who is training the inference model, or there can be many other conditions that trigger the generation of the exchange request. These will not be listed here.

[0163] S620: The second processor 300 sends an exchange command to the first processor 200.

[0164] Optionally, if the second processor 300 stores a first address table, or the second address table 320 is stored in the memory of the computing device 400, the second processor 300 can determine the starting address of the first memory block according to the first address table, determine the starting address of the second memory block according to the second address table, obtain the address interval between miniblocks in the first processor 200, and generate the exchange instruction. The exchange instruction includes the starting address of the first memory block, the starting address of the second memory block, and the address interval.

[0165] Optionally, if the second processor 300 does not store the first address table, but only stores it in the first processor 200, the second processor 300 can also determine the starting address of the second memory block based on the second address table, and then generate a swap instruction. The swap instruction includes the block identifier of the first memory block and the starting address of the second memory block. The first processor 200 then queries the first address table 220 to determine the starting address and address interval of the first memory block based on its block identifier. Both of these methods are acceptable, and this application does not impose a specific limitation.

[0166] S630: The first processor 200 reads the first memory block into the second memory block according to the starting address of the first memory block and the second memory block and the address interval.

[0167] In a specific implementation, the first processor 200 can determine the starting address and address interval of the first memory block according to the swap instruction, and then use an interface or method for equal-interval transport to read the first memory block from the first KV cache 210. In a specific implementation, if the swap instruction includes the IDs of multiple blocks, multiple blocks can be read one block at a time as described above.

[0168] If the first KV cache 210 is in accordance with Figure 2 The data is divided in a specific way: a contiguous storage space is used to store the K vector or V vector of one network layer, and a block includes the K vectors and V vectors of X network layers, which is... Figure 2 In a column of miniblocks, the address interval is the length of M miniblocks (Mb). At this time, based on the address interval, 2X miniblocks can be read at once to obtain the first storage block.

[0169] If the first KV cache 210 is in accordance with Figure 4 The data is divided in a contiguous storage space to store the K vector and V vector of a network layer. A block includes X K vectors or V vectors of network layers. However, the IDs of the blocks corresponding to the K vectors and V vectors of the same network layer are the same. That is, a block during data exchange includes a K block and a V block. A K block includes multiple mini blocks that store K vectors, and a V block includes multiple mini blocks that store V vectors.

[0170] In this scenario, if the address interval is 2M miniblocks (2Mb), two reads are required: first, read the K block based on its starting address, and second, read the V block based on its starting address to obtain the first storage block. If the address interval is M miniblocks (Mb), one read is required, directly retrieving the data from two blocks with the same ID, namely the K block and the V block, to obtain the first storage block.

[0171] For example, with Figure 4 For example, if we need to read data from block0, which includes the Kblock in column 1 and the Vblock in column 7, we can use the length of Add0-Add5 as the address interval (M = 6 miniblocks) and read Add0, Add6, Add12, Add18... at equal intervals to obtain the data of two blocks0 at once, thus obtaining the first storage block. Alternatively, we can use the length of Add0-Add11 as the address interval (2M = 12 miniblocks) and read at equal intervals twice. The first time, we read X miniblocks from Kblock at equal intervals (Add0, Add12...), and the second time, we read X miniblocks from Vblock at equal intervals (Add6, Add18...), thus obtaining the first storage block. The above examples are for illustration only and are not intended to limit the scope of this application.

[0172] In specific implementations, the interface for equally spaced data transfer can include the aclrtMemcpy2d interface, which is an affinity-based equally spaced small storage block transfer interface. This interface can be called to transfer the data of the entire block at once, thereby reducing the number of read operations and avoiding discrete reading of a large number of small miniblocks, thus improving the efficiency of data exchange.

[0173] Optionally, data exchange efficiency can be further optimized based on actual business needs. For example, before calling the interface, process-core binding can be performed, binding the process or thread calling the data exchange interface to the processor core of the first processor 200. This ensures that the data transfer capability between the processor core of the second processor and the first processor can be maximized, thereby ensuring the efficient transfer of the first storage block and further improving data exchange efficiency.

[0174] For another example, when initializing the KV cache in the second processor 300, the pin-memory option can be enabled to ensure that data is placed in pin-memory. The memory pages in pin-memory will not be swapped out to virtual memory (disk page file) by the operating system, but will always remain in physical memory. This avoids data being stored in memory pages and then swapped to disk pages, which would result in performance loss.

[0175] It should be understood that the above examples provide some possible operational steps for optimizing data exchange efficiency. The model inference method of this application can also be combined with other operational steps to further optimize data exchange efficiency, which will not be illustrated here.

[0176] It should be understood that traditional technology requires reading multiple discrete miniblocks and sending them to a second processor. During data transmission, multiple messages need to be sent, and the header of each message includes the source address and destination address of each miniblock. However, using the technical solution provided in this application, since the addresses and destination addresses of the read miniblocks are equally spaced, multiple miniblocks in the first storage block can be packaged into a single message and transmitted using a single message header. This improves data transmission efficiency, resulting in high data exchange efficiency, low resource consumption, and ultimately, improved model inference efficiency.

[0177] Optionally, after the first processor 200 obtains the first storage block according to S630, it can also use data transfer technologies such as Direct Memory Access (DMA) and Remote Direct Memory Access (RDMA) to directly write the first storage block into the second storage block of the second processor 300 according to the destination address information of the first storage block, which is the identifier of the second storage block, without the CPU intervention of the second processor 300, thereby achieving efficient data transfer.

[0178] It should be understood that after the first processor 200 executes S630, it releases the space of the first storage block of the KV cache. The first storage block can be used to store the KV vectors in the inference process. After that, if the KV cache space of the first processor 200 is sufficient, the second processor 300 can execute S640 to swap the data in the third storage block of the second processor 300 with the data in the fourth storage block of the first processor 200.

[0179] S640: The second processor 300 reads the third memory block to the fourth memory block according to the starting address of the third memory block and the address interval.

[0180] In a specific implementation, the second processor 300 can also obtain an exchange request carrying representations of the third and fourth storage blocks. The third storage block is the source block, located in the second KV cache 310 of the second processor 300, and the fourth storage block is the target block, located in the first KV cache 210 of the first processor 200. The second processor 300 can obtain the starting address and address interval of the third storage block from the second starting address table 320 according to the ID of the third storage block. Then, through an interface or method for equal-interval transport, it reads multiple mini blocks in the third storage block at equal intervals and writes them into the fourth storage block of the first processor 200. For details, please refer to the relevant description of S630, which will not be repeated here.

[0181] It should be noted that S630 and S640 can be executed simultaneously or sequentially. In the sequential execution case, S630 can be executed followed by S640. In this case, the exchange request in S610 also includes the identifiers of the third and fourth storage blocks, for the second processor 300 to execute S640. Alternatively, S630 can be executed, and S640 can be executed after a certain period of time. In this case, the second processor 300 can execute S610 again. This time, the exchange request in S610 includes the identifiers of the third and fourth storage blocks, for the second processor 300 to execute S640.

[0182] In simple terms, the first processor 200 and the second processor 300 can simultaneously exchange memory blocks to each other's memory. Alternatively, depending on the actual processing situation, S630 can be executed when the first processor 200 has insufficient memory, and S640 can be executed when the first processor has sufficient memory or when the first processor needs to use data from the third storage block. This application does not make any specific limitations.

[0183] In summary, the model inference method provided in this application involves the computing device first allocating a key-value cache for the inference model. This key-value cache includes M storage blocks in the memory of the first processor and N blocks in the memory of the second processor of the computing device. Each block includes multiple sub-blocks, and the addresses and intervals between adjacent mini-blocks are equal. Each block is used to store intermediate data generated by each layer in the order of the X layers included in the inference model. Thus, when the computing device receives an exchange request to exchange the data of the first block of M blocks to the second storage block of N storage blocks, it can generate an exchange instruction including the starting address of the first block and the starting address of the second block according to the exchange request, and send the exchange instruction to the first processor. This allows the first processor to read the data of the first block to the second block according to the starting address of the first block, the starting address of the second block, and the address interval. When the first processor and the second processor exchange data in blocks, the data of a block can be read at once using an interface or method with equal spacing, resulting in high data exchange efficiency, low resource consumption, and thus improved model inference efficiency.

[0184] The above text describes in detail the model inference method provided in this application. The following section will combine... Figure 7 The software structure of the computing device provided in this application will be explained and described. This computing device can be... Figures 1-6 The computing device 400 shown.

[0185] Figure 7 This is a schematic diagram of the software structure of a computing device 400 provided in this application, such as... Figure 7 As shown, the computing device 400 includes an allocation unit 410, an acquisition unit 420, a starting address confirmation unit 430, and an equally spaced transport unit 440. It should be understood that... Figure 7 This is an exemplary partitioning method. The computing device 400 can be divided into more or fewer unit modules, and this application does not make any specific limitations.

[0186] The allocation unit 410, acquisition unit 420, first address confirmation unit 430, and equal-spacing transport unit 440 can all be implemented in software or in hardware. For example, the implementation of allocation unit 410 will be described below. Similarly, the implementation of acquisition unit 420, first address confirmation unit 430, and equal-spacing transport unit 440 can refer to the implementation of allocation unit 410.

[0187] As an example of a software functional unit, allocation unit 410 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, allocation unit 410 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0188] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0189] As an example of a hardware functional unit, allocation unit 410 may include at least one computing device, such as a server. Alternatively, allocation unit 410 may be implemented using a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The aforementioned PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system-on-a-chip (SoC), an offload card, an inference card, or any combination thereof.

[0190] The multiple computing devices included in the allocation unit 410 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the allocation unit 410 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the allocation unit 410 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offload cards, and inference cards.

[0191] It should be noted that, in other embodiments, the steps implemented by the allocation unit 410, the acquisition unit 420, the first address confirmation unit 430, and the equidistant transport unit 440 can be specified as needed. The allocation unit 410, the acquisition unit 420, the first address confirmation unit 430, and the equidistant transport unit 440 respectively implement different steps in the model inference method to realize all the functions of the computing device 400.

[0192] The functions of the allocation unit 410, acquisition unit 420, first address confirmation unit 430, and equidistant transport unit 440 are explained below. The computing device 400 includes a first processor 200 and a second processor 300.

[0193] Allocation unit 410 is used to allocate key-value cache for the inference model. The key-value cache includes M storage blocks located in the memory of the first processor of the computing device and N storage blocks located in the memory of the second processor of the computing device. Each storage block includes multiple sub-storage blocks, and the address interval between two adjacent sub-storage blocks is equal. Each storage block is used to store intermediate data generated by each layer in the order of X layers included in the inference model. The allocation unit can be implemented by the first processor 200 and the second processor 300. For details, please refer to [reference needed]. Figure 5 S530A to S560B of the embodiments and their optional steps.

[0194] The acquisition unit 420 is used to acquire a swap request, generate a swap instruction based on the swap request, and send the swap instruction to the first processor. The swap request requests the swap of data from the first storage block out of M storage blocks to the second storage block out of N storage blocks. The swap instruction includes the starting address of the first storage block and the starting address of the second storage block, instructing the first processor to read intermediate data from the first storage block to the second storage block based on the starting address of the first storage block, the starting address of the second storage block, and the address interval. The acquisition unit 420 can be implemented by the second processor 300; see reference for details. Figure 6 S610 to S620 and their optional steps in the embodiments.

[0195] The equidistant transfer unit 440 is used to read intermediate data from the first storage block to the second storage block according to the starting address of the first storage block, the starting address of the second storage block, and the address interval. The equidistant transfer unit 440 can be implemented by the first processor 200 or the second processor 300; see reference for details. Figure 6 S630, S640 and their optional steps in the embodiments.

[0196] In one possible implementation, allocation unit 410 is used to allocate a contiguous first storage space in the memory of the first processor and a contiguous second storage space in the memory of the second processor. Contiguous storage space is allocated in both the first and second storage spaces according to the order of the inference model layers. Each contiguous storage space is used to store the K vector or V vector of one layer. M and N are obtained, and the starting addresses of the M and N storage blocks, as well as the address intervals, are determined based on M, N, and the size of the sub-storage blocks. In the M and N storage blocks, each storage block is used to store the K vector and V vector of the intermediate data of the X layer of the inference model. Specifically, this is used to implement... Figure 5 S530A to S560B and their optional steps in the embodiment, this implementation method corresponds to Figure 2 The division method shown.

[0197] In another possible implementation, allocation unit 410 is used to allocate a contiguous first storage space in the memory of the first processor and a contiguous second storage space in the memory of the second processor. Contiguous storage space is allocated in both the first and second storage spaces according to the order of the inference model layers. Each contiguous storage space is used to store the K vector and V vector of one layer. M and N are obtained, and the starting addresses of M and N storage blocks, the starting addresses of N storage blocks, and the address intervals are determined based on M, N, and the size of the sub-storage blocks. In the M and N storage blocks, each storage block is used to store the K vector or V vector of the intermediate data of the X layer of the inference model. Specifically, this is used to implement... Figure 5 S530A to S560B and their optional steps in the embodiment, this implementation method corresponds to Figure 4 The division method shown.

[0198] In one possible implementation, the first processor is a general-purpose processor and the second processor is an inference card, or the first processor is an inference card and the second processor is a general-purpose processor, or both the first processor and the second processor are inference cards.

[0199] In one possible implementation, the first address confirmation unit 430 is used to obtain the identifiers of the first and second memory blocks from the exchange request, obtain the first address of the first memory block and the first address of the second memory block based on the identifiers of the first and second memory blocks, and generate an exchange instruction. The first address confirmation unit 430 can be implemented by the second processor 300, specifically for implementing S620 and its optional steps.

[0200] In one possible implementation, the lengths of the first and second storage spaces are obtained based on model information and user configuration information. The model information includes the number of network layers X, and the user configuration information includes the number of storage blocks M in the memory of the first processor, the number of storage blocks N in the memory of the second processor, and the size b of the sub-storage blocks.

[0201] In summary, the computing device provided in this application first allocates a key-value cache for the inference model. This key-value cache includes M storage blocks in the memory of the first processor and N blocks in the memory of the second processor of the computing device. Each block includes multiple sub-blocks, and the addresses and intervals of two adjacent mini-blocks are equal. Each block is used to store the intermediate data generated by each layer according to the order of the X layers included in the inference model. Thus, when the computing device receives an exchange request to exchange the data of the first block of M blocks to the second storage block of N storage blocks, it can generate an exchange instruction including the starting address of the first block and the starting address of the second block according to the exchange request, and send the exchange instruction to the first processor. This allows the first processor to read the data of the first block to the second block according to the starting address of the first block, the starting address of the second block, and the address interval. When the first processor and the second processor exchange data in blocks, the data of a block can be read at once using an interface or method with equal spacing, resulting in high data exchange efficiency, low resource consumption, and thus improving the efficiency of model inference.

[0202] The following is combined with Figures 8-9 The hardware structure of the computing device provided in this application will be explained.

[0203] Figure 8 This is a schematic diagram of the hardware structure of a computing device provided in this application, such as... Figure 8As shown, the computing device 800 includes: a bus 802, a processor 804, an inference card 807, a memory 806, and a communication interface 808. The processor 804, memory 806, inference card 807, and communication interface 808 communicate with each other via the bus 802. The computing device 800 can be either the first processor 200 or the second processor 300 described above. It should be understood that this application does not limit the number of processors, memory, and inference cards in the computing device 800.

[0204] The 802 bus can be a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL) bus, a Cache Coherent Interconnect for Accelerators (CCIX) bus, etc. The Unified Bus is also known as the Lingqu bus. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 8 The bus 802 is represented by only one line, but this does not mean that there is only one bus or one type of bus. The bus 802 may include a path for transmitting information between various components of the computing device 800 (e.g., memory 806, processor 804, communication interface 808). The unified bus may also be called the Lingqu bus.

[0205] The processor 804 may include any one or more computing devices such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP) or a digital signal processor (DSP), an ASIC, an FPGA, a CPLD, an NPU, a SoC, an offload card, or an inference card.

[0206] Memory 806 may include volatile memory, such as random access memory (RAM). Memory 806 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD). Furthermore, memory 806 may also be implemented using storage class memory (SCM), phase change memory (PCM), or other types of storage media.

[0207] It is worth noting that the same type of storage medium can be configured in the same computing device to realize the function of memory 806, or two or more types of storage media can be configured to realize the function of memory 806. This application does not limit this.

[0208] The inference card 807 can be implemented using computing devices or AI chips such as DPU, NPU, GPU, XPU, and SoC, and this application does not make any specific limitations.

[0209] The communication interface 808 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 800 and other devices, such as communication between the second processor 300 and the first processor 200.

[0210] The memory 806 stores executable program code, and the processor 804 executes the executable program code to achieve the above. Figures 5-6 The embodiments describe the operational steps of the model inference method. The program code may include the code of one or more software units, which may be... Figure 7 The diagram shows an allocation unit 410, an acquisition unit 420, a first address confirmation unit 430, and an equally spaced transport unit 440. The allocation unit 410 is used to implement… Figure 5 In the embodiments S510A to S560B and their optional steps, the acquisition unit 420 is used to implement... Figure 6 In the embodiment of S610 and its optional steps, the first address confirmation unit 430 is used to implement... Figure 6 In the embodiment of S620 and its optional steps, the equally spaced transport unit 440 is used to implement... Figure 6 The steps S630 to S640 and their optional steps in the embodiment will not be described in detail here.

[0211] The inference card 807 includes a processing unit and a power supply circuit. The power supply circuit supplies power to the processing unit, enabling the processing unit to perform the above-mentioned operations. Figures 5-6 The operational steps of the model reasoning method described in the embodiment are as follows. For example, the processing unit can implement the functions of the allocation unit 410 and the equidistant transport unit 440 described above. The functional descriptions of the above-mentioned multiple units can be found in the relevant content above, and will not be repeated here.

[0212] In one possible implementation, processor 804 is the second processor 300, and inference card 807 is the first processor 200. Processor 804 can read program code from memory 806 to implement the above. Figures 5-6 The operation steps that the second processor 300 needs to implement in the embodiment can be implemented by the inference card 807 through the processing unit. Figure 9 The operation steps that the first processor 200 needs to implement in the embodiment.

[0213] This application also provides a cluster of computing devices. For example... Figure 9 As shown, Figures 5-6 This is an example diagram of a computing device cluster provided in this application, which includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0214] Optionally, the memory 806 of one or more computing devices 800 in the computing device cluster may store instructions for the same operation steps that the second processor 300 needs to implement in the implementation model inference method. The inference card 807 in each computing device 800 is used to implement the above... Figures 5-6 The first processor 200 in the embodiment needs to implement the same operation steps.

[0215] Optionally, the memory 806 of one or more computing devices 800 in the computing device cluster may also store partial instructions of different implementation model inference methods that the second processor 300 needs to implement. In other words, a combination of one or more computing devices 800 can jointly execute instructions for implementing the operation steps that the second processor 300 needs to implement in the data exchange method. The inference card 807 in each computing device 800 is used to implement the above. ​ The first processor 200 in the embodiment needs to implement different operation steps.

[0216] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to execute the model inference method provided in this application.

[0217] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute a reasoning method based on a large language model, or instruct the computing device to execute a reasoning method.

[0218] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A model reasoning method, characterized in that, The method includes: A key-value cache is allocated for the inference model. The key-value cache includes M storage blocks in the memory of the first processor of the computing device and N storage blocks in the memory of the second processor of the computing device. Each storage block includes multiple sub-storage blocks. The address interval between two adjacent sub-storage blocks is equal. Each storage block is used to store the intermediate data generated by each layer in the order of X layers included in the inference model. Obtain an exchange request, the exchange request being used to request the exchange of data from the first storage block among the M storage blocks to the second storage block among the N storage blocks; A swap instruction is generated based on the swap request and sent to the first processor. The swap instruction includes the starting address of the first storage block and the starting address of the second storage block, and is used to instruct the first processor to read intermediate data in the first storage block into the second storage block according to the starting address of the first storage block, the starting address of the second storage block and the address interval.

2. The method according to claim 1, characterized in that, The allocation of key-value cache for the inference model includes: Allocate a first memory space with contiguous addresses in the memory of the first processor, and allocate a second memory space with contiguous addresses in the memory of the second processor. In the first storage space and the second storage space respectively, contiguous storage space is allocated for each layer according to the order of each layer of the inference model, wherein each contiguous storage space is used to store the K vector or V vector of one layer; Obtain M and N, and determine the starting address of the M storage blocks, the starting address of the N storage blocks, and the address interval based on M, N, and the size of the sub-storage blocks. In the M and N storage blocks, each storage block is used to store the K vector and V vector of the intermediate data of the X layer of the inference model.

3. The method according to claim 1, characterized in that, The allocation of key-value cache for the inference model includes: Allocate a first memory space with contiguous addresses in the memory of the first processor, and allocate a second memory space with contiguous addresses in the memory of the second processor. In the first storage space and the second storage space respectively, contiguous storage space is allocated for each layer according to the order of each layer of the inference model, wherein each contiguous storage space is used to store the K vector and V vector of one layer; Obtain M and N, and determine the starting address of the M storage blocks, the starting address of the N storage blocks, and the address interval based on M, N, and the size of the sub-storage blocks. In the M and N storage blocks, each storage block is used to store the K vector or V vector of the intermediate data of the X layer of the inference model.

4. The method according to any one of claims 1 to 3, characterized in that, The first processor is a general-purpose processor and the second processor is an inference card, or the first processor is an inference card and the second processor is a general-purpose processor, or both the first processor and the second processor are inference cards.

5. The method according to any one of claims 1 to 4, characterized in that, The step of generating a swap instruction based on the swap request includes: Obtain the identifiers of the first storage block and the second storage block from the exchange request; The starting address of the first storage block and the starting address of the second storage block are obtained based on the identifier of the first storage block and the identifier of the second storage block, and the exchange instruction is generated.

6. The method according to any one of claims 2 to 5, characterized in that, The lengths of the first storage space and the second storage space are obtained based on model information and user configuration information. The model information includes the number of network layers X, and the user configuration information includes the number of storage blocks M in the memory of the first processor, the number of storage blocks N in the memory of the second processor, and the size b of the sub-storage block.

7. A computing device, characterized in that, The computing device includes a first processor and a second processor. The first processor is configured to allocate a key-value cache for the inference model. The key-value cache includes M storage blocks located in the memory of the first processor. Each storage block includes multiple sub-storage blocks. The address interval between two adjacent sub-storage blocks is equal. Each storage block is used to store intermediate data generated by each layer in the order of X layers included in the inference model. The second processor is configured to allocate a key-value cache for the inference model, the key-value cache comprising N storage blocks located in the memory of the second processor; The second processor is configured to acquire a swap request, generate a swap instruction based on the swap request, and send the swap instruction to the first processor. The swap request is used to request the swap of data from the first storage block among the M storage blocks to the second storage block among the N storage blocks. The swap instruction includes the starting address of the first storage block and the starting address of the second storage block. The first processor is configured to read intermediate data from the first storage block into the second storage block according to the starting address of the first storage block, the starting address of the second storage block, and the address interval.

8. The computing device according to claim 7, characterized in that, The first processor is configured to allocate a contiguous first storage space in the memory of the first processor, allocate contiguous storage space for each layer in the first storage space according to the order of each layer of the inference model, wherein each contiguous storage space is used to store the K vector or V vector of a layer, obtain the M, and determine the starting address and the address interval of the M storage blocks according to the M and the size of the sub-storage blocks, wherein each storage block is used to store the K vector and V vector of the intermediate data of the X layer of the inference model; The second processor is configured to allocate a contiguous second storage space in the memory of the second processor, and allocate contiguous storage space for each layer in the second storage space according to the order of each layer of the inference model. Each contiguous storage space is used to store the K vector or V vector of a layer. The N is obtained, and the starting address and address interval of the N storage blocks are determined according to the N and the size of the sub-storage blocks. Each storage block is used to store the K vector and V vector of the intermediate data of the X layer of the inference model.

9. The computing device according to claim 7, characterized in that, The first processor is configured to allocate a contiguous first storage space in its memory, allocate contiguous storage space for each layer in the first storage space according to the order of the layers of the inference model, wherein each contiguous storage space is used to store the K vector and V vector of a layer, obtain the M, and determine the starting address and address interval of the M storage blocks according to the M and the size of the sub-storage blocks, wherein in the M storage blocks, each storage block is used to store the K vector or V vector of the intermediate data of the X layer of the inference model; The second processor is configured to allocate a contiguous second storage space in its memory, and allocate contiguous storage space for each layer in the second storage space according to the order of the layers of the inference model, wherein each contiguous storage space is used to store the K vector and V vector of a layer, obtain the N, and determine the starting address and the address interval of the N storage blocks according to the N and the size of the sub-storage blocks, wherein in the N storage blocks, each storage block is used to store the K vector or V vector of the intermediate data of the X layer of the inference model.

10. The computing device according to any one of claims 7 to 9, characterized in that, The first processor is a general-purpose processor and the second processor is an inference card, or the first processor is an inference card and the second processor is a general-purpose processor, or both the first processor and the second processor are inference cards.

11. The computing device according to any one of claims 7 to 10, characterized in that, The second processor is configured to obtain the identifiers of the first storage block and the second storage block from the exchange request; The second processor is configured to obtain the starting address of the first storage block and the starting address of the second storage block based on the identifier of the first storage block and the identifier of the second storage block, and generate the exchange instruction.

12. The computing device according to any one of claims 8 to 11, characterized in that, The lengths of the first storage space and the second storage space are obtained based on model information and user configuration information. The model information includes the number of network layers X, and the user configuration information includes the number of storage blocks M in the memory of the first processor, the number of storage blocks N in the memory of the second processor, and the size b of the sub-storage block.

13. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a computing device, cause the computing device to perform the operational steps of the method as described in any one of claims 1 to 6.

14. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device, the computing device performs the operational steps of the method as described in any one of claims 1 to 6.

Images