Data transmission, reception method, transmission system, device, medium and program product

Abstract

Embodiments of the present application relate to the technical field of artificial intelligence chip, and provide a data sending and receiving method, a transmission system, a device, a medium and a program product, to improve the transmission efficiency of key value cache and the inference efficiency of large model. The data sending method comprises: reading key value vectors of at least one attention layer from a video memory to obtain key value vectors of a plurality of page blocks, wherein the key value vectors cached in the video memory are key value vectors of each attention layer in the large model calculated by the pre-filled node when processing an inference request, and the key value vectors of each attention layer in the video memory are stored in a plurality of page blocks; rearranging the plurality of page blocks in a set manner into a pre-applied memory space, and sending the key value vectors in the memory space to the decoding node, wherein the plurality of rearranged page blocks in the memory space are distributed in continuous storage units.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence chip technology, and in particular to a data transmission and reception method, transmission system, device, medium and program product. Background Technology

[0002] Separating the pre-filling and decoding stages in large model inference (PD separation) is a key technique for optimizing throughput. In the PD separation scenario, after the pre-filling node completes its computation, it obtains the key-value vectors of each attention layer in the large model. These key-value vectors are then stored in the GPU memory cache, forming the key-value cache. The key-value cache is then sent to the decoding node so that it can continue working. The transmission performance of the key-value cache is crucial during this process.

[0003] The key-value cache grows dynamically during its generation. To manage memory fragmentation, the key-value cache is usually divided into fixed-size page blocks. Each page block corresponds to a contiguous area of ​​video memory. In a PD separation scenario, the key-value cache is transferred at the page block level.

[0004] Currently, in some hardware, the key-value cache storage layout must be a three-dimensional matrix (matrix3d) with a shape of [sequence dimension N, attention head dimension H, feature dimension W]. The H and W dimensions are very large, such as 2048 or 8192. The page block shape is [page size (page_size), head dimension (head_dim)] (page blocks are segmented by head), and page_size and head_dim are mapped to the H and W dimensions of the key-value cache, respectively. Since H and W are much larger than page_size and head_dim, each page block is not stored contiguously in video memory but is divided into multiple non-contiguous segments. That is, the storage units occupied by a single page block in video memory are not contiguous. Therefore, when transferring the key-value cache at the page block level, the starting position of the key-value cache for a single page block in video memory is found based on the storage unit identifier. Then, the key-value vector of each page block is transferred segment by segment until the end.

[0005] like Figure 1 As shown, assume that the size of each BF16 memory cell 10 is 32. 32. When caching data, cache it in video memory in column-major order. That is, first traverse the first column (H dimension) from top to bottom. After completing the H traversal, traverse the second column (H dimension) from top to bottom, and so on. If the page block size is 128... If the value is 128, then page block 11 will occupy 4 bytes in the cache. 4 storage units, namely, 1, 2, 3, 4, 7, 8, 9, 10, 13, 14, 15, 16, 19, 20, 21, and 22, and the storage units of this part of data are discontinuous and are divided into four segments: 1-4, 7-10, 13-16, and 19-22. If sending is performed with a page block as a basic granularity, a single page block needs to be sent 4 times (1-4, 7-10, 13-16, and 19-22 are sent in turn), and the transmission efficiency is relatively low. If all data is sent (1-24 is sent), a large amount of dirty data (5, 6, 11, 12, 17, 18, 23, and 24) will be carried in the sent data, which not only has poor performance but also has the risk of polluting the data on the decoding side.

[0006] It is assumed that an input sequence contains 512 tokens, a large model contains 48 layers of Transformers, each layer of the Transformer contains an attention mechanism, each attention mechanism has 32 attention heads, each attention head has a key-value vector dimension of 128, and the page_size of a page block is 128. At this time, the input sequence contains 512 tokens, and the page_size is 128. Therefore, all tokens need 4 page blocks to store the key-value vectors. Each layer and each attention head in the large model needs an independent key-value cache. For a large model with 48 layers and 32 attention heads, the total number of cache regions required is 48 32=1536, and each cache region needs 4 page blocks to store the key-value vectors of the entire sequence. Therefore, a total of 4 1536=6144 page blocks are required to store the key-value vectors. If each page block is divided into 4 discontinuous storage units in the video memory, a total of 6144 x 4 = 24576 transmissions are required when sending the key-value cache, and the amount of data transmitted each time is 128 x 32 x sizeof(BF16), where sizeof is a function representing the number of bytes occupied by a data type, and sizeof(BF16)=2.

[0007] This transmission mode with high frequency and low data volume cannot fully utilize the network bandwidth and introduces a large amount of transmission overhead such as a protocol header and network delay, which seriously reduces the overall transmission efficiency and further affects the inference efficiency of the large model. SUMMARY

[0008] Embodiments of the present application provide a data sending and receiving method, a transmission system, a device, a medium, and a program product to improve the transmission efficiency of key-value cache and the inference efficiency of a large model.

[0009] In a first aspect, an embodiment of the present application provides a data sending method applied to a pre-filling node in a large model in which the pre-filling node and a decoding node are separated, and the method comprises:

[0010] read the key-value vectors of at least one attention layer from the video memory to obtain key-value vectors of a plurality of page blocks, wherein the key-value vectors cached in the video memory are key-value vectors of each attention layer in the large model calculated by the pre-population node when processing an inference request, and the key-value vectors of each attention layer in the video memory are stored in a plurality of page blocks;

[0011] rearrange the plurality of page blocks in a set manner into a pre-applied memory space, and send the key-value vectors in the memory space to the decoding node, wherein the plurality of page blocks rearranged in the memory space are distributed in continuous storage units.

[0012] In a second aspect, an embodiment of the present application provides a data receiving method applied to a decoding node in a large model in which a pre-population node and the decoding node are separated, and the method comprises:

[0013] receiving the key-value vectors sent by the pre-population node in a pre-applied memory space;

[0014] splitting the received key-value vectors into a plurality of page blocks based on a pre-determined page block shape;

[0015] caching the plurality of page blocks in a video memory in a layout manner of a three-dimensional matrix.

[0016] In a third aspect, an embodiment of the present application provides a data sending device applied to a pre-population node in a large model in which a pre-population node and a decoding node are separated, and the device comprises:

[0017] a reading module configured to read the key-value vectors of at least one attention layer from the video memory to obtain key-value vectors of a plurality of page blocks, wherein the key-value vectors cached in the video memory are key-value vectors of each attention layer in the large model calculated by the pre-population node when processing an inference request, and the key-value vectors of each attention layer in the video memory are stored in a plurality of page blocks;

[0018] a sending module configured to rearrange the plurality of page blocks in a set manner into a pre-applied memory space, and send the key-value vectors in the memory space to the decoding node, wherein the plurality of page blocks rearranged in the memory space are distributed in continuous storage units.

[0019] Optionally, the page block is a two-dimensional tensor, a size of a first dimension of the page block is P, and a size of a second dimension of the page block is D.

[0020] The sending module is specifically configured to:

[0021] create a key-value vector cache pool in the memory space, the key-value vector cache pool being a three-dimensional tensor, a size of a first dimension of the key-value vector cache pool being a first preset value, a size of a second dimension being greater than or equal to B / N P, and a size of a third dimension being greater than or equal to D N, wherein B is a number of page blocks to be stored, N is a number of page blocks stored in each column of the cache pool, and P, D, B, and N are positive integers, B is divisible by N;

[0022] rearrange the plurality of page blocks in a column-major order on the second dimension and the third dimension of the key-value vector cache pool.

[0023] Optionally, the key-value vector includes quantized key values and a scaling factor.

[0024] The sending module is specifically configured to:

[0025] create the key-value vector cache pool and a scaling factor cache pool in the memory space, a starting address of the scaling factor cache pool being adjacent to an ending address of the key-value vector cache pool, the scaling factor cache pool being a one-dimensional array, and a length of the scaling factor cache pool being B P.

[0026] rearrange the plurality of page blocks in a column-major order on the second dimension and the third dimension of the key-value vector cache pool, and arrange the scaling factor in the scaling factor cache pool.

[0027] Optionally, the reading module is further configured to determine a starting address and a total data amount of key-value vectors of at least one attention layer to be read before reading the key-value vectors of the at least one attention layer from the video memory.

[0028] The sending module is specifically configured to send the key-value vectors in the memory space to the decoding node, including sending the starting address, the total data amount, and the key-value vectors in the memory space to the decoding node.

[0029] Optionally, the sending module pre-applies the memory space in the following manner:

[0030] obtain configuration parameters of the large model and inference parameters pre-configured by a user, the inference parameters including a batch number and a context inference length, and the configuration parameters including an attention head dimension and a page size of a page block;

[0031] based on the batch number, the context inference length, the attention head dimension, and the page size, calculate a size of a memory space to be applied, and apply a continuous memory space with the size.

[0032] In a fourth aspect, an embodiment of the present application provides a data receiving device, applied to a decoding node in a large model in which a pre-filling node and the decoding node are separated, and the device comprises:

[0033] A receiving module, configured to receive a key-value vector sent by the pre-filling node in a pre-applied memory space;

[0034] A processing module, configured to split the received key-value vector into a plurality of page blocks based on a pre-determined page block shape;

[0035] A caching module, configured to cache the plurality of page blocks in a three-dimensional matrix layout in a display memory.

[0036] Optionally, the receiving module is specifically configured to receive a start address, a total data amount, and a key-value vector sent by the pre-filling node, the start address being a start address of a key-value vector of at least one attention layer read by the pre-filling node, and the total data amount being a total data amount of the key-value vector of the at least one attention layer read by the pre-filling node.

[0037] The caching module is specifically configured to:

[0038] In the display memory, the plurality of page blocks are cached in a three-dimensional matrix layout starting from the start address, and a cached data amount is equal to the total data amount.

[0039] Optionally, the receiving module is configured to pre-apply the memory space in the following manner:

[0040] Obtain configuration parameters of the large model and user-preconfigured inference parameters, the inference parameters comprising a batch number and a context inference length, and the configuration parameters comprising an attention head dimension and a page size of a page block;

[0041] Based on the batch number, the context inference length, the attention head dimension, and the page size, calculate a size of a to-be-applied memory space, and apply a continuous memory space with the size of the memory space.

[0042] In a fifth aspect, an embodiment of the present application provides a data transmission system, applied to a large model in which a pre-filling node and a decoding node are separated, and the system comprises a sending module and a receiving module, the sending module is deployed in the pre-filling node, and the receiving module is deployed in the decoding node;

[0043] The sending module is configured to read key-value vectors of at least one attention layer from a video memory corresponding to the pre-population node, obtain key-value vectors of a plurality of page blocks, rearrange the key-value vectors of the plurality of page blocks in a set manner into a pre-applied sending memory space, and send the key-value vectors in the memory space to the receiving module, wherein the key-value vectors cached in the video memory are key-value vectors of each attention layer in the large model calculated by the pre-population node when processing an inference request, the key-value vectors of each attention layer in the video memory are stored in a plurality of page blocks, and the rearranged plurality of page blocks in the sending memory space are distributed in continuous storage units.

[0044] The receiving module is configured to receive the key-value vectors sent by the sending module in a pre-applied memory space, split the received key-value vectors into a plurality of page blocks based on a pre-determined page block shape, and cache the plurality of page blocks in a three-dimensional matrix layout manner into a video memory corresponding to the decoding node.

[0045] In a sixth aspect, an embodiment of the present application provides a computer device, including a memory, a processor chip, and a computer program stored in the memory and executable on the processor chip, and the processor chip implements the steps of the method in any one of the first aspect and the second aspect when executing the program.

[0046] In a seventh aspect, an embodiment of the present application provides a computer readable storage medium storing a computer program executable by a computer device, and when the program is executed on the computer device, the computer device executes the steps of the method in any one of the first aspect and the second aspect.

[0047] In an eighth aspect, an embodiment of the present application provides a computer program product, including a computer program stored in a computer readable storage medium, and the computer program includes program instructions, and when the program instructions are executed by a computer device, the computer device executes the steps of the method in any one of the first aspect and the second aspect.

[0048] In the embodiment of the present application, after the pre-population node of the PD separated large model reads the key-value vectors of at least one attention layer from the video memory, obtains the key-value vectors of the plurality of page blocks, rearranges the plurality of page blocks in a set manner into a pre-applied memory space, and then sends the key-value vectors in the memory space to the decoding node.

[0049] In this embodiment, when the pre-fill node sends the key-value vector cached in the video memory to the decoding node, it rearranges at least one attention layer of multiple page blocks (in related technologies, each page block is divided into multiple segments and distributed in non-contiguous storage units within the video memory) into a pre-allocated memory space. The multiple page blocks are distributed in contiguous storage units within the memory space. Thus, when sending the key-value vector of multiple page blocks in the memory space, the key-value vector of at least one attention layer of multiple page blocks can be sent to the decoding node in only one transmission. This reduces the number of transmissions and allows the key-value vector of at least one attention layer of multiple page blocks to be sent in one transmission, increasing the amount of data sent in a single transmission. This improves the transmission efficiency of the key-value cache and, consequently, the inference efficiency of large models. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a schematic diagram illustrating the non-contiguous storage units occupied by page blocks in video memory in related technologies;

[0052] Figure 2 This is a schematic diagram of the structure of an artificial intelligence chip provided in an embodiment of this application;

[0053] Figure 3 A flowchart illustrating a data transmission method provided in an embodiment of this application;

[0054] Figure 4 A schematic diagram illustrating the principle of a page block rearrangement method provided in an embodiment of this application;

[0055] Figure 5 A schematic diagram illustrating the principle of another page block rearrangement method provided in this application embodiment;

[0056] Figure 6 A flowchart illustrating a data receiving method provided in an embodiment of this application;

[0057] Figure 7 This is a schematic diagram illustrating the interaction between a pre-filled node and a decoding node, provided as an embodiment of this application.

[0058] Figure 8 This is a schematic diagram of the structure of a data transmission system provided in an embodiment of this application;

[0059] Figure 9A structural schematic diagram of a data sending device provided by an embodiment of the present application is shown in FIG. 1.

[0060] Figure 10 A structural schematic diagram of a data receiving device provided by an embodiment of the present application is shown in FIG. 2.

[0061] Figure 11 A structural schematic diagram of a computer device provided by an embodiment of the present application is shown in FIG. 3. DETAILED DESCRIPTION

[0062] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions of the present application will be described below in detail with the drawings of the embodiments of the present application. Obviously, the described embodiments are only some but not all of the embodiments of the present application. Based on the embodiments described in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0063] Some concepts involved in the embodiments of the present application will be introduced below.

[0064] A large model, i.e., a large language model, refers to an artificial intelligence language model generated by training a large number of parameters (usually reaching tens of billions or even hundreds of billions) on a large-scale data set based on deep learning technology, which can understand and generate human language and can process various natural language processing tasks such as text generation, translation, and question answering.

[0065] Pre-padding stage and decoding stage separation (abbreviated as PD separation) is a key technology for optimizing throughput in large model inference. The pre-padding stage processes the input sequence in the entire inference request and generates the first token through the attention mechanism, which is computationally intensive but can be calculated in parallel; the decoding stage then generates subsequent tokens one by one, which requires frequent memory access and serial computation.

[0066] PD separation allows separate optimization for different stage characteristics. The pre-padding stage focuses on computational efficiency and can call hardware acceleration; the decoding stage focuses on memory bandwidth and needs to optimize key-value caching. In actual deployment, different computing resources are usually allocated to the two stages, and even heterogeneous hardware is used, so as to significantly improve the overall throughput performance while ensuring low latency, which is particularly important in long text generation and high-concurrency service scenarios.

[0067] The design idea of the embodiments of the present application will be briefly introduced below:

[0068] In view of the transmission mode with high frequency and low data volume in the related art, network bandwidth cannot be fully utilized, and a large amount of transmission overhead such as a protocol header and network delay is introduced, which seriously reduces the overall transmission efficiency and further affects the inference efficiency of the large model. Embodiments of the present application provide a data sending and receiving method, a transmission system, a device, a medium and a program product. On the pre-population node side of the PD separated large model, the key value vectors of at least one attention layer are read from the video memory, and after obtaining the key value vectors of a plurality of page blocks, the plurality of page blocks are rearranged in a set manner in the pre-applied memory space. The plurality of page blocks rearranged in the memory space are distributed in continuous storage units, and then the key value vectors in the memory space are sent to the decoding node.

[0069] In the embodiments of the present application, when the pre-population node sends the key value vectors cached in the video memory to the decoding node, the plurality of page blocks of at least one attention layer (each page block is divided into multiple segments and distributed in non-continuous storage units in the video memory in the related art) are rearranged in the pre-applied memory space, and the plurality of page blocks are distributed in continuous storage units in the memory space. In this way, when the key value vectors of the plurality of page blocks in the memory space are sent, the key value vectors of the plurality of page blocks of at least one attention layer can be sent to the decoding node only once, which reduces the number of transmissions on the one hand, and the key value vectors of the plurality of page blocks of at least one attention layer can be sent at one time, which increases the amount of data transmitted at one time, thereby improving the transmission efficiency of the key value cache and further improving the inference efficiency of the large model.

[0070] The preferred embodiments of the present application are described below in conjunction with the accompanying drawings of the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application, and the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0071] Reference Figure 2 which is a structure diagram of an artificial intelligence chip to which the embodiments of the present application are applicable. The artificial intelligence chip 200 at least includes a plurality of video memories 201 and a plurality of computing units 202. The computing unit 202 can be a streaming processor cluster (SPC for short). The video memory 201 can be a high bandwidth memory (HBM for short), or other types of memories.

[0072] In the embodiment of the present application, the large model can include multiple sub-networks, each of which is deployed in a computing unit 202, and the parameters (such as key-value vectors) required for the pre-filling stage and the decoding stage of the large model can be saved in different video memories 201 respectively. The multiple sub-networks can be deployed in the computing units 202 contained in one artificial intelligence chip 200, or can be deployed in the computing units 202 contained in multiple artificial intelligence chips 200, and the present application does not make specific limitations on this.

[0073] When the large model executes the inference request, after the pre-filling node completes the calculation, the key-value vectors of each attention layer in the large model are obtained, and the key-value vectors are stored in the cache space of the pre-filling node corresponding video memory. Then the pre-filling node reads the key-value vectors of at least one attention layer from the video memory, obtains the key-value vectors of the multiple page blocks, rearranges the multiple page blocks in a set manner to the pre-applied memory space, and distributes the rearranged multiple page blocks in the continuous storage units in the memory space, and sends the key-value vectors in the memory space to the decoding node.

[0074] After the decoding node receives the key-value vectors sent by the pre-filling node, the multiple page blocks are cached in the cache space of the pre-filling node corresponding video memory in the opposite manner, and subsequent inference processing is performed.

[0075] Compared with the transmission scheme of the traditional key-value cache (each page block is divided into multiple segments and distributed in non-continuous storage units in the video memory, and needs to be sent multiple times), the embodiment of the present application rearranges the multiple page blocks of at least one attention layer in the video memory to the pre-applied memory space, and the multiple page blocks are distributed in the continuous storage units in the memory space. In this way, when the key-value vectors of the multiple page blocks in the memory space are actually sent, the key-value vectors of the multiple page blocks of at least one attention layer can be sent to the decoding node only once, which can reduce the number of video memory data copying between the pre-filling node corresponding video memory and the decoding node corresponding video memory, improve the data transmission efficiency, and thus improve the inference efficiency of the large model.

[0076] The artificial intelligence chip 200 in the present application can include other structures in addition to the above structure, and the present application does not make specific limitations on this.

[0077] The artificial intelligence chip 200 can be a graphics processing unit (GPU), a general-purpose computing on graphics processing units (GPGPU), a domain specific architecture (DSA), etc.

[0078] Specifically, the large model mentioned in the embodiments of the present application can be a large model in various application scenarios, such as an image processing scenario, a speech processing scenario, a text processing scenario, etc. The large model can be used to process various inference tasks. The sending and receiving of the key-value vector between the pre-filling node and the decoding node in the PD separation large model can also be performed in the process of processing various tasks.

[0079] For example, in the text processing scenario, the input tensor of the large model can be text data used in text generation, text recognition, etc. After receiving the text data, in the process of processing the text data using the large model, the pre-filling node calculates the key-value vector of each attention layer after processing the entire input sequence. Then, the pre-filling node can use the data sending method provided in the embodiments of the present application to send the key-value vector in the video memory of the pre-filling node to the decoding node. After receiving the key-value vector, the decoding node can use the data receiving method provided in the embodiments of the present application to cache the received key-value vector in the video memory corresponding to the decoding node, and perform subsequent inference calculation, so as to reduce the number of data transmission, improve the data transmission efficiency, and thus improve the inference efficiency and inference speed of the large model.

[0080] For example, in the speech processing scenario, the input tensor of the large model can be speech data used in speech enhancement, speech recognition, speech synthesis, etc. After receiving the speech data, in the process of processing the speech data using the large model, the pre-filling node calculates the key-value vector of each attention layer after processing the entire input sequence. Then, the pre-filling node can use the data sending method provided in the embodiments of the present application to send the key-value vector in the video memory of the pre-filling node to the decoding node. After receiving the key-value vector, the decoding node can use the data receiving method provided in the embodiments of the present application to cache the received key-value vector in the video memory corresponding to the decoding node, and perform subsequent inference calculation, so as to reduce the number of data transmission, improve the data transmission efficiency, and thus improve the inference efficiency and inference speed of the large model.

[0081] For example, in the image processing scenario, the input tensor of the large model can be image data used in image preprocessing, image segmentation, target detection, etc. After receiving the image data, in the process of processing the image data using the large model, the pre-filling node calculates the key-value vector of each attention layer after processing the entire input sequence. Then, the pre-filling node can use the data sending method provided in the embodiments of the present application to send the key-value vector in the video memory of the pre-filling node to the decoding node. After receiving the key-value vector, the decoding node can use the data receiving method provided in the embodiments of the present application to cache the received key-value vector in the video memory corresponding to the decoding node, and perform subsequent inference calculation, so as to reduce the number of data transmission, improve the data transmission efficiency, and thus improve the inference efficiency and inference speed of the large model.

[0082] The following is based on Figure 2 The diagram shows the architecture of the artificial intelligence chip. It details the flow of the data transmission and reception method provided in this application's embodiments. For the pre-filled node side, see [link / reference]. Figure 3 This method is performed by a computer device, which includes... Figure 2 The artificial intelligence chip shown includes the following steps:

[0083] S301, read the key value vector of at least one attention layer from the video memory to obtain the key value vector of multiple page blocks. The key value vector cached in the video memory is the key value vector of each attention layer in the large model calculated by the pre-filled node when processing the inference request. The key value vector of each attention layer is stored in the video memory in multiple page blocks.

[0084] In practice, when processing inference requests, the pre-filled node calculates the key-value vector of each attention layer in the large model and caches the key-value vector in the pre-filled node's GPU memory. The calculation of the key-value vector by the pre-filled node can employ methods found in related technologies, and this application does not limit this approach.

[0085] In some hardware devices, the key-value cache must be stored in matrix3d. When key-value vectors are cached in video memory, they are stored in page blocks. This results in the storage units occupied by a single page block being non-contiguous in video memory (e.g., ...). Figure 1 As shown, this will not be repeated here.

[0086] In specific implementation, in order to improve the transmission efficiency of key-value vectors, the embodiments of this application first read the key-value vectors of at least one attention layer from the video memory to obtain the key-value vectors of multiple page blocks when sending the key-value vectors to the decoding node. Then, the key-value vectors of the multiple page blocks are rearranged so that they are distributed in consecutive storage units before being sent.

[0087] S302, rearranges multiple page blocks into a pre-allocated memory space in a set manner, and sends the key-value vector in the memory space to the decoding node, wherein the multiple page blocks rearranged in the memory space are distributed in contiguous storage units.

[0088] In practice, in order to rearrange multiple page blocks, this application embodiment pre-allocates contiguous memory space. After reading the key value vector of at least one attention layer and obtaining the key value vectors of multiple page blocks, the multiple page blocks are rearranged into the pre-allocated memory space in a set manner.

[0089] It should be noted that if the key-value vectors of the M attention layers are rearranged in the pre-allocated memory space each time, the pre-allocated memory space only needs to be sufficient to store the key-value vectors of the M attention layers, and the memory space can be reused in the sending of multiple rounds of key-value vectors.

[0090] Specifically, after the key-value vectors of the first round of M attention layers are rearranged in the memory space, the key-value vectors in the memory space can be sent to the decoding node, and after the sending is completed, the key-value vectors of the second round of M attention layers can be rearranged in the memory space and then sent, and the same is true for the subsequent rounds. In this way, the memory space can be reused, and memory waste caused by the application of a large memory space can be avoided.

[0091] Of course, in other embodiments of the present application, if the key-value vectors of the M attention layers are rearranged in the pre-allocated memory space each time, the pre-allocated memory space can also be sufficient to store the key-value vectors of all attention layers of the large model.

[0092] In this embodiment, the memory space does not need to be reused in the sending of multiple rounds of key-value vectors. Specifically, after the key-value vectors of the first round of M attention layers are rearranged in the memory space, the key-value vectors of the first round of M attention layers in the memory space can be sent to the decoding node. During the sending process, the key-value vectors of the second round of M attention layers can continue to be rearranged in the memory space, and then after the key-value vectors of the second round of M attention layers are completely rearranged, the key-value vectors of the second round of M attention layers are sent, the key-value vectors of the third round of M attention layers are rearranged, and the same is true for the subsequent rounds. In this way, rearrangement and data sending can be performed simultaneously, further improving transmission efficiency.

[0093] The application manner of the pre-allocated continuous memory space in the embodiments of the present application will be described below with reference to the case where the memory space is reused and one attention layer of key-value vectors is stored in the memory space.

[0094] When the continuous memory space is specifically applied, the configuration parameters of the large model and the inference parameters pre-configured by the user are first obtained, wherein the inference parameters include the batch size and the context inference length, and the configuration parameters include the attention head dimension and the page size of the page block. Then, based on the batch size, the context inference length (input sequence (previous context) length + output sequence (next context) length), the attention head dimension, and the page size, the size of the memory space to be applied is calculated, and the continuous memory space is applied with the size of the memory space.

[0095] It should be noted that, in order to ensure that the pre-applied continuous memory space can store key-value vectors of various sizes, the batch size and context reasoning length in the user-preconfigured reasoning parameters can be set to the maximum value or a larger value.

[0096] In one example, assuming that in the configuration parameters of the large model, the number of attention heads is represented by heads, the dimension of the attention head is represented by head dim, and the page size is represented by page size, in the user-preconfigured reasoning parameters, the batch size is represented by max batch, and the context reasoning length is represented by max context length, then the maximum number of page blocks num blocks of one attention layer is max batch (max context length+page size-1) / / pagesize heads ( / / for quotient operation), if the required memory block bytes of each page block is page size head dim sizeof (BF16), wherein sizeof is a function representing the number of bytes occupied by a data type, and sizeof (BF16)=2.

[0097] In the key-value non-separated scenario, the required continuous memory space for storing the key-value vector of one attention layer is: total bytes=num blocks block bytes; in the key-value separated scenario, the required continuous memory space for storing the key-value vector of one attention layer is: total bytes=num blocks block bytes 2.

[0098] In actual applications, if the applied continuous memory space needs to store key-value vectors of multiple attention layers, then the number of attention layers is multiplied by the memory size calculated above.

[0099] It should be noted that, if the key-value vector is quantized to INT8 and FP32, the scaling factor when quantizing the key-value vector also needs to be stored. Since the storage amount of the quantized key-value vector is half of the original, and the number of scaling factors is the same as the number of page blocks, and the memory of a single scaling factor is page size sizeof (FP32), sizeof (FP32) = 4, far less than block bytes / 2, so the memory space block bytes applied for each page block in the application is sufficient to store the quantized key value and scaling factor, therefore, the application is also applicable to the scenario that the key value vector is quantized.

[0100] The specific implementation of rearranging the plurality of page blocks in the set manner into the pre-applied memory space in the embodiments of the application is described below in combination with specific examples.

[0101] The page block in the embodiments of the application is a two-dimensional tensor, assuming that the size of the first dimension of the page block is P and the size of the second dimension is D, when the plurality of page blocks is rearranged in the set manner into the pre-applied memory space, a key value vector cache pool can be created in the memory space, the shape of the key value vector cache pool is a three-dimensional tensor, the size of the first dimension of the key value vector cache pool is a first preset value (such as fixed as 1), the size of the second dimension is greater than or equal to B / N P, and the size of the third dimension is greater than or equal to D N, wherein B is the number of page blocks to be stored, N is the number of page blocks stored in each column in the cache pool, and P, D, B and N are all positive integers, B can be divided by N, and then the plurality of page blocks is rearranged in the column-major order manner on the second dimension and the third dimension of the key value vector cache pool.

[0102] It should be noted that for the key value separation scenario, since K and V need to be stored separately, the size of the second dimension of the key value vector cache pool needs to be greater than or equal to B / N P 2.

[0103] In one example, as Figure 4 shown, assuming that B = 8, N = 1, P and D are both 128, the size of the second dimension of the key value vector cache pool is 8 128, and the size of the third dimension is equal to 128. In this example, when the page blocks 1~8 read from the video memory are rearranged in the key value vector cache pool in the memory space, the data of the page block 1 is stored in [0:127, 0:127], the data of the page block 2 is stored in [128:255, 0:127], and so on, then the 8 rearranged page blocks are distributed in the continuous storage units.

[0104] In another example, as Figure 5 shown, assuming that B = 8, N = 2, P and D are both 128, the size of the second dimension of the key value vector cache pool is 8 / 2 128, and the size of the third dimension is equal to 2 128. In this example, when the page blocks 1-8 read from the video memory are rearranged in the key-value vector cache pool in the memory space, the data of the page block 1 is stored in [0:127, 0:127], the data of the page block 2 is stored in [128:255, 0:127], and so on. The data of the page block 5 is stored in [0:127, 128:255], the data of the page block 6 is stored in [128:255, 128:255], and so on. Thus, the eight rearranged page blocks are also distributed in the continuous storage units.

[0105] For the scenario in which the key-value vector is quantized, the key-value vector includes the quantized key-value and the scaling factor. When the plurality of page blocks are rearranged in the pre-applied memory space in a set manner, in addition to creating the key-value vector cache pool, the embodiments of the present application can also create a scaling factor cache pool in the memory space, and the memory start address of the scaling factor cache pool is adjacent to the memory end address of the key-value vector cache pool. The shape of the scaling factor cache pool is a one-dimensional array, and the length of the scaling factor cache pool is B P, and then the plurality of page blocks are rearranged in the column-major order manner in the second dimension and the third dimension of the key-value vector cache pool, and the scaling factor is arranged in the scaling factor cache pool.

[0106] In specific implementation, after the plurality of page blocks are rearranged in the pre-applied memory space in a set manner, the key-value vector in the memory space can be sent to the decoding node. The plurality of rearranged page blocks in the memory space are distributed in the continuous storage units. Thus, the key-value vector in the memory space can be sent to the decoding node through one sending process.

[0107] In actual application, to ensure the synchronization of the transmission data between the pre-populated node and the decoding node, before reading the key-value vector of at least one attention layer from the video memory, the embodiments of the present application also need to determine the start address and the total data amount of the key-value vector of the at least one attention layer to be read, so as to inform the decoding node of the start address and the total data amount of the key-value vector to be sent this time when the key-value vector is sent to the decoding node in the subsequent process, so that the decoding node can be stored in the corresponding video memory. The specific way of determining the start address can adopt the way in the related art, which is not limited in the embodiments of the present application.

[0108] Specifically, in determining the total data amount to be sent, for the scenario in which the key-value vector is not quantized, the total data amount to be sent is equal to the storage space occupied by the key-value vector cache pool; for the scenario in which the key-value vector is quantized, the total data amount to be sent is equal to the sum of the storage space occupied by the key-value vector cache pool and the storage space occupied by the scaling factor cache pool.

[0109] Suppose the shape of the key-value vector cache pool is [N, H, W], wherein the first dimension N = 1, the second dimension H = B P 2, the third dimension W = D. According to practical experience, W is usually not more than 8192, and when H > 8192, H needs to be folded into the N dimension.

[0110] The specific folding manner is that candidate values [1024, 2048, 4096] of a folding factor (fold) are traversed, for each candidate value, NN = (H + fold - 1) / fold (upward rounding) is calculated, and if the calculated NN is not greater than 1024, the fold value is selected as the final folding factor. Through the folding factor, the original tensor can be reshaped from the shape [N, H, W] to the new shape [NN, fold, W].

[0111] For the key-value vector cache pool with the shape [N, H, W], the storage space occupied is: N alignH (H) alignW (W) sizeof (T).

[0112] alignH (x) is: if the data T is of the BF16 type, alignH (x) = (x + 31) / 32; if the data T is of the S8 type, alignH (x) = (x + 63) / 64. alignW (x) is: (x + 31) / 32. When the data T is of the BF16 type, sizeof (T) = 2; and when the data T is of the S8 type, sizeof (T) = 1.

[0113] The scaling factor cache pool is a one-dimensional array, and is assumed to have a shape [NV, W], NV = 1, and W = B P, and when W > 8192, W needs to be folded into the NV dimension, and the folding manner is the same as above. After folding, the shape of the scaling factor cache pool changes to [NN, fold].

[0114] For the scaling factor cache pool with the shape [NV, W], the storage space occupied is: NV alignW (W) sizeof (T). Wherein alignW (x) is: (x + 511) / 512. When the data T is of the FP32 type, sizeof (T) = 4.

[0115] Through the above manner, the total amount of data to be sent each time can be calculated to inform the decoding node to reserve space for receiving data.

[0116] The pre-filled node side data sending method provided by the embodiments of the present application is described in detail above in combination with specific embodiments. Correspondingly, at the decoding node side, referring to Figure 6 The data receiving method provided by the embodiments of the present application is executed by a computer device, and the computer device includes Figure 2 an artificial intelligence chip as shown in the figure, and the method includes the following steps:

[0117] S601, receiving a key-value vector sent by a pre-filled node in a pre-applied memory space.

[0118] The pre-applied memory space is applied in the following manner: obtaining configuration parameters of the large model and user pre-configured inference parameters, the inference parameters including a batch quantity and a context inference length, and the configuration parameters including an attention head dimension and a page size of a page block; based on the batch quantity, the context inference length, the attention head dimension, and the page size, calculating a size of a to-be-applied memory space, and applying a continuous memory space with the size of the memory space.

[0119] In specific implementation, the specific application manner of the pre-applied memory space can be the same as that of the pre-filled node side, which is not described herein again.

[0120] S602, splitting the received key-value vector into a plurality of page blocks based on a pre-determined page block shape.

[0121] In specific implementation, the pre-applied memory space can be divided into a key-value vector cache pool and / or a scaling factor cache pool in the same manner as that of the pre-filled node side, and then be split, and the specific splitting manner can be the reverse of storage.

[0122] S603, caching the plurality of page blocks in a three-dimensional matrix layout manner into a video memory.

[0123] In specific implementation, to ensure synchronization of transmission data between the pre-filled node and the decoding node, the starting address of caching needs to be determined when the plurality of page blocks are cached into the video memory in the three-dimensional matrix layout manner.

[0124] In the embodiments of the present application, the pre-filled node also notifies the decoding node of the starting address and the total data quantity of the key-value vector sent this time when sending the key-value vector in the memory space. Therefore, after the decoding node receives the starting address, the total data quantity, and the key-value vector sent by the pre-filled node, when the plurality of page blocks are cached into the video memory in the three-dimensional matrix layout manner, the plurality of page blocks are cached into the video memory in the three-dimensional matrix layout manner starting from the starting address in the video memory, and the data quantity of caching is equal to the total data quantity.

[0125] In other embodiments of the present application, the decoding node can also determine the starting address and the total data amount of the key-value vector of this transmission according to the transmission condition. The specific way of determining the starting address can adopt the way in the related art, and the specific way of determining the total data amount can adopt the same way as that of the pre-filling node side, which will not be described here.

[0126] The data receiving method of the decoding node side provided by the embodiments of the present application is simply introduced above, and the specific implementation process of the pre-filling node side and the decoding node side in transmitting the key-value vector is described below in combination with the interaction process of the pre-filling node and the decoding node. Figure 7

[0127] As shown in FIG. 7, the specific implementation process of the data sending method provided by the embodiments of the present application includes the following steps. Figure 7

[0128] Step 701, receiving an inference request.

[0129] Step 702, processing the inference request, calculating the key-value vector of each attention layer, and caching the calculated key-value vector to the video memory of the pre-filling node.

[0130] Step 703, determining the starting address of the key-value vector to be sent this time when transmitting the key-value vector to the decoding node.

[0131] Step 704, reading the key-value vector to be sent this time from the video memory, and rearranging the key-value vector to the pre-applied continuous memory space.

[0132] Step 705, determining the total data amount of the key-value vector to be sent this time.

[0133] Step 706, sending the key-value vector in the memory space to the decoding node.

[0134] Meanwhile, the pre-filling node can inform the decoding node of the starting address and the total data amount of the key-value vector to be sent this time.

[0135] Step 707, judging whether the key-value vector is completely sent, if yes, executing step 708, otherwise, continuing to execute step 703 to continue to send the key-value vector next time.

[0136] Step 708, ending the sending of the key-value vector under the condition that the key-value vector is completely sent.

[0137] The specific implementation process of the data sending method provided by the embodiments of the present application in the decoding node includes the following steps.

[0138] Step 711, receiving an inference request.

[0139] ​​Step 712, when receiving the key-value vector transmitted by the pre-population node, determining the starting address of the received key-value vector.

[0140] Step 713, receiving the key-value vector transmitted by the pre-population node in the pre-applied continuous memory space.

[0141] Step 714, based on the pre-determined page block shape, splitting the key-value vector in the memory space into multiple page blocks.

[0142] Step 715, determining the total data amount of the received key-value vector.

[0143] Step 716, starting from the determined starting address, caching the multiple page blocks in the decoding node corresponding video memory in a three-dimensional matrix layout, and the cached data amount is equal to the total data amount.

[0144] The above describes the data sending and receiving method provided by the embodiment of the application in combination with the interaction process of the pre-population node and the decoding node. The embodiment of the application effectively solves the problem of low key-value vector transmission efficiency caused by the discontinuous storage units occupied by the page blocks in the related art, although a new data rearrangement process is added, but compared with the method in the related art, the time consumption is greatly reduced.

[0145] Based on the same technical concept, as shown in Figure 8 The embodiment of the application provides a data transmission system applied to a large model with separated pre-population nodes and decoding nodes, and the system comprises a sending module 801 and a receiving module 802. The sending module is deployed in the pre-population node, and the receiving module is deployed in the decoding node.

[0146] The sending module 801 is used for reading the key-value vector of at least one attention layer from the video memory corresponding to the pre-population node, obtaining the key-value vector of multiple page blocks, rearranging the key-value vector of the multiple page blocks in a set manner in the pre-applied sending memory space, and sending the key-value vector in the memory space to the receiving module 802. The key-value vector cached in the video memory is the key-value vector of each attention layer in the large model calculated by the pre-population node when processing an inference request. The key-value vector of each attention layer in the video memory is stored in multiple page blocks. The multiple page blocks rearranged in the sending memory space are distributed in continuous storage units.

[0147] The receiving module 802 is used for receiving the key-value vector sent by the sending module 801 in the pre-applied memory space, splitting the received key-value vector into multiple page blocks based on the pre-determined page block shape, and caching the multiple page blocks in the decoding node corresponding video memory in a three-dimensional matrix layout.

[0148] Based on the same technical concept, on the pre-filled node side, the embodiment of the present application provides a structural diagram of a data sending device, as shown in Figure 9 The data processing device 900 includes:

[0149] The reading module 901 is configured to read the key-value vector of at least one attention layer from the video memory to obtain the key-value vector of a plurality of page blocks, wherein the key-value vector cached in the video memory is the key-value vector of each attention layer in the large model calculated by the pre-filled node when processing the inference request, and the key-value vector of each attention layer in the video memory is stored in a plurality of page blocks.

[0150] The sending module 902 is configured to rearrange the plurality of page blocks in a set manner into a pre-applied memory space and send the key-value vector in the memory space to the decoding node, wherein the plurality of rearranged page blocks in the memory space are distributed in continuous storage units.

[0151] Optionally, the page block is a two-dimensional tensor, the size of the first dimension of the page block is P, and the size of the second dimension is D.

[0152] The sending module 902 is specifically configured to:

[0153] create a key-value vector cache pool in the memory space, the shape of the key-value vector cache pool is a three-dimensional tensor, the size of the first dimension of the key-value vector cache pool is a first preset value, the size of the second dimension is greater than or equal to B / N P, and the size of the third dimension is greater than or equal to D N, wherein B is the number of page blocks to be stored, N is the number of page blocks stored in each column of the cache pool, and P, D, B and N are positive integers, B is divisible by N.

[0154] Rearrange the plurality of page blocks in column-major order on the second dimension and the third dimension of the key-value vector cache pool.

[0155] Optionally, the key-value vector includes a quantized key-value and a scaling factor.

[0156] The sending module 902 is specifically configured to:

[0157] create a key-value vector cache pool and a scaling factor cache pool in the memory space, the memory start address of the scaling factor cache pool is adjacent to the memory end address of the key-value vector cache pool, the shape of the scaling factor cache pool is a one-dimensional array, and the length of the scaling factor cache pool is B P.

[0158] Rearrange the plurality of page blocks in column-major order on the second dimension and the third dimension of the key-value vector cache pool, and arrange the scaling factor in the scaling factor cache pool.

[0159] Optionally, the reading module 901 is further configured to determine a starting address and a total data amount of the key-value vector of the at least one attention layer to be read before reading the key-value vector of the at least one attention layer from the video memory.

[0160] The sending module 902 is configured to send the key-value vector in the memory space to the decoding node, including sending the starting address, the total data amount, and the key-value vector in the memory space to the decoding node.

[0161] Optionally, the sending module 902 applies the memory space in the following manner:

[0162] Obtain the configuration parameters of the large model and the user-preconfigured inference parameters, the inference parameters including a batch number and a context inference length, and the configuration parameters including an attention head dimension and a page size of a page block;

[0163] Based on the batch number, the context inference length, the attention head dimension, and the page size, calculate a size of the memory space to be applied, and apply a continuous memory space with the size of the memory space.

[0164] Based on the same technical concept, on the decoding node side, the present application provides a structural diagram of a data receiving device, as shown in Figure 10 The data processing device 1000 includes:

[0165] The receiving module 1001 is configured to receive the key-value vector sent by the pre-populating node in the pre-applied memory space.

[0166] The processing module 1002 is configured to split the received key-value vector into a plurality of page blocks based on a pre-determined page block shape.

[0167] The caching module 1003 is configured to cache the plurality of page blocks in a three-dimensional matrix layout manner in the video memory.

[0168] Optionally, the receiving module 1001 is configured to receive the starting address, the total data amount, and the key-value vector sent by the pre-populating node, the starting address being a starting address of the key-value vector of the at least one attention layer read by the pre-populating node, and the total data amount being a total data amount of the key-value vector of the at least one attention layer read by the pre-populating node.

[0169] The caching module 1003 is configured to:

[0170] In the video memory, the plurality of page blocks are cached in a three-dimensional matrix layout manner with the starting address as the starting point, and a cached data amount is equal to the total data amount.

[0171] Optionally, the receiving module 1001 applies the memory space in the following manner:

[0172] Obtain the configuration parameters of the large model and the inference parameters pre-configured by the user. The inference parameters include the batch size and context inference length, while the configuration parameters include the attention head dimension and the page size of the page block.

[0173] Based on the batch size, context inference length, attention head dimension, and page size, calculate the size of the memory space to be requested, and request contiguous memory space according to the memory space size.

[0174] Based on the same technical concept, embodiments of this application provide a computer device, such as... Figure 11 As shown, it includes at least one processor chip 1101 and a memory 1102 connected to at least one processor chip. In this embodiment, the specific connection medium between the processor chip 1101 and the memory 1102 is not limited. Figure 11 Taking the connection between processor chip 1101 and memory 1102 via a bus as an example, the bus can be divided into address bus, data bus, control bus, etc.

[0175] In this embodiment of the application, the memory 1102 stores instructions that can be executed by at least one processor chip 1101. By executing the instructions stored in the memory 1102, at least one processor chip 1101 can perform the steps of the above-described data transmission and reception method.

[0176] The processor chip 1101 is the control center of the computer device. It can connect to various parts of the computer device using various interfaces and lines, and performs data processing by running or executing instructions stored in the memory 1102 and calling data stored in the memory 1102. Optionally, the processor chip 1101 may include one or more processing units. The processor chip 1101 may integrate an application processor and a modem processor. The application processor mainly handles the operating system, user interface, and applications, while the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor chip 1101. In some embodiments, the processor chip 1101 and the memory 1102 may be implemented on the same chip; in some embodiments, they may also be implemented on separate chips.

[0177] The processor chip 1101 can be a general-purpose processor, such as a Graphics Processing Unit (GPU), a General-Purpose computing on Graphics Processing Units (GPGPU), a Central Processing Unit (CPU), a Digital Signal Processor, an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present application can be directly embodied as completed by a hardware processor, or completed by a combination of hardware and software modules in the processor.

[0178] The memory 1102 is a non-volatile computer-readable storage medium, and can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. The memory 1102 can include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Programmable Read-Only Memory (PROM), a Read-Only Memory (ROM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), a magnetic memory, a magnetic disk, an optical disk, and the like. The memory 1102 is any other medium capable of carrying or storing desired program codes in the form of instructions or data structures and capable of being accessed by a computer device, but is not limited thereto. The memory 1102 in the embodiments of the present application can also be a circuit or any other device capable of realizing a storage function, used to store program instructions and / or data.

[0179] Based on the same inventive concept, the embodiments of the present application provide a computer-readable storage medium storing a computer program executable by a computer device, which, when the program is executed on the computer device, causes the computer device to perform the steps of the data sending and receiving method described above.

[0180] Based on the same inventive concept, the embodiment of the present application provides a computer program product, which comprises a computer program stored on a computer readable storage medium, and the computer program comprises program instructions, which, when executed by a computer device, cause the computer device to perform the steps of the above-mentioned data sending and receiving method.

[0181] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) containing computer-usable program code.

[0182] The present application is described with reference to the flowcharts and / or block diagrams of the method, device (system) and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer device or other programmable data processing device produce the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for performing the function specified by the one or more blocks.

[0183] These computer program instructions can also be stored in a computer readable memory capable of directing the computer device or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce a product comprising instruction means, which implement the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for performing the function specified by the one or more blocks.

[0184] These computer program instructions can also be loaded into the computer device or other programmable data processing device, so that a series of operation steps are performed on the computer device or other programmable device to produce a processing implemented by the computer device, so that the instructions executed on the computer device or other programmable device provide the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for performing the function specified by the one or more blocks.

[0185] While the preferred embodiments of the application have been described, additional variations and modifications can be made to these embodiments by those skilled in the art once they have the benefit of the present disclosure without departing from the spirit and scope of the application. Accordingly, it is intended that the appended claims include all such modifications and variations as fall within the scope of the present application.

[0186] It is apparent that those skilled in the art can make various changes and modifications to the application without departing from the spirit and scope of the application. It is therefore intended that the present application cover all such changes and modifications that are within its scope.

Claims

1. A data sending method applied to a pre-filling node in a large model with a pre-filling node and a decoding node separated, characterized in that, The method comprises: reading key-value vectors of at least one attention layer from a video memory to obtain key-value vectors of a plurality of page blocks, wherein the key-value vectors cached in the video memory are key-value vectors of each attention layer in the large model calculated by the pre-filling node when processing an inference request, and the key-value vectors of each attention layer in the video memory are stored in a plurality of page blocks; rearranging the plurality of page blocks in a set manner into a pre-applied memory space, and sending the key-value vectors in the memory space to the decoding node, wherein the plurality of rearranged page blocks in the memory space are distributed in continuous storage units.

2. The method of claim 1, wherein, The page block is a two-dimensional tensor, the size of the first dimension of the page block is P, and the size of the second dimension is D; The method comprises: A key-value vector cache pool is created in the memory space, the key-value vector cache pool is a three-dimensional tensor, a size of a first dimension of the key-value vector cache pool is a first preset value, a size of a second dimension is greater than or equal to B / N P, and a size of a third dimension is greater than or equal to D N, wherein B is a number of page blocks to be stored, N is a number of page blocks stored in each column of the cache pool, and P, D, B, and N are positive integers, and B is divisible by N. rearranging the plurality of page blocks in a column-major order manner on the second dimension and the third dimension of the key-value vector cache pool.

3. The method of claim 2, wherein, The key-value vector comprises a quantized key-value and a scaling factor; The method comprises: creating the key-value vector cache pool and a scale factor cache pool in the memory space, a memory start address of the scale factor cache pool being adjacent to a memory end address of the key-value vector cache pool, the scale factor cache pool being shaped as a one-dimensional array, a length of the scale factor cache pool being B P; rearranging the plurality of page blocks in a column-major order manner on the second dimension and the third dimension of the key-value vector cache pool, and arranging the scaling factor in the scaling factor cache pool.

4. The method according to any one of claims 1-3, characterized in that, Before the step of reading the key-value vectors of at least one attention layer from the video memory, the method further comprises: determining a starting address and a total data amount of the key-value vectors of at least one attention layer to be read; The step of sending the key-value vectors in the memory space to the decoding node comprises: sending the starting address, the total data amount, and the key-value vectors in the memory space to the decoding node.

5. The method according to any one of claims 1-3, characterized in that, The pre-applied memory space is applied in the following manner: obtaining configuration parameters of the large model and user-preconfigured inference parameters, the inference parameters comprising a batch size and a context inference length, and the configuration parameters comprising an attention head dimension and a page size of a page block; based on the batch size, the context inference length, the attention head dimension, and the page size, calculating a size of a memory space to be applied, and applying a continuous memory space with the size.

6. A data receiving method applied to a decoding node in a large model with a pre-filling node and a decoding node separated, wherein the decoding node receives data sent by the pre-filling node as claimed in any one of claims 1-5. The method comprises: receiving the key-value vectors sent by the pre-filling node in the pre-applied memory space; based on a pre-determined page block shape, splitting the received key-value vectors into a plurality of page blocks; caching the plurality of page blocks in a three-dimensional matrix layout manner in a video memory.

7. The method of claim 6, wherein, The step of receiving the key-value vectors sent by the pre-filling node comprises: receiving a starting address, a total data amount, and key-value vectors sent by the pre-filling node, wherein the starting address is a starting address of the key-value vectors of at least one attention layer read by the pre-filling node, and the total data amount is a total data amount of the key-value vectors of at least one attention layer read by the pre-filling node; The step of caching the plurality of page blocks in a three-dimensional matrix layout manner in a video memory comprises: In the display memory, the plurality of page blocks are cached in a three-dimensional matrix layout starting from the start address, and an amount of cached data is equal to the total data amount.

8. The method according to claim 6 or 7, characterized in that, The pre-applied memory space is applied in the following manner: Obtain configuration parameters of the large model and user-preconfigured inference parameters, the inference parameters including a batch quantity and a context inference length, and the configuration parameters including an attention head dimension and a page size of a page block. Based on the batch quantity, the context inference length, the attention head dimension, and the page size, calculate a size of a to-be-applied memory space, and apply a continuous memory space with the size.

9. A data transmission system applied to a large model with a pre-filling node and a decoding node separated, characterized in that, The system includes a sending module and a receiving module, the sending module is deployed in the pre-filling node, and the receiving module is deployed in the decoding node. The sending module reads key-value vectors of at least one attention layer from a display memory corresponding to the pre-filling node to obtain key-value vectors of a plurality of page blocks, rearranges the key-value vectors of the plurality of page blocks in a set manner into a pre-applied sending memory space, and sends the key-value vectors in the memory space to the receiving module. The receiving module receives the key-value vectors sent by the sending module in the pre-applied memory space, splits the received key-value vectors into a plurality of page blocks based on a pre-determined page block shape, and caches the plurality of page blocks in a three-dimensional matrix layout into a display memory corresponding to the decoding node.

10. A computer device comprising a memory, a processor chip and a computer program stored on the memory and executable on the processor chip, characterized in that, The processor chip implements the steps of the method of any one of claims 1-8 when executing the program.

11. A computer readable storage medium, characterized in that, The computer program product includes a computer program stored on a computer readable storage medium, and the computer program includes program instructions which, when executed by a computer device, cause the computer device to perform the steps of the method of any one of claims 1-8.

12. A computer program product, characterised in that, The computer program product includes a computer program stored on a computer readable storage medium, and the computer program includes program instructions which, when executed by a computer device, cause the computer device to perform the steps of the method of any one of claims 1-8.

Images