Method and device for transmitting key-value cache data

Abstract

The application discloses a kind of transmission method and equipment of key value cache data, it is related to artificial intelligence technical field, the method is applied to the memory as first shared memory on the computing unit of prefill node, is applied to the memory as second shared memory on the computing unit of decoding node.Prefill node copies key value cache data to first shared memory, by key value cache data is transmitted from first shared memory to the second shared memory of decoding node application, key value cache data only needs to be transmitted once, and key value cache data can be transmitted from prefill node to decoding node, and transmission efficiency can be greatly improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and device for transmitting key-value cached data. Background Technology

[0002] Prefill / decode (P / D) separation deployment refers to separating prefill inference and decode inference onto different server nodes. When deploying high-concurrency inference services, P / D separation deployment can significantly improve throughput and reduce first-token latency, making it a primary deployment strategy. However, after P / D separation deployment, the prefill node needs to transfer massive amounts of KV-cache data to the decode node. Related technologies require multiple transfers between the prefill and decode nodes during KV-cache data transmission, resulting in low transmission efficiency.

[0003] Therefore, how to improve transmission efficiency has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] This invention provides a method, device, storage medium, and program product for transmitting key-value cache data, so as to at least solve the problem of low data transmission efficiency of KV-cache in related technologies.

[0005] This invention provides a method for transmitting key-value cached data, applied to pre-filled nodes, comprising:

[0006] Request contiguous memory on the computing unit of the pre-filled node as the first shared memory;

[0007] Receive a request sent by the distribution service, and allocate pre-filled memory space for inference on the computing unit according to the request;

[0008] Perform pre-filled inference and cache the key-value cache data generated by the pre-filled inference in the memory space;

[0009] The key-value cache data in the memory space is copied to the first shared memory so that the key-value cache data can be transferred from the first shared memory to the second shared memory; the second shared memory is a contiguous memory requested by the decoding node on the computing unit of the decoding node.

[0010] This invention also provides a method for transmitting key-value cached data, applied to a decoding node, comprising:

[0011] The decoding node allocates contiguous memory on its computing unit as a second shared memory to transfer key-value cache data generated by the pre-filling node's pre-filling inference from the first shared memory to the second shared memory; the first shared memory is the contiguous memory allocated by the pre-filling node on its computing unit; the pre-filling node copies the key-value cache data generated by the pre-filling inference from the memory space used for pre-filling inference in the computing unit to the first shared memory;

[0012] The key-value cache data in the second shared memory is copied to the memory space used for decoding inference in the computing unit.

[0013] The present invention also provides a transmission device for key-value cached data, comprising:

[0014] The first application module is used to apply for contiguous memory on the computing unit of the pre-filled node as a first shared memory;

[0015] A receiving module is used to receive requests sent by the distribution service and allocate pre-filled memory space for inference on the computing unit according to the requests;

[0016] The inference module is used to perform pre-filled inference and cache the key-value cache data generated by the pre-filled inference in the memory space;

[0017] The first copy module is used to copy the key-value cache data in the memory space to the first shared memory, so that the key-value cache data can be transferred from the first shared memory to the second shared memory; the second shared memory is a contiguous memory requested by the decoding node on the computing unit of the decoding node.

[0018] The present invention also provides a transmission device for key-value cached data, comprising:

[0019] The second application module is used to apply for contiguous memory on the computing unit of the decoding node as a second shared memory, so as to transfer the key-value cache data generated by the pre-filling node performing pre-filling inference from the first shared memory to the second shared memory; the first shared memory is the contiguous memory applied for by the pre-filling node on the computing unit of the pre-filling node; the pre-filling node copies the key-value cache data generated by performing pre-filling inference from the memory space used for pre-filling inference in the computing unit to the first shared memory;

[0020] The second copy module is used to copy the key-value cache data in the second shared memory to the memory space used for decoding and inference in the computing unit.

[0021] The present invention also provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described key-value cache data transmission methods.

[0022] The present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described key-value cache data transmission methods.

[0023] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of any of the above-described key-value cache data transmission methods.

[0024] The beneficial effects are as follows: In this invention, the pre-filled node allocates contiguous memory on its computing unit as the first shared memory, and the decoding node allocates contiguous memory on its computing unit as the second shared memory. The pre-filled node copies the key-value cache data to the first shared memory. By transferring the key-value cache data from the first shared memory to the second shared memory allocated by the decoding node, the key-value cache data only needs to be transferred once to achieve the transfer from the pre-filled node to the decoding node, which can greatly improve the transmission efficiency. Attached Figure Description

[0025] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A flowchart illustrating a method for transmitting key-value cached data according to an embodiment of the present invention;

[0027] Figure 2 This is a schematic diagram of a key-value cache data transmission structure provided in an embodiment of the present invention;

[0028] Figure 3 A flowchart illustrating another method for transmitting key-value cached data provided in an embodiment of the present invention;

[0029] Figure 4 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0031] It should be noted that, in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0032] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0033] The embodiments of the present invention provide a method for transmitting key-value cached data. The following describes the method for transmitting key-value cached data in detail, in conjunction with the execution flow of the method.

[0034] refer to Figure 1 As shown, Figure 1 This is a flowchart illustrating a method for transmitting key-value cached data according to an embodiment of the present invention. The method is applied to pre-filled nodes and includes:

[0035] S101: Allocate contiguous memory on the computing unit of the pre-filled node as first shared memory;

[0036] S102: Receive a request sent by the distribution service, and allocate memory space for pre-filled inference on the computing unit according to the request;

[0037] S103: Perform pre-filled inference and cache the key-value cache data generated by the pre-filled inference in the memory space;

[0038] S104: Copy the key-value cache data in the memory space to the first shared memory so that the key-value cache data can be transferred from the first shared memory to the second shared memory; the second shared memory is a contiguous memory requested by the decoding node on the computing unit of the decoding node.

[0039] Prefill nodes are server nodes that perform prefill inference. Prefill inference involves understanding the user's input, generating the first token, and initializing the KV-cache based on the user's input. The computational unit can be a GPU (Graphics Processing Unit). Prefill inference relies on GPU computing power and is computationally intensive. Generally, the longer the input, the longer it takes to generate the first token.

[0040] In language model inference, a token represents the inference word and is usually composed of a number. Generally, the model generates one token per inference, and one token represents one throughput. These tokens can be matched with corresponding words in the vocabulary.

[0041] Key-value (KV) caching, used in language models to improve inference speed and reduce redundant computation, involves caching the key-value matrices of already inferred tokens during inference. When inferring the next token, the cached KV matrix is ​​reused, and only the key and value of the next token are calculated. This significantly reduces computational load.

[0042] A decode node is a server node that performs decode inference. Decode inference, based on the inference results of the prefill inference and the KV-cache memory information, iterates through the inference process, generating a token with each inference iteration and expanding the corresponding KV-cache cache. Decode inference relies on GPU memory bandwidth and is memory-intensive.

[0043] This invention provides a method for transmitting KV-Cache data (key-value cache data) under P / D separation in a large-scale language-based artificial intelligence model, with the prefill node as the execution entity. P / D separation refers to separating prefill inference and decode inference onto different server nodes. Since prefill inference takes a relatively long time to generate the first token, while decode inference generates a token much faster, to achieve a better user experience and improve the throughput of the inference service, the inference phases of the language model are decoupled based on their characteristics, and prefill inference and decode inference are deployed separately on different service nodes. After decoupling, prefill inference will not preempt the computing power of decode inference, and decode inference will not preempt the memory bandwidth of prefill inference; the two no longer interfere with each other.

[0044] To achieve efficient transmission of key-value cached data and improve transmission efficiency, based on high-speed cross-node interconnection (e.g., InfiniBand or RoCE technology), in this embodiment of the invention, the prefill inference service on the prefill node requests a contiguous block of memory as shared memory on the computing unit of the prefill node. This shared memory requested by the prefill inference service is called the first shared memory. The first shared memory is dedicated to the transmission of key-value cached data. The inference service is the inference application; after the user's request is inferred through this application, the result is returned to the user.

[0045] The prefill node receives requests from the distribution service, allocates memory space for prefill inference on its computing unit according to the request length, and performs prefill inference. After prefill inference is complete, it generates the first token and caches the computation result of the prefill inference (key-value cache data) in the allocated memory space. The prefill node copies the key-value cache data from the memory space to the first shared memory, so that the key-value cache data can be transferred from the first shared memory to the second shared memory requested by the decoding node on its computing unit. The second shared memory requested by the decoding node is specifically used for the transfer of key-value cache data. The distribution service refers to the request distribution and node management service, designed to enable connections between the prefill node and the decode node. When a user request is received, the distribution service distributes the request to both the prefill node and the decode node, and binds their communication IPs and ports, thus establishing a connection between them.

[0046] In some embodiments, transferring the key-value cache data from the first shared memory to the second shared memory requested by the decoding node includes:

[0047] Write the key-value cache data in the first shared memory to the second shared memory;

[0048] The decoding node is notified that the key-value cache data has been written to the second shared memory, so that the decoding node copies the key-value cache data in the second shared memory to the memory space used for decoding inference on the computing unit.

[0049] In this embodiment, the prefill node writes the key-value cache data in the first shared memory to the second shared memory of the decoding node. After writing the key-value cache data to the second shared memory, the prefill node sends a notification to the decoding node, informing it that the key-value cache data has been written to the second shared memory. Then, the decoding node copies the key-value cache data in the second shared memory to the memory space allocated for decoding inference on the computing unit in the decoding node and begins decoding inference.

[0050] The pre-fill node writes the key-value cache data from the first shared memory to the second shared memory and notifies the decoding node upon completion. By transferring the key-value cache data from the first shared memory to the second shared memory allocated by the decoding node, the key-value cache data only needs to be transferred once to achieve the transfer from the pre-fill node to the decoding node, which greatly improves the transmission efficiency.

[0051] In some embodiments, the pre-filled node and the decoding node communicate via socket communication.

[0052] Therefore, the way to notify the decoding node that the key-value cache data has been written to the second shared memory can be: by notifying the decoding node that the key-value cache data has been written to the second shared memory through socket communication.

[0053] After the prefill node writes the key-value cache data into the second shared memory, it sends a notification to the decoding node via socket communication, informing the decoding node that the key-value cache data has been written into the second shared memory. Then, the decoding node copies the key-value cache data in the second shared memory to the memory space allocated for decoding inference in the computing unit and begins decoding inference.

[0054] Socket communication ensures seamless and efficient communication between the prefill node and the decoding node.

[0055] In some embodiments, it also includes:

[0056] The address of the second shared memory sent by the decoding node is received so that the key-value cache data in the first shared memory can be written to the second shared memory according to the address of the second shared memory.

[0057] The decode node's inference service requests contiguous memory as a second shared memory. The decode node sends the address of the second shared memory to the prefill node. The prefill node receives the address of the second shared memory sent by the decode node, and then writes the key-value cache data in the first shared memory into the second shared memory according to the address of the second shared memory, completing the transfer of key-value cache data between the prefill node and the decode node.

[0058] In some embodiments, transferring the key-value cache data from the first shared memory to the second shared memory requested by the decoding node includes:

[0059] The decoding node is notified that the key-value cache data has been written to the first shared memory, so that the decoding node reads the key-value cache data from the first shared memory and writes it to the second shared memory, and copies the key-value cache data in the second shared memory to the memory space used for decoding inference on the computing unit.

[0060] In this embodiment, after the prefill node copies the key-value cache data to the first shared memory, the prefill node sends a notification to the decoding node, informing the decoding node that the key-value cache data has been written to the first shared memory. The decoding node then reads the key-value cache data from the first shared memory and writes it to the second shared memory, and copies the key-value cache data in the second shared memory to the memory space used for decoding inference on the computing unit of the decoding node and begins decoding inference.

[0061] The pre-fill node writes the key-value cache data to the first shared memory and notifies the decoding node upon completion. The decoding node then reads the key-value cache data from the first shared memory into the second shared memory. By transferring the key-value cache data from the first shared memory to the second shared memory allocated by the decoding node, the key-value cache data only needs to be transferred once to achieve the transfer from the pre-fill node to the decoding node, which greatly improves the transmission efficiency.

[0062] In some embodiments, the pre-filled node and the decoding node communicate via socket communication.

[0063] Therefore, the way to notify the decoding node that the key-value cache data has been written to the first shared memory can be: by notifying the decoding node that the key-value cache data has been written to the first shared memory through socket communication.

[0064] After the prefill node copies the key-value cache data to the first shared memory, the prefill node sends a notification to the decoding node via socket communication, informing the decoding node that the key-value cache data has been written to the first shared memory. The decoding node then reads the key-value cache data from the first shared memory and writes it to the second shared memory, and copies the key-value cache data in the second shared memory to the memory space used for decoding inference in the computing unit and begins decoding inference.

[0065] In some embodiments, it also includes:

[0066] The address of the first shared memory is sent to the decoding node so that the decoding node can read the key-value cache data in the first shared memory according to the address of the first shared memory.

[0067] The prefill node sends the address of the first shared memory to the decoding node. The decoding node receives the address of the first shared memory sent by the prefill node, and then reads the key-value cache data in the first shared memory according to the address of the first shared memory.

[0068] In some embodiments, it also includes:

[0069] After transferring the key-value cache data in the first shared memory to the decoding node, delete the key-value cache data in the first shared memory and release the first shared memory.

[0070] Combination Figure 2 As shown, the following describes a specific implementation method:

[0071] Step 1: On the prefill node, start the prefill inference service. The prefill inference service requests a large block of contiguous memory as the first shared memory.

[0072] Step 2: On the decode node, start the decode inference service. The decode inference service requests a large block of contiguous memory as a second shared memory.

[0073] Step 3: Start the distribution service and begin receiving user requests.

[0074] Step 4: The distribution service receives the user request, sends the request to the prefill node and the decode node respectively, and informs the prefill node and the decode node of the connection method.

[0075] Step 5: After receiving the user request, the decode node pre-allocates the memory space needed for inference on the GPU card according to the length of the request, and sends the address of the memory requested in Step 2, i.e. the address of the second shared memory, to the prefill node.

[0076] Step 6: After receiving the user request, the prefill node allocates the memory space required for inference on the GPU card according to the length of the request, and begins to execute the prefill inference calculation.

[0077] Step 7: After the prefill inference calculation is completed, the first token will be generated, and the calculation result will be cached in the memory space allocated in step 6. This calculation result is the KV-cache data. The prefill node copies the KV-cache data to the first shared memory allocated in step 1.

[0078] Step 8: The prefill node uses the inter-card communication protocol to write the KV-cache data in the first shared memory to the second shared memory of the decode node. After the writing is complete, it sends a notify message via socket communication to inform the decode node that the KV-cache data has been transferred.

[0079] Step 9: After receiving the notify from the prefill node, the decode node copies the KV-cache data in the second shared memory to the memory space allocated in step 5, and then starts to execute the user-requested decode stage inference until the inference is completed.

[0080] The following describes another specific implementation method:

[0081] Step 1: On the prefill node, start the prefill inference service. The prefill inference service requests a large block of contiguous memory as the first shared memory.

[0082] Step 2: On the decode node, start the decode inference service. The decode inference service requests a large block of contiguous memory as a second shared memory.

[0083] Step 3: Start the distribution service and begin receiving user requests.

[0084] Step 4: The distribution service receives the user request, sends the request to the prefill node and the decode node respectively, and informs the prefill node and the decode node of the connection method.

[0085] Step 5: After receiving the user request, the decode node pre-allocates the memory space needed for inference on the GPU card according to the length of the request.

[0086] Step 6: After receiving the user request, the prefill node allocates the memory space required for inference on the GPU card according to the length of the request, and begins to execute the prefill inference calculation. It then sends the address of the memory requested in Step 1, i.e., the address of the first shared memory, to the decoding node.

[0087] Step 7: After the prefill inference calculation is completed, the first token will be generated, and the calculation result will be cached in the memory space allocated in step 6. This calculation result is the KV-cache data. The prefill node copies the KV-cache data to the first shared memory allocated in step 1.

[0088] Step 8: After the Prefill node copies the KV-cache data to the first shared memory requested in Step 1, it sends a notify via socket communication to inform the decode node that the KV-cache data has been copied to the first shared memory.

[0089] Step 9: After receiving the notify from the prefill node, the decode node reads the KV-cache data from the first shared memory to the second shared memory, copies the KV-cache data in the second shared memory to the memory space allocated in step 5, and then starts to execute the decode stage inference requested by the user until the inference is completed.

[0090] In summary, in this invention, the pre-fill node allocates contiguous memory on its computing unit as the first shared memory, and the decoding node allocates contiguous memory on its computing unit as the second shared memory. The pre-fill node copies the key-value cache data to the first shared memory. By transferring the key-value cache data from the first shared memory to the second shared memory allocated by the decoding node, the key-value cache data only needs to be transferred once to achieve the transfer from the pre-fill node to the decoding node, which can greatly improve the transmission efficiency.

[0091] refer to Figure 3 As shown, Figure 3 This is a flowchart illustrating a method for transmitting key-value cached data according to an embodiment of the present invention. The method is applied to a decoding node and includes:

[0092] S201: Request contiguous memory on the computing unit of the decoding node as a second shared memory, so as to transfer the key-value cache data generated by the pre-filling node performing pre-filling inference from the first shared memory to the second shared memory; the first shared memory is the contiguous memory requested by the pre-filling node on the computing unit of the pre-filling node; the pre-filling node copies the key-value cache data generated by performing pre-filling inference from the memory space used for pre-filling inference in the computing unit to the first shared memory;

[0093] S202: Copy the key-value cache data in the second shared memory to the memory space used for decoding inference.

[0094] To achieve efficient transmission of key-value cached data and improve transmission efficiency, based on high-speed cross-node interconnection (e.g., InfiniBand or RoCE technology), in this embodiment of the invention, the prefill inference service on the prefill node requests a contiguous block of memory as shared memory. This shared memory requested by the prefill inference service is called the first shared memory. The first shared memory is dedicated to the transmission of key-value cached data. Similarly, the decode inference service on the decode node requests a contiguous block of memory as shared memory. This shared memory requested by the decode inference server is called the second shared memory. The second shared memory is also dedicated to the transmission of key-value cached data.

[0095] The prefill node receives a request from the distribution service, allocates memory space for prefilling inference on the computing unit according to the request length, and performs prefilling inference. After prefilling inference is complete, the first token is generated, and the computation result of the prefilling inference, i.e., key-value cache data, is cached in the allocated memory space. The prefill node copies the key-value cache data from the memory space to the first shared memory. The decoding node requests contiguous memory as a second shared memory. The key-value cache data is transferred from the first shared memory to the second shared memory requested by the decoding node. After the key-value cache data is transferred to the second shared memory, the decoding node copies the key-value cache data from the second shared memory to the memory space used for decoding inference and begins decoding inference.

[0096] In some embodiments, transferring key-value cache data generated by pre-filling inference from a pre-filled node from a first shared memory to a second shared memory includes:

[0097] The system receives a first notification message from the pre-filled node; the first notification message indicates that the pre-filled node has written the key-value cache data in the first shared memory to the second shared memory.

[0098] The prefill node writes the key-value cache data in the first shared memory to the second shared memory of the decoding node. After writing the key-value cache data to the second shared memory, the prefill node sends a first notification to the decoding node, informing the decoding node that the key-value cache data has been written to the second shared memory. Then, the decoding node copies the key-value cache data in the second shared memory to the memory space allocated for decoding inference in the computing unit and begins decoding inference.

[0099] In some embodiments, it also includes:

[0100] The address of the second shared memory is sent to the pre-fill node so that the pre-fill node writes the key-value cache data in the first shared memory into the second shared memory according to the address of the second shared memory.

[0101] The decoding node sends the address of the second shared memory to the prefill node. The prefill node receives the address of the second shared memory sent by the decoding node, and then writes the key-value cache data in the first shared memory into the second shared memory according to the address of the second shared memory, thus completing the transmission of key-value cache data between the prefill node and the decoding node.

[0102] In some embodiments, transferring key-value cache data generated by pre-filling inference from a pre-filled node from a first shared memory to a second shared memory includes:

[0103] The second notification information sent by the pre-filled node indicates that the pre-filled node has written the key-value cache data into the first shared memory.

[0104] Read the key-value cache data from the first shared memory and write the read key-value cache data into the second shared memory.

[0105] After the prefill node copies the key-value cache data to the first shared memory, the prefill node sends a second notification to the decoding node, informing the decoding node that the key-value cache data has been written to the first shared memory. The decoding node then reads the key-value cache data from the first shared memory and writes it to the second shared memory, and copies the key-value cache data in the second shared memory to the memory space used for decoding inference in the computing unit and begins decoding inference.

[0106] In some embodiments, it also includes:

[0107] The address of the first shared memory sent by the pre-filled node is received so as to read the key-value cache data in the first shared memory according to the address of the first shared memory.

[0108] The prefill node sends the address of the first shared memory to the decoding node. The decoding node receives the address of the first shared memory sent by the prefill node, and then reads the key-value cache data in the first shared memory according to the address of the first shared memory.

[0109] In some embodiments, it also includes:

[0110] After completing the decoding and inference, delete the key-value cache data in the second shared memory and release the second shared memory.

[0111] In summary, in this invention, the pre-fill node allocates contiguous memory on its computing unit as the first shared memory, and the decoding node allocates contiguous memory on its computing unit as the second shared memory. The pre-fill node copies the key-value cache data to the first shared memory. By transferring the key-value cache data from the first shared memory to the second shared memory allocated by the decoding node, the key-value cache data only needs to be transferred once to achieve the transfer from the pre-fill node to the decoding node, which can greatly improve the transmission efficiency.

[0112] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0113] Embodiments of the present invention also provide a transmission apparatus for key-value cached data, the apparatus comprising:

[0114] The first application module is used to apply for contiguous memory on the computing unit of the pre-filled node as a first shared memory;

[0115] The receiving module is used to receive requests sent by the distribution service and allocate memory space for pre-filled inference on the computing unit according to the requests.

[0116] The inference module is used to perform pre-filled inference and cache the key-value cache data generated by the pre-filled inference in the memory space;

[0117] The first copy module is used to copy the key-value cache data in the memory space to the first shared memory, so that the key-value cache data can be transferred from the first shared memory to the second shared memory; the second shared memory is a contiguous memory requested by the decoding node on the computing unit of the decoding node.

[0118] In some embodiments, the device includes:

[0119] The first write module is used to write the key-value cache data in the first shared memory to the second shared memory;

[0120] The first notification module is used to notify the decoding node that the key-value cache data has been written to the second shared memory, so that the decoding node copies the key-value cache data in the second shared memory to the memory space used for decoding inference on the computing unit.

[0121] In some embodiments, the first notification module is used to notify the decoding node via socket communication that the key-value cache data has been written to the second shared memory.

[0122] In some embodiments, it also includes:

[0123] The first address receiving module is used to receive the address of the second shared memory sent by the decoding node, so as to write the key-value cache data in the first shared memory into the second shared memory according to the address of the second shared memory.

[0124] In some embodiments, the device includes:

[0125] The second notification module is used to notify the decoding node that the key-value cache data has been written to the first shared memory, so that the decoding node reads the key-value cache data from the first shared memory and writes it to the second shared memory, and copies the key-value cache data in the second shared memory to the memory space used for decoding inference in the computing unit.

[0126] In some embodiments, the second notification module is used to notify the decoding node via socket communication that the key-value cache data has been written to the first shared memory.

[0127] In some embodiments, it also includes:

[0128] The first address sending module is used to send the address of the first shared memory to the decoding node, so that the decoding node can read the key-value cache data in the first shared memory according to the address of the first shared memory.

[0129] The transmission device provided by this invention allocates contiguous memory as first shared memory at the pre-filling node and contiguous memory as second shared memory at the decoding node. Key-value cache data is copied to the first shared memory at the pre-filling node. By transferring the key-value cache data from the first shared memory to the second shared memory allocated at the decoding node, the key-value cache data only needs to be transferred once to achieve the transfer from the pre-filling node to the decoding node, which can greatly improve transmission efficiency.

[0130] Embodiments of the present invention also provide a transmission apparatus for key-value cached data, the apparatus comprising:

[0131] The second application module is used to apply for contiguous memory on the computing unit of the decoding node as a second shared memory, so as to transfer the key-value cache data generated by the pre-filling node performing pre-filling inference from the first shared memory to the second shared memory; the first shared memory is the contiguous memory applied for by the pre-filling node on the computing unit of the pre-filling node; the pre-filling node copies the key-value cache data generated by performing pre-filling inference from the memory space used for pre-filling inference in the computing unit to the first shared memory;

[0132] The second copy module is used to copy the key-value cache data in the second shared memory to the memory space used for decoding and inference in the computing unit.

[0133] In some embodiments, the device includes:

[0134] The first receiving module is used to receive a first notification message sent by the pre-filled node; the first notification message indicates that the pre-filled node has written the key-value cache data in the first shared memory into the second shared memory.

[0135] In some embodiments, the device further includes:

[0136] The sending module is used to send the address of the second shared memory to the pre-fill node, so that the pre-fill node writes the key-value cache data in the first shared memory into the second shared memory according to the address of the second shared memory.

[0137] In some embodiments, the device includes:

[0138] The second receiving module is used to receive a second notification message sent by the pre-filled node; the second notification message indicates that the pre-filled node has written the key-value cache data into the first shared memory;

[0139] Read the key-value cache data from the first shared memory and write the read key-value cache data into the second shared memory.

[0140] In some embodiments, it also includes:

[0141] The second address receiving module is used to receive the address of the first shared memory sent by the pre-filled node, so as to read the key-value cache data in the first shared memory according to the address of the first shared memory.

[0142] For a description of the features of the key-value cache data transmission device in the corresponding embodiment, please refer to the relevant description of the key-value cache data transmission method in the corresponding embodiment, which will not be repeated here.

[0143] The transmission device provided by this invention allocates contiguous memory as first shared memory at the pre-filling node and contiguous memory as second shared memory at the decoding node. Key-value cache data is copied to the first shared memory at the pre-filling node. By transferring the key-value cache data from the first shared memory to the second shared memory allocated at the decoding node, the key-value cache data only needs to be transferred once to achieve the transfer from the pre-filling node to the decoding node, which can greatly improve transmission efficiency.

[0144] Embodiments of the present invention also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above embodiments of the key-value cache data transmission method.

[0145] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above embodiments of the key-value cache data transmission method when running.

[0146] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0147] Embodiments of the present invention also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described key-value cache data transmission method embodiments.

[0148] Embodiments of the present invention also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described key-value cache data transmission method embodiments.

[0149] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0150] The present invention has provided a detailed description of a method and device for transmitting key-value cached data. Specific examples have been used to illustrate the principles and implementation methods of the invention. The descriptions of these embodiments are only intended to aid in understanding the method and core ideas of the invention. It should be noted that those skilled in the art can make various improvements and modifications to the invention without departing from its principles, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A method for transmitting key-value cached data, characterized in that, Applied to pre-filled nodes, including: A contiguous block of memory is allocated on the computing unit of the pre-filled node as a first shared memory; the first shared memory is specifically used for the transmission of key-value cache data; Receive a request sent by the distribution service, and allocate pre-filled memory space for inference on the computing unit according to the request; Perform pre-filled inference and cache the key-value cache data generated by the pre-filled inference in the memory space; The key-value cache data in the memory space is copied to the first shared memory so that the key-value cache data can be transferred from the first shared memory to the second shared memory; the second shared memory is a contiguous memory requested by the decoding node on the computing unit of the decoding node; the second shared memory is dedicated to the transfer of key-value cache data. Transferring the key-value cache data from the first shared memory to the second shared memory includes: Write the key-value cache data in the first shared memory to the second shared memory; The decoding node is notified that the key-value cache data has been written to the second shared memory, so that the decoding node copies the key-value cache data in the second shared memory to the memory space used for decoding inference on the computing unit. Alternatively, transferring the key-value cache data from the first shared memory to the second shared memory includes: The decoding node is notified that the key-value cache data has been written to the first shared memory, so that the decoding node reads the key-value cache data from the first shared memory and writes it to the second shared memory, and copies the key-value cache data in the second shared memory to the memory space used for decoding inference in the computing unit. The pre-filled node and the decoding node communicate via socket communication.

2. The transmission method according to claim 1, characterized in that, Also includes: The address of the second shared memory sent by the decoding node is received so that the key-value cache data in the first shared memory can be written to the second shared memory according to the address of the second shared memory.

3. The transmission method according to claim 1, characterized in that, Also includes: The address of the first shared memory is sent to the decoding node so that the decoding node can read the key-value cache data in the first shared memory according to the address of the first shared memory.

4. A method for transmitting key-value cached data, characterized in that, Applied to decoding nodes, including: A contiguous block of memory is allocated on the computing unit of the decoding node as a second shared memory, so that key-value cache data generated by the pre-filling node performing pre-filling inference can be transferred from the first shared memory to the second shared memory; the first shared memory is the contiguous block of memory allocated by the pre-filling node on the computing unit of the pre-filling node; the pre-filling node copies the key-value cache data generated by performing pre-filling inference from the memory space used for pre-filling inference in the computing unit to the first shared memory; the first shared memory is dedicated to the transfer of key-value cache data; the second shared memory is dedicated to the transfer of key-value cache data. Copy the key-value cache data in the second shared memory to the memory space used for decoding inference in the computing unit; Transferring the key-value cache data generated by pre-filling inference from the pre-filled nodes from the first shared memory to the second shared memory includes: Receive a first notification message sent by the pre-filled node; the first notification message indicates that the pre-filled node has written the key-value cache data in the first shared memory into the second shared memory; Alternatively, transferring the key-value cache data generated by pre-filling inference from the pre-filled nodes from the first shared memory to the second shared memory includes: The system receives a second notification message from the pre-filled node; the second notification message indicates that the pre-filled node has written the key-value cache data into the first shared memory. Read the key-value cache data from the first shared memory and write the read key-value cache data into the second shared memory; The pre-filled node and the decoding node communicate via socket communication.

5. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the key-value cache data transmission method as described in any one of claims 1 to 3 or the key-value cache data transmission method as described in claim 4 when executing the computer program.

Images