Cold and hot data exchange method, device and system for key value cache, medium and product

Abstract

The invention discloses a key value cache cold and hot data exchange method, device and system, a medium and a product, and the method comprises the steps that a calculation core determines a target historical key value cache of each target historical lexical element required for generating a current lexical element, and sends a corresponding key value identification list to a data access control unit; when the data access control unit determines that the first memory does not store all the target historical key value caches according to the key value identification list and the remaining space cannot completely store the missing target historical key value caches, the multiple historical key value caches are eliminated, and the missing target historical key value caches are carried to the first memory from the second memory. And when the calculation core detects that the current system reaches the generation time point of the current lexical element, each target historical key value cache is obtained from the first memory at a high speed, and the current lexical element is generated and output. According to the technical scheme, the low cost of storing the target historical key value cache data can be considered, and meanwhile it is guaranteed that the computing core has enough high bandwidth to access the target historical key value cache.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence (AI) technology, and in particular to a method, apparatus, system, medium and product for key-value caching of cold and hot data exchange. Background Technology

[0002] With the continuous evolution of LLM (Large Language Model) technology, the ability of models to output long sequences has gradually increased, driving the increasing demand for contextual understanding in reasoning scenarios. When outputting the current word (also referred to as a token) in reasoning, the large language model relies on the key-value cache (KV cache) of all previously generated words to implement attention calculation between words. However, the storage footprint of the key-value cache is quadratically proportional to the length of the output word sequence. In long sequence generation scenarios, the amount of key-value cache data explodes, posing new challenges to the capacity and bandwidth of storage media.

[0003] To address the issue of the explosive growth of key-value caches, some models have adopted a key-value cache sparsity processing scheme to implement the model inference process. The core idea of ​​this scheme is that, at the stage of determining the current word, instead of using the key-value cache of all historical words, it selects to use the key-value cache of a fixed number of words from the historical words that are highly relevant to the current word to perform the calculation.

[0004] In the process of realizing this invention, the inventors discovered that when the key-value caching sparsity processing scheme is directly applied to the existing artificial intelligence chip storage system, it will either bring a large economic cost or introduce a large inference latency, and will not be able to truly give full play to the technical advantages of the key-value caching sparsity processing scheme. Summary of the Invention

[0005] This invention provides a method, apparatus, system, medium, and product for exchanging hot and cold data in a key-value cache. This method can simultaneously achieve high-bandwidth computation and low-cost storage in scenarios where large model inference is performed using a key-value cache sparsity processing scheme. According to one aspect of this invention, a method for exchanging hot and cold data in a key-value cache is provided, applied to scenarios where large model inference tasks are performed using a dedicated data processing chip. The dedicated data processing chip is communicatively connected to a first memory, a second memory, and a data access control unit. The communication bandwidth of the first memory is greater than that of the second memory, and the storage capacity of the first memory is smaller than that of the second memory. The dedicated data processing chip includes a computing core. The method includes:

[0006] When the computing core determines the target historical key value cache of each target historical word required to generate the current word, it sends the key value identifier list of each target historical key value cache to the data access control unit.

[0007] When the data access control unit determines, based on the key value identifier list, that the first memory does not store all target historical key value caches and that the remaining space in the first memory is insufficient to fully store the missing target historical key value caches, it controls the first memory to discard multiple historical key value caches and then moves the missing target historical key value caches from the second memory to the first memory.

[0008] When the computing core detects that the current system time has reached the generation time of the current word, it rapidly retrieves the historical key-value cache of each target from the first memory, and generates and outputs the current word based on the historical key-value cache of each target.

[0009] According to another aspect of the present invention, a key-value cache cold and hot data exchange device is provided, applied in scenarios where a dedicated data processing chip is used to perform large model inference tasks. The dedicated data processing chip is communicatively connected to a first memory, a second memory, and a data access control unit, respectively. The communication bandwidth of the first memory is greater than that of the second memory, and the storage capacity of the first memory is smaller than that of the second memory. The dedicated data processing chip includes a computing core. The device includes:

[0010] The key-value cache confirmation module is used to send the list of key-value identifiers of each target historical key-value cache to the data access control unit when the calculation core determines the target historical key-value caches of each target historical word required to generate the current word;

[0011] The key value cache transfer module is used to control the first memory to eliminate multiple historical key value caches and transfer the missing target historical key value caches from the second memory to the first memory when the data access control unit determines, based on the key value identifier list, that the first memory does not store all target historical key value caches and the remaining space of the first memory is insufficient to fully store the missing target historical key value caches.

[0012] The word generation module is used to quickly retrieve the historical key value cache of each target from the first memory when the current system time is detected to be the generation time point of the current word, and generate and output the current word based on the historical key value cache of each target.

[0013] According to another aspect of the present invention, a key-value caching cold and hot data exchange system is provided, comprising: a dedicated data processing chip, a first memory, a second memory, and a data access control unit. The dedicated data processing chip is communicatively connected to the first memory, the second memory, and the data access control unit, respectively. The data access control unit is communicatively connected to the first memory and the second memory, respectively. The communication bandwidth of the first memory is greater than that of the second memory, and the storage capacity of the first memory is smaller than that of the second memory. The dedicated data processing chip includes a computing core.

[0014] The method for exchanging hot and cold data in a key-value cache, as described in any one of the embodiments of the present invention, is achieved through the coordinated execution of the computing core and the data access control unit.

[0015] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement a hot and cold data exchange method for key-value caching as described in any embodiment of the present invention.

[0016] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps of the key-value caching cold and hot data exchange method as described in any embodiment of the present invention.

[0017] The technical solution of this invention is based on a key-value cache sparsity processing scheme that divides all historical lexical units into hot and cold data. In scenarios where a dedicated data processing chip is used to perform large-scale model inference tasks, a high-bandwidth first memory and a high-capacity second memory are used to store the hot and cold data in the key-value cache, respectively. Then, during the brief time difference between determining all the historical key-value caches required for the current lexical unit and actually generating the current lexical unit, the data access control unit moves the missing historical key-value cache from the second memory to the first memory. Thus, the computing core can retrieve the historical key-value cache from the first memory for each lexical unit calculation. This technical solution provides a tiered key-value cache storage design for large-scale model inference scenarios, ensuring low-cost storage of a large amount of historical key-value cache while also guaranteeing high-bandwidth access to the hot data in the historical key-value cache by the computing core.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram illustrating the implementation process of a key-value caching sparsity processing scheme provided by existing technology;

[0021] Figure 2 This is a flowchart of a method for exchanging hot and cold data in a key-value cache according to an embodiment of the present invention;

[0022] Figure 3 This is a flowchart of another method for exchanging hot and cold data in key-value caching according to an embodiment of the present invention;

[0023] Figure 4 This is a schematic diagram of a key-value caching cold and hot data exchange device according to an embodiment of the present invention;

[0024] Figure 5 This is a schematic diagram of a key-value caching cold and hot data exchange system according to an embodiment of the present invention;

[0025] Figure 6 This is a schematic diagram of a specific key-value caching cold and hot data exchange system provided according to an embodiment of the present invention. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] To facilitate understanding of the technical solutions in the various embodiments of the present invention, a simplified flowchart illustrating the implementation process of the key-value caching sparsity processing scheme is provided below. Specifically, as shown... Figure 1 As shown, when using a key-value caching sparsity scheme in the inference phase of a large model, the large model first obtains the prompt words input by the user. Based on these prompt words, the large model completes the model prefilling operation at time T0. In this prefilling stage, the large model uses the prompt words input by the user to generate key-value caches for M tokens, that is, key-value caches for token 0', key-value caches for token 1', ..., key-value caches for token M'.

[0029] After model pre-filling is completed, the large model enters the model decoding stage. At this time, the large model will be processed from T1 to T2. n+1 At each time point, the output sequentially starts from word 1, word 2, ..., word N, and word (N+1). Specifically, taking time point T1 as an example, when word 1 needs to be generated, the relevance calculation needs to be performed using the key-value cache of all historical words from the previous time point (i.e., time T0). After obtaining word 1, the key-value cache of word 1 needs to be cached synchronously at time point T1, that is, the key-value cache of word (M'+1).

[0030] Similarly, when the calculation reaches a certain point in time and finds that the total number of cached key-value pairs for all historical word elements exceeds a set threshold (e.g., 2048), the calculation of new word elements will no longer use the key-value pairs of all historical word elements, but will only use a fixed number (e.g., 2048) of historical word elements with high relevance to perform the calculation. Specifically, taking T... n+1 Taking a time point as an example, when generating lexical units (N+1), it is necessary to refer to T. nThe entire historical word set at the given time, that is, the historical key-value cache from word 0' to word (M'+N), is assumed to have a value greater than a preset fixed number K. Therefore, when calculating word (N+1), it is necessary to select the K most relevant historical word key-value caches from the historical key-value caches of word 0' to word (M'+N) to calculate the final word (N+1).

[0031] As can be seen from the implementation mechanism of the key-value caching sparsity processing scheme, in the process of implementing model inference based on the key-value caching sparsity processing scheme, it is indeed necessary to store all historical key-value caches. However, when the computing core calculates the next word to be output each time, it will only use a portion of all historical key-value caches. Therefore, based on whether the historical key-value cache is used during word calculation, it can be divided into hot data and cold data. In this embodiment, the inventors considered that under the existing artificial intelligence chip (typically, dedicated data processing chip) storage system, either storing a massive amount of historical key-value cache in a large-capacity, low-cost memory introduces a large inference latency, or storing a massive amount of historical key-value cache in a high-bandwidth, high-cost memory leads to a sharp increase in cost. Neither implementation method can meet the demand for low latency and low cost in large model inference. Based on this, the embodiments of the present invention creatively propose an implementation scheme that effectively balances high-bandwidth computing and low-cost storage to meet the actual needs of users.

[0032] Figure 2 This is a flowchart illustrating a method for exchanging hot and cold data in a key-value cache, provided by an embodiment of the present invention. This embodiment is applicable to large-model inference scenarios employing a key-value cache sparsity processing scheme. The method can be executed by a key-value cache hot and cold data exchange device, which can be implemented in hardware and / or software and is generally configured within a key-value cache hot and cold data exchange system. Specifically, this device includes a dedicated data processing chip, a data access control unit, a first memory, and a second memory. The method can be executed by the computing core and data access control unit within the dedicated data processing chip. Furthermore, the various functional modules within the device can be correspondingly configured within the computing core and data access control unit. The dedicated data processing chip can be understood as data processing hardware customized for a specific task. In various embodiments of the present invention, this dedicated data processing chip is mainly used to carry a specific large model and execute the large model to achieve model inference operations. In a specific example, a user inputs a question text into the large model, and the large model sequentially outputs the inferred words to the user, forming a question response result matching the question text.

[0033] Correspondingly, such as Figure 2As shown, the method may include:

[0034] S210. When the calculation core determines the target historical key value cache of each target historical word required to generate the current word, it sends the key value identifier list of each target historical key value cache to the data access control unit.

[0035] In this context, a lexical unit refers to the smallest semantic and computational unit that a model can process after text has been segmented by a word segmenter in the field of natural language processing and large-scale modeling. The current lexical unit refers to the lexical unit that the large model is currently reasoning about based on historical lexical information, and is expected to obtain. The target historical lexical unit refers to the historical lexical units that the large model needs to rely on to obtain the current lexical unit. The target historical key-value cache refers to the key-value cache corresponding to the historical lexical units that the large model uses to reason about the current lexical unit.

[0036] Specifically, the aforementioned large model can be understood as a pre-trained large language model with a large number of parameters (e.g., exceeding 1 billion), based on a deep neural network architecture (e.g., Transformer). This large model can be a generative AI system, that is, after being trained on massive amounts of data, it can achieve natural language understanding, generation, and reasoning to obtain new content from multi-source input data such as text, images, or audio and video. Typically, this large model can be an autoregressive language model that supports setting the context size.

[0037] The key-value identifier list is used to store the lexical identifier information of each target historical lexical corresponding to the target historical key-value cache. The lexical identifier information can be understood as the unique numerical number of each lexical in the preset model word table. Based on the lexical identifier information, the target historical lexical that is adapted to it can be uniquely located, and then the target historical key-value cache corresponding to the target historical lexical can be uniquely located.

[0038] The data access control unit can be understood as a hardware unit independent of the dedicated data processing chip. The data access control unit is used to manage the data in the first memory and the second memory, and to realize the data transfer function between the first memory and the second memory.

[0039] Understandably, large models need to rely on historical lexical units during the reasoning process of generating the current lexical unit. Therefore, before reasoning about the current lexical unit, the large model needs to identify the target historical lexical units upon which the generation of the current lexical unit is based through its computational core, determine the target historical key-value cache corresponding to each target historical lexical unit, and after generating a key-value identifier list that can identify all target historical key-value caches, send this key-value identifier list to the data access control unit as the basis for scheduling the target historical key-value caches. The target historical key-value cache can be understood as the hot data used in the current lexical unit generation process, while the non-target historical key-value cache can be understood as cold data.

[0040] S220. When the data access control unit determines, based on the key value identifier list, that the first memory does not store all target historical key value caches and the remaining space of the first memory is insufficient to fully store the missing target historical key value caches, it controls the first memory to eliminate multiple historical key value caches and then moves the missing target historical key value caches from the second memory to the first memory.

[0041] The first type of memory can refer to a type of memory used to store hot data in the historical key-value cache, characterized by high communication bandwidth, small storage capacity, and high cost. Examples include HBM (High Bandwidth Memory) or GDDR (Graphics Double Data Rate Memory). The second type of memory can refer to another type of memory used to store cold data in the historical key-value cache, characterized by low communication bandwidth, large storage capacity, and low cost. Examples include LPDDR (Low Power Double Data Rate Memory) or DDR (Double Data Rate Memory).

[0042] Understandably, when the large model generates the current lexical term through inference, it can use the data access control unit to determine whether all target historical key-value caches are stored in the first memory based on the key-value identifier list. Since the second memory has a larger storage capacity and lower cost, all historical key-value caches generated during the model inference phase are stored in the second memory. Once the computational core determines that a historical key-value cache is the target historical key-value cache required for the current lexical term calculation, it needs to ensure that these target historical key-value caches are located in the first memory, which has a higher communication bandwidth, so that the computational core can access each target historical key-value cache at high speed when calculating the current lexical term.

[0043] Therefore, before reasoning about the current word based on the target historical key value cache, the data access control unit first checks whether the first memory stores all the target historical key value caches according to the key value identifier list. If not all the target historical key value caches are stored and the remaining space of the first memory cannot completely store the missing target historical key value caches, the first memory is controlled to eliminate multiple non-target historical key value caches. After the space requirement is met, the missing target historical key value caches are moved from the second memory to the first memory.

[0044] In an optional implementation of this embodiment, a data management table can be maintained locally in the data access control unit. This table records which historical key-value caches are stored in the first memory, and the storage addresses of each historical key-value cache within that first memory. Furthermore, through simple data comparison, the data access control unit can quickly determine whether all target historical key-value caches are stored in the first memory without needing to interact with the first memory.

[0045] S230. When the computing core detects that the current system time has reached the generation time of the current word, it rapidly retrieves the historical key value cache of each target from the first memory, and generates and outputs the current word based on the historical key value cache of each target.

[0046] Understandably, the process of the large model using the target historical key-value cache to infer and generate the current word is strictly executed according to the preset large model inference process at each time point. When the computing core detects that the current large model inference system time has reached the preset current word generation time point, it performs the operation of high-speed retrieval of each target historical key-value cache from the first memory, and infers and generates the current word based on each target historical key-value cache, and finally outputs the corresponding inference result.

[0047] Furthermore, after the large model outputs the current word element, this current word element will be used as the latest historical word element, and the key-value cache of this historical word element will also be stored as the latest historical key-value cache. Then, it can return to execute S210 to prepare for the calculation of the latest current word element, and so on, until the large model outputs all words as the response result for a user's input question.

[0048] The technical solution of this invention is based on a key-value cache sparsity processing scheme that divides all historical lexical units into hot and cold data. In scenarios where a dedicated data processing chip is used to perform large-scale model inference tasks, a high-bandwidth first memory and a high-capacity second memory are used to store the hot and cold data in the key-value cache, respectively. Then, during the brief time difference between determining all the historical key-value caches required for the current lexical unit and actually generating the current lexical unit, the data access control unit moves the missing historical key-value cache from the second memory to the first memory. Thus, the computing core can retrieve the historical key-value cache from the first memory for each lexical unit calculation. This technical solution provides a tiered key-value cache storage design for large-scale model inference scenarios, ensuring low-cost storage of a large amount of historical key-value cache while also guaranteeing high-bandwidth access to the hot data in the historical key-value cache by the computing core.

[0049] Figure 3This is a flowchart of another method for cold and hot data exchange of key-value cache provided by an embodiment of the present invention. This embodiment is a specific implementation based on the above embodiment. In this embodiment, the implementation of the data access control unit detecting the target historical key-value cache of the first memory and transferring data from the second memory to the first memory is specified.

[0050] Correspondingly, such as Figure 3 As shown, the method may include:

[0051] S310. When the calculation core determines the target historical key value cache of each target historical word required to generate the current word, it sends the key value identifier list of each target historical key value cache to the data access control unit.

[0052] Specifically, the determination time point of the target historical key value cache for each target historical word required to generate the current word word is earlier than the generation time point of the current word word, and there is a preset time interval (typically on the order of microseconds) between the determination time point and the generation time point.

[0053] Understandably, in order to ensure inference efficiency, the large model inference process needs to perform operations at each stage according to the preset time points. Therefore, before reaching the generation time point of the current word in the large model inference process, it is necessary to ensure that the target historical key-value cache of each target historical word required to generate the current word has been determined.

[0054] After detailed analysis of the model inference process of various large models, the inventors discovered that, approximately a few microseconds before each target historical key-value cache is used for attention calculation to obtain the current word, the computational core has already determined which target historical key-value caches will be selected for attention calculation. In most cases, this lead time of a few microseconds is sufficient to complete the data transfer of dozens of historical key-value caches. The specific reason is that during the model inference process of large models, the generation of each word requires several layers of computation, and the combination of historical words required for each layer of computation changes slowly. The ability to transfer dozens of historical key-value caches can, in most scenarios, completely transfer the target historical key-value caches missing in the first memory from the second memory to the first memory, allowing the computational core to enjoy high bandwidth when retrieving all target historical key-value caches.

[0055] S320: The data access control unit compares each key value identifier in the key value identifier list with each historical key value cache stored in the first memory to identify the missing target historical key value cache in the first memory.

[0056] Understandably, before reasoning about the current word based on the target historical key value cache, the data access control unit can first compare each key value identifier in the key value identifier list with each historical key value cache stored in the first memory, thereby identifying and marking the missing target historical key value cache in the first memory.

[0057] Of course, when comparing each key value in the key value identifier list with the historical key value caches stored in the first memory, there may be a situation where the first memory does not lack any target historical key value caches. In this case, no additional processing is required; it is sufficient to directly retrieve the historical key value caches from the first memory at high speed when the current word generation time point is reached. Therefore, this embodiment mainly details the complex situation where some target historical key value caches are missing in the first memory.

[0058] S330. The data access control unit detects the remaining space of the first memory, and when it is determined that the remaining space of the first memory cannot completely store the missing target historical key value cache, it removes multiple historical key value caches from the first memory according to a preset data removal strategy.

[0059] Among them, the data eviction strategy can refer to the strategy of deleting non-target historical key-value cache stored in the first memory.

[0060] Understandably, after the data access control unit identifies the missing target historical key value cache in the first memory, it checks the remaining space of the first memory. If the remaining space of the first memory cannot completely store the missing target historical key value cache, it can eliminate at least a number of non-target historical key value caches occupying the corresponding storage space from the first memory according to the preset data elimination strategy and the storage space required by the missing target historical key value cache.

[0061] Of course, if the remaining space in the first memory can completely store the missing target historical key value cache, the missing target historical key value cache can be moved from the second memory to the first memory directly by executing data transfer instructions.

[0062] Optionally, the data access control unit may, according to a preset data eviction policy, evict multiple historical key-value caches from the first memory, which may include:

[0063] The data access control unit calculates the space required for the missing target historical key-value cache.

[0064] The data access control unit sorts the historical key-value caches in the first memory according to the historical access count and / or data lifetime of each historical key-value cache in the first memory;

[0065] Based on the sorting results, the data access control unit removes multiple historical key-value caches that match the space occupancy from the first memory.

[0066] Among them, data lifetime can refer to the entire data lifespan of a key-value cache for a certain word, from the time it is generated until the reasoning ends and it becomes invalid.

[0067] Understandably, in order to meet the space requirements for storing the target historical key value cache missing in the first memory in the first memory, the data access control unit can eliminate multiple historical key value caches from the first memory according to a preset data elimination strategy.

[0068] Specifically, it is necessary to first calculate the specific space required to store the missing target historical key-value cache in the first memory, and then sort the historical key-value caches in the first memory based solely on the number of historical accesses, the data lifetime, or both the number of historical accesses and the data lifetime.

[0069] For example, historical key-value caches with more historical accesses and shorter data lifespans are more likely to be the target historical key-value caches in the entire inference process of the large model. Based on the sorting results, starting with historical key-value caches with fewer historical accesses or longer data lifespans, multiple historical key-value caches that match the calculated required space occupancy can be eliminated from the first memory.

[0070] S340: Generate a data transfer instruction that matches the missing target historical key value cache through the data access control unit, and transfer the missing target historical key value cache from the second memory to the first memory by executing the data transfer instruction.

[0071] Among them, the data transfer instruction can refer to the operation instruction executed by the data access control unit when transferring data to the target historical key value cache. It can include the data identifier, data size and storage location of the target historical key value cache to be transferred in the second memory, as well as the instruction information such as the target storage location to be transferred in the first memory.

[0072] Understandably, after determining the missing target historical key value cache in the first memory required for the large model inference to generate the current word, the data access control unit generates a data transfer instruction that matches the missing target historical key value cache. Based on the data transfer instruction information, the identification information of the missing target historical key value cache data and its storage location in the second memory, as well as the target storage location to be transferred to the first memory, can be determined. Finally, the missing target historical key value cache is transferred from the second memory to the first memory by executing the data transfer instruction.

[0073] S350: When the computing core detects that the current system time has reached the generation time of the current word, it sends multiple data read requests for the historical key value cache of each target to the data access control unit.

[0074] Specifically, after the computing core determines the identification information of each target historical key-value cache a few microseconds before the current word's generation time, the data access control unit can move all target historical key-value caches to the first memory within those microseconds to the greatest extent possible. Furthermore, when the computing core detects that the current system time has reached the current word's generation time, it can directly send a data read request for each target historical key-value cache to the data access control unit, thereby rapidly retrieving each target historical key-value cache from the first memory to perform attention calculations for the current word.

[0075] S360: The data access control unit retrieves the historical key value cache of each target from the first memory at high speed according to the multiple data read requests received, and feeds back the historical key value cache of each target to the computing core.

[0076] Specifically, the computing core can send a matching data read request to the first memory for each target historical key-value cache.

[0077] Understandably, when the data access control unit receives multiple data read requests from the computing core, it can quickly retrieve the corresponding multiple target historical key value caches from the first memory according to the read request information, and feed back each target historical key value cache to the computing core for the computing core to use for reasoning to generate the current word.

[0078] Optionally, the data access control unit, based on multiple received data read requests, rapidly retrieves the historical key-value caches of each target from the first memory and feeds back the historical key-value caches of each target to the computing core. Specifically, this may include:

[0079] S3601. The data access control unit retrieves the target historical key value cache corresponding to each data read request from the first memory at high speed according to the received data read requests.

[0080] S3602. If the data access control unit determines that the current target historical key value cache that matches the current data read request is not currently stored in the first memory, then it temporarily stores the current data read request.

[0081] S3603. When the data access control unit receives the current target historical key value cache from the second memory, it provides the current target historical key value cache as the feedback result of the current data read request directly to the computing unit, and synchronously moves the current target historical key value cache to the first memory.

[0082] Understandably, when the computing core detects that the current system time has reached the generation time of the current word, it sends a data read request for each target historical key-value cache to the data access control unit. Upon receiving the data read request, the data access control unit needs to perform a high-speed retrieval operation from the first memory to obtain the target historical key-value cache corresponding to each data read request. If no target historical key-value cache matching the current data read request is detected in the first memory, it indicates a special situation. That is, when the current system time reaches the generation time of the current word, the data access control unit has not successfully transferred all the missing target historical key-value caches from the second memory to the first memory; in other words, some missing target historical key-value caches are still being transferred.

[0083] At this point, the data access control unit can temporarily store the current data read requests corresponding to these target historical key-value caches that have not yet been moved to the first memory (i.e., the current target historical key-value cache mentioned above). Then, when the data access control unit receives the current target historical key-value cache from the second memory, it can synchronously send the received current target historical key-value cache to the computing unit and the first memory in real time to reduce read request latency and enable the computing core to obtain the required target historical key-value cache for computation more quickly.

[0084] Based on the above embodiments, after generating and outputting the current lexical unit by calculating the historical key values ​​of each target through the computing core, it may further include:

[0085] The computational core uses the current key value cache that matches the current word as the latest historical key value cache, and synchronously stores it in the first and second memories for use when generating the next word.

[0086] Understandably, during the large model inference process, the computational core generates and outputs the current lexical unit based on the historical key-value cache of each target. The generated current lexical unit can be used as the basis for generating lexical units in subsequent inferences. The current key-value cache that matches the current lexical unit can be used as the latest historical key-value cache and synchronously stored in the first memory and the second memory. The second memory is used to ensure that all historical key-value cache data can be detected and retrieved. The first memory is used because the newly generated key-value cache will be used in the inference calculation of one or several lexical units with a high probability. Through the above operations, the number of times hot data is moved can be reduced to the greatest extent.

[0087] Of course, it is understandable that if the storage space of the first memory is found to be full before storing the current key value cache of the current word matching, the data access control unit can also eliminate one or more historical key value caches from the first memory according to the preset data elimination strategy, so as to free up space in the first memory to store the current key value cache.

[0088] Furthermore, considering that each time a current word is generated, its corresponding current key-value cache is stored in the first memory, when it is detected that the remaining space in the first memory is insufficient to completely store the missing target historical key-value cache, more historical key-value caches can be evicted, reserving storage space in the first memory for the current key-value cache. This reduces the number of times the data eviction policy is applied and lowers the excessive occupancy of the computing core.

[0089] Optionally, after generating and outputting the current lexical unit based on the historical key-value cache of each target through the computation core, it may also include:

[0090] After a complete large model inference process is completed, the data access control unit identifies all associated historical key-value caches stored in the first memory that match the current large model inference process.

[0091] The data access control unit marks the invalidation status of each associated historical key-value cache in the locally managed dataset; the cache space occupied by the data marked as invalid will be reclaimed and reused.

[0092] The local management dataset can refer to the data set maintained locally by the data access control unit to mark the status of all historical key-value caches in the first memory. An invalid state can refer to a state in the first memory where the key-value cache is marked as no longer needed by the large model when generating the next lexical after the large model has completed its inference process.

[0093] Understandably, after a complete large-scale model inference process generates the current lexical unit, the data access control unit can identify all associated historical key-value caches stored in the first memory that match the current large-scale model inference process. Understandably, the data lifespan of these associated historical key-value caches ends after the large-scale model inference is completed. The data access control unit can update the status of each corresponding associated historical key-value cache to an invalid state in the local management dataset. The cache space occupied by the historical key-value cache data marked as invalid in the first memory will be reclaimed and used in subsequent hot data storage processes.

[0094] The above technical solution effectively ensures the rationality of hot data stored in the first memory by evicting data in the first memory based on the historical access count and / or data lifetime of each historical key-value cache in the first memory. This can minimize the number of data eviction operations. At the same time, during the process of the data access control unit moving data from the first memory for the computing core to calculate the current word, if it is determined that the current target historical key-value cache matching the current data read request is not currently stored in the first memory, the implementation method of directly providing the current target historical key-value cache to the computing unit when receiving the current target historical key-value cache feedback from the second memory can reduce the time for the computing core to obtain the target historical key-value cache to a certain extent in special scenarios where the data access control unit cannot complete the data moving of all target historical key-value caches within a few microseconds, thereby minimizing inference latency.

[0095] Optionally, after sending the list of key-value identifiers of each target's historical key-value cache to the data access control unit via the computing core, it may also include:

[0096] When the data access control unit determines, based on the key value identifier list, that the first memory fully stores the entire target historical key value cache, it does not perform the operation of transferring data from the second memory to the first memory; and

[0097] When the data access control unit determines, based on the key value identifier list, that the first memory does not store all target historical key value caches, and that the remaining space in the first memory can completely store the missing target historical key value caches, it directly moves the missing target historical key value caches from the second memory to the first memory.

[0098] Understandably, since all historical key-value cache data generated by the large model is stored in the second memory, and during the inference process, the data access control unit will move the target historical key-value cache determined by the computing core from the second memory to the first memory, before the large model performs inference on the current word based on the target historical key-value cache, the data access control unit first checks whether the first memory stores all target historical key-value caches according to the key-value identifier list. If all target historical key-value caches are stored, the operation of moving data from the second memory to the first memory is not performed. If the first memory does not store all target historical key-value caches, and the remaining space in the first memory can completely store the missing target historical key-value caches, then the missing target historical key-value caches can be directly moved from the second memory to the first memory to ensure that the computing core can obtain the hot data in the target historical key-value caches from the first memory with high bandwidth.

[0099] Based on the above embodiments, there are multiple first memories and multiple second memories. Each first memory is pre-divided into a non-exchangeable data area and an exchangeable data area, wherein historical key-value cache is stored in the exchangeable data area of ​​each first memory.

[0100] In particular, considering that the number of historical key-value caches that need to be stored or used for attention calculations during large model inference is often large, in scenarios where dedicated data processing chips are used to perform large model inference tasks, multiple primary memories and multiple secondary memories are often required to cooperate in storage to meet the actual model inference requirements.

[0101] The exchangeable data area can refer to the area in the first memory used to store historical key-value caches, and can store key-value cache data transferred from the second memory. The non-exchangeable data area can refer to the area in the first memory other than the exchangeable data area, and can be used to store model weights, operator code, and software stack management data, etc.

[0102] Generally, the data required to be stored in the non-exchangeable data area is usually fixed. Therefore, after determining the required storage space size of the non-exchangeable data area in the first memory, the remaining area in the first memory can be used as the exchangeable data area.

[0103] Figure 4 This is a schematic diagram of a key-value caching cold and hot data exchange device provided in an embodiment of the present invention. Figure 4 As shown, the device includes: a key-value cache confirmation module 410, a key-value cache transport module 420, and a lexical generation module 430.

[0104] The key value cache confirmation module 410 is used to send the key value identifier list of each target historical key value cache to the data access control unit when the calculation core determines the target historical key value cache of each target historical word required to generate the current word;

[0105] The key value cache transfer module 420 is used to control the first memory to eliminate multiple historical key value caches and transfer the missing target historical key value caches from the second memory to the first memory when the data access control unit determines, based on the key value identifier list, that the first memory does not store all target historical key value caches and the remaining space of the first memory is insufficient to completely store the missing target historical key value caches.

[0106] The word generation module 430 is used to quickly retrieve the historical key value cache of each target from the first memory when the current system time is detected to have reached the generation time point of the current word through the computing core, and generate and output the current word based on the historical key value cache of each target.

[0107] The technical solution of this invention is based on a key-value cache sparsity processing scheme that divides all historical lexical units into hot and cold data. In scenarios where a dedicated data processing chip is used to perform large-scale model inference tasks, a high-bandwidth first memory and a high-capacity second memory are used to store the hot and cold data in the key-value cache, respectively. Then, during the brief time difference between determining all the historical key-value caches required for the current lexical unit and actually generating the current lexical unit, the data access control unit moves the missing historical key-value cache from the second memory to the first memory. Thus, the computing core can retrieve the historical key-value cache from the first memory for each lexical unit calculation. This technical solution provides a tiered key-value cache storage design for large-scale model inference scenarios, ensuring low-cost storage of a large amount of historical key-value cache while also guaranteeing high-bandwidth access to the hot data in the historical key-value cache by the computing core.

[0108] Based on the above embodiments, the determination time point of the target historical key value cache of each target historical word required to generate the current word word is determined earlier than the generation time point of the current word word, and there is a time interval of a preset time magnitude between the determination time point and the generation time point.

[0109] Based on the above embodiments, the key-value cache transfer module 420 can be specifically used to: compare each key-value identifier in the key-value identifier list with each historical key-value cache stored in the first memory through the data access control unit, and identify the missing target historical key-value cache in the first memory; detect the remaining space of the first memory through the data access control unit, and when it is determined that the remaining space of the first memory cannot completely store the missing target historical key-value cache, remove multiple historical key-value caches from the first memory according to a preset data removal strategy; generate a data transfer instruction matching the missing target historical key-value cache through the data access control unit, and transfer the missing target historical key-value cache from the second memory to the first memory by executing the data transfer instruction.

[0110] Based on the above embodiments, the key-value cache transport module 420 can also be specifically used for:

[0111] The data access control unit calculates the space required for the missing target historical key-value cache. The data access control unit then sorts the historical key-value caches in the first memory based on their historical access counts and / or data lifetimes. Based on the sorting results, the data access control unit removes multiple historical key-value caches from the first memory that match the required space usage.

[0112] Based on the above embodiments, the lexical generation module 430 can be specifically used for:

[0113] When the computing core detects that the current system time has reached the generation time of the current word, it sends multiple data read requests for the historical key-value caches of each target to the data access control unit. Based on the received data read requests, the data access control unit rapidly retrieves the historical key-value caches of each target from the first memory and feeds them back to the computing core.

[0114] Based on the above embodiments, the lexical generation module 430 can also be specifically used for:

[0115] The data access control unit rapidly retrieves the target historical key-value cache corresponding to each received data read request from the first memory. If the data access control unit determines that the current target historical key-value cache matching the current data read request is not currently stored in the first memory, it temporarily stores the current data read request. When the data access control unit receives the current target historical key-value cache from the second memory, it provides the current target historical key-value cache as feedback result of the current data read request directly to the computing unit, and simultaneously moves the current target historical key-value cache to the first memory.

[0116] Based on the above embodiments, the lexical generation module 430 can also be specifically used for:

[0117] The computational core uses the current key value cache that matches the current word as the latest historical key value cache, and synchronously stores it in the first and second memories for use when generating the next word.

[0118] Based on the above embodiments, a key-value cache supplementation module may also be included, for:

[0119] When the data access control unit determines, based on the key value identifier list, that the first memory fully stores all target historical key value caches, it does not perform the operation of transferring data from the second memory to the first memory. Conversely, when the data access control unit determines, based on the key value identifier list, that the first memory does not store all target historical key value caches, and that the remaining space in the first memory is sufficient to fully store the missing target historical key value caches, it directly transfers the missing target historical key value caches from the second memory to the first memory.

[0120] Based on the above embodiments, a data management module may also be included, for:

[0121] After a complete large model inference process is completed, the data access control unit identifies all associated historical key-value caches stored in the first memory that match the current large model inference process. The data access control unit then marks the invalidation status of each associated historical key-value cache in the locally managed dataset. The cache space occupied by data marked as invalid will be reclaimed.

[0122] The cold and hot data exchange device for key-value caching provided in the embodiments of the present invention can execute the cold and hot data exchange method for key-value caching provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0123] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solutions of this invention comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0124] Figure 5This is a schematic diagram of a key-value caching cold and hot data exchange system according to an embodiment of the present invention. The system includes: a dedicated data processing chip 510, a first memory 520, a second memory 530, and a data access control unit 540. The dedicated data processing chip 510 is communicatively connected to the first memory 520, the second memory 530, and the data access control unit 540. The data access control unit 540 is communicatively connected to the first memory 520 and the second memory 530. The communication bandwidth of the first memory 520 is greater than that of the second memory 530, and the storage capacity of the first memory 520 is smaller than that of the second memory 530. The dedicated data processing chip 510 includes a computing core 5101. That is, the cost of the first memory 520 is much higher than that of the second memory 530.

[0125] It is understandable that, in order to effectively meet the needs of large model inference tasks and improve the system's data processing efficiency, the system may include multiple first memories 520 (not shown in the figure) and multiple second memories 530 (not shown in the figure).

[0126] In addition, each first memory 520 is pre-divided into a non-interchangeable data area and an interchangeable data area. The non-interchangeable data area can be used to store management data that does not need to be changed, while the interchangeable data area can be used to store key-value cache (i.e., hot data) transferred from the second memory 530, so as to meet the high-bandwidth acquisition of target historical key-value cache data by the computing core 5101.

[0127] In the key-value cache cold and hot data exchange system provided in this embodiment, the key-value cache cold and hot data exchange method as described in any one of the embodiments of the present invention is jointly implemented by the computing core 5101 and the data access control unit 540.

[0128] Figure 6 This is a schematic diagram of a specific key-value caching cold and hot data exchange system according to an embodiment of the present invention. Specifically, in this specific application scenario, the first memory can be HBM or GDDR, and the second memory can be LPDDR or DDR. It should be noted that... Figure 6 The number of the first and second memories shown is for illustrative purposes only, and the number of both is not limited in this embodiment.

[0129] This invention combines a first memory for high-speed storage and a second memory for low-cost storage, achieving the dual goals of low-cost large-capacity storage and high bandwidth. For example... Figure 6As shown, in this key-value cache hot and cold data exchange system, the storage space of the first memory is divided. The non-swappable data area is used to store data outside the key-value cache, such as model weights, operator code, and software stack management data. This data will always be stored in the first memory and will not participate in swapping in or out relative to the second memory. The swappable data area and the second memory are both used to store the key-value cache. Data in the key-value cache stored in the swappable data area may be swapped in or out.

[0130] A data access control unit is configured on the data bus. Memory accesses issued by the computing core are processed by this control unit before being forwarded to the data bus. Accesses to non-swappable data areas are directly forwarded to the high-speed storage interface of the first memory, while accesses to swappable data areas require selection between the second and first memory, depending on the situation. The data access control unit is responsible for swapping key-value caches between the second and first memories. It records information and controls the data transfer strategy in units of N bytes (N is the value of one key-value cache for a single term, e.g., 1.5MB). Each data block records information including: the starting address of the data block, whether the data block is located in the first memory, the access count of the data block, and the term identifier of the data block. This information is used to calculate the data block swapping strategy. Each term's information record occupies only a few bytes. Even with a key-value cache of several hundred GB, its management volume is only a few MB in size. All this information can be stored in the storage within the data access control unit, achieving high-speed retrieval.

[0131] For each inference task, the software needs to pre-allocate the size of the exchangeable data area in the first memory, and this size is optimized according to the characteristics of the model. In the large language model, once a key-value cache for a word is generated, there is no need to rewrite it until it is discarded at the end of the inference process. Based on this characteristic, the embodiments of the present invention design the key-value cache in the second memory and the key-value cache in the first memory to have an inclusion relationship. That is, if the key-value cache of a word can be found in the first memory, then there must be a copy of it in the second memory, and vice versa. This can improve the efficiency of the key-value cache being evicted from the first memory. When evicted, it is only necessary to mark it in the data management structure of the data access control unit, without having to move the data from the first memory to the second memory.

[0132] During a model inference process, the key-value caches of initially generated terms are simultaneously written to both the first and second memory. As the inference process progresses, the number of output terms increases, eventually exceeding the size of the swappable data area. At this point, when a new term's key-value cache is generated, the data access control unit needs to select a historical key-value cache and evict it from the first memory to make room for the new term's key-value cache. This is because newly generated terms have a higher probability of being relevant to future outputs and are more likely to be reused in subsequent inferences; therefore, they should be prioritized for placement in high-speed storage. The selection of which historical term's key-value cache to evict requires a weighted judgment considering both the term's lifetime and the number of times it has been accessed. The weights of each factor are configurable hardware parameters.

[0133] For large language models that apply key-value caching sparsity techniques, at each layer of computation for each new word, a fixed number (e.g., 2048) of the key-value caches with the highest relevance will be selected from all historical key-value caches for attention computation. Therefore, each computation of a new word will bring about an update to the access count of the historical key-value cache. The data access control unit needs to maintain an access count for each word to serve as a reference for the eviction policy.

[0134] In most current sparse key-value caching models, it's possible to determine which historical lexical tokens will be selected for attention computation a few microseconds before the key-value cache is used. The software can then send a list of selected token identifiers to the data access control unit in advance. The data access control unit compares this list with its managed data. For token identifiers located in the second memory, it sends a data transfer instruction to move them from the second memory to the first memory in advance. If the space in the exchangeable data area is full, a corresponding number of key-value caches need to be evicted according to a strategy. A few microseconds of advance time are sufficient to move dozens of key-value caches. During model operation, each token generation involves several layers of computation, and the required combination of historical lexical tokens changes slowly at each layer. The ability to move dozens of key-value caches ensures that, in most scenarios, the computing core can find them in the first memory when reading the key-value cache, enjoying high bandwidth.

[0135] Of course, there is another situation: if the data access control unit does not have time to move all the required key-value caches within the prefetch time, the computing core has already sent the data access request to the data access control unit. The data access unit needs to temporarily store these read requests. After the data is returned from the second memory, it is directly returned to the computing core and updated in the first memory. This can reduce the read request latency and allow the computing core to get the data faster.

[0136] At the end of a complete inference cycle, all lexical units no longer need to be used, and the initial memory space they occupy can be released to cache the next inference lexical units. At this point, the software needs to send a refresh command to the data access control unit to mark the key-value cache space of all lexical units as obsolete, so that their management information can be rebuilt in the next inference cycle.

[0137] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0138] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0139] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0140] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0141] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0142] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0143] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for exchanging hot and cold data using key-value caching, characterized in that, In scenarios where a dedicated data processing chip is used to perform large model inference tasks, the dedicated data processing chip is communicatively connected to a first memory, a second memory, and a data access control unit. The communication bandwidth of the first memory is greater than that of the second memory, and the storage capacity of the first memory is smaller than that of the second memory. The dedicated data processing chip includes a computing core. The method includes: When the computing core determines the target historical key value cache of each target historical word required to generate the current word, it sends the key value identifier list of each target historical key value cache to the data access control unit. When the data access control unit determines, based on the key value identifier list, that the first memory does not store all target historical key value caches and that the remaining space in the first memory is insufficient to fully store the missing target historical key value caches, it controls the first memory to discard multiple historical key value caches and then moves the missing target historical key value caches from the second memory to the first memory. When the computing core detects that the current system time has reached the generation time of the current word, it rapidly retrieves the historical key-value cache of each target from the first memory, and generates and outputs the current word based on the historical key-value cache of each target.

2. The method according to claim 1, characterized in that, The time point at which the target historical key value cache of each target historical word required to generate the current word is determined is earlier than the time point at which the current word is generated, and there is a time interval of a preset time magnitude between the determined time point and the generation time point.

3. The method according to claim 1, characterized in that, When the data access control unit determines, based on the key value identifier list, that the first memory does not store all target historical key value caches, and the remaining space in the first memory is insufficient to completely store the missing target historical key value caches, it controls the first memory to discard multiple historical key value caches and then moves the missing target historical key value caches from the second memory to the first memory, including: The data access control unit compares each key value identifier in the key value identifier list with each historical key value cache stored in the first memory to identify the missing target historical key value cache in the first memory. The data access control unit detects the remaining space of the first memory, and when it is determined that the remaining space of the first memory cannot completely store the missing target historical key value cache, it removes multiple historical key value caches from the first memory according to the preset data removal strategy. The data access control unit generates a data transfer instruction that matches the missing target historical key value cache, and by executing the data transfer instruction, the missing target historical key value cache is transferred from the second memory to the first memory.

4. The method according to claim 3, characterized in that, The data access control unit, according to a preset data eviction policy, evicts multiple historical key-value caches from the first memory, including: The data access control unit calculates the space required for the missing target historical key-value cache. The data access control unit sorts the historical key-value caches in the first memory according to the historical access count and / or data lifetime of each historical key-value cache in the first memory; Based on the sorting results, the data access control unit removes multiple historical key-value caches from the first memory that match the space occupancy.

5. The method according to claim 1, characterized in that, After the computing core sends the list of key-value identifiers cached for each target's historical key-values ​​to the data access control unit, it also includes: When the data access control unit determines, based on the key value identifier list, that the first memory fully stores the entire target historical key value cache, it does not perform the operation of transferring data from the second memory to the first memory; and When the data access control unit determines, based on the key value identifier list, that the first memory does not store all target historical key value caches, and that the remaining space in the first memory can completely store the missing target historical key value caches, it directly moves the missing target historical key value caches from the second memory to the first memory.

6. The method according to any one of claims 1-5, characterized in that, When the computational core detects that the current system time has reached the generation time of the current word, it rapidly retrieves the historical key-value cache of each target from the first memory, specifically including: When the computing core detects that the current system time has reached the generation time of the current word, it sends multiple data read requests for the historical key value cache of each target to the data access control unit. The data access control unit retrieves the historical key-value cache of each target from the first memory at high speed according to the multiple data read requests received, and feeds back the historical key-value cache of each target to the computing core.

7. The method according to claim 6, characterized in that, The data access control unit, based on multiple received data read requests, rapidly retrieves the historical key-value caches of each target from the first memory and feeds these caches back to the computing core. Specifically, this includes: The data access control unit retrieves the target historical key value cache corresponding to each data read request from the first memory at high speed according to the received data read requests; If the data access control unit determines that the current target historical key-value cache that matches the current data read request is not currently stored in the first memory, then it temporarily stores the current data read request. When the data access control unit receives the current target historical key value cache from the second memory, it provides the current target historical key value cache as the feedback result of the current data read request directly to the computing unit, and synchronously moves the current target historical key value cache to the first memory.

8. The method according to any one of claims 1-5, characterized in that, After the computational core generates and outputs the current lexical unit based on the historical key values ​​of each target, it also includes: The computational core uses the current key value cache that matches the current word as the latest historical key value cache, and synchronously stores it in the first and second memories for use when generating the next word.

9. The method according to any one of claims 1-5, characterized in that, After the computational core generates and outputs the current lexical unit based on the historical key values ​​of each target, it also includes: After a complete large model inference process is completed, the data access control unit identifies all associated historical key-value caches stored in the first memory that match the current large model inference process. The data access control unit marks the invalidation status of each associated historical key-value cache in the locally managed dataset; the cache space occupied by the data marked as invalid will be reclaimed and reused.

10. The method according to claim 1, characterized in that, There are multiple first memories and multiple second memories. Each first memory is pre-divided into a non-exchangeable data area and an exchangeable data area, wherein historical key-value cache is stored in the exchangeable data area of ​​each first memory.

11. A key-value caching cold and hot data exchange device, characterized in that, This device is applied to scenarios where a dedicated data processing chip is used to perform large model inference tasks. The dedicated data processing chip is communicatively connected to a first memory, a second memory, and a data access control unit. The communication bandwidth of the first memory is greater than that of the second memory, and the storage capacity of the first memory is smaller than that of the second memory. The dedicated data processing chip includes a computing core. The device comprises: The key-value cache confirmation module is used to send the list of key-value identifiers of each target historical key-value cache to the data access control unit when the calculation core determines the target historical key-value caches of each target historical word required to generate the current word; The key value cache transfer module is used to control the first memory to eliminate multiple historical key value caches and transfer the missing target historical key value caches from the second memory to the first memory when the data access control unit determines, based on the key value identifier list, that the first memory does not store all target historical key value caches and the remaining space of the first memory is insufficient to fully store the missing target historical key value caches. The word generation module is used to quickly retrieve the historical key value cache of each target from the first memory when the current system time is detected to be the generation time point of the current word, and generate and output the current word based on the historical key value cache of each target.

12. A key-value caching system for exchanging hot and cold data, characterized in that, include: The system includes a dedicated data processing chip, a first memory, a second memory, and a data access control unit. The dedicated data processing chip is communicatively connected to the first memory, the second memory, and the data access control unit. The data access control unit is communicatively connected to the first memory and the second memory. The communication bandwidth of the first memory is greater than that of the second memory, and the storage capacity of the first memory is smaller than that of the second memory. The dedicated data processing chip includes a computing core. The method for exchanging hot and cold data in key-value caching as described in any one of claims 1-10 is achieved through the coordinated execution of the computing core and the data access control unit.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the hot and cold data exchange method of the key-value cache as described in any one of claims 1-10.

14. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the cold and hot data exchange method for key-value caching according to any one of claims 1-10.

Images