Host system and data caching method for key-value cache

Abstract

The present application provides a data caching method and a host system for key-value caching. In one embodiment, the data caching method comprises: obtaining a cache segment, setting a first segment order of the cache segment in a plurality of segment orders; filling a redundant segment in a second segment order before the first segment order; inputting the redundant segment in the second segment order and the cache segment in the first segment order to an artificial intelligence model to calculate key-value data corresponding to the cache segment; and writing the key-value data corresponding to the cache segment to a rewritable non-volatile memory module, wherein the key-value data corresponding to the redundant segment is not written to the rewritable non-volatile memory module.

Description

Technical Field

[0001] This disclosure relates to a host system and data caching method for key-value caching, which can be applied to artificial intelligence models. Background Technology

[0002] With the development of deep learning technology, large language models (LLMs) have been widely used in natural language understanding and generation tasks. To support the reasoning requirements of long sequence inputs, most current large language models adopt a model architecture based on attention mechanisms and introduce key-value cache (KV Cache) technology during the inference process to reuse previously computed keys and values, thereby avoiding redundant calculations, reducing computational burden, and shortening inference latency.

[0003] In existing technologies, the reuse of key-value (KV) caches is typically based on the condition of exact prefix matching. Specifically, the system can directly reuse the prefix KV cache only when the prefix portion of the input sequence to be processed is exactly the same as the input sequence that has been processed and cached in the previous inference stage, thereby accelerating subsequent attention computation. This design can indeed effectively improve inference performance in scenarios such as conversational generation or fixed contexts.

[0004] However, in Retrieval-Augmented Generation (RAG) applications, the context used for model inference is typically dynamically selected and combined from multiple chunks by an external retrieval system. Because the text chunks retrieved for different queries vary greatly in their combination and order, even with similar query topics, the resulting sequences input to the language model often have different prefixes, making it difficult to meet the condition of perfect prefix matching.

[0005] Therefore, current key-value caching and reuse techniques based on prefix full matching generally have a low hit rate in RAG inference scenarios, making it difficult to effectively reuse already computed key-value pairs. As a result, the model still needs to recalculate the key-value pairs of the complete context in most inference requests, causing the inference process to often be in a state similar to a cold start, which in turn increases computational resource consumption, inference latency, and limits the overall performance of the system. Summary of the Invention

[0006] This disclosure proposes a host system including a memory storage device and a processor. The processor is electrically connected to the memory storage device and is used to perform multiple steps: acquiring cached segments; setting a first segment order for the cached segments among multiple segment orders; filling redundant segments in a second segment order preceding the first segment order; inputting the redundant segments in the second segment order and the cached segments in the first segment order into an artificial intelligence model to calculate key-value data for the corresponding cached segments; and writing the key-value data for the corresponding cached segments into a rewritable non-volatile memory module, wherein the key-value data for the corresponding redundant segments is not written to the rewritable non-volatile memory module.

[0007] In one embodiment disclosed herein, before inputting the redundant fragments located in the second fragment order and the cached fragments located in the first fragment order into the artificial intelligence model, the above steps further include: embedding the position corresponding to the first fragment order into the cached fragments.

[0008] In one embodiment disclosed herein, the above steps further include: setting the attention score corresponding to redundant segments to zero in the self-attention layer of the artificial intelligence model.

[0009] In one embodiment of this disclosure, the above steps further include: determining the number of segments and the number of segment sequences based on the key-value cache space of the rewritable non-volatile memory module, wherein the size of the key-value cache space is greater than or equal to the space required by multiplying the number of segments by the number of segment sequences.

[0010] In one embodiment disclosed herein, the above steps further include: generating redundant segments based on a random function.

[0011] In one embodiment of this disclosure, the above steps further include: obtaining a plurality of reference segments; for a first reference segment among these reference segments, obtaining key-value data of a cache segment from a rewritable non-volatile memory module according to the segment order of the first reference segment, wherein the content of the first reference segment is the same as the content of the cache segment; and executing an artificial intelligence model based on the obtained key-value data.

[0012] In one embodiment of this disclosure, the step of obtaining a reference fragment includes: querying a database based on a query to obtain the reference fragment.

[0013] An embodiment of the present invention proposes a data caching method executed by a host system. This data caching method includes: obtaining cached fragments; setting a first fragment order among multiple fragment orders; filling redundant fragments into a second fragment order preceding the first fragment order; inputting the redundant fragments in the second fragment order and the cached fragments in the first fragment order into an artificial intelligence model to calculate the key-value data of the corresponding cached fragments; and writing the key-value data of the corresponding cached fragments into a rewritable non-volatile memory module, wherein the key-value data of the corresponding redundant fragments are not written to the rewritable non-volatile memory module.

[0014] An embodiment of the present invention proposes a data caching method executed by a host system. This data caching method includes: acquiring multiple reference segments, wherein a rewritable non-volatile memory module stores key-value data of the multiple cache segments in a segment order, and the content of a first reference segment is identical to the content of a first cache segment; for the first reference segment, reading the key-value data of the first cache segment in a segment order according to the first reference segment among the multiple reference segments; and executing an artificial intelligence model based on the read key-value data.

[0015] In one embodiment of this disclosure, the step of obtaining a reference fragment includes: querying a database based on a query to obtain the reference fragment.

[0016] In one embodiment of this disclosure, the lengths of the aforementioned cached segments are the same.

[0017] The host system and data caching methods described above can improve the hit rate of key-value data and inference latency.

[0018] To make the above features and advantages of the present invention more apparent and understandable, specific embodiments are described below in conjunction with the accompanying drawings. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of a host system and input / output (I / O) devices according to an exemplary embodiment of the present invention;

[0020] Figure 2 This is a schematic diagram of a host system, a memory storage device, and an I / O device according to an exemplary embodiment of the present invention;

[0021] Figure 3 This is a schematic diagram of a memory storage device according to an exemplary embodiment of the present invention;

[0022] Figure 4 This is a schematic diagram illustrating a neural network in the inference phase according to an embodiment;

[0023] Figure 5This is a schematic diagram illustrating the calculation of self-attention according to one embodiment;

[0024] Figure 6 This is a schematic diagram illustrating the calculation of self-attention according to one embodiment;

[0025] Figure 7 This is a schematic diagram illustrating matrix operations related to attention according to one embodiment;

[0026] Figure 8 This is a schematic diagram illustrating the operation of applying a KV cache according to an embodiment;

[0027] Figure 9 This is a schematic diagram illustrating the calculation of key-value data according to one embodiment;

[0028] Figure 10 This is a schematic diagram illustrating the key-value cache during the inference phase according to one embodiment;

[0029] Figure 11 This is a flowchart illustrating a data caching method according to one embodiment;

[0030] Figure 12 This is a flowchart illustrating a data caching method according to one embodiment. Detailed Implementation

[0031] Some embodiments of the present invention will now be described in detail with reference to the accompanying drawings. Component symbols used in the following description, when appearing in different drawings, are considered to be the same or similar components. These embodiments are only a part of the present invention and do not disclose all possible implementations of the invention. More precisely, these embodiments are merely examples of systems and methods within the scope of the present invention's patent application.

[0032] The terms "first," "second," etc., used in this article do not specifically refer to order or sequence; they are merely used to distinguish elements or operations described using the same technical terms.

[0033] Figure 1 This is a schematic diagram of a host system and input / output (I / O) devices according to an exemplary embodiment of the present invention. Figure 2 This is a schematic diagram of a host system, a memory storage device, and an I / O device according to an exemplary embodiment of the present invention.

[0034] Please refer to Figure 1 and Figure 2The host system 11 is a computer system, which can be a personal computer, server, distributed system, laptop computer, etc., and this invention is not limited thereto. The host system 11 may include a processor 111, random access memory (RAM) 112, read-only memory (ROM) 113, and data transfer interface 114. The processor 111, RAM 112, ROM 113, and data transfer interface 114 may be coupled to a system bus 110. The processor 111 may be a graphics processing unit (GPU), tensor processing unit (TPU), neural processing unit (NPU), central processing unit, etc. In some embodiments, the processor 111 also includes memory.

[0035] In one exemplary embodiment, processor 111 may be coupled to memory storage device 10 via data transfer interface 114. For example, processor 111 may store data to or read data from memory storage device 10 via data transfer interface 114. Furthermore, host system 11 may be coupled to I / O device 12 via system bus 110. For example, host system 11 may transmit output signals to or receive input signals from I / O device 12 via system bus 110. In other embodiments, processor 111 may also be electrically connected to memory storage device 10 via a dedicated data transfer interface 114, instead of via system bus 110.

[0036] In one exemplary embodiment, the processor 111, random access memory 112, read-only memory 113, and data transfer interface 114 may be disposed on the motherboard 20 of the host system 11. The number of data transfer interfaces 114 may be one or more. Through the data transfer interface 114, the motherboard 20 may be coupled to the memory storage device 10 via wired or wireless means.

[0037] In one exemplary embodiment, the memory storage device 10 may be, for example, a USB flash drive 201, a memory card 202, or a solid state drive (SSD) 203. In some embodiments, the memory storage device 10 may be located externally to the host system 11 as a wireless memory storage device 204. The wireless memory storage device 204 may be, for example, a Near Field Communication (NFC) memory storage device, a WiFi wireless fax memory storage device, a Bluetooth memory storage device, or a Bluetooth Low Energy (BLE) memory storage device (e.g., iBeacon), or other memory storage devices based on various wireless communication technologies. Furthermore, the motherboard 20 may also be coupled to various I / O devices such as a Global Positioning System (GPS) module 205, a network interface card 206, a wireless transmission device 207, a keyboard 208, a screen 209, and a speaker 210 via the system bus 110. For example, in one exemplary embodiment, the motherboard 20 may access the wireless memory storage device 204 via the wireless transmission device 207.

[0038] Figure 3 This is a schematic diagram of a memory storage device according to an exemplary embodiment of the present invention. Please refer to... Figure 3 The memory storage device 10 includes a connection interface unit 31, a memory control circuit unit 32, and a rewritable non-volatile memory module 33.

[0039] The interface unit 31 is used to couple to the processor 111. The memory storage device 10 can communicate with the processor 111 via the interface unit 31. In one exemplary embodiment, the interface unit 31 is compatible with the Peripheral Component Interconnect Express (PCI Express) standard. In one exemplary embodiment, the connection interface unit 31 may also conform to the Serial Advanced Technology Attachment (SATA) standard, the Parallel Advanced Technology Attachment (PATA) standard, the Institute of Electrical and Electronics Engineers (IEEE) 1394 standard, the Universal Serial Bus (USB) standard, the SD interface standard, the Ultra High Speed-I (UHS-I) interface standard, the Ultra High Speed-II (UHS-II) interface standard, the Memory Stick (MS) interface standard, the MCP interface standard, the MMC interface standard, the eMMC interface standard, the Universal Flash Storage (UFS) interface standard, the eMCP interface standard, the CF interface standard, the Integrated Device Electronics (IDE) standard, or other suitable standards. The connection interface unit 31 can be packaged in a chip with the memory control circuit unit 32, or the connection interface unit 31 can be disposed outside a chip containing the memory control circuit unit 32.

[0040] The memory control circuit unit 32 is coupled to the connection interface unit 31 and the rewritable non-volatile memory module 33. The memory control circuit unit 32 is used to execute multiple logic gates or control instructions implemented in hardware or firmware, and to perform operations such as writing, reading and erasing data in the rewritable non-volatile memory module 33 according to the instructions of the processor 111.

[0041] The rewritable non-volatile memory module 33 is used to store data written by the processor 111. The rewritable non-volatile memory module 33 may include a single-level cell (SLC) NAND flash memory module (i.e., a flash memory module that can store 1 bit in one memory cell), a multi-level cell (MLC) NAND flash memory module (i.e., a flash memory module that can store 2 bits in one memory cell), a triple-level cell (TLC) NAND flash memory module (i.e., a flash memory module that can store 3 bits in one memory cell), a quadruple-level cell (QLC) NAND flash memory module (i.e., a flash memory module that can store 4 bits in one memory cell), other flash memory modules, or other memory modules with the same characteristics.

[0042] Each memory cell in the rewritable nonvolatile memory module 33 stores one or more bits by changing a voltage (hereinafter also referred to as the threshold voltage). Specifically, each memory cell has a charge trapping layer between its control gate and the channel. By applying a write voltage to the control gate, the amount of electrons in the charge trapping layer can be changed, thereby changing the threshold voltage of the memory cell. This operation of changing the threshold voltage of the memory cell is also called "writing data to the memory cell" or "programming the memory cell". As the threshold voltage changes, each memory cell in the rewritable nonvolatile memory module 33 has multiple storage states. By applying a read voltage, it can be determined which storage state a memory cell belongs to, thereby retrieving the one or more bits stored in that memory cell.

[0043] In one exemplary embodiment, the memory cells of the rewritable non-volatile memory module 33 can constitute multiple physical programming units, and these physical programming units can constitute multiple physical erase units. Specifically, memory cells on the same word line can form one or more physical programming units. If each memory cell can store more than two bits, then physical programming units on the same word line can be classified into at least lower physical programming units and upper physical programming units. For example, the least significant bit (LSB) of a memory cell belongs to the lower physical programming unit, and the most significant bit (MSB) of a memory cell belongs to the upper physical programming unit. Generally, in MLC NAND flash memory, the write speed of the lower physical programming unit is greater than that of the upper physical programming unit, and / or the reliability of the lower physical programming unit is higher than that of the upper physical programming unit.

[0044] In some embodiments, the processor 111 or the memory control circuit unit 32 may determine the programming mode for writing data to the rewritable non-volatile memory module 33. When using a single-level memory cell programming mode, data is written to the lower physical programming unit. When using a multi-level (including second-level, third-level, fourth-level, etc.) memory cell programming mode, data is written to at least the lower physical programming unit and the upper physical programming unit.

[0045] In this embodiment, the processor 111 runs an artificial intelligence model, which includes a neural network. Figure 4 This is a schematic diagram illustrating the inference phase of a neural network according to an embodiment. Please refer to... Figure 4 The neural network 500 comprises a first layer 501, a second layer 502, ..., up to an Nth layer 503, where N is a positive integer. The input to the neural network 500 consists of multiple symbols 511-513. Each symbol can be text data, speech data, image data, or other types of data, which is not limited to this invention. Each symbol is a vector generated through encoding or other processing. Here, the operation of the neural network 500 at different time points is described. Specifically, the neural network 500 outputs a next-generation symbol 514, which is fed back as input to the neural network 500 to generate a next-generation symbol 515. Similarly, the next-generation symbol 515 is fed back as output to the neural network 500 to generate a next-generation symbol 516. This mechanism is also known as autoregressive (AT).

[0046] Each of layers 501 through 503 is also called a decoder, and these decoders have the same architecture. The output of layer 501 can also be called a symbol, embedding, or representation. The output of layer 501 is passed to layer 502, and the output of layer 502 is passed to the next layer, until it reaches layer 503. Figure 4 The architecture of the second layer 502 is scaled up as an example.

[0047] The second layer 502 comprises steps 521-526. In general, step 521 involves projection to obtain the query matrix Q, key matrix K, and value matrix V. Step 522 calculates multiple attention scores based on the query matrix Q and key matrix K. Step 523 normalizes these attention scores, changing their numerical range; for example, soft-max can be used, or in other embodiments, sparsemax can be used. Then, in step 524, the normalized attention scores are multiplied by the value matrix V and summed to calculate an attention output. In step 525, this attention output is projected again and the current symbol is added; this is also called a residual layer. Finally, in step 526, the output is passed through a fully connected layer to obtain the output of this layer. Steps 521-524 are also called self-attention layers. Figure 4 The neural network in this example is merely an illustration; in other embodiments, the artificial intelligence model includes a self-attention layer, which can be combined with any other network architecture.

[0048] Figure 5 This is a schematic diagram illustrating the calculation of self-attention according to one embodiment. Please refer to... Figure 4 and Figure 5 Assume there are three symbols Among them, the symbol Also known as the current token. The token... Multiply by a key weight matrix The bond vector can be obtained. , put the symbol Multiply by a value weight matrix Value vectors can be obtained Similarly, use the symbol... Multiply by the key weight matrix The bond vector can be obtained. , put the symbol Multiply by the value weight matrix Value vectors can be obtained Put the symbol Multiply by the key weight matrix The bond vector can be obtained. , put the symbol Multiply by the value weight matrix Value vectors can be obtained In addition, the substitute symbol Multiply by the query weight matrix The query vector can be obtained. .

[0049] Next, calculate the query vector. Bond vector The inner product of these terms yields an attention score, denoted here as... Similarly, calculate the query vector. Bond vector The inner product can be used to obtain the attention score. Calculate the query vector Bond vector The inner product can be used to obtain the attention score. Next, we will analyze the attention scores. , ,as well as Normalization may be performed, such as soft-max, but this invention is not limited thereto.

[0050] Then, the normalized attention score Multiply by value vector Normalized attention scores Multiply by value vector Normalized attention scores Multiply by value vector Finally, these products are summed to obtain an attention output, expressed as the following mathematical formula 1, where This is a vector, called the attention output.

[0051] [Mathematical Expression 1]

[0052]

[0053] The attention output calculated above It corresponds to the current symbol. For symbols , The corresponding attention output can also be calculated. , .like Figure 6 As shown, using symbols For example, symbol Multiply by the query weight matrix The query vector can be obtained later. The rest of the operations and Figure 5 Same. In particular, querying vectors... It will not be related to the key vector Multiplication, because of the substitution symbol Occurred in the symbol Previously, calculations were performed regarding the symbol. There are no symbols when the attention is output. The information. In other words, the query vector for each symbol is only multiplied by the key vector of the previously occurring symbol.

[0054] Because for each symbol The corresponding attention output will be calculated. Here, the symbol can be... Form a matrix X, and use the matrix to represent the above calculation. Figure 7 This is a schematic diagram illustrating matrix operations related to attention according to one embodiment. Please refer to... Figure 5 and Figure 7 Each row of matrix X represents a symbol. In this example, there are L symbols, and each symbol (a vector) has a length of D, where L and D are positive integers. On the other hand, the query asks for the weight matrix... Key weight matrix and value weight matrix The size is , where d is a positive integer.

[0055] Each symbol Both will query the weight matrix Multiplication. Once represented as a matrix, these calculations are equivalent to multiplying matrix X with the query weight matrix. Multiply them to obtain a query matrix Q. Similarly, for each symbol... Metropolitan area and key weight matrix Multiplication, expressed as a matrix, is equivalent to multiplying matrix X and the key weight matrix. Multiply them to obtain a key matrix K. Each symbol... The sum weight matrix Multiplication, expressed as a matrix, is equivalent to multiplying matrix X by the value weight matrix. Multiply them to obtain a value matrix V. The query matrix Q, the key matrix K, and the value matrix V all have the same size. Each column corresponds to a symbol, and each row corresponds to a feature. In other words, the query matrix Q contains L query vectors, such as the query vectors mentioned above. Similarly, the key matrix K contains L key vectors, such as the key vectors mentioned above. The value matrix V contains L value vectors, such as the value vectors mentioned above. .

[0056] Query vector Bond vector The multiplication of the query matrix Q and the key matrix K, when represented as matrices, is equivalent to multiplying the transposes of the query matrix Q and the key matrix K, such as... Figure 7 The matrix 800 in the image has a size of 800. Each element in matrix 800 is an attention score. However, multiple elements in matrix 800 are " This means that this element does not need to be calculated, because the query vector of each symbol is only multiplied by the key vector of the previous symbol. Next, normalization is performed by column, that is, the elements in the same column of matrix 800 are normalized (e.g., soft-max), from which matrix S can be obtained.

[0057] Next, the attention score will be combined with the value vector. After multiplying, calculate the sum. Representing it as a matrix, this calculation is equivalent to multiplying matrix S and the query matrix V to obtain the output matrix. Its size is In other words, the output matrix It contains L attention outputs, such as the attention outputs mentioned above. They correspond to the symbol respectively. .

[0058] When multiple symbols are input into a neural network, some computations are repeated. For example, when processing symbols... The bond vector must be calculated. Sum value vector When processing The key vector must also be calculated. Sum value vector Therefore, after processing the symbols... In the future, the key vector can be... Sum value vector Write to the rewritable non-volatile memory module 33, during the processing of symbols The key vector can be read from the rewritable non-volatile memory module 33. Sum value vector This approach is called KV caching. In some embodiments, the key vector and value vector stored in the rewritable non-volatile memory module 33 are collectively referred to as key-value data.

[0059] Figure 8 This is a schematic diagram illustrating the operation of a KV cache application according to one embodiment. Please refer to... Figure 8 First, obtain the current symbol 901. This current symbol 901 is... Figure 7 The last column of matrix X. Next, multiply the current symbol by the query weight matrix. This yields query vector 902. Then, the current symbol 901 is multiplied by the key weight matrix. The bond vector 903 is obtained; this bond vector 903 is... Figure 7 The last column of the middle key matrix K. Similarly, multiply the current symbol 901 by the value weight matrix. The value vector 904 is obtained; this value vector 904 is... Figure 7 The last column of the median matrix V.

[0060] Then, key-value data, containing the previous key matrix 911 and the previous value matrix 931, is read from the rewritable non-volatile memory module 33. Key vector 903 and the previous key matrix 911 are combined to generate the current key matrix 912. Specifically, key vector 903 can be transposed and then appended to the last row of the previous key matrix 911 to generate the current key matrix 912. The current key matrix 912 is identical to... Figure 7 The transpose of the middle key matrix K. Next, multiply the query vector 902 and the current key matrix 912 to obtain a temporary vector 920, where each element is an attention score. Normalizing all elements of the temporary vector 920 (e.g., soft-max) yields the temporary vector 921.

[0061] On the other hand, combining the value vector 904 with the previous value matrix 931 produces the current value matrix 932. Specifically, the value vector 904 is appended to the last column of the previous value matrix 931 to produce the current value matrix 932. This current value matrix 932 is identical to... Figure 7 The value matrix V is then used. The temporary vector 921 and the current value matrix 932 are then multiplied to obtain the attention output. .

[0062] After processing the current symbol, the current key matrix 912 and the current value matrix 932 are written to the rewritable non-volatile memory module 33. When processing the next symbol, the current key matrix 912 and the current value matrix 932 are read from the rewritable non-volatile memory module 33, thereby avoiding redundant calculations.

[0063] In Retrieval-Augmented Generation (RAG) applications, the user's query along with the retrieved data is input into an artificial intelligence model, which contains at least one self-attention layer. If the key-value data of the retrieved data is pre-stored in a rewritable non-volatile memory module 33, the key-value caching technique during inference can significantly improve computational speed. Since the retrieved data has multiple possible permutations, in this embodiment, the key-value data of this data at multiple locations is stored in the rewritable non-volatile memory module 33.

[0064] Figure 9This is a schematic diagram illustrating the calculation of key-value data according to one embodiment. First, a cache fragment 941 is obtained. In this embodiment, the cache fragment is data used in the retrieval enhancement application or frequently used data; however, in other embodiments, cache fragment 941 can also be a frequently asked question or its corresponding answer in a question-answering system. Cache fragment 941 contains multiple symbols, which can be text data, voice data, image data, or other types of data, and this invention is not limited thereto. Before inputting cache fragment 941 into the artificial intelligence model, position embedding is added to cache fragment 941. Since different position embeddings result in different key-value data, this embodiment generates key-value data corresponding to multiple fragment orders. The fragment order represents the relevance of cache fragment 941 to the query; the earlier the fragment order, the more relevant cache fragment 941 is to the query. For example, the query and cache fragment 942 are respectively obtained as vectors using an existing transformer model. The distance between the two vectors can determine whether the relevance between the cache fragment and the query is high or low. For example, a distance greater than a threshold value is defined as low relevance, and the distance between the two vectors can be calculated using existing methods such as Euclidean distance.

[0065] Specifically, cached fragments 941-943 contain identical content, but cached fragment 941 is the first in sequence, cached fragment 942 is the second, and cached fragment 943 is the third. This fragment order refers to the order in which it is input into the AI ​​model (although all data is input together, there is still an order for the AI ​​model), or the arrangement order of multiple fragments before they are input into the AI ​​model. Before inputting into the AI ​​model, if a cached fragment is not in the first fragment order, redundant fragments need to be added before it. For example, cached fragment 941 is in the first fragment order, and there is no other data before cached fragment 941, so cached fragment 941 can be added with the position embedding corresponding to the first fragment order before being input into the AI ​​model to produce key-value data 961. In some embodiments, this position embedding is generated using a sine function or any other function. In some embodiments, the position embedding is trainable.

[0066] Since cache segment 942 is located in the second segment sequence, redundant segment 951 is inserted into the previous segment sequence. In this embodiment, the length of redundant segment 951 is the same as the length of cache segment 942. In some embodiments, redundant segment 951 is generated based on a random function, but in some embodiments, all values ​​of redundant segment 951 may be set to zero. Alternatively, redundant segment 951 is a segment with low correlation to cache segment 942. For example, as described above using vectors to determine correlation, the vector distance between cache segment 942 and redundant segment 951 is greater than a threshold value. Next, the position corresponding to the first segment sequence is embedded into redundant segment 951, and the position corresponding to the second segment sequence is embedded into cache segment 942. Then, the redundant segment 951 in the first segment sequence and the cache segment 942 in the second segment sequence are input into the artificial intelligence model to calculate the key-value data 962 corresponding to cache segment 942. This key-value data 962 is written into the rewritable non-volatile memory module 33. It is worth noting that the AI ​​model outputs key-value data corresponding to redundant segment 951, but this key-value data is not written to the rewritable non-volatile memory module 33. In other words, after inputting redundant segment 951 and cache segment 942 into the AI ​​model, the AI ​​model can also output complete key-value data. The complete key-value data includes the key-value data corresponding to redundant segment 951 and the key-value data 962 corresponding to cache segment 942; only the key-value data 962 of cache segment 942 is written to the rewritable non-volatile memory module 33.

[0067] Since cache segment 943 is located in the third segment sequence, redundant segments 952 and 953 are filled into the previous two segment sequences, respectively. The length of redundant segment 952 (or 953) is the same as the length of cache segment 943. Redundant segments 952 and 953 can be generated based on a random function or set to zero. Next, the positions corresponding to the first segment sequence are embedded into redundant segment 952, the positions corresponding to the second segment sequence are embedded into redundant segment 953, and the positions corresponding to the third segment sequence are embedded into cache segment 943. Then, redundant segments 952 and 953 and cache segment 943 are input into an artificial intelligence model to calculate the key-value data 963 corresponding to cache segment 943. This key-value data 963 is written to the rewritable non-volatile memory module 33. Similarly, the key-value data corresponding to redundant segments 952 and 953 are not written to the rewritable non-volatile memory module 33.

[0068] In the above embodiment, redundant segments are added before the cached segments. In other embodiments, redundant segments can also be added after the cached segments. For example, redundant segments can be added in the sequence of two segments following cached segment 941. Another example is adding redundant segments in the sequence of segments following cached segment 942.

[0069] In some embodiments, since the content of the redundant segments is randomly generated, the attention score calculated in the self-attention layer will be very small, and therefore will not significantly affect the final attention output. In some embodiments, the attention scores corresponding to the redundant segments 951-953 can also be set to zero.

[0070] Different cache segments can generate key-value data corresponding to different segment orders. For example, the content of cache segment 970 is different from the content of cache segment 941. Following the above approach, cache segment 970 will generate corresponding key-value data 971~973 in the first, second, and third segment orders, respectively. For simplicity... Figure 9 The redundant segments preceding cache segment 970 are not shown. Similarly, the content of cache segment 980 differs from that of cache segments 941 and 970. Following the above procedure, cache segment 980 generates corresponding key-value data 981~983 in the order of the first, second, and third segments, respectively. These key-value data 971~973 and 981~983 are all written to the rewritable non-volatile memory module 33.

[0071] In the rewritable non-volatile memory module 33, a key-value cache space 990 is set up to store these key-value data 961~963, 971~973, and 981~983. In some embodiments, the number of segments (i.e., a positive integer N) and the order of the segments can be determined according to the size of the key-value cache space 990. The number of segments N refers to the number of multiple cached segments 941, 970, and 980 with different content. Figure 9 In the example, the number of segments N is 3. Additionally, in... Figure 9 In the example, there are three different fragment orders, so the number of fragment orders is 3. The size of the key-value cache space 990 will be greater than or equal to the space required by multiplying the number of fragments (i.e., 3) by the number of fragment orders (i.e., 3). In some embodiments, the size of the key-value cache space 990 is predetermined, so the appropriate number of fragment orders can be calculated based on the number of fragments N, or the number of fragments N can be calculated based on the number of fragment orders.

[0072] In existing technologies, if N cache segments are to store all possible combinations of key-value data, there will be a total of N! key-value data entries. In practice, the number of segments N will be much larger than this. Figure 9In the example, the number 3 makes such calculation and storage impractical. In contrast, if the fragment order is 3, this embodiment only requires storing 3*N key-value data, significantly reducing both computational load and storage space compared to existing technologies.

[0073] In some embodiments, in addition to writing key-value data to the rewritable non-volatile memory module 33, the corresponding cache segments 941, 970, and 980 can also be written to the rewritable non-volatile memory module 33. In some embodiments, the artificial intelligence model has multiple layers (see...). Figure 4 (As explained in the description), each layer has its own self-attention layer, and the key-value data of each attention layer will be written to the rewritable non-volatile memory module 33.

[0074] Figure 10 This is a schematic diagram illustrating the key-value cache during the inference phase, based on one embodiment. Please refer to... Figure 10 In the inference phase, multiple reference fragments 1001-1003 are first obtained. These reference fragments 1001-1003 are, for example, reference data retrieved in a search-enhanced application scenario. These reference fragments 1001-1003 may contain text data, voice data, image data, or other types of data, which are not limited to this invention. In other words, the reference fragments 1001-1003 are obtained after querying one or more databases based on a query. In some embodiments, these reference fragments 1001-1003 are ordered according to their relevance to the query, with the content of reference fragment 1001 being the most relevant to the query content, followed by reference fragments 1002 and 1003. In this embodiment, the reference fragments 1001-1003 are of the same length. For each reference fragment, key-value data corresponding to the same fragment order and content is read from the rewritable non-volatile memory module 33 according to the fragment order of this reference fragment among all reference fragments 1001-1003.

[0075] Specifically, please refer to Figure 9 and Figure 10First, the contents of reference segments 1001-1003 are compared with the contents of cache segments 941, 970, and 980. In this example, the content of reference segment 1001 is the same as the content of cache segment 970, the content of reference segment 1002 is the same as the content of cache segment 941, and the content of reference segment 1003 is the same as the content of cache segment 980. For each reference segment 1001-1003, the key-value data of the corresponding cache segment in the same segment order is read from the rewritable non-volatile memory module 33. Specifically, since reference segment 1001 is the first segment in the segment order, the key-value data 971 of cache segment 970 in the first segment order is read from the rewritable non-volatile memory module 33. Furthermore, since reference segment 1002 is the second segment in the segment order, the key-value data 962 of cache segment 941 in the second segment order is read from the rewritable non-volatile memory module 33. Since reference segment 1003 is the third segment in the sequence, the key-value data 983 of cache segment 980 in the third segment sequence will be read from the rewritable non-volatile memory module 33. In other words, when selecting to read the key-value data 962, 971, and 983 of the aforementioned cache segments 941, 970, and 980, it is based on the segment order of each cache segment 941, 970, and 980, not on the cache segment in the previous segment order. That is, when reading the key-value data of the Pth cache segment, the contents of the first to (P-1)th cache segments will not affect the key-value data being read, where P is a positive integer. Taking the cache segment 941 in the second segment order as an example, regardless of whether the first segment order is cache segment 970 or cache segment 980, the key-value data 962 of cache segment 941 in the second segment order will be read.

[0076] Next, the inference phase of the artificial intelligence model is executed based on the obtained key-value data 971, 962, and 983. It is worth noting that the positions of key-value data 971, 962, and 983 correspond to the segment order of their respective reference segments. Therefore, key-value data 971 is in the first position, key-value data 962 is in the second position, and key-value data 983 is in the third position. When the artificial intelligence model performs inference, the different positions of the key-value data will lead to different results.

[0077] In existing technologies, a prefix complete match is required to read (reuse) key-value data, but this results in a very low hit rate. For example, in... Figure 10In the previous example, key-value data with the exact same prefix and the same order as reference fragments 1001-1003 must be stored in order to use key-value caching. However, in this embodiment, a perfect prefix match is not required. The key-value data of reference fragments 1001-1003 at their respective positions are pre-stored, so these key-value data 962, 971, and 983 can be reused.

[0078] Figure 11 This is a flowchart illustrating a data caching method according to one embodiment. Please refer to... Figure 11 This data caching method is used to calculate key-value data for caching. In step 1101, a cache fragment is obtained, and a first fragment order is set for this cache fragment. In step 1102, redundant fragments are filled into a second fragment order preceding the first fragment order. In step 1103, the redundant fragments in the second fragment order and the cache fragments in the first fragment order are input into an artificial intelligence model to calculate the key-value data for the corresponding cache fragments. In step 1104, the key-value data for the corresponding cache fragments is written to a rewritable non-volatile memory module 33, wherein the key-value data for the corresponding redundant fragments is not written to the rewritable non-volatile memory module 33.

[0079] Figure 12 This is a flowchart illustrating a data caching method according to one embodiment. Please refer to... Figure 12 This data caching method is used to read key-value data. In step 1201, multiple reference fragments are obtained, including a first reference fragment whose content is the same as the content of a first cache fragment. This first cache fragment can be stored in the rewritable non-volatile memory module 33 or in storage space outside the host system 11. In step 1202, for the first reference fragment, the key-value data of the first cache fragment in the same fragment order is read according to the fragment order of the first reference fragment. In step 1203, an artificial intelligence model is executed based on the read key-value data.

[0080] Figure 11 and Figure 12 Each step has been explained in detail above and will not be repeated here. It is worth noting that... Figure 11 and Figure 12 Each step can be implemented as multiple program codes or circuits, but this invention is not limited thereto. Furthermore, Figure 11 and Figure 12 The method can be used in conjunction with the above embodiments or alone; in other words, Figure 11 and Figure 12 Other steps can also be added between the various steps.

[0081] In the host system and data caching method described above, key-value data from multiple cache fragments are stored sequentially across multiple fragments. This can improve the hit rate of key-value data, effectively reuse key-value data, and reduce inference latency.

[0082] Although the present invention has been disclosed above by way of embodiments, it is not intended to limit the present invention. Any person skilled in the art may make some modifications and refinements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the scope of the appended claims.

Claims

1. A host system, characterized in that, Include: Memory storage device, including a rewritable non-volatile memory module; and A processor, electrically connected to the memory storage device, is used to perform multiple steps: Retrieve the cached fragment and set the first fragment order of the cached fragment among multiple fragment orders; Fill redundant segments into the second segment sequence that precedes the first segment sequence; The redundant segments in the second segment order and the cached segments in the first segment order are input into an artificial intelligence model to calculate the key-value data corresponding to the cached segments; and The key-value data corresponding to the cached segment is written to the rewritable non-volatile memory module, wherein the key-value data corresponding to the redundant segment is not written to the rewritable non-volatile memory module.

2. The host system according to claim 1, characterized in that, Before inputting the redundant segments in the second segment order and the cached segments in the first segment order into the artificial intelligence model, the multiple steps further include: The position corresponding to the first segment order is embedded and added to the cache segment.

3. The host system according to claim 1, characterized in that, The plurality of steps further include: In the self-attention layer of the artificial intelligence model, the attention score corresponding to the redundant segment is set to zero.

4. The host system according to claim 1, characterized in that, The plurality of steps further include: The number of segments and the number of the plurality of segments are determined based on the key-value cache space of the rewritable non-volatile memory module, wherein the size of the key-value cache space is greater than or equal to the space required by multiplying the number of segments by the number of the plurality of segments.

5. The host system according to claim 1, characterized in that, The plurality of steps further include: The redundant segments are generated based on a random function.

6. The host system according to claim 1, characterized in that, The plurality of steps further include: Obtain multiple reference segments; For a first reference segment among the plurality of reference segments, the key-value data of the cached segment is obtained from the rewritable non-volatile memory module according to the segment order of the first reference segment among the plurality of reference segments, wherein the content of the first reference segment is the same as the content of the cached segment; and The artificial intelligence model is executed based on the obtained key-value data.

7. The host system according to claim 6, characterized in that, The steps for obtaining the plurality of reference segments include: The multiple reference fragments are obtained by querying the database.

8. A data caching method, characterized in that, Executed by the host system, the data caching method includes: Retrieve the cached fragment and set the first fragment order of the cached fragment among multiple fragment orders; Fill redundant segments into the second segment sequence that precedes the first segment sequence; The redundant segments in the second segment order and the cached segments in the first segment order are input into an artificial intelligence model to calculate the key-value data corresponding to the cached segments; and The key-value data corresponding to the cached segment is written to the rewritable non-volatile memory module, wherein the key-value data corresponding to the redundant segment is not written to the rewritable non-volatile memory module.

9. The data caching method according to claim 8, characterized in that, Before inputting the redundant segments in the second segment order and the cached segments in the first segment order into the artificial intelligence model, the data caching method further includes: The position corresponding to the first segment order is embedded and added to the cache segment.

10. The data caching method according to claim 8, characterized in that, Also includes: In the self-attention layer of the artificial intelligence model, the attention score corresponding to the redundant segment is set to zero.

11. The data caching method according to claim 8, characterized in that, Also includes: The number of segments and the number of the plurality of segments are determined based on the key-value cache space of the rewritable non-volatile memory module, wherein the size of the key-value cache space is greater than or equal to the space required by multiplying the number of segments by the number of the plurality of segments.

12. The data caching method according to claim 8, characterized in that, Also includes: The redundant segments are generated based on a random function.

13. A data caching method, characterized in that, Executed by the host system, the data caching method includes: Multiple reference segments are obtained, wherein a rewritable non-volatile memory module stores key-value data of multiple cache segments in multiple segment order, and the content of the first reference segment among the multiple reference segments is the same as the content of the first cache segment among the multiple cache segments; For the first reference segment, the key-value data of the first cached segment in the segment order of the plurality of reference segments is read according to the segment order of the first reference segment; An artificial intelligence model is executed based on the read key-value data.

14. The data caching method according to claim 13, characterized in that, The steps for obtaining the plurality of reference segments include: The multiple reference fragments are obtained by querying the database.

15. The data caching method according to claim 13, characterized in that, The lengths of the multiple cached segments are the same.

Images