Data caching method and host system

Abstract

This disclosure proposes a data caching method and a host system. The host system includes a rewritable non-volatile memory module which includes multiple channels. The data caching method includes: obtaining a current token during an inference phase; reading a previous key matrix and a previous value matrix from the rewritable non-volatile memory module; generating a current key matrix according to the current token and the previous key matrix; generating a current value matrix according to the current token and the previous value matrix; writing multiple portions of the current key matrix into different channels; and writing multiple portions of the current value matrix into different channels.

Description

Data retrieval methods and host systems This disclosure relates to a data caching method for artificial intelligence algorithms, a memory storage device, and a memory control circuit unit. The rapid growth of portable electronic devices such as mobile phones and laptops in recent years has led to a surge in consumer demand for storage media. Rewritable non-volatile memory modules (e.g., flash memory) are ideally suited for integration into the aforementioned portable electronic devices due to their non-volatile data, low power consumption, small size, and lack of mechanical structure. In recent years, with the development of artificial intelligence, related applications have become increasingly numerous. Artificial intelligence involves a large amount of computation, accompanied by frequent data writing and reading. When using rewritable non-volatile memory modules to perform artificial intelligence-related calculations, how to improve read and write speeds is a topic of concern for those in this field. To address the aforementioned issues, this disclosure proposes a data caching method and a host system. This disclosure proposes a data caching method for a rewritable non-volatile memory module comprising multiple channels. Each channel includes multiple physical cells. The data caching method includes: obtaining the current symbol during the inference phase; reading the previous key matrix and the previous value matrix from the rewritable non-volatile memory module; generating the current key matrix based on the current symbol and the previous key matrix; generating the current value matrix based on the current symbol and the previous value matrix; writing a portion of the current key matrix to physical cells in one channel and storing another portion of the current key matrix to physical cells in another channel; and writing a portion of the current value matrix to physical cells in one channel and storing another portion of the current value matrix to physical cells in another channel. In one embodiment of this disclosure, the aforementioned portion of the current key matrix, another portion of the current key matrix, a portion of the current value matrix, and another portion of the current value matrix are written to the corresponding entity unit in a single-order memory cell programmed mode. In one embodiment of this disclosure, a portion of the current key matrix belongs to the first generation of symbols, and another portion of the current key matrix belongs to the second generation of symbols. The first generation of symbols is different from the second generation of symbols. In one embodiment of this disclosure, a portion of the current key matrix belongs to a first feature, and another portion of the current key matrix belongs to a second feature, wherein the first feature is different from the second feature. In one embodiment of this disclosure, the above-described data retrieval method further includes: calculating at least one query vector, at least one key vector, and at least one value vector based on the current symbol; multiplying the query vector and the current key matrix to obtain a temporary vector; and multiplying the temporary vector and the current value matrix to obtain an attention output. The step of generating the current key matrix based on the current symbol and the previous key matrix includes: combining the key vector and the previous key matrix to generate the current key matrix. The step of generating the current value matrix based on the current symbol and the previous value matrix includes: combining the value vector and the previous value matrix to generate the current value matrix. In one embodiment of this disclosure, the query vector includes a first query vector and a second query vector, the key vector includes a first key vector and a second key vector, the value vector includes a first value vector and a second value vector, the previous key matrix includes a first previous key matrix and a second previous key matrix, and the previous value matrix includes a first previous value matrix and a second previous value matrix. The step of combining the key vector and the previous key matrix to generate the current key matrix includes: combining the first key vector and the first previous key matrix to generate a first current key matrix, and combining the second key vector and the second previous key matrix to generate a second current key matrix. The step of combining the value vector and the previous value matrix to generate at least one current value matrix includes: combining the first value vector and the first previous value matrix to generate a first current value matrix, and combining the second value vector and the second previous value matrix to generate a second current value matrix. A portion of the current key matrix belongs to the first current key matrix, and another portion belongs to the second current key matrix; similarly, a portion of the current value matrix belongs to the first current value matrix, and another portion belongs to the second current value matrix. In one embodiment of this disclosure, the aforementioned current key symbol is input to the first layer of the neural network, which also includes a second layer. The data caching method further includes writing a portion of the current key matrix corresponding to the second layer and a portion of the current key matrix corresponding to the first layer into an entity unit in the same channel. In one embodiment of this disclosure, the above-described data retrieval method further includes: obtaining the next generation symbol; reading a portion of the current key matrix and another portion of the current key matrix in parallel from the channel; and reading a portion of the current value matrix and another portion of the current value matrix in parallel from the channel. In one embodiment of this disclosure, the steps of calculating the query vector, key vector, and value vector based on the current symbol include: multiplying the current symbol by a query weight matrix to obtain the query vector; multiplying the current symbol by a key weight matrix to obtain the key vector; and multiplying the current symbol by a value weight matrix to obtain the value vector. From another perspective, embodiments of the present invention provide a host system comprising a memory storage device and a processor. The memory storage device includes a rewritable non-volatile memory module comprising multiple channels, each channel including multiple physical cells. The processor is electrically connected to the memory storage device to perform multiple steps: obtaining the current symbol during an inference phase; reading a previous key matrix and a previous value matrix from the rewritable non-volatile memory module; generating a current key matrix based on the current symbol and the previous key matrix; and generating a current value matrix based on the current symbol and the previous value matrix. The memory storage device is used to write a portion of the current key matrix to physical cells in one channel, store another portion of the current key matrix to physical cells in another channel, write a portion of the current value matrix to physical cells in one channel, and store another portion of the current value matrix to physical cells in another channel. In one embodiment of this disclosure, the memory storage device described above is used to write a portion of the current key matrix, another portion of the current key matrix, a portion of the current value matrix, and another portion of the current value matrix to the corresponding physical unit in a single-level memory cell programmed mode. In one embodiment of this disclosure, the aforementioned current key symbol is input to the first layer of the neural network, which also includes a second layer. A memory storage device is used to write portions of the current key matrix corresponding to the second layer and portions of the current key matrix corresponding to the first layer to physical units in the same channel. In one embodiment of this disclosure, the host system described above is also used to obtain next-generation symbols. The memory storage device reads a portion of the current key matrix and another portion of the current key matrix in parallel from the channel, and reads a portion of the current value matrix and another portion of the current value matrix in parallel from the channel. To make the above features and advantages of the present invention more apparent and understandable, specific embodiments are described below in conjunction with the accompanying drawings. Some embodiments of the present invention will now be described in detail with reference to the accompanying drawings. Component symbols used in the following description are considered identical or similar when they appear in different drawings. These embodiments are only a part of the present invention and do not disclose all possible implementations of the invention. More precisely, these embodiments are merely examples of systems and methods within the scope of the present invention's patent application. The terms "first," "second," etc., used in this article do not specifically refer to order or sequence; they are merely used to distinguish elements or operations described using the same technical terms. Generally, a memory storage device (also known as a memory storage system) includes a rewritable non-volatile memory module and a controller (also known as a control circuit). The memory storage device can be used with a host system to enable the host system to write data to or read data from the memory storage device. Figure 1 is a schematic diagram of a host system and input / output (I / O) devices according to an exemplary embodiment of the present invention. Figure 2 is a schematic diagram of a host system, memory storage device, and I / O devices according to an exemplary embodiment of the present invention. Referring to Figures 1 and 2, the host system 11 is a computer system, which can be a desktop computer, server, distributed system, laptop computer, etc., and this invention is not limited thereto. The host system 11 may include a processor 111, random access memory (RAM) 112, read-only memory (ROM) 113, and data transfer interface 114. The processor 111, RAM 112, ROM 113, and data transfer interface 114 may be coupled to a system bus 110. The processor 111 may be a graphics processing unit (GPU), tensor processing unit (TPU), neural processing unit (NPR), central processing unit, etc. In some embodiments, the processor 111 also includes memory. In one exemplary embodiment, processor 111 may be coupled to memory storage device 10 via data transfer interface 114. For example, processor 111 may store data to or read data from memory storage device 10 via data transfer interface 114. Furthermore, host system 11 may be coupled to I / O device 12 via system bus 110. For example, host system 11 may transmit output signals to or receive input signals from I / O device 12 via system bus 110. In other embodiments, processor 111 may also be electrically connected to memory storage device 10 via a dedicated data transfer interface 114, instead of via system bus 110. In one exemplary embodiment, the processor 111, random access memory 112, read-only memory 113, and data transfer interface 114 may be disposed on the motherboard 20 of the host system 11. The number of data transfer interfaces 114 may be one or more. Through the data transfer interface 114, the motherboard 20 may be coupled to the memory storage device 10 via wired or wireless means. In one exemplary embodiment, the memory storage device 10 may be, for example, a USB flash drive 201, a memory card 202, or a solid state drive (SSD) 203. In some embodiments, the memory storage device 10 may be located externally to the host system 11 as a wireless memory storage device 204. The wireless memory storage device 204 may be, for example, a Near Field Communication (NFC) memory storage device, a WiFi wireless fax memory storage device, a Bluetooth memory storage device, or a Bluetooth Low Energy (BLE) memory storage device (e.g., iBeacon), or other memory storage devices based on various wireless communication technologies. Furthermore, the motherboard 20 may also be coupled to various I / O devices such as a Global Positioning System (GPS) module 205, a network interface card 206, a wireless transmission device 207, a keyboard 208, a screen 209, and a speaker 210 via the system bus 110. For example, in one exemplary embodiment, motherboard 20 may access wireless memory storage device 204 via wireless transmission device 207. Figure 3 is a schematic diagram of a memory storage device according to an exemplary embodiment of the present invention. Referring to Figure 3, the memory storage device 10 includes a connection interface unit 31, a memory control circuit unit 32, and a rewritable non-volatile memory module 33. The interface unit 31 is used to couple to the processor 111. The memory storage device 10 can communicate with the processor 111 via the interface unit 31. In an exemplary embodiment, the interface unit 31 is compatible with the Peripheral Component Interconnect Express (PCI Express) standard. In one exemplary embodiment, the connection interface unit 31 may also conform to the Serial Advanced Technology Attachment (SATA) standard, the Parallel Advanced Technology Attachment (PATA) standard, the Institute of Electrical and Electronics Engineers (IEEE) 1394 standard, the Universal Serial Bus (USB) standard, the SD interface standard, the Ultra High Speed-I (UHS-I) interface standard, the Ultra High Speed-II (UHS-II) interface standard, the Memory Stick (MS) interface standard, the MCP interface standard, the MMC interface standard, the eMMC interface standard, the Universal Flash Storage (UFS) interface standard, the eMCP interface standard, the CF interface standard, the Integrated Device Electronics (IDE) standard, or other suitable standards. The interface unit 31 may be packaged in the same chip as the memory control circuit unit 32, or the interface unit 31 may be disposed outside the chip containing the memory control circuit unit 32. The memory control circuit unit 32 is coupled to the connection interface unit 31 and the rewritable non-volatile memory module 33. The memory control circuit unit 32 is used to execute multiple logic gates or control instructions implemented in hardware or firmware, and to perform operations such as writing, reading and erasing data in the rewritable non-volatile memory module 33 according to the instructions of the processor 111. The rewritable non-volatile memory module 33 is used to store the data written by the processor 111. The rewritable non-volatile memory module 33 may include a single-level cell (SLC) NAND flash memory module (i.e., a flash memory module that can store 1 bit in one memory cell), a multi-level cell (MLC) NAND flash memory module (i.e., a flash memory module that can store 2 bits in one memory cell), a triple-level cell (TLC) NAND flash memory module (i.e., a flash memory module that can store 3 bits in one memory cell), a quad-level cell (QLC) NAND flash memory module (i.e., a flash memory module that can store 4 bits in one memory cell), other flash memory modules, or other memory modules with the same characteristics. Each memory cell in the rewritable non-volatile memory module 33 stores one or more bits by changing a voltage (hereinafter also referred to as the threshold voltage). Specifically, each memory cell has a charge trapping layer between its control gate and channel. By applying a write voltage to the control gate, the amount of electrons in the charge trapping layer can be changed, thereby changing the threshold voltage of the memory cell. This operation of changing the threshold voltage of the memory cell is also called "writing data to the memory cell" or "programming the memory cell". With the change of the threshold voltage, each memory cell in the rewritable non-volatile memory module 33 has multiple storage states. By applying a read voltage, it is possible to determine which storage state a memory cell belongs to, thereby retrieving the one or more bits stored in that memory cell. In one exemplary embodiment, the memory cells of the rewritable nonvolatile memory module 33 can constitute multiple physical programming units, and these physical programming units can constitute multiple physical erase units. Specifically, memory cells on the same word line can form one or more physical programming units. If each memory cell can store more than two bits, the physical programming units on the same word line can be classified into at least lower physical programming units and upper physical programming units. For example, the least significant bit (LSB) of a memory cell belongs to a lower physical programming unit, and the most significant bit (MSB) of a memory cell belongs to an upper physical programming unit. Generally, in MLC NAND flash memory, the write speed of lower physical programming units is greater than that of upper physical programming units, and / or the reliability of lower physical programming units is higher than that of upper physical programming units. In some embodiments, the processor 111 or the memory control circuit unit 32 may determine the programming mode for writing data to the rewritable non-volatile memory module 33. When using a single-level memory cell programming mode, data is written to the lower physical programming unit. When using a multi-level (including second-level, third-level, fourth-level, etc.) memory cell programming mode, data is written to at least the lower physical programming unit and the upper physical programming unit. Figure 4 is a schematic diagram illustrating a multi-channel configuration according to one embodiment. Referring to Figure 4, the rewritable non-volatile memory module 33 includes multiple channels 401-404, each channel having multiple physical units. In one exemplary embodiment, a physical unit refers to a physical address or a physical programming unit. In one exemplary embodiment, a physical unit may also consist of multiple consecutive or non-consecutive physical addresses. In one exemplary embodiment, a physical unit may also refer to a virtual block (VB). A virtual block may include multiple physical addresses or multiple physical programming units. In one exemplary embodiment, a virtual block may include one or more physical erase units. Each channel 401-404 operates independently. For example, each channel belongs to an independent chip and is controlled by a specific chip enable (CE) signal. The memory control circuit unit 32 can write data to or read data from channels 401-404 in parallel. In this embodiment, the processor 111 executes an inference phase of a neural network, during which data is generated and cached in the rewritable non-volatile memory module 33. Specifically, depending on the operation, architecture, and data characteristics of the neural network, different data can be written in parallel to different channels 401-404, thereby improving read and write speeds. The following data writing and reading processes are handled by the processor 111 transmitting write or read instructions to the memory control circuit unit 32, which then reads data from or writes data to the rewritable non-volatile memory module 33. Figure 5 is a schematic diagram illustrating the neural network in the inference phase according to an embodiment. Referring to Figure 5, the neural network 500 includes a first layer 501, a second layer 502, ..., up to the Nth layer 503, where N is a positive integer. The input to the neural network 500 is a plurality of symbols 511-513. Each symbol can be text data, voice data, image data, or other types of data, which is not limited to this invention. Each symbol is a vector generated through encoding or other processing. Here, the operation of the neural network 500 at different points in time is unfolded. Specifically, the neural network 500 outputs the next generation symbol 514, and the next generation symbol 514 is fed back as input to the neural network 500 to generate the next generation symbol 515. Similarly, the next generation symbol 515 is fed back as output to the neural network 500 to generate the next generation symbol 516. This mechanism is also called autoregressive (AT). Each of layers 501 through 503 is also called a decoder, and these decoders have the same architecture. The output of layer 501 can also be called a symbol, embedding, or feature vector. The output of layer 501 is passed to layer 502, and the output of layer 502 is passed to the next layer, until it reaches layer 503. The architecture of layer 502 is enlarged in Figure 5 as an example. The second layer 502 comprises steps 521-526. In general, step 521 involves projection to obtain the query matrix Q, key matrix K, and value matrix V. Step 522 calculates an attention score based on the query matrix Q and key matrix K. Step 523 performs normalization on these attention scores, altering their numerical range; for example, soft-max can be used, or in other embodiments, sparsemax can be used. Then, in step 524, the normalized attention scores are multiplied by the value matrix V and summed to calculate an attention output. In step 525, this attention output is projected again and a current symbol is added; this is also called a residual layer. Finally, in step 526, the output is obtained by passing it through a fully connected layer. Steps 522-524 are also referred to as self-attention calculation. Figure 6 is a schematic diagram illustrating the calculation of self-attention according to one embodiment. Referring to Figures 5 and 6, assume there are three symbols in total. , , Among them, the symbol Also known as the current generation symbol. Substitute symbol Multiply by a key weight matrix The key vector can be obtained. , put the symbol Multiply by a value weight matrix Value vectors can be obtained Similarly, use the symbol... Multiply by the key weight matrix The key vector can be obtained. , put the symbol Multiply by the value weight matrix Value vectors can be obtained Put the symbol Multiply by the key weight matrix The key vector can be obtained. , put the symbol Multiply by the value weight matrix Value vectors can be obtained In addition, the substitute symbol Multiply by the query weight matrix The query vector can be obtained. . Next, calculate the query vector. and key vector The inner product of these terms yields an attention score, denoted here as... Similarly, calculate the query vector. and key vector The inner product can be used to obtain the attention score. Calculate the query vector and key vector The inner product can be used to obtain the attention score. Next, we will analyze the attention scores. , ,as well as The normalization may be performed, for example, by a soft-max normalization, but this invention is not limited thereto. Then, the normalized attention score Multiply by value vector The normalized attention score Multiply by value vector The normalized attention score Multiply by value vector Finally, these products are summed to obtain an attention output, expressed as the following mathematical formula 1, where This is a vector called the attention output. [Mathematical Formula 1] The attention output calculated above It corresponds to the current symbol. For symbols , The corresponding attention output can also be calculated. , As shown in Figure 7, the symbol is used as a substitute. For example, symbol Multiply by the query weight matrix The query vector can be obtained later. The remaining operations are the same as in Figure 6. The difference is that the query vector... It will not be related to the key vector Multiplication, because of the substitution symbol Occurred in the symbol Previously, calculations were performed regarding the symbol. There are no symbols when the attention output is performed. The information. In other words, the query vector for each symbol is only multiplied by the key vector of the previously occurring symbol. Because for each symbol , , The corresponding attention output will be calculated. Here, the symbol can be... , , A matrix X is constructed, and the above calculations are represented by a matrix. Figure 8 is a schematic diagram illustrating matrix operations related to attention according to one embodiment. Referring to Figures 6 and 8, each row of matrix X represents a symbol. In this example, there are L symbols, and each symbol (a vector) has a length of D, where L and D are positive integers. On the other hand, the weight matrix is ​​queried. Key weight matrix and value weight matrix The size is , where d is a positive integer. Each symbol , , Both will query the weight matrix Multiplication. Once represented as a matrix, these calculations are equivalent to multiplying matrix X with the query weight matrix. Multiply them to obtain a query matrix Q. Similarly, for each symbol... , , Metropolitan area and key weight matrix Multiplication, represented as a matrix, is equivalent to multiplying matrix X and the query weight matrix. Multiply them to obtain a key matrix K. Each symbol... , , The sum weight matrix Multiplication, represented as a matrix, is equivalent to multiplying matrix X and the query weight matrix. Multiply them to obtain a value matrix V. The query matrix Q, the key matrix K, and the value matrix V all have the same size. Each column corresponds to a symbol, and each row corresponds to a feature. In other words, the query matrix Q contains L query vectors, such as the query vectors mentioned above. , , Similarly, the key matrix K contains L key vectors, such as the key vectors mentioned above. , , The value matrix V contains L value vectors, such as the value vectors mentioned above. , , . Query vector , , and key vector , , The multiplication of the query matrix Q and the key matrix K, when represented as matrices, is equivalent to multiplying the transposes of the query matrix Q and the key matrix K, as shown in matrix 800 in Figure 8, which has a size of 1. Each element in matrix 800 is an attention score. However, multiple elements in matrix 800 are " This means that this element does not need to be calculated, because the query vector of each symbol is only multiplied by the key vector of the previous symbol. Next, normalization is performed column-wise, that is, normalization (e.g., soft-max) is performed on the elements of the same column in matrix 800, thereby obtaining matrix S. Next, the attention score will be combined with the value vector. , , After multiplying, calculate the sum. Representing it as a matrix, this calculation is equivalent to multiplying matrix S and the query matrix V to obtain the output matrix. Its size is In other words, the output matrix It contains L attention outputs, such as the attention outputs mentioned above. , , They correspond to the symbol respectively. , , . When multiple symbols are input into a neural network, some computations are repeated. For example, when processing symbols... The key vector must be calculated. Sum value vector When processing The key vector must also be calculated. Sum value vector Therefore, after processing the symbols... In the future, the key vector can be... Sum value vector Write to rewritable non-volatile memory module 33, in processing symbols The key vector can be read from the rewritable nonvolatile memory module 33. Sum value vector This practice is called KV caching. Figure 9 is a schematic diagram illustrating the operation of applying a KV cache according to an embodiment. Referring to Figure 9, first, the current identifier 901 is obtained. This current identifier 901 is the last column of matrix X in Figure 8. Next, the current identifier is multiplied by the query weight matrix. This yields query vector 902. Then, the current symbol 901 is multiplied by the key weight matrix. This yields key vector 903, which is the last column of key matrix K in Figure 8. Similarly, the current symbol 901 is multiplied by the value weight matrix. This yields value vector 904, which is the last column of value matrix V in Figure 8. Then, the previous key matrix 911 is read from the rewritable nonvolatile memory module 33. The key vector 903 is combined with the previous key matrix 911 to produce the current key matrix 912. Specifically, the key vector 903 can be transposed and then appended to the last row of the previous key matrix 911 to produce the current key matrix 912. The current key matrix 912 is identical to the transpose of the key matrix K in Figure 8. Next, the query vector 902 is multiplied by the current key matrix 912 to obtain a temporary vector 920, where each element is an attention score. Normalizing all elements of the temporary vector 920 (e.g., soft-max) yields the temporary vector 921. On the other hand, the previous value matrix 931 is read from the rewritable non-volatile memory module 33. Combining the value vector 904 with the previous value matrix 931 produces the current value matrix 932. Specifically, the value vector 904 is appended to the last column of the previous value matrix 931 to produce the current value matrix 932. This current value matrix 932 is identical to the value matrix V in Figure 8. Then, the temporary vector 921 and the current value matrix 932 are multiplied to obtain the attention output. . After processing the current symbol, the current key matrix 912 and the current value matrix 932 are written to the rewritable non-volatile memory module 33. When processing the next symbol, the current key matrix 912 and the current value matrix 932 are read from the rewritable non-volatile memory module 33. Therefore, the rewritable non-volatile memory module 33 is frequently written to and read from. To improve the performance of the KV cache, multiple channels in the rewritable non-volatile memory module 33 can be used for parallel processing. Specifically, a portion of the current key matrix 912 can be written to the physical cells in one channel, while another portion can be written to the physical cells in another channel. In this way, multiple parts of the current key matrix 912 can be written to multiple channels in parallel, thereby increasing the write speed. Similarly, a portion of the current value matrix 932 can be written to one channel, and another portion to another channel, which also increases the write speed. In addition, when processing the next generation symbol, multiple parts of the current key matrix 912 can be read in parallel from multiple channels, and multiple parts of the current value matrix 932 can be read in parallel from multiple channels, which can increase the reading speed. The current key matrix 912 has two dimensions: a symbol dimension (corresponding to L symbols) and a feature dimension (corresponding to d features). In some embodiments, the portion of the current key matrix 912 belonging to the first symbol can be written to one channel, and the portion belonging to another second symbol can be written to another channel, where the first symbol is different from the second symbol. In some embodiments, different features can also be written to different channels, for example, the portion of the current key matrix 912 belonging to the first feature can be written to one channel, and the portion belonging to another second feature can be written to another channel, where the first feature is different from the second feature. Similarly, the current value matrix 932 also has two dimensions: a symbol dimension and a feature dimension. In some embodiments, the portion of the current value matrix 932 belonging to the first symbol can be written to one channel, and the portion belonging to another second symbol can be written to another channel. In some embodiments, the portion of the current value matrix 932 belonging to the first feature can be written to one channel, and the portion belonging to another second feature can be written to another channel. In some embodiments, multiple portions of the current key matrix 912 and multiple portions of the current value matrix 932 are written to physical cells in the rewritable non-volatile memory module 33 in a single-level memory cell programmed mode. In other words, each memory cell in the written physical cell stores only one bit. Because the single-level memory cell programmed mode has a faster write and read speed and can be erased more times (compared to the multi-level memory cell programmed mode), the single-level memory cell programmed mode is more suitable for the frequent write / read operations of the KV cache described above. Referring to Figure 5, the above operation pertains to the key-value (KV) caching of a specific layer in the neural network 500. When processing another layer in the neural network 500, since different layers must be processed sequentially (not in parallel), data from different layers can be written to the same channel. Figure 10 is a schematic diagram illustrating caching of multiple layers according to an embodiment. Referring to Figure 10, using current key matrix 912-1 and current key matrix 912-2 as examples, current key matrix 912-1 belongs to the first layer, and current key matrix 912-2 belongs to the second layer. Here, multiple portions corresponding to different symbols in the current key matrix are written to different channels. For example, the portion of the current key matrix 912-1 belonging to the first symbol is written to channel 401, the portion belonging to the second symbol is written to channel 402, the portion belonging to the third symbol is written to channel 403, and the portion belonging to the fourth symbol is written to channel 404. On the other hand, the portion of the current key matrix 912-2 belonging to the first generation symbol is written to channel 401, the portion belonging to the second generation symbol is written to channel 402, the portion belonging to the third generation symbol is written to channel 403, and the portion belonging to the fourth generation symbol is written to channel 404. When the number of channels is greater than or equal to the number of symbols, the portion of the current key matrix 912-2 belonging to the fifth generation symbol can be written to the fifth channel. When the number of channels is less than the number of symbols, the portion of the current key matrix 912-2 belonging to the fifth generation symbol can be written to the first channel. The current key matrix in Figure 9 can be replaced with the current value matrix; the rest of the operation is the same and will not be described further here. The attention calculation described above is for single-head systems, but in other embodiments, the KV cache can also be applied to multi-head systems. When using a multi-head system, each symbol can generate multiple query vectors, multiple key vectors, and multiple value vectors, which are then divided into multiple heads. Query vectors, key vectors, and value vectors belonging to the same head are calculated as shown in Figures 6 and 7. Figure 11 is a schematic diagram illustrating a multi-head configuration according to one embodiment. Referring to Figure 11, for simplicity, only two heads are used as an example here. (The symbols are not included in the provided text.) After projection, a key vector can be generated. and value vector Key vector The key vector can be obtained through further projection (multiplying by the two weight matrices respectively). and key vector Similarly, value vectors The value vector can be obtained through further projection. Sum value vector The key vector Sum value vector It belongs to the first head, while the key vector Sum value vector It belongs to the second head. For symbol In generating query vectors Key vector Value vector Later, query the vector Further projection can generate a query vector. , key vector Further projection can generate key vectors. and key vector value vector Further projection can produce a value vector. Sum value vector The query vector is... Key vector and value vectors Belongs to the first head, query vector Key vector and value vector It belongs to the second head. Query vector Meeting key vectors Multiply to obtain the attention score, which will be summed with the value vector. Multiply. Query vectors. It will also be related to the key vector. Multiply to obtain the attention score, which will be summed with the value vector. Multiply them. The two products are added together to obtain a vector, which serves as the first input for the projection in step 1110. Query vector Meeting key vectors Multiply to obtain the attention score, which will be summed with the value vector. Multiply. Query vectors. It will also be related to the key vector. Multiply to obtain the attention score, which will be summed with the value vector. Multiply them. The two products are added together to obtain a vector, which serves as the second input for the projection in step 1110. In step 1110, the two input vectors are concatenated and multiplied by a matrix to obtain the attention output. For each head, the KV cache is handled in the same way as for a single head. Specifically, the key vector... and key vector This forms a first-key matrix and is stored in a rewritable, non-volatile memory module 33. When processing the next symbol, this first-key matrix is ​​called the first prior-key matrix. On the other hand, the key vector... and key vector A second key matrix is ​​formed and stored in the rewritable non-volatile memory module 33. This second key matrix is ​​called the second previous key matrix when processing the next symbol. In some embodiments, key matrices belonging to different headers can be written in parallel to different channels, so previous key matrices can be read in parallel from different channels. Subsequent calculations refer to FIG9. For the next symbol, the first previous key matrix is ​​combined with the first key vector belonging to the first header to generate the first current key matrix. In addition, the second previous key matrix is ​​combined with the second key vector belonging to the second header to generate the second current key matrix. After processing the next symbol, the first current key matrix and the second current key matrix can be written to different channels. Similarly, value vectors Sum value vector This will form a first-value matrix and be stored in a rewritable non-volatile memory module 33. When processing the next symbol, this first-value matrix is ​​called the first prior-value matrix. On the other hand, the value vector... Sum value vector This forms a second value matrix, which is stored in the rewritable non-volatile memory module 33. This second value matrix is ​​called the second prior value matrix when processing the next symbol. In some embodiments, value matrices belonging to different headers can be written in parallel to different channels, thus allowing prior value matrices to be read in parallel from different channels. Subsequent calculations refer to Figure 9. For the next symbol, the first prior value matrix is ​​combined with the first value vector belonging to the first header to produce the first current value matrix. Furthermore, the second prior value matrix is ​​combined with the second value vector belonging to the second header to produce the second current value matrix. After processing the next symbol, the first and second current value matrices can be written to different channels. Based on the above embodiments, after generating at least one current key matrix, a portion of the current key matrix can be written to one channel, and another portion can be written to another channel. These portions can be different headers, symbols, or features. Similarly, after generating at least one current value matrix, a portion of the current value matrix can be written to one channel, and another portion can be written to another channel. These portions can be different headers, symbols, or features. When processing different layers of the neural network, symbols, features, or headers from different layers can be written to the same channel. In some embodiments, the processor 111 may transmit arrangement information of the current key matrix and the current value matrix to the memory control circuit unit 32. This arrangement information can be used to calculate which header, symbol, and feature the data in each logical address belongs to. For example, the arrangement information is used to indicate that the first header is transmitted first, followed by the second header, and within the same header, the first symbol is transmitted first, followed by the second symbol. In this way, the memory control circuit unit 32 will first receive the data belonging to the first header, the first symbol, and the first feature, then receive the data belonging to the first header, the first symbol, and the second feature, and so on. The memory control circuit unit 32 can write different parts of the current key matrix and the current value matrix to different channels according to this arrangement information. Figure 12 is a flowchart illustrating a data retrieval method according to an embodiment. Referring to Figure 12, in step 1201, the current identifier is obtained during the inference phase. In step 1202, the previous key matrix and the previous value matrix are read from the rewritable non-volatile memory module. In step 1203, the current key matrix is ​​generated based on the current identifier and the previous key matrix. For example, the query vector, key vector, and value vector are first calculated based on the current identifier, and then the key vector and the previous key matrix are combined to generate the current key matrix. In step 1204, the current value matrix is ​​generated based on the current identifier and the previous value matrix. For example, the value vector and the previous value matrix are combined to generate the current value matrix. In step 1205, the query vector and the current key matrix are multiplied to obtain a temporary vector. In step 1206, the temporary vector and the current value matrix are multiplied to obtain an attention output. In step 1207, a portion of the current key matrix is ​​written to a physical cell in one channel, and another portion of the current key matrix is ​​stored in a physical cell in another channel. In step 1208, a portion of the current value matrix is ​​written into an entity cell in one channel, and another portion of the current value matrix is ​​stored in an entity cell in another channel. The steps in Figure 12 have been described in detail above and will not be repeated here. It is worth noting that each step in Figure 12 can be implemented as multiple pieces of code or circuits, and this invention is not limited thereto. Furthermore, the method in Figure 12 can be used in conjunction with the above embodiments or used independently; in other words, other steps can be added between the steps in Figure 12. In some embodiments, steps 1205 and 1206 can also be omitted. This invention does not limit the order of the steps in Figure 12. For example, steps 1203 and 1204 can be executed simultaneously, or steps 1205 and 1206 can be executed after steps 1207 and 1208. When performing KV caching, data needs to be written and read frequently. The techniques disclosed above can improve the speed of writing and reading. Although the present invention has been disclosed above by way of embodiments, it is not intended to limit the present invention. Anyone skilled in the art can make some modifications and refinements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims. 10: Memory storage device; 11: Host system; 12: I / O device; 20: Motherboard; 31: Connection interface unit; 32: Memory control circuit unit; 33: Rewritable non-volatile memory module; 110: System bus; 111: Processor; 112: Random access memory; 113: Read-only memory; 114: Data transfer interface; 201: USB flash drive; 202: Memory card; 203: Solid state drive; 204: Wireless memory storage device; 205: Global positioning system module; 206: Network interface card; 207: Wireless transmission device; 208: Keyboard; 209: Screen; 210: Speaker; 401~404: Channel; 500: Neural network; 501: First layer; 502: Second layer; 503: Nth layer; 511~513: Symbol; 514~516: Next generation symbol; 521~526, 1110, 1201~1208: Step 800: Matrix 901: Current symbol 902, , , , , :Query vector 903, , , , , , , :Key vector 904, , , , , , , Value vector 911: Previous key matrix 912, 912-1, 912-2: Current key matrix 920, 921: Temporary vector 931: Previous value matrix 932: Current value matrix X, S: Matrix : Query the weight matrix :Key weight matrix Value weight matrix Q: Query matrix K: Key matrix V: Value matrix Output matrices L, D, d: positive integers Figure 1 is a schematic diagram of a host system and input / output (I / O) devices according to an exemplary embodiment of the present invention. Figure 2 is a schematic diagram of a host system, memory storage device, and I / O devices according to an exemplary embodiment of the present invention. Figure 3 is a schematic diagram of a memory storage device according to an exemplary embodiment of the present invention. Figure 4 is a schematic diagram of a multi-channel according to an embodiment. Figure 5 is a schematic diagram of a neural network in the inference phase according to an embodiment. Figure 6 is a schematic diagram of self-attention calculation according to an embodiment. Figure 7 is a schematic diagram of self-attention calculation according to an embodiment. Figure 8 is a schematic diagram of matrix operations related to attention according to an embodiment. Figure 9 is a schematic diagram of operation when applying KV caching according to an embodiment. Figure 10 is a schematic diagram of caching multiple layers according to an embodiment. Figure 11 is a schematic diagram of a multi-head system according to an embodiment. Figure 12 is a flowchart of a data caching method according to an embodiment. Steps 1201~1208

Claims

1. A data caching method for a rewritable non-volatile memory module, wherein the rewritable non-volatile memory module includes multiple channels, each of which includes multiple physical units, and the data caching method includes: In an inference phase, a current identifier is obtained; at least one previous key matrix and at least one previous value matrix are read from the rewritable nonvolatile memory module; at least one current key matrix is ​​generated based on the current identifier and the at least one previous key matrix; at least one current value matrix is ​​generated based on the current identifier and the at least one previous value matrix; a portion of the at least one current key matrix is ​​written to the physical cells in one of the channels, and another portion of the at least one current key matrix is ​​stored in parallel to the physical cells in the other of the channels; and a portion of the at least one current value matrix is ​​written to the physical cells in one of the channels, and another portion of the at least one current value matrix is ​​stored in parallel to the physical cells in the other of the channels. The data caching method as described in claim 1, wherein the portion of the at least one current key matrix, the other portion of the at least one current key matrix, the portion of the at least one current value matrix, and the other portion of the at least one current value matrix are written to the corresponding entity units in a single-level memory cell programmed mode. The data caching method as described in claim 1, wherein the portion of the at least one current key matrix belongs to a first-generation symbol, and the other portion of the at least one current key matrix belongs to a second-generation symbol, wherein the first-generation symbol is different from the second-generation symbol. The data caching method as described in claim 1, wherein the portion of the at least one current key matrix belongs to a first feature, and the other portion of the at least one current key matrix belongs to a second feature, wherein the first feature is different from the second feature. The data caching method described in request item 1 further includes: Calculate at least one query vector, at least one key vector, and at least one value vector based on the current symbol; The step of multiplying the at least one query vector and the at least one current key matrix to obtain at least one temporary vector; and multiplying the at least one temporary vector and the at least one current value matrix to obtain at least one attention output, wherein the step of generating at least one current key matrix based on the current symbol and the at least one previous key matrix includes: combining the at least one key vector and the at least one previous key matrix to generate the at least one current key matrix, wherein the step of generating at least one current value matrix based on the current symbol and the at least one previous value matrix includes: combining the at least one value vector and the at least one previous value matrix to generate the at least one current value matrix. The data retrieval method as described in claim 5, wherein the at least one query vector includes a first query vector and a second query vector, the at least one key vector includes a first key vector and a second key vector, the at least one value vector includes a first value vector and a second value vector, the at least one previous key matrix includes a first previous key matrix and a second previous key matrix, and the at least one previous value matrix includes a first previous value matrix and a second previous value matrix, wherein the step of combining the at least one key vector and the at least one previous key matrix to generate the at least one current key matrix includes: The step of combining the first key vector and the first previous key matrix to generate a first current key matrix, and combining the second key vector and the second previous key matrix to generate a second current key matrix, wherein combining the at least one value vector and the at least one previous value matrix to generate at least one current value matrix includes: combining the first value vector and the first previous value matrix to generate a first current value matrix, and combining the second value vector and the second previous value matrix to generate a second current value matrix, wherein a portion of the at least one current key matrix belongs to the first current key matrix, and another portion of the at least one current key matrix belongs to the second current key matrix, wherein a portion of the at least one current value matrix belongs to the first current value matrix, and another portion of the at least one current value matrix belongs to the second current value matrix. The data retrieval method as described in claim 5, wherein the step of calculating the at least one query vector, the at least one key vector, and the at least one value vector based on the current identifier includes: Multiply the current symbol by a query weight matrix to obtain the at least one query vector; multiply the current symbol by a key weight matrix to obtain the at least one key vector; and multiply the current symbol by a value weight matrix to obtain the at least one value vector. The data caching method as described in claim 1, wherein the current identifier is input to a first layer of a neural network, the neural network further comprising a second layer, the data caching method further comprising: A portion of the current key matrix corresponding to the second layer and a portion of the current key matrix corresponding to the first layer are written to the entity units in the same channel. The data caching method described in request item 1 further includes: Obtain the next generation symbol; read the portion of the at least one current key matrix and the other portion of the at least one current key matrix in parallel from the channels; and read the portion of the at least one current value matrix and the other portion of the at least one current value matrix in parallel from the channels. A host system includes: a memory storage device including a rewritable non-volatile memory module, wherein the rewritable non-volatile memory module includes a plurality of channels, each of the channels including a plurality of physical cells; and a processor electrically connected to the memory storage device for performing a plurality of steps: obtaining a current identifier in an inference phase; reading at least one previous key matrix and at least one previous value matrix from the rewritable non-volatile memory module; generating at least one current key matrix based on the current identifier and the at least one previous key matrix; and generating at least one current value matrix based on the current identifier and the at least one previous value matrix, wherein the memory storage device is configured to write a portion of the at least one current key matrix to the physical cells in one of the channels, and store another portion of the at least one current key matrix in parallel to the physical cells in another of the channels. The memory storage device is also used to write a portion of the at least one current value matrix to the physical cells in one of the channels, and to store another portion of the at least one current value matrix to the physical cells in the other of the channels in parallel. The host system as described in claim 10, wherein the memory storage device is used to write the portion of the at least one current key matrix, the other portion of the at least one current key matrix, the portion of the at least one current value matrix, and the other portion of the at least one current value matrix to the corresponding physical units in a single-level memory cell programmed mode. The host system as described in claim 10, wherein the portion of the at least one current key matrix belongs to a first-generation symbol, and the other portion of the at least one current key matrix belongs to a second-generation symbol, the first-generation symbol being different from the second-generation symbol. The host system as described in claim 10, wherein the portion of the at least one current key matrix belongs to a first feature, and the other portion of the at least one current key matrix belongs to a second feature, the first feature being different from the second feature. The host system as described in claim 10, wherein the steps further include: Calculate at least one query vector, at least one key vector, and at least one value vector based on the current symbol; The step of multiplying the at least one query vector and the at least one current key matrix to obtain at least one temporary vector; and multiplying the at least one temporary vector and the at least one current value matrix to obtain at least one attention output, wherein the step of generating at least one current key matrix based on the current symbol and the at least one previous key matrix includes: combining the at least one key vector and the at least one previous key matrix to generate the at least one current key matrix, wherein the step of generating at least one current value matrix based on the current symbol and the at least one previous value matrix includes: combining the at least one value vector and the at least one previous value matrix to generate the at least one current value matrix. The host system as described in claim 14, wherein the at least one query vector includes a first query vector and a second query vector, the at least one key vector includes a first key vector and a second key vector, the at least one value vector includes a first value vector and a second value vector, the at least one previous key matrix includes a first previous key matrix and a second previous key matrix, and the at least one previous value matrix includes a first previous value matrix and a second previous value matrix, wherein the step of combining the at least one key vector and the at least one previous key matrix to generate the at least one current key matrix includes: The step of combining the first key vector and the first previous key matrix to generate a first current key matrix, and combining the second key vector and the second previous key matrix to generate a second current key matrix, wherein combining the at least one value vector and the at least one previous value matrix to generate at least one current value matrix includes: combining the first value vector and the first previous value matrix to generate a first current value matrix, and combining the second value vector and the second previous value matrix to generate a second current value matrix, wherein a portion of the at least one current key matrix belongs to the first current key matrix, and another portion of the at least one current key matrix belongs to the second current key matrix, wherein a portion of the at least one current value matrix belongs to the first current value matrix, and another portion of the at least one current value matrix belongs to the second current value matrix. The host system as described in claim 14, wherein the step of calculating the at least one query vector, the at least one key vector, and the at least one value vector based on the current identifier includes: multiplying the current identifier by a query weight matrix to obtain the at least one query vector; multiplying the current identifier by a key weight matrix to obtain the at least one key vector; and multiplying the current identifier by a value weight matrix to obtain the at least one value vector. The host system as described in claim 10, wherein the current key is input to a first layer of a neural network, the neural network further comprising a second layer, wherein the memory storage device is used to write a portion of a current key matrix corresponding to the second layer and a portion of the current key matrix corresponding to the first layer to the same entity units in the channels. The host system as described in claim 10, wherein the host system is further configured to obtain next-generation symbols, wherein the memory storage device reads, in parallel from the channels, a portion of the at least one current key matrix and another portion of the at least one current key matrix, and in parallel from the channels, a portion of the at least one current value matrix and another portion of the at least one current value matrix.

Images