Data caching method and host system

Abstract

The present disclosure proposes a data caching method and a host system. The method includes: inputting multiple tokens into a neural network and obtaining key-value data corresponding to multiple layers; partitioning a first storage device into a cache block and a shared block; storing first key-value data corresponding to all the layers in the cache block; storing second key-value data corresponding to a first layer in the shared block; storing second key-value data corresponding to at least a portion of layers other than the first layer in a second storage device; and, during an inference process for the first layer, reading the first key-value data corresponding to the first layer from the cache block and reading the second key-value data corresponding to the first layer from the shared block.

Description

[Technical Field]

[0001] This disclosure relates to a data caching method and host system applied to key-value caching. [Previous Technology]

[0002] With the rapid development of artificial intelligence technology, especially in the field of natural language processing, large-scale language models have been widely used in various tasks, such as machine translation and dialogue generation. These models typically employ a self-attention mechanism to improve their ability to process long texts. During inference, these models often require key-value caching to avoid redundant computation and improve inference efficiency. Key-value caching refers to temporarily storing the keys and values ​​generated for past tokens in memory for quick lookup during subsequent token generation, avoiding redundant computation. However, as the length of tokens supported by the model increases, the memory requirement for key-value caching also grows rapidly. Insufficient memory can limit the length of tokens supported by the model. [Summary of the Invention]

[0003] In order to solve the above problems, this disclosure proposes a data caching method and a host system.

[0004] This disclosure proposes a data caching method applicable to a first storage device and a second storage device. The data caching method includes: inputting a plurality of symbols into a neural network and obtaining a plurality of key-value data corresponding to a plurality of layers; dividing the first storage device into cache blocks and shared blocks; storing first key-value data corresponding to all layers in the key-value data into the respective cache blocks corresponding to these layers; storing second key-value data corresponding to the first layer in the key-value data into the shared blocks; storing second key-value data corresponding to at least a portion of other layers besides the first layer in the key-value data into the second storage device; and, when performing an inference procedure about the first layer, reading the first key-value data corresponding to the first layer from the cache blocks and reading the second key-value data corresponding to the first layer from the shared blocks.

[0005] In one embodiment of this disclosure, the access speed of the first storage device is greater than that of the second storage device, and the size of all key-value data is greater than the capacity of the first storage device.

[0006] In one embodiment of this disclosure, the above-described data caching method further includes: after executing the inference procedure of the first layer, moving the second key-value data corresponding to the new symbol from the shared block to the second storage device; reading the second key-value data corresponding to the second layer from the second storage device to the shared block; and when performing the inference procedure about the second layer, reading the first key-value data corresponding to the second layer from the cache block and reading the second key-value data corresponding to the second layer from the shared block.

[0007] In one embodiment of this disclosure, the above-described data caching method further includes: obtaining the symbol length supported by the neural network; and determining the number of cached blocks and the number of shared blocks based on the symbol length.

[0008] In one embodiment of this disclosure, the step of determining the number of cache blocks and the number of shared blocks based on the symbol length includes: obtaining the total number of blocks in the first storage device; setting the sum of the number of cache blocks and the number of shared blocks to be the same as the total number of blocks; dividing the symbol length by the block size to obtain the block length; and calculating a minimum value of the number of shared blocks under a condition that the number of shared blocks multiplied by the number of upper layers plus the number of cache blocks is greater than or equal to the block length.

[0009] In one embodiment of this disclosure, the above-mentioned data caching method further includes: as the neural network continuously generates multiple output symbols, the symbol length is gradually increased, wherein the block length is greater than the total number of blocks.

[0010] In one embodiment of this disclosure, the above-described data caching method further includes: generating multiple output symbols by a neural network, connecting the output symbols to symbols to obtain multiple historical symbols; obtaining a first prefix of the historical symbols; calculating a first hash value of the first prefix; and storing the first hash value and the key value data corresponding to the first prefix to a second storage device.

[0011] In one embodiment of this disclosure, the above-described data retrieval method further includes: obtaining a second prefix of a history symbol, wherein the second prefix covers the first prefix; calculating a second hash value of the second prefix; and storing the second hash value and the key-value data corresponding to the second prefix into a second storage device.

[0012] In one embodiment of this disclosure, the step of storing the first hash value and the key value data corresponding to the first prefix to the second storage device includes: determining whether the first hash value matches a first previous hash value of the third storage device, wherein the access speed of the third storage device is greater than the access speed of the second storage device and the access speed of the third storage device is less than the access speed of the first storage device; if the first hash value does not match the first previous hash value, determining whether the third storage device is full; if the third storage device is not full, writing the first hash value and the key value data corresponding to the first prefix to the third storage device; if the third storage device is full, determining whether the second storage device is full; if the second storage device is full, clearing the data in the second storage device; if the second storage device is not full, selecting a previous key value data in the third storage device; determining whether a hash value of the previous key value data matches a second previous hash value of the second storage device; and if the hash value of the previous key value data does not match the second previous hash value, moving the previous key value data to the second storage device.

[0013] In one embodiment of this disclosure, the data caching method further includes: when performing an inference procedure about one layer, the cache block stores the first key-value data of layer M, and the shared block stores the second key-value data of layer N. The cache block and the shared block are distributed on the same memory chip, M and N are positive integers, and M is greater than N.

[0014] In one embodiment of this disclosure, obtaining key value data includes: obtaining a prefix of a symbol; calculating a hash value of the prefix; determining whether the hash value matches a previous hash value of a second storage device; and if the hash value matches a previous hash value, reading key value data corresponding to the prefix from the second storage device.

[0015] From another perspective, embodiments of the present invention provide a host system comprising a processor and a second storage device, the processor including a first storage device. The processor is electrically connected to the second storage device and is used to execute the data caching method described above.

[0016] In order to make the above features and advantages of the present invention more apparent and understandable, specific embodiments are described below, and detailed descriptions are given in conjunction with the accompanying drawings.

Implementation Method

[0017] Some embodiments of the present invention will now be described in detail with reference to the accompanying drawings. Component symbols used in the following description are considered the same or similar components when they appear in different drawings. These embodiments are only a part of the present invention and do not disclose all possible implementations of the invention. More precisely, these embodiments are merely examples of systems and methods within the scope of the present invention's patent application.

[0018] The terms "first" and "second" used in this document do not specifically refer to order or sequence, but are used only to distinguish elements or operations described with the same technical terms.

[0019] FIG1 is a schematic diagram of a host system and input / output (I / O) device according to an exemplary embodiment of the present invention. FIG2 is a schematic diagram of a host system, storage device and I / O device according to an exemplary embodiment of the present invention.

[0020] Referring to Figures 1 and 2, the host system 11 is a computer system, which can be a desktop computer, server, distributed system, laptop computer, etc., and this invention is not limited thereto. The host system 11 may include a processor 111, a storage device 112, a read-only memory (ROM) 113, and a data transfer interface 114. The storage device 112 is, for example, random access memory (RAM).

[0021] The processor 111, storage device 112, read-only memory 113, and data transfer interface 114 may be coupled to the system bus 110. In some embodiments, the storage device 112 may also be connected to the processor 111 via a dedicated transfer interface, instead of via the bus 110. The processor 111 may be a graphics processing unit (GPU), a tensor processing unit (TPU), a neural processing unit (NPR), etc. The processor 111 has a storage device 120, which may be, for example, video random access memory (VRAM).

[0022] In one exemplary embodiment, the processor 111 may be coupled to the storage device 10 via a data transfer interface 114. For example, the processor 111 may store data to or read data from the storage device 10 via the data transfer interface 114. Furthermore, the host system 11 may be coupled to the I / O device 12 via a system bus 110. For example, the host system 11 may transmit output signals to or receive input signals from the I / O device 12 via the system bus 110. In other embodiments, the processor 111 may also be electrically connected to the storage device 10 via a dedicated data transfer interface 114, instead of via the system bus 110.

[0023] In one exemplary embodiment, the processor 111, storage device 112, read-only memory 113, and data transfer interface 114 may be disposed on the motherboard 20 of the host system 11. The number of data transfer interfaces 114 may be one or more. Through the data transfer interface 114, the motherboard 20 may be coupled to the storage device 10 via wired or wireless means.

[0024] In one exemplary embodiment, the storage device 10 may be, for example, a USB flash drive 201, a memory card 202, or a solid state drive (SSD) 203. In some embodiments, the storage device 10 may be located externally to the host system 11 as a wireless storage device 204. The wireless storage device 204 may be, for example, a Near Field Communication (NFC) storage device, a WiFi storage device, a Bluetooth storage device, or a Bluetooth Low Energy (BLE) storage device (e.g., iBeacon), or other storage devices based on various wireless communication technologies. Furthermore, the motherboard 20 may also be coupled to various I / O devices such as a Global Positioning System (GPS) module 205, a network interface card 206, a wireless transmission device 207, a keyboard 208, a screen 209, and a speaker 210 via the system bus 110. For example, in one exemplary embodiment, the motherboard 20 may access the wireless storage device 204 via the wireless transmission device 207.

[0025] FIG3 is a schematic diagram of a storage device according to an exemplary embodiment of the present invention. Referring to FIG3, the storage device 10 includes a connection interface unit 31, a memory control circuit unit 32, and a rewritable non-volatile memory module 33.

[0026] The connection interface unit 31 is used to couple to the processor 111. The storage device 10 can communicate with the processor 111 via the connection interface unit 31. In an exemplary embodiment, the connection interface unit 31 is compatible with the Peripheral Component Interconnect Express (PCI Express) standard. In one exemplary embodiment, the connection interface unit 31 may also conform to the Serial Advanced Technology Attachment (SATA) standard, the Parallel Advanced Technology Attachment (PATA) standard, the Institute of Electrical and Electronics Engineers (IEEE) 1394 standard, the Universal Serial Bus (USB) standard, the SD interface standard, the Ultra High Speed-I (UHS-I) interface standard, the Ultra High Speed-II (UHS-II) interface standard, the Memory Stick (MS) interface standard, the MCP interface standard, the MMC interface standard, the eMMC interface standard, the Universal Flash Storage (UFS) interface standard, the eMCP interface standard, the CF interface standard, the Integrated Device Electronics (IDE) standard, or other suitable standards. The interface unit 31 may be packaged in the same chip as the memory control circuit unit 32, or the interface unit 31 may be disposed outside the chip containing the memory control circuit unit 32.

[0027] The memory control circuit unit 32 is coupled to the connection interface unit 31 and the rewritable non-volatile memory module 33. The memory control circuit unit 32 is used to execute multiple logic gates or control instructions implemented in hardware or firmware, and to perform operations such as writing, reading and erasing data in the rewritable non-volatile memory module 33 according to the instructions of the processor 111.

[0028] The rewritable non-volatile memory module 33 is used to store the data written by the processor 111. The rewritable non-volatile memory module 33 may include a single-level cell (SLC) NAND flash memory module (i.e., a flash memory module that can store 1 bit in one memory cell), a multi-level cell (MLC) NAND flash memory module (i.e., a flash memory module that can store 2 bits in one memory cell), a triple-level cell (TLC) NAND flash memory module (i.e., a flash memory module that can store 3 bits in one memory cell), a quad-level cell (QLC) NAND flash memory module (i.e., a flash memory module that can store 4 bits in one memory cell), other flash memory modules, or other memory modules with the same characteristics.

[0029] Each memory cell in the rewritable non-volatile memory module 33 stores one or more bits by changing a voltage (hereinafter also referred to as the threshold voltage). Specifically, each memory cell has a charge trapping layer between its control gate and channel. By applying a write voltage to the control gate, the amount of electrons in the charge trapping layer can be changed, thereby changing the threshold voltage of the memory cell. This operation of changing the threshold voltage of the memory cell is also called "writing data to the memory cell" or "programming the memory cell". With the change of the threshold voltage, each memory cell in the rewritable non-volatile memory module 33 has multiple storage states. By applying a read voltage, it can be determined which storage state a memory cell belongs to, thereby obtaining one or more bits stored in that memory cell.

[0030] In one exemplary embodiment, the memory cells of the rewritable non-volatile memory module 33 can constitute multiple physical programming units, and these physical programming units can constitute multiple physical erase units. Specifically, memory cells on the same word line can form one or more physical programming units. If each memory cell can store more than two bits, the physical programming units on the same word line can be classified into at least lower physical programming units and upper physical programming units. For example, the least significant bit (LSB) of a memory cell belongs to a lower physical programming unit, and the most significant bit (MSB) of a memory cell belongs to an upper physical programming unit. Generally, in MLC NAND flash memory, the write speed of lower physical programming units is greater than that of upper physical programming units, and / or the reliability of lower physical programming units is higher than that of upper physical programming units.

[0031] In some embodiments, the processor 111 or the memory control circuit unit 32 may determine the programming mode for writing data to the rewritable non-volatile memory module 33. When using a single-level memory cell programming mode, data is written to the lower physical programming unit. When using a multi-level (including second-level, third-level, fourth-level, etc.) memory cell programming mode, data is written to at least the lower physical programming unit and the upper physical programming unit.

[0032] Figure 4 is a schematic diagram illustrating a memory hierarchy according to one embodiment. Referring to Figure 4, in this embodiment, the access speeds of storage devices 120, 112, and 10 are different. The access speed of storage device 120 is greater than that of storage device 112, and the access speed of storage device 112 is greater than that of storage device 10. In some embodiments, the capacity of storage device 120 is less than that of storage device 112, and the capacity of storage device 112 is less than that of storage device 10, but this is not a limitation of the present invention. A data caching method is proposed here, in which storage device 120 is used to store data required by a neural network during the inference phase. However, since the capacity of storage device 120 may be insufficient, some data may be stored in storage device 112 or storage device 10. In this embodiment, three storage devices 120, 112, and 10 are used, but more or fewer storage devices may be used in other embodiments. For example, in some embodiments, storage device 112 may be omitted, and data that cannot be stored in storage device 120 may be stored in storage device 10.

[0033] Figure 5 is a schematic diagram illustrating the neural network in the inference phase according to an embodiment. Referring to Figure 5, the neural network 500 includes a first layer 501, a second layer 502, ..., up to the Nth layer 503, where N is a positive integer. The input of the neural network 500 consists of multiple symbols 511-513. Each symbol can be text data, voice data, image data, or other types of data, which is not limited to this invention. Each symbol is a vector generated through encoding or other processing. Here, the operation of the neural network 500 at different points in time is unfolded. Specifically, the neural network 500 outputs the next generation symbol 514, and the next generation symbol 514 is fed back as input to the neural network 500 to generate the next generation symbol 515. Similarly, the next generation symbol 515 is fed back as output to the neural network 500 to generate the next generation symbol 516. This mechanism is also called autoregressive (AT).

[0034] Each of layers 501 through 503 is also called a decoder, and these decoders have the same architecture. The output of layer 501 can also be called a symbol, embedding, or feature vector. The output of layer 501 is passed to layer 502, and the output of layer 502 is passed to the next layer, until it is passed to layer 503. The architecture of layer 502 is enlarged in Figure 5 as an example.

[0035] The operation of the second layer 502 includes steps 521-526. Generally speaking, in step 521, projection is performed to obtain the query matrix Q, the key matrix K, and the value matrix V. In step 522, the query matrix Q and the key matrix K are multiplied to calculate multiple attention scores. In step 523, these attention scores are transformed, for example, by soft-max, or in other embodiments, by sparsemax. Then, in step 524, the transformed attention scores are multiplied by the value matrix V and summed to calculate an attention output. In step 525, this attention output can be projected again and the input symbol added; this is also called a residual layer. Finally, in step 526, the output of this layer is obtained by passing it through a fully connected layer. Steps 522-524 are also referred to as the calculation of self-attention.

[0036] Here, the key matrix K and the value matrix V are collectively referred to as key-value data. The inference stage can be roughly divided into two stages: prefilling and decoding. During prefilling, the key-value data of all symbols 511~513 are calculated and stored in the storage device. During decoding, the key-value data is read from the storage device to avoid duplicate calculations.

[0037] Figure 6 is a schematic diagram of dividing a storage device into multiple blocks according to an embodiment. Referring to Figure 6, the memory space of the storage device 120 is divided into multiple blocks B1 to B10, each block being a vertical memory space. It is also assumed that the neural network 500 has 6 layers L1 to L6, corresponding to the horizontal memory space. When multiple symbols are input into the neural network 500, the key-value data corresponding to these layers and symbols is first calculated. The size of the key-value data corresponding to each symbol is 2*6*D, where 2 represents the key and value, 6 represents the number of layers L1 to L6, and D represents the vector length. Each empty space in Figure 6 can store key-value data corresponding to one or more symbols. For simplicity, it is assumed that each empty space can store key-value data corresponding to one symbol. Therefore, the entire storage device 120 can store 10 symbols and key-value data corresponding to 6 layers. For example, in Figure 6, the first space in block B1 can store the key-value data of the first symbol in the first level L1, the second space in block B1 can store the key-value data of the first symbol in the second level L2, the first space in block B2 can store the key-value data of the second symbol in the first level L1, and so on.

[0038] When the number of generation symbols is less than or equal to 10, the storage device 120 can store the key-value data of all symbols. However, when the number of generation symbols is greater than 10, these key-value data cannot be stored in the storage device 120 simultaneously. Therefore, in this embodiment, these blocks B1~B10 are divided into multiple cache blocks 610 and at least one shared block 620. In this example, blocks B1~B8 belong to cache blocks 610, while blocks B9 and B10 belong to shared blocks 620. In addition, all key-value data are also divided into two parts, namely first key-value data and second key-value data. For example, the key-value data corresponding to the 1st to 8th symbols belongs to the first key-value data, while the key-value data corresponding to the 9th symbol and thereafter belongs to the second key-value data. The first key-value data corresponding to all layers L1~L6 are stored in the respective cache blocks 610 of these layers. For example, the first key-value data of the first layer L1 is stored in blocks B1 to B8 of the first layer L1; the first key-value data of the second layer L2 is stored in blocks B1 to B8 of the second layer L2, and so on. In particular, only the second key-value data of a certain layer is stored in the shared block 620, thereby storing all the key-value data of this layer in the storage device 120. For example, in Figure 6, the second key-value data KV9 to KV17 of the first layer L1 are stored in the shared block 620, where the second key-value data KV9 to KV17 correspond to the 9th to 17th symbols, respectively. On the other hand, the second key-value data of at least some of the other layers L2 to L6 besides the first layer L1 are stored in another storage device (e.g., storage device 112 or storage device 10). For example, the second key value data of layers L2 to L6 can be stored in storage device 112; or a portion of the second key value data of layers L2 to L6 can be stored in storage device 112, and another portion of the second key value data of layers L2 to L6 can be stored in storage device 10; or when there is still free space in shared block 620, a portion of the second key value data of layers L2 to L6 can be stored in shared block 620, and the remaining second key value data can be stored in storage device 112.

[0039] Figure 7 is a schematic diagram illustrating the data during the inference process of the first layer according to an embodiment. Referring to Figures 6 and 7, when the inference process of the first layer L1 is performed, the first key-value data KV1~KV8 corresponding to the first layer L1 can be read from cache block 610, and the second key-value data KV9~KV17 corresponding to the first layer L1 can be read from shared block 620. Here, the second key-value data KV9~KV17 is shown in the position of the first layer L1, but the actual storage location is shown in Figure 6. A new symbol (i.e., the 18th symbol) is also generated during the inference process. For example, when the next symbol 514 in Figure 5 is input to the neural network, the corresponding key-value data KV18 is also generated. This key-value data KV18 is also stored in shared block 620. After the inference process of the first layer L1 is completed, the second key-value data KV9~KV18 in shared block 620 is moved to storage device 112.

[0040] Figure 8 is a schematic diagram illustrating the data during the inference process of the second layer according to an embodiment. When the inference process of the second layer L2 is to be performed, the second key-value data KV9~KV17 corresponding to the second layer L2 is read from the storage device 112 to the shared block 620. During the inference process of the second layer L2, the first key-value data KV1~KV8 corresponding to the second layer L2 can be read from the cache block 610, and the second key-value data KV9~KV17 corresponding to the second layer L2 can be read from the shared block 620. Similarly, a new key-value data KV18 is generated in the inference process of the second layer L2. After the inference process of the second layer L2 is completed, the key-value data KV18 corresponding to the second layer L2 is written to the storage device 112.

[0041] In other words, during the inference process at a certain level, the first key-value data is read from cache block 610 and the second key-value data is read from shared block 620. After the inference process is completed, the key-value data (including the key-value data of the new symbol) in shared block 620 is written to storage device 112. Storage device 120 originally could only handle symbols with a length of 10, but through this method, the length of symbols it can handle can be increased to 20.

[0042] From another perspective, when performing a certain level of inference, cache block 610 stores the first key-value data of level M, and shared block 620 stores the second key-value data of level N. M and N are positive integers, and M is greater than N. In this example, M=6 and N=1, but in other embodiments, M=6 and N=2. The present invention does not limit the values ​​of positive integers M and N. On the other hand, cache block 610 and shared block 620 are distributed on the same memory chip.

[0043] In some embodiments, the number of shared blocks 620 can be dynamically determined. Specifically, the symbol length supported by the neural network can be obtained first, and then the number of cache blocks 610 and the number of shared blocks 620 can be determined based on this symbol length. When the symbol length is large, more shared blocks 620 are needed to store symbols of the same layer; when the symbol length is small, fewer shared blocks 620 can be designed to reduce read and write operations to the storage device 112.

[0044] Assume the number of cached blocks 610 is X, and the number of shared blocks 620 is Y. The total number of blocks in storage device 120 is represented by Z. In some embodiments, X and Y can be determined according to the following mathematical formulas. [Mathematical Formula 1][Mathematical Formula 2]

[0045] Where is the number of all layers in the neural network. is the symbol length supported by the neural network. is the block size, indicating how many symbols each block can store, in this example. is the block length, representing how many blocks the neural network can support. In Equation 1, the sum of the number of cache blocks X and the number of shared blocks Y is set to be equal to the total number of blocks Z. In Equation 2, the minimum value of Y is calculated under a condition that the value obtained by multiplying the number of shared blocks Y by the number of upper layers plus the number of cache blocks X must be greater than or equal to the block length. Taking the example in Figure 8, X=8, Y=2, and Z=10 can be substituted into Equation 1 and Equation 2. Equation 2 can be rewritten as , where Y=2 is the minimum value that meets the condition. Such a value can support a longer symbol length while avoiding too much reading and writing to the storage device 112.

[0046] In some embodiments, the symbol length can also be increased progressively. The symbol length supported by a language model may be several thousand or tens of thousands. Setting the symbol length to such a large value would generate too many shared blocks. Therefore, as the neural network continues to generate output symbols, the symbol length can be gradually increased when the shared blocks are insufficient. For example, the symbol length can be initially set to 100, then increased to 200, and gradually increased to the supported upper limit. At this point, the calculated block length will be greater than the total number of blocks Z, but because of the shared blocks, the storage device 120 can still store so many symbols.

[0047] Referring to Figure 6, in this example, key-value data KV9~KV17 are stored interleaved in blocks B9 and B10. When key-value data in shared block 620 is written to storage device 112 or storage device 10, key-value data within the same block are stored in the same file. Therefore, the first file stores key-value data KV9, KV11, KV13, KV15, and KV17, while the second file stores key-value data KV10, KV12, KV14, and KV16. Each layer corresponds to two files, thus generating a total of 2*6=12 files. However, in other embodiments, key-value data of the same layer can be stored in the same file, or each key-value data can be stored in a separate file; this invention is not limited to this. On the other hand, the key-value data KV9~KV17 can be stored sequentially in block B9 first, and then stored in block B10. This invention does not specify the arrangement of key-value data KV9~KV17.

[0048] Figure 9 is a schematic diagram illustrating the merging of key-value data of cache blocks and shared blocks according to an embodiment. Referring to Figure 9, in this example, storage device 120 has eight blocks B1 to B8, where blocks B1 to B7 are cache blocks and block B8 is a shared block. In this example, a total of six files (e.g., files 901 to 903) will be generated to store the second key-value data corresponding to the six layers L1 to L6 respectively. In particular, in this embodiment, the key-value data in the cache blocks can also be stored in storage device 112 or storage device 10. In such an example, the key-value data of the same layer (including the first key-value data and the second key-value data) belong to the same file. Since the first key-value data of the same layer is arranged horizontally in storage device 120, but the second key-value data is arranged vertically, format conversion is required. In other words, the file 901 corresponding to the first layer L1 must first be read from storage device 112 or storage device 10. The second key-value data in file 901 is then converted into a horizontally arranged sequence following the first key-value data of the first layer L1 to form file 911. File 911 is then written to storage device 112 or storage device 10. Similarly, for the second layer L2, file 902 is first read. The second key-value data in file 902 is then converted into a horizontally arranged sequence following the first key-value data of the second layer L2 to form file 912. File 912 is then written to storage device 112 or storage device 10. Therefore, a total of 6 files (including files 911-913) are generated to store all key-value data corresponding to all layers L1-L6.

[0049] After storing the first key-value data and the second key-value data in storage device 112 or storage device 10, if the same content is input subsequently, the corresponding key-value data can be directly read from storage device 112 or storage device 10 to storage device 120, thereby reducing computation time. In some embodiments, the hash value of the corresponding symbol can also be calculated, and these hash values ​​are also stored in storage device 112 or storage device 10. The corresponding key-value data can then be found by comparing the hash values.

[0050] Figure 10 is a schematic diagram illustrating a dialogue record according to an embodiment. In the prior art, only the key-value data of the current dialogue is temporarily stored, but the key-value data of all dialogue records are used when making inferences, so the inference speed of the model will become slower and slower. In this example, a long-term key-value cache is to be established. For example, user 1020 and chatbot 1030 are having a dialogue. The first sentence 1021 entered by user 1020 is "What is the longest river in the world?". This sentence 1021 will be converted into symbols 1001~1004, and the corresponding key-value data will also be calculated. When chatbot 1030 makes inferences, it will use the key-value data of sentence 1021. Chatbot 1030 will generate multiple output symbols 1011~1014. These output symbols 1011~1014 will be converted into sentence 1031, the content of which is "The longest river in the world is the Nile River". These output symbols 1011~1014 are concatenated after symbols 1001~1004, collectively referred to as multiple historical symbols. Next, multiple prefixes of these historical symbols are obtained, their hash values ​​are calculated, and finally, the hash values ​​and corresponding key-value data are stored in storage device 112. For example, symbol 1001 forms the first prefix, so the hash value 1041 of symbol 1001 and the corresponding key-value data are written to storage device 112. Symbols 1001 and 1002 form the second prefix, so the hash value 1042 of symbols 1001 and 1002 and the corresponding key-value data are written to storage device 112. Symbols 1001~1003 form the third prefix, so the hash value 1043 of symbols 1001~1003 and the corresponding key-value data are written to storage device 112. The second prefix covers the first prefix, and the third prefix covers the second prefix. From another perspective, the first prefix is ​​a subset of the second prefix, and the second prefix is ​​a subset of the third prefix. The above process is repeated until the hash values ​​of all symbols 1001~1004 and 1011~1014 are calculated. That is, in the example of Figure 10, there are 10 prefixes. The hash value of each prefix and the corresponding key value data are written to the storage device 112.

[0051] Next, user 1020 inputs sentence 1022. Inference requires key-value data for sentences 1021 and 1031. At this point, the hash value of sentences 1021 and 1031 can be calculated. This hash value hits the hash value (called the previous hash value) in storage device 112, allowing the key-value data corresponding to sentences 1021 and 1031 to be read from storage device 112. In this way, when user 1020 inputs sentence 1022, it is not necessary to repeatedly calculate the key-value data for sentences 1021 and 1031. This approach saves more and more time as the conversation progresses because the conversation record becomes longer and the amount of key-value data that needs to be calculated increases.

[0052] In this embodiment, a prefix hash value is calculated, so a sentence does not need to be exactly the same as a previous sentence to find matching key-value data. For example, FIG11 is a schematic diagram illustrating another dialogue record according to an embodiment. Referring to FIG10 and FIG11, user 1120 is different from user 1020, or they are the same but FIG10 and FIG11 belong to different dialog boxes. User 1120 inputs sentence 1121, which has a common prefix with sentence 1021, namely "What is the longest". Sentence 1121 is converted into multiple symbols 1101 to 1104. Next, multiple prefixes are obtained from these symbols 1101 to 1104, the hash values ​​of these prefixes are calculated, and it is determined whether these hash values ​​match the previous hash values ​​in storage device 112. For example, symbol 1101 forms the first prefix; symbols 1101 and 1102 form the second prefix; symbols 1101-1103 form the third prefix; and symbols 1101-1104 form the fourth prefix. In this example, the hash values ​​of symbols 1101 and 1102 (corresponding to "What is the longest") will match the previous hash value in storage device 112, so the key-value data 1150 corresponding to the second prefix can be read from storage device 112. This eliminates the need to repeatedly calculate the key-value data of symbols 1101 and 1102.

[0053] In some embodiments, the practices of Figures 10 and 11 can be applied to Retrieval Augmented Generation (RAG). On the one hand, in some applications, users may frequently enter the same questions, such as frequently asking "What should I do if my credit card is lost?" in financial customer service or asking about the same symptoms in medical customer service. On the other hand, if the chatbot 1030 searches a fixed database, the same references will be obtained for the same questions, and the key-value data of these references will be stored in the storage device 112, thus reducing computation. In other words, the aforementioned historical symbols can also include symbols of the text obtained after searching the database.

[0054] Figure 12 is a flowchart illustrating the reading of key-value data according to an embodiment. Referring to Figures 4 and 12, in this example there are three storage devices 120, 112, and 10. When reading or writing key-value data, it is sequentially determined whether the data has been stored in these three storage devices 120, 112, and 10. For a certain prefix, a hash value is first calculated. In step 1201, it is determined whether this hash value matches a previous hash value stored in storage device 120. If yes, in step 1202, the corresponding key-value data is read from storage device 120. If the result of step 1201 is no, in step 1203, it is determined whether the hash value matches a previous hash value stored in storage device 112. If the result of step 1203 is yes, step 1204 is executed, and the corresponding key-value data is read from storage device 112. If the result of step 1203 is no, step 1205 is executed, and it is determined whether the hash value matches a previous hash value stored in storage device 10. If the result of step 1205 is yes, proceed to step 1206 to read the corresponding key-value data from storage device 10. If the result of step 1205 is no, proceed to step 1207 to calculate the key-value data. After calculating the key-value data of all symbols, in step 1208, the neural network performs the decoding procedure.

[0055] After the neural network completes the decoding process, multiple historical symbols are obtained (see Figure 10), and a prefix hash value is calculated. Next, please refer to Figure 13, which is a flowchart of writing key-value data according to an embodiment. In step 1301, it is determined whether this hash value matches a previous hash value in storage device 112. If yes, it means that the corresponding key-value data has been stored in storage device 112, so this process can end. If the result of step 1301 is no, it is determined in step 1302 whether storage device 112 is full. If storage device 112 is not full, in step 1303, the hash value and the corresponding key-value data are written to storage device 112. If storage device 112 is full, it is determined in step 1304 whether storage device 10 is full. If the result of step 1304 is yes, a portion of the data in storage device 10 is cleared, and the process returns to step 1304. If storage device 10 is not full, in step 1306, previously used key-value data is selected from storage device 112, for example, the least recently used key-value data can be selected. In step 1307, it is determined whether the hash value of the previously used key-value data matches a previously used hash value in storage device 10. If the result of step 1307 is yes, it means that the corresponding key-value data has been stored in storage device 10, so the process can end. If the result of step 1307 is no, in step 1308, the previously used key-value data is moved to storage device 10, and then the process returns to step 1302.

[0056] Figure 14 is a flowchart illustrating a data retrieval method according to an embodiment. Referring to Figure 14, in step 1401, multiple symbols are input into the neural network, and multiple key-value data corresponding to multiple layers are obtained. In some embodiments, the key-value data of all layers can be directly calculated; in other embodiments, some or all of the key-value data can be obtained from the storage device by calculating the hash value of the prefix. In step 1402, the first storage device is divided into cache blocks and shared blocks. In step 1403, the first key-value data corresponding to all layers is stored in the cache blocks corresponding to each layer. In step 1404, the second key-value data corresponding to the first layer is stored in the shared block. In step 1405, the second key-value data corresponding to at least some of the other layers besides the first layer is stored in the second storage device. In step 1406, during the inference procedure for the first layer, the first key-value data corresponding to the first layer is read from the cache block, and the second key-value data corresponding to the first layer is read from the shared block.

[0057] The data caching method and host system described above can solve the problem of not being able to process more symbols due to insufficient storage capacity. In addition, the above-described method also provides long-term key-value caching, thereby reducing redundant calculations.

[0058] Although the present invention has been disclosed above by way of embodiments, it is not intended to limit the present invention. Anyone skilled in the art can make some modifications and refinements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims. [Simplified Explanation of the Diagram]

[0059] Figure 1 is a schematic diagram of a host system and input / output (I / O) device according to an exemplary embodiment of the present invention. Figure 2 is a schematic diagram of a host system, storage device, and I / O device according to an exemplary embodiment of the present invention. Figure 3 is a schematic diagram of a storage device according to an exemplary embodiment of the present invention. Figure 4 is a schematic diagram of a memory hierarchy according to an embodiment. Figure 5 is a schematic diagram of a neural network in the inference phase according to an embodiment. Figure 6 is a schematic diagram of dividing a storage device into multiple blocks according to an embodiment. Figure 7 is a schematic diagram of data during the first-level inference process according to an embodiment. Figure 8 is a schematic diagram of data during the second-level inference process according to an embodiment. Figure 9 is a schematic diagram of merging key-value data of cache blocks and shared blocks according to an embodiment. Figure 10 is a schematic diagram of a dialog record according to an embodiment. Figure 11 is a schematic diagram of another dialog record according to an embodiment. Figure 12 is a flowchart of reading key-value data according to an embodiment. Figure 13 is a flowchart of writing key-value data according to an embodiment. Figure 14 is a flowchart illustrating a data caching method according to an embodiment.

Claims

1. A data caching method applicable to a first storage device and a second storage device, the data caching method comprising: inputting a plurality of symbols into a neural network and obtaining a plurality of key-value data corresponding to a plurality of layers; dividing the first storage device into a plurality of cache blocks and at least one shared block; storing first key-value data corresponding to the layers in the key-value data into cache blocks corresponding to each layer in the cache blocks; storing second key-value data corresponding to a first layer in the key-value data into the at least one shared block; storing second key-value data corresponding to at least a portion of other layers besides the first layer in the key-value data into the second storage device; and, when performing an inference procedure about the first layer, reading the first key-value data corresponding to the first layer from the cache blocks and reading the second key-value data corresponding to the first layer from the at least one shared block.

2. The data caching method as described in claim 1, wherein the access speed of the first storage device is greater than the access speed of the second storage device, and the size of the key-value data is greater than a capacity of the first storage device.

3. The data caching method as described in claim 1 further includes: After the inference procedure of the first layer is executed, the second key value data corresponding to a new symbol is moved from the at least one shared block to the second storage device; the second key value data corresponding to a second layer in the layers is read from the second storage device to the at least one shared block; and when performing an inference procedure about the second layer, the first key value data corresponding to the second layer is read from the cache blocks, and the second key value data corresponding to the second layer is read from the at least one shared block.

4. The data caching method as described in claim 1 further includes: Obtain the generation length supported by this neural network; And determine the number of cached blocks and the number of the at least one shared block based on the length of the symbol.

5. The data caching method as described in claim 4, wherein the step of determining the number of cached blocks and the number of the at least one shared block based on the symbol length includes: Obtain a total number of blocks in the first storage device; set the sum of the number of cache blocks and the number of at least one shared block to be the same as the total number of blocks; divide the symbol length by a block size to obtain a block length; and calculate a minimum value of the number of at least one shared block under a condition, wherein the condition is that the number of at least one shared block multiplied by the number of layers plus the number of cache blocks is greater than or equal to the block length.

6. The data caching method as described in claim 5 further includes: As the neural network continues to generate multiple output symbols, the length of the symbol is gradually increased, where the length of the block is greater than the total number of blocks.

7. The data caching method as described in claim 1 further includes: The neural network generates multiple output symbols, and connects these output symbols to these symbols to obtain multiple historical symbols; a first prefix of these historical symbols is obtained; a first hash value of the first prefix is ​​calculated; and the first hash value and the key value data corresponding to the first prefix are stored in the second storage device.

8. The data caching method as described in claim 7 further includes: Obtain a second prefix of the historical symbols, wherein the second prefix covers the first prefix; calculate a second hash value of the second prefix; and store the second hash value and the key value data corresponding to the second prefix in the second storage device.

9. The data caching method as described in claim 7, wherein the step of storing the first hash value and the key-value data corresponding to the first prefix to the second storage device includes: The system determines whether the first hash value matches a first previous hash value of a third storage device, wherein the access speed of the third storage device is greater than the access speed of the second storage device, and the access speed of the third storage device is less than the access speed of the first storage device; if the first hash value does not match the first previous hash value, the system determines whether the third storage device is full; if the third storage device is not full, the system writes the first hash value and the key value data corresponding to the first prefix to the third storage device; if the third storage device is full, the system determines whether the second storage device is full; if the second storage device is full, the system clears the data in the second storage device; if the second storage device is not full, the system selects a previous key value data in the third storage device; the system determines whether a hash value of the previous key value data matches a second previous hash value of the second storage device; and if the hash value of the previous key value data does not match the second previous hash value, the system moves the previous key value data to the second storage device.

10. The data caching method as described in claim 1, wherein retrieving the key-value data corresponding to the layers includes: Obtain a prefix of these symbols; Calculate a hash value of the prefix; Determine whether the hash value matches a previous hash value stored in the second storage device; and if the hash value matches the previous hash value, read the key value data corresponding to the prefix from the second storage device.

11. The data caching method as described in claim 1 further includes: When performing an inference procedure about one of the layers, the cache blocks store first key-value data of layer M, and the shared block stores second key-value data of layer N, wherein the cache blocks and the at least one shared block are distributed on the same memory chip, M and N are positive integers, and M is greater than N.

12. A host system comprising: a processor including a first storage device; and a second storage device; wherein the processor is electrically connected to the second storage device and configured to perform a plurality of steps: inputting a plurality of symbols into a neural network and obtaining a plurality of key-value data corresponding to a plurality of layers; dividing the first storage device into a plurality of cache blocks and at least one shared block; storing first key-value data corresponding to the layers in the key-value data into cache blocks corresponding to each layer in the cache blocks; storing second key-value data corresponding to a first layer in the key-value data into the at least one shared block; storing second key-value data corresponding to at least a portion of the layers other than the first layer in the key-value data into the second storage device; and, during an inference procedure about the first layer, reading the first key-value data corresponding to the first layer from the cache blocks and reading the second key-value data corresponding to the first layer from the at least one shared block.

13. The host system as claimed in claim 12, wherein the access speed of the first storage device is greater than the access speed of the second storage device, and the size of the key-value data is greater than a capacity of the first storage device.

14. The host system as described in claim 12, wherein the steps further include: After the inference procedure of the first layer is executed, the second key value data corresponding to a new symbol is moved from the at least one shared block to the second storage device; the second key value data corresponding to a second layer in the layers is read from the second storage device to the at least one shared block; and when performing an inference procedure about the second layer, the first key value data corresponding to the second layer is read from the cache blocks, and the second key value data corresponding to the second layer is read from the at least one shared block.

15. The host system as described in claim 12, wherein the steps further include: Obtain the generation length supported by the neural network; and determine the number of cached blocks and the number of at least one shared block based on the generation length.

16. The host system of claim 15, wherein the steps of determining the number of cache blocks and the number of the at least one shared block based on the cipher length include: Obtain a total number of blocks in the first storage device; set the sum of the number of cache blocks and the number of at least one shared block to be the same as the total number of blocks; divide the symbol length by a block size to obtain a block length; and calculate a minimum value of the number of at least one shared block under a condition, wherein the condition is that the number of at least one shared block multiplied by the number of layers plus the number of cache blocks is greater than or equal to the block length.

17. The host system as described in claim 16, wherein the steps further include: As the neural network continues to generate multiple output symbols, the length of the symbol is gradually increased, where the length of the block is greater than the total number of blocks.

18. The host system as described in claim 12, wherein the steps further include: The neural network generates multiple output symbols, and connects these output symbols to these symbols to obtain multiple historical symbols; a first prefix of these historical symbols is obtained; a first hash value of the first prefix is ​​calculated; and the first hash value and the key value data corresponding to the first prefix are stored in the second storage device.

19. The host system as described in claim 18, wherein the steps further include: Obtain a second prefix of the historical symbols, wherein the second prefix covers the first prefix; calculate a second hash value of the second prefix; and store the second hash value and the key value data corresponding to the second prefix in the second storage device.

20. The host system as claimed in claim 18, further comprising a third storage device, wherein the step of storing the first hash value and the key-value data corresponding to the first prefix to the second storage device includes: The system determines whether the first hash value matches a first previous hash value of the third storage device, wherein the access speed of the third storage device is greater than the access speed of the second storage device, and the access speed of the third storage device is less than the access speed of the first storage device; if the first hash value does not match the first previous hash value, the system determines whether the third storage device is full; if the third storage device is not full, the system writes the first hash value and the key value data corresponding to the first prefix to the third storage device; if the third storage device is full, the system determines whether the second storage device is full; if the second storage device is full, the system clears the data in the second storage device; if the second storage device is not full, the system selects a previous key value data in the third storage device; the system determines whether a hash value of the previous key value data matches a second previous hash value of the second storage device; and if the hash value of the previous key value data does not match the second previous hash value, the system moves the previous key value data to the second storage device.

21. The host system as described in claim 12, wherein obtaining the key-value data corresponding to those layers includes: Obtain a prefix of the symbols; calculate a hash value of the prefix; determine whether the hash value matches a previous hash value stored in the second storage device; and if the hash value matches the previous hash value, read the key value data corresponding to the prefix from the second storage device.

22. The host system as described in claim 12, wherein the steps further include: When performing an inference procedure about one of the layers, the cache blocks store first key-value data of layer M, and the shared block stores second key-value data of layer N, wherein the cache blocks and the at least one shared block are distributed on the same memory chip, M and N are positive integers, and M is greater than N.

Images