A data cache compression method and decompression method for large models

Abstract

A data cache compression method for large models, comprising the following steps: 1) writing data in double-byte units, in parallel with the first double-byte and user-specified double-byte or; 2) judging the number of zero values after the data itself, the first double-byte or and the user-specified double-byte or; 3) selecting the zero value exceeding the preset threshold byte mode, and encoding into ZVC, XOR or USR corresponding to the storage in the TAG state bit corresponding to the cache line; at the same time, a certain number of bit state bits are added at the tail of the compressed data to indicate the zero value position; 4) if none of the three modes has zero value greater than the preset threshold byte, encode into the RAW state without compression, and store in the TAG state bit corresponding to the cache line. And provide a data cache decompression method for large models. The present application reduces hardware overhead and improves compression efficiency.

Description

Technical Field

[0001] This invention belongs to the field of integrated circuit design and relates to a data caching compression method and decompression method for large models. Background Technology

[0002] Existing foreign technologies employ general BDI (Base Delta Immediate) residual compression and ZVC (Zero Value Compression) zero-value detection compression. These two methods have the following drawbacks: 1) They require a base value and use subtraction to calculate the residual, resulting in significant storage and hardware overhead; 2) In specific application scenarios, the compression effect is generally poor. Summary of the Invention

[0003] To overcome the shortcomings of existing technologies, this invention provides a data caching compression and decompression method for large models. It analyzes the characteristics of large model data caching data structures and summarizes the patterns that can be utilized. It also provides hardware-friendly XOR residual calculation, adopts an improved CacheLine length ZVC algorithm, fixes the position of compression checksums, reduces hardware overhead, and improves compression efficiency.

[0004] The technical solution adopted by this invention to solve its technical problem is: A data caching and compression method for large models includes the following steps: 1) Data is written in units of two bytes, either in parallel with the first two bytes or the user-specified two bytes; 2) Determine the number of zero values ​​following the data itself, the first two bytes, and the user-specified two bytes; 3) Select the byte method for zero values ​​exceeding the preset threshold and encode it into zero-crossing detection (ZVC), XOR, or user-defined (USR), and store it in the status bit of the corresponding cache tag (TAG) of the cache line; at the same time, add a set number of bit status bits to the end of the compressed data to indicate the position of the zero value; 4) If none of the three methods have a byte value greater than the preset threshold zero, then it is encoded as uncompressed raw (RAW) state and stored in the corresponding TAG state bit of the cache line.

[0005] Furthermore, in step 3), the ZVC compression process is as follows: Step 3.1.1: Before writing cache data, perform a OR operation on each byte and count the number of 1s after the OR operation, which is to count the number of zero values ​​in the byte data of the set bit length. Step 3.1.2: If the number of zero values ​​is less than the preset threshold, no compression is performed; if it is greater than the preset threshold, compression is performed. Step 3.1.3: Mark the ZVC compression code in the Type segment of the Tag. If the corresponding Tag0 is invalid, update Tag0 first. If both Tag0 and Tag1 are valid, according to the Cache update algorithm, remove the Tag0 and update the corresponding Tag0. Step 3.1.4: Calculate the positions of non-zero data and mark them in Sym[63:0]; Step 3.1.5: Write the non-zero value and Sym[63:0] into the half CL corresponding to the Tag in step 3.1.3; Step 3.1.6: Direct compression writing and tag update process completed.

[0006] Furthermore, in step 3), the XOR / USR compression process is as follows: Step 3.2.1: Before writing to the cache, perform an OR operation on each byte, or the first two bytes, or the user-specified offset two bytes, or all subsequent bytes, and count the number of 1s in the whole or the last two bytes, that is, count the number of zero values ​​in the set bit length byte data; Step 3.2.2: If the number of zero values ​​is less than the preset threshold, no compression is performed; if it is greater than the preset threshold, compression is performed. Step 3.2.3: Mark the XOR or USR compression code in the Type segment of the Tag. If the corresponding Tag0 is invalid, update Tag0 first. If both Tag0 and Tag1 are valid, according to the Cache update algorithm, remove the Tag0 and update the corresponding Tag0. Step 3.2.4: Calculate the positions of non-zero data and mark them in Sym[63:0]; Step 3.2.5: Write the non-zero value and Sym[63:0] into the half CL corresponding to the Tag in step 3; Step 3.2.6: Direct compression writing and tag update process completed.

[0007] A data cache decompression method for large models includes the following steps: 1) Cache line hit, read data and cache tag; 2) Determine the data compression type based on the standard caching compression status identifier; 3) If it is in RAW mode, output directly; 4) If it is ZVC, XOR or USR, then perform an XOR operation first to restore the compressed data; 5) Based on the last set bit zero value indicator, pad the corresponding positions in the compressed data with zeros; Restore the complete cache line of the specified number of bytes and output it.

[0008] Furthermore, in step 4), the ZVC decompression process is as follows: Step 4.1.1: After a cache hit, read the Type and Valid status bits of each Tag. Type[1:0]=RAW indicates directly compressed CL data; Step 4.1.2: Read the half-CL data corresponding to the Tag in Step 1, the lowest 64 bits are Sym[63:0]; Step 4.1.3: According to the Sym[63:0] instruction, fill the zero position corresponding to the half CL with zeros to restore it to the full CL of the set number of bytes; Step 4.1.4: Output the set number of bytes CL to complete the output of the corresponding data when the cache is hit.

[0009] Furthermore, in section 4), the XOR / USR decompression process is as follows: Step 4.2.1: After a cache hit, read the Type and Valid status bits of each tag. Type[1:0]=XOR / USR is the XOR compressed half-CL data; Step 4.2.2: Read the half-CL data corresponding to the Tag in Step 1, with the lowest preset threshold (e.g., 64 bits) set to Sym[63:0]; Step 4.2.3: Based on Type[1:0] in Step 1, perform a corresponding XOR operation on the high-order bits (e.g., 24) of the half-CL data to restore the data; Step 4.2.4: According to the Sym[63:0] instruction, pad the zero position corresponding to the half CL with the first double byte or the user-specified double byte to restore it to the full CL with the set number of bytes; Step 4.2.5: Output the set number of bytes CL to complete the output of the corresponding data when the cache is hit.

[0010] The technical concept of this invention is as follows: Statistical analysis reveals statistical patterns in the weights and intermediate data of large models. These data are divided into cache lines according to a set number of bytes. After performing an XOR operation between the first two bytes or a user-specified byte value and subsequent bytes of the set number of bytes, a large number of zero values ​​appear in the cache line. A zero-crossing check method is then used for compression, losslessly compressing the original set number of bytes (CL) to half the set number of bytes (CL), reducing data cache usage, or doubling the CL for the same cache capacity, thus improving cache hit rate. This method only adds an XOR circuit encoding circuit before traditional cache writing and identifies the compression method in the cache tag. Simultaneously, it adds a cache tag compression status judgment and corresponding decoding circuit after cache reading. Key operations such as cache hit, replacement, and maintenance are completely identical to classic caches, exhibiting excellent compatibility. Each write and read operation introduces an additional one-cycle delay.

[0011] The beneficial effects of this invention are mainly reflected in: 1) The algorithm is hardware-friendly, requiring only XOR operations and no complex addition or subtraction of bias calculations, resulting in a small hardware area overhead. This area is negligible compared to the total area of ​​L2 or L3 cache, and the newly added TAG segment accounts for 4.29% of the total area. 2) The algorithm does not modify the traditional cache architecture, but only adds relatively independent compression and decompression circuits at the cache write and read points. It is highly practical and only introduces a latency of 1 cycle for each write and read. Compared with the typical L2's 8-cycle write or read latency, it increases the latency by 12.5%, while compared with the typical L3's 16-cycle write or read latency, it only increases the latency by 6.25%. 3) The algorithm achieves the best compression efficiency of 50% when using sparse weights and the worst compression efficiency of 0% when using fully random weights, without any negative compression rate. In actual tests, the weight compression of each layer of large models such as FP16, FP8 and INT8 after conventional sparsification is between 10% and 30%. Attached Figure Description

[0012] Figure 1 This is a classic 8-way 64KB cache structure diagram.

[0013] Figure 2 It improves the tag entry format.

[0014] Figure 3 It is an improved dual-tag mapping structure that supports data compression.

[0015] Figure 4 It is in the IEEE-754 standard FP16 / BF16 / BF8 / INT16 / INT8 format.

[0016] Figure 5 It is a 64-byte block ZVC / USR compressed data and symbol storage format.

[0017] Figure 6 It is a 64-byte block XOR compressed data and symbol storage format.

[0018] Figure 7 It is a format for storing compressed data and indicator symbols.

[0019] Figure 8 It is a hardware implementation circuit for encoding.

[0020] Figure 9 It is the hardware implementation circuit for decoding.

[0021] Figure 10 It is a typical large model weight BF16 format data distribution. Detailed Implementation

[0022] The present invention will now be further described with reference to the accompanying drawings.

[0023] Reference Figures 1-10 A data caching and compression method for large models includes the following steps: 1) Data is written in units of two bytes, either in parallel with the first two bytes or the user-specified two bytes; 2) Determine the number of zero values ​​following the data itself, the first two bytes, and the user-specified two bytes; 3) Select the method of zero value exceeding the preset threshold (40 bytes) and encode it into ZVC, XOR or USR, and store it in the TAG status bit of the corresponding cache line; at the same time, add a set number of bits (64 bits) of status bit to the end of the compressed data to indicate the position of the zero value. 4) If none of the three methods have a byte value greater than the preset threshold zero, then it is encoded as an uncompressed RAW state and stored in the corresponding TAG state bit of the cache line.

[0024] A data cache decompression method for large models includes the following steps: 1) Cache line hit, read data and cache tag; 2) Determine the data compression type based on the standard caching compression status identifier; 3) If it is in RAW mode, output directly; 4) If it is ZVC, XOR or USR, then perform an XOR operation first to restore the compressed data; 5) Based on the last set bit zero value indicator, pad the corresponding positions in the compressed data with zeros; Restore the complete cache line of the specified number of bytes and output it.

[0025] Figure 1 In this architecture, a classic cache line (CL) is 64 bytes long, with 128 sets and an 8-way cache structure, resulting in a total cache capacity of 64B × 128 × 8 = 64KB. The cache tag stores only the physical address [31:13], dirty, and valid bits, totaling 21 bits. During random access to the cache, these three segments in the tag are read and compared to determine if the CL has been hit and whether it is readable and writable.

[0026] Figure 2 In the classic Cache, the two bits Type[1:0] are added to the Tag status to support four compression states. The specific encoding method is as follows: it marks the compression state and mode of CL, which is used for CL multiplexing and data decompression during reading.

[0027] Table 1 is a statistical table; Reference Figure 3This improves the Tag and CL mapping relationship of the classic cache. A new Tag1 group is added, with each CL group corresponding to both Tag0 and Tag1. When Type[1:0]=0, Tag1 is invalid, and CL stores the number of uncompressed bytes (64 bytes). When Type[1:0]!=0, the original CL is compressed to half the set number of bytes, and the remaining half byte can be used to store the compressed CL of another CL. Tag0 maintains the state of the lower-order compressed CL, and Tag1 maintains the state of the higher-order compressed CL. The cache access and Tag maintenance mechanism is exactly the same as the classic cache, effectively expanding from 8-way to 16-way. Each Tag is 23 bits, each CL is 512 bits, and the newly added Tag1 accounts for 23 / (23+512) = 4.29%.

[0028] Reference Figure 4 IEEE-754 defines the commonly used double-byte floating-point data formats FP16 and BF16 for large models, single-byte FP8 for low-precision floating-point quantization, and INT16 and INT8 for integer quantization. These data formats and widths are not sensitive in the cache; as long as the data follows a regular pattern, compression becomes possible.

[0029] Reference Figure 5 , Figure 6 and Figure 7 After sparsification of the large model, a large number of zero values ​​exist in the weight values. Therefore, when there are a large number of zero values ​​in the CL (Combined Lead Count), these zero values ​​can be compressed. CL compression is achieved by using a single bit in Symbol[63:0] to indicate the position of the zero value in the set bit-length byte. Simultaneously, analysis of a large number of large model weights and intermediate data reveals that in a sequence with a set bit-length byte as the unit, the values ​​of each set bit-length byte fluctuate around the average value. Subtracting the average value also results in a large number of zero values. Utilizing this pattern while avoiding subtraction, this scheme innovatively uses either the first byte, the first two bytes (for better results, which will be illustrated below using this configuration as an example), or a user-defined biased two-byte to achieve the DC component removal effect.

[0030] Reference Figure 8 The input data is a set number of bytes (CL). A parallel OR operation is performed with the first two bytes and a user-defined bias, generating three data streams. Zero-value detection is performed in parallel on each stream. The three data streams are checked for any zero-value exceeding a preset threshold (40). If any stream detects a zero-value exceeding the threshold, the compression method is recorded in Type[1:0], and half of the compressed set number of bytes is written to the high or low half of the set number of bytes in CL. If no zero-value exceeds the preset threshold in any of the three data streams, no compression is performed, Type[1:0] is set to 0, and CL uses the full set number of bytes.

[0031] Reference Figure 9After CL is hit and read, the compression status is first determined from the Type[1:0] of the Tag. If Type[1:0] is 0, then CL is not compressed, and the original data can be output directly; if Type[1:0] != 0, then CL is compressed, and the compressed data is restored according to the value of Type[1:0], or the first two bytes, or the user-specified bias. At the same time, zeros are padded according to the corresponding position of Sym[63:0] to restore the set number of bytes of data for output. Considering that non-zero value data is placed from the lowest position, the decompression process utilizes this storage rule to reduce selector input. For example, regardless of whether it is compressed or not, Byte63 can only be stored in the Byte63~40 positions, and cannot be placed in a lower position. When decoding and calculating Byte63, Byte39~0 should not be included in the selector calculation to reduce unnecessary logic.

[0032] Reference Figure 10 A typical 64-byte BF16 value distribution is as follows: Figure 10 As shown in Table 2, the array values ​​fluctuate around -4 to 4. After performing an XOR operation on the first two bytes with the other 31 BF16 data, 41 single-byte 0 values ​​appear, indicating that Type[1:0]=XOR compression was successful.

[0033] Table 2 shows the weight data in BF16 format with a typical 64-byte CL length for large model tensor processors. The decimal floating-point values ​​are shown in the second column (BF16), and the hexadecimal encoded values ​​are shown in the third column (HEX). The 0th to 30th data points are XORed with the first two bytes, resulting in the fourth column (XOR). The high byte is recorded as zero in the fifth column (ZVC1), and the low byte is recorded as zero in the sixth column (ZVC0). Ultimately, after XORing this set of BF16 data, there are 29 high-byte zero values ​​and 12 low-byte zero values, totaling 41 bytes of zero values, which can be compressed.

[0034] In this embodiment, the cached tag entry format is improved as follows: Figure 2 As shown, the improved Cache Tag and CL mapping relationship is as follows: Figure 3 As shown, the three compressed data preservation formats of CL are improved as follows: Figure 5 As shown in / 6 / 7, the XOR operation is used instead of the subtraction operation in the BDI algorithm, reducing the logic area. The RAW, XOR, and USR modes are used in parallel to enhance the applicability of the compression algorithm.

[0035] The ZVC compression process in this embodiment is as follows: Step 3.1.1: Before writing cache data, perform a OR operation on each byte and count the number of zero values ​​after the OR operation, that is, count the number of zero values ​​in the set number of bytes (64 bytes). Step 3.1.2: If the number of zero values ​​is less than the preset threshold (40), no compression is performed; if it is greater than the preset threshold, compression is performed. Step 3.1.3: Mark the ZVC compression code in the Type segment of the Tag. If the corresponding Tag0 is invalid, update Tag0 first. If both Tag0 and Tag1 are valid, according to the Cache update algorithm, remove the Tag0 and update the corresponding Tag0. Step 3.1.4: Calculate the positions of non-zero data and mark them in Sym[63:0]; Step 3.1.5: Write the non-zero value and Sym[63:0] into the half CL corresponding to the Tag in Step 3; Step 3.1.6: Direct compression writing and tag update process completed.

[0036] The XOR / USR compression process in this embodiment is as follows: Step 3.2.1: Before writing to the cache, perform an OR operation on each byte, or the first two bytes, or the user-specified offset two bytes, or all subsequent bytes, and count the number of zero values ​​in the whole or the last two bytes, that is, count the number of zero values ​​in the set bit length bytes of data; Step 3.2.2: If the number of zero values ​​is less than the preset threshold, no compression is performed; if it is greater than the preset threshold, compression is performed. Step 3.2.3: Mark the XOR or USR compression code in the Type segment of the Tag. If the corresponding Tag0 is invalid, update Tag0 first. If both Tag0 and Tag1 are valid, according to the Cache update algorithm, remove the Tag0 and update the corresponding Tag0. Step 3.2.4: Calculate the positions of non-zero data and mark them in Sym[63:0]; Step 3.2.5: Write the non-zero value and Sym[63:0] into the half CL corresponding to the Tag in step 3; Step 3.2.6: Direct compression writing and tag update process completed.

[0037] The ZVC decompression process in this embodiment is as follows: Step 4.1.1: After a cache hit, read the Type and Valid status bits of each Tag. Type[1:0]=RAW indicates directly compressed CL data; Step 4.1.2: Read the half-CL data corresponding to the Tag in Step 1, the lowest 64 bits are Sym[63:0]; Step 4.1.3: According to the Sym[63:0] instruction, fill the zero position corresponding to the half CL with zeros to restore it to the full CL of the set number of bytes; Step 4.1.4: Output the set number of bytes CL to complete the output of the corresponding data when the cache is hit.

[0038] The XOR / USR decompression process in this embodiment is as follows: Step 4.2.1: After a cache hit, read the Type and Valid status bits of each tag. Type[1:0]=XOR / USR is the XOR compressed half-CL data; Step 4.2.2: Read the half-CL data corresponding to the Tag in Step 1, the lowest 64 bits are Sym[63:0]; Step 4.2.3: Perform a corresponding XOR operation on the high 24 bytes of the half-CL data according to Type[1:0] in Step 1 to restore the data; Step 4.2.4: According to the Sym[63:0] instruction, pad the zero position corresponding to the half CL with the first double byte or the user-specified double byte to restore it to the full CL with the set number of bytes; Step 4.2.5: Output the set number of bytes CL to complete the output of the corresponding data when the cache is hit.

[0039] The method in this embodiment is implemented in the large model Transformer hardware acceleration module L2 / L3 Cache modeling, and its effectiveness has been demonstrated.

[0040] The embodiments described in this specification are merely examples of implementations of the inventive concept and are for illustrative purposes only. The scope of protection of this invention should not be considered limited to the specific forms described in these embodiments; rather, it extends to equivalent technical means conceived by those skilled in the art based on the inventive concept.

Claims

1. A data caching and compression method for large models, characterized in that, Includes the following steps: 1) Data is written in units of two bytes, either in parallel with the first two bytes or the user-specified two bytes; 2) Determine the number of zero values ​​following the data itself, the first two bytes, and the user-specified two bytes; 3) Select the byte method where the number of zero values ​​exceeds the preset threshold, and encode it into ZVC, XOR or USR, and store it in the TAG status bit of the corresponding cache line; at the same time, add a set number of bit status bits to the end of the compressed data to indicate the position of the zero value; 4) If none of the three methods have a byte value greater than the preset threshold zero, then it is encoded as an uncompressed RAW state and stored in the corresponding TAG state bit of the cache line.

2. The data caching and compression method for large models as described in claim 1, characterized in that, In step 3), the ZVC compression process is as follows: Step 3.1.1: Before writing cache data, perform a OR operation on each byte and count the number of 1s after the OR operation, which is to count the number of zero values ​​in the byte data of the set bit length. Step 3.1.2: If the number of zero values ​​is less than the preset threshold, no compression is performed; if it is greater than the preset threshold, compression is performed. Step 3.1.3: Mark the ZVC compression code in the Type segment of the Tag. If the corresponding Tag0 is invalid, update Tag0 first. If both Tag0 and Tag1 are valid, according to the Cache update algorithm, remove the Tag0 and update the corresponding Tag0. Step 3.1.4: Calculate the positions of non-zero data and mark them in Sym[63:0]; Step 3.1.5: Write the non-zero value and Sym[63:0] into the half CL corresponding to the Tag in step 3.1.3; Step 3.1.6: Direct compression writing and tag update process completed.

3. The data caching and compression method for large models as described in claim 1, characterized in that, In step 3), the XOR / USR compression process is as follows: Step 3.2.1: Before writing to the cache, perform an OR operation on each byte, or the first two bytes, or the user-specified offset two bytes, or all subsequent bytes, and count the number of 1s in the whole or the last two bytes, that is, count the number of zero values ​​in the set bit length byte data; Step 3.2.2: If the number of zero values ​​is less than the preset threshold, no compression is performed; if it is greater than the preset threshold, compression is performed. Step 3.2.3: Mark the XOR or USR compression code in the Type segment of the Tag. If the corresponding Tag0 is invalid, update Tag0 first. If both Tag0 and Tag1 are valid, according to the Cache update algorithm, remove the Tag0 and update the corresponding Tag0. Step 3.2.4: Calculate the positions of non-zero data and mark them in Sym[63:0]; Step 3.2.5: Write the non-zero value and Sym[63:0] into the half CL corresponding to the Tag in step 3; Step 3.2.6: Direct compression writing and tag update process completed.

4. A data caching and decompression method for large models, characterized in that, Includes the following steps: 1) Cache line hit, read data and cache tag; 2) Determine the data compression type based on the standard caching compression status identifier; 3) If it is in RAW mode, output directly; 4) If it is ZVC, XOR or USR, then perform an XOR operation first to restore the compressed data; 5) Based on the last set bit zero value indicator, pad the corresponding positions in the compressed data with zeros; 6) Restore the complete cache line of the set number of bytes and output it.

5. The data caching and decompression method for large models as described in claim 4, characterized in that, In step 4), the ZVC decompression process is as follows: Step 4.1.1: After a cache hit, read the Type and Valid status bits of each Tag. Type[1:0]=RAW indicates directly compressed CL data; Step 4.1.2: Read the half-CL data corresponding to the Tag in Step 1, with the minimum set bit length as Sym[63:0]; Step 4.1.3: According to the Sym[63:0] instruction, fill the zero position corresponding to the half CL with zeros to restore it to the full CL of the set number of bytes; Step 4.1.4: Output the set number of bytes CL to complete the output of the corresponding data when the cache is hit.

6. The data caching and decompression method for large models as described in claim 4, characterized in that, In section 4), the XOR / USR decompression process is as follows: Step 4.2.1: After a cache hit, read the Type and Valid status bits of each tag. Type[1:0] = XOR / USR is the XOR compressed half-CL data; Step 4.2.2: Read the half-CL data corresponding to the Tag in Step 1, with the minimum set bit length as Sym[63:0]; Step 4.2.3: Based on Type[1:0] in Step 1, set the high value of the half CL data (by performing a corresponding OR operation on the corresponding number of bits and bytes to restore the data; Step 4.2.4: According to the Sym[63:0] instruction, pad the zero position corresponding to the half CL with the first double byte or the user-specified double byte to restore it to the full CL with the set number of bytes; Step 4.2.5: Output the set number of bytes CL to complete the output of the corresponding data when the cache is hit.

Images