A data encoding method and apparatus suitable for neural network bit-slice processing

Abstract

This invention belongs to the field of neural network data processing technology, and relates to a data encoding method and apparatus suitable for bit-slice processing in neural networks. The method involves: online conversion of the bit-slice data format during both data loading and vector calculation in the neural network to obtain SBR and RSINT formats; encoding of activation value data of the network layer using the RSINT format; and encoding of weight data of the network layer using the restricted RSINT format. This invention supports asymmetric quantization of activation values, expands the data range of symmetric quantization weights, relaxes constraints on activation and weight quantization algorithms, and unifies the underlying hardware multiplier structure through equivalent transformation, effectively improving the performance of neural network models for bit-slice processing, while balancing hardware overhead, flexibility, and model accuracy.

Description

Technical Field

[0001] This invention belongs to the field of neural network data processing technology, and relates to a data encoding method and apparatus suitable for neural network bit slice processing. Background Technology

[0002] Deep neural networks have achieved superior performance compared to humans in many fields such as computer vision and natural language processing. However, they require substantial computational resources, especially for multiply-and-acculus (MAC) operations. Quantization is a mainstream model compression method, the core idea of ​​which is to reduce the numerical precision of weights and activation values, thereby minimizing computational and storage overhead while maintaining model performance as much as possible. Since different layers have varying sensitivities to model performance, using variable precision for different layers can further reduce computational and storage costs. To address this, bit-slicing architecture has been proposed. Its characteristic is that integer data is split into multiple bit slices, and bit-slice-level computation supports different levels of precision and leverages finer-grained data sparsity.

[0003] The bit slice data format is closely related to the quantization algorithm. The existing bit slice architecture mainly adopts the following bit slice data formats:

[0004] 1) Conventional Bit-slice Representation (CBR): The binary two's complement fixed-point data is split into a high-order signed bit slice and at least one low-order unsigned bit slice. That is, INT8 will be split into a high-order SINT4 and a low-order UINT4. A 5-bit multiplier is required to support 4-bit bit slices.

[0005] 2) Signed Bit-slice Representation (SBR): A sign bit is added to each unsigned bit slice to generate an SBR bit slice. If the data is negative, the bit slice will borrow a bit from the lower-order SBR bit slice and add one. That is, INT7 will be split into a higher-order SINT4 and a lower-order SINT4. A 4-bit multiplier is required to support 3-bit bit slices.

[0006] 3) Range-Shifted Integer (RSINT): This is achieved by adding the sign bit or carry bit of a lower-order bit slice to the next higher-order bit slice. For 8-bit data, the format range of RSINT is [-136, 119]. RSINT8 will be split into higher-order SINT4 and lower-order SINT4, requiring a 4-bit multiplier to support 4-bit bit slices.

[0007] 4) Unsigned Bit-slice Representation (UBR): This format splits an (n*k+k) bit unsigned number into (n+1) k-bit slices. For 8-bit UINT8 data, n and k are 1 and 4, respectively. Most bit-slice architectures are implemented directly using multipliers, while a few use LUTs to implement multipliers.

[0008] Existing bit-slice data formats mainly suffer from the following problems: 1) CBR and SBR formats: The multiplier bit width needs to be one bit larger than the bit-slice bit width, resulting in greater hardware overhead; 2) SBR format: The data range is significantly limited, unable to support the full bit width of the original data, which has a significant impact on model accuracy, relying on Quantization-Aware Training (QAT) to compensate; 3) RSINT and UBR formats: Asymmetric quantization, with a significantly different data range from INT8, relying on Quantization-Aware Training to readjust the data range. Directly using Post-Training Quantization (PTQ) would introduce additional computational overhead; 4) CBR: Multiple multiplication types exist, resulting in a larger hardware multiplier bit width. Furthermore, existing bit-slice architectures use the same symmetric quantization algorithm for activation values ​​and weights, ignoring the asymmetric distribution of activation values ​​and the symmetric distribution of weights. Summary of the Invention

[0009] To address the aforementioned technical problems in existing technologies, this invention proposes a data encoding method and apparatus suitable for bit-slicing processing in neural networks. Considering the differences in activation value and weight quantization algorithms, it unifies the underlying hardware multiplier structure, achieving a better balance between hardware overhead and model accuracy. The specific technical solution is as follows:

[0010] A data encoding method suitable for bit slice processing in neural networks is proposed, which performs online conversion of bit slice data format during data loading and vector calculation in neural networks to obtain SBR and RSINT formats.

[0011] The activation value data of the network layer is encoded using the RSINT format;

[0012] For the weight data of the network layer, a restricted RSINT format is set, called the restricted RSINT format, and the data is encoded using the restricted RSINT format.

[0013] Furthermore, in the online conversion process, the input is binary two's complement data, which undergoes initial bit slice generation and bit slice update, and the final output is a bit slice in CBR, SBR or RSINT format;

[0014] Furthermore, the binary two's complement data is initially bit-sliced ​​to generate a CBR format, and when the conversion control signal is low, the final output is a bit slice in CBR format; the binary two's complement data is initially bit-sliced ​​to generate an SBR format, and when the conversion control signal is high, the final output is a bit slice in SBR format; the binary two's complement data is initially bit-sliced ​​to generate a CBR format, and when the conversion control signal is high, the final output is a bit slice in RSINT format.

[0015] Furthermore, in the initial bit slice generation stage, for CBR format generation, the input data can be directly truncated by k bits; for SBR format generation, the highest-order bit slice is directly truncated by k bits, and subsequent lower-order bit slices are directly truncated by (k-1) bits, and the sign position of the bit slice is the sign bit of the input data, where k is the bit width of the bit slice.

[0016] Furthermore, during the bit slice update stage, the internal logic uses an adder, an AND gate, and an OR gate. The OR gate performs a logical OR operation on the sign bit and carry of the previous input bit slice. When the conversion control signal is high, the result of the logical OR is used to update the current input bit slice. The output of the adder is the updated bit slice, which includes one carry. The lowest-order bit slice is output directly without going through the adder, and its carry value is 0. The final output result only contains bit slice information and does not contain carry information.

[0017] Furthermore, the restricted RSINT format limits the data range to [-2]. （n+1）k-1 +1+△,2 （n+1）k-1 [-1-△], (n+1) represents the number of bit slices obtained from the data using RSINT format, with each bit slice having a width of k. This is the data range offset value.

[0018] Furthermore, the encoding via the restricted RSINT format specifically refers to:

[0019] In the quantization-aware training scenario, the quantization step size is learned by defining quantization values, wherein floating quantization values ​​are converted and rounded to the nearest integer and restricted to a threshold range; the threshold range is between the minimum and maximum values ​​of the restricted RSINT format data range.

[0020] In post-training quantization scenarios, the quantization step size is calculated first, and then the quantized value is calculated.

[0021] In a quantized model, the data is directly extracted to the range of restricted RSINT format, or the quantization step size is adjusted based on the maximum value of the restricted RSINT and CBR format data ranges.

[0022] Furthermore, the encoding via RSINT format specifically refers to:

[0023] In the quantization-aware training scenario, the quantization step size is learned by defining quantization values, wherein floating quantization values ​​are rounded to the nearest integer and restricted to a threshold range, which is between the minimum and maximum values ​​of the RSINT format data range;

[0024] In post-training quantization scenarios, the quantization step size is calculated first, then the activation zeros are calculated, and finally the quantized value is obtained.

[0025] In a quantized model, when the data range width of the original format and the RSINT format are the same, the original format can be equivalently converted to the RSINT format by subtracting a data range offset value from the activation quantization value and its zeros.

[0026] A data encoding device suitable for neural network bit slice processing includes: a control module, a data loading module, a matrix calculation module, a vector calculation module, a data storage module, and multiple local memories;

[0027] The control module receives configuration information from an external host and, after decoding the information, sends control signals to the data loading module, matrix calculation module, vector calculation module, and data storage module.

[0028] The data loading module and data storage module are used for data transmission between local storage and external storage, and the data is transmitted in blocks; the data loading module has a built-in online conversion unit that supports CBR to SBR and RSINT formats for preprocessing the activation values ​​of the first operator input of the neural network.

[0029] The matrix calculation module is used to perform computationally intensive operations and includes at least one 2D processing unit array. The array performs multiplication operations in the form of outer product or inner product, and the bit slices are iteratively accumulated between the bit slices in the time dimension.

[0030] The vector calculation module has a built-in online conversion unit that supports CBR to SBR and RSINT formats. This online conversion unit is used to convert the results of bias, quantization and activation function operations performed by the vector calculation module from CBR format to SBR and RSINT formats and provide them to the matrix calculation module.

[0031] Beneficial effects: This invention supports asymmetric quantization of activation values, expands the data range of symmetric quantization weights, solves the problem of limited weight data range in existing bit-slice processing architectures, relaxes the constraints on activation and weight quantization algorithms, and unifies the underlying hardware multiplier structure through equivalent transformation, effectively improving the performance of neural network models for bit-slice processing, and balancing hardware overhead, flexibility and model accuracy. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the neural network bit-slicing processing architecture in this embodiment.

[0033] Figure 2 This is a schematic diagram of the data processing principle of the online conversion unit supporting CBR to SBR and RSINT formats in this embodiment.

[0034] Figure 3 This is an example diagram of the INT8 and RSINT8 data formats and their equivalent conversion in this embodiment.

[0035] Figure 4 The local slicing processing architecture of this embodiment supports neural network inference graphs in SBR and RSINT formats. Detailed Implementation

[0036] To make the objectives, technical solutions, and technical effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0037] This embodiment discloses a data encoding method suitable for bit-slice processing in neural networks, as detailed below:

[0038] Design a neural network bit-slicing processing device that supports SBR and RSINT formats, such as... Figure 1 As shown, the system architecture includes a control module, a data loading module, a matrix calculation module, a vector calculation module, a data storage module, and multiple local storage devices.

[0039] The control module receives configuration information from an external host and stores it in a register. Once enabled, it initiates a request to retrieve instructions, stores them in an instruction queue, and then sends control signals to the data loading module, matrix calculation module, vector calculation module, and data storage module after fetching and decoding the instructions.

[0040] The data loading module and data storage module are responsible for data transfer between the local memory and external storage, typically in blocks, and support 2D or 3D Direct Memory Access (DMA). Similar to the padding function, the data loading module has a built-in online conversion unit that supports CBR to SBR and RSINT formats, mainly used for preprocessing the activation values ​​of the first operator input in the neural network.

[0041] The matrix calculation module is responsible for performing computationally intensive operations such as convolution. It contains at least one 2D processing element (PE) array. The PE array can be executed in either outer or inner product mode. The PE array internally supports SINTk×SINTk multiplication operations with bit slices as the unit, where k is the bit width of the bit slice. Bit slices are iteratively accumulated along the time dimension.

[0042] The vector computation module mainly performs operations such as biasing, quantization, and activation functions. The data format of the results of these operations is CBR format. In order to provide the SBR or RSINT format required by the matrix computation module, the vector computation module also includes an online conversion unit that supports CBR to SBR and RSINT formats. The mode of this online conversion unit is controlled by software configuration registers and is automatically triggered when writing the results back to the output memory.

[0043] like Figure 2 As shown, the online conversion unit takes binary two's complement data as input, generates an initial bit slice, updates the bit slice, and finally outputs a bit slice in CBR, SBR, or RSINT format. When the output is a CBR format bit slice, the initial bit slice format is CBR, and the Ctrl signal is low; when the output is an SBR format bit slice, the initial bit slice format is SBR, and the Ctrl signal is high; when the output is an RSINT format bit slice, the initial bit slice format is CBR, and the Ctrl signal is high.

[0044] In the initial bit slice generation stage, for CBR format generation, the input data can be directly truncated by k bits; for SBR format generation, the highest-order bit slice is directly truncated by k bits, and subsequent lower-order bit slices are directly truncated by (k-1) bits, and the sign position of the bit slice is the sign bit of the input data.

[0045] During the bit slice update phase, the internal logic uses an adder, an AND gate, and an OR gate. The OR gate performs a logical OR operation on the sign bit S and carry C of the previous input bit slice. When the Ctrl signal is high, the result of the logical OR is used to update the current input bit slice. The adder outputs the updated bit slice, which includes one carry. Since bit slice updates propagate from lower to higher orders, the lowest-order bit slice is output directly without going through the adder, and its carry C is 0. The final output contains only bit slice information and no carry information.

[0046] This embodiment uses a data-range-limited RSINT format: the data encoding method of the limited RSINT format is the same as that of the RSINT format, the only difference being the data range. Assuming the original data can be processed into (n+1) bit slices using the RSINT format, and each bit slice has a width of k, then the range that the RSINT format can represent is [-2...]. （n+1）k-1 -△,2 （n+1）k-1 -1-△], where The restricted RSINT format limits the data range to [-2]. （n+1）k-1 +1+△,2 （n +1）k-1 -1-△], on the one hand, can avoid [2 （n+1）k-1 +△,2 （n+1）k-1 Original data within the range of -1] was converted to RSINT format and overflowed into [-2]. （n+1）k-1 ,2 （n+1）k-1 The problem of -△] is addressed, while the representation of [-2] is discarded. （n+1）k-1 -2 （n+1）k-1 The original data within the range of +△] avoids the bias problem of the RSINT format range and achieves the symmetry of the data range. The advantage of the restricted RSINT format is that it unifies the multiplication type, supports symmetric quantization, and supports the full bit width of the original data. The cost is the loss of about 7% of the data range.

[0047] The existing bit-slice data format and the restricted RSINT format are compared in Table 1 below:

[0048]

[0049] The bit slice width is k-bit, and the number of blocks is n+1; in the table, k=4 and n=1.

[0050] Weight data encoding based on the restricted RSINT format has the following characteristics: Considering the symmetry of the weight distribution, mainstream quantization techniques employ symmetrical quantization, with weight zeros being 0. Therefore, this invention proposes encoding weight data using the restricted RSINT format. For quantization-aware training scenarios, quantization values ​​are defined... Let's learn the quantization step size S, where clip(z, α, β) restricts the quantization of z to the threshold range [α, β], round(z) rounds z to the nearest integer, and α and β are the minimum and maximum values ​​of the restricted RSINT format data range. For post-training quantization scenarios, we first calculate the quantization step size. Then calculate the quantization value. For quantized models, you can choose to directly extract data within the limited RSINT format range, or you can adjust the quantization step size. , where β and γ are the maximum values ​​of the restricted RSINT and CBR format data ranges, respectively.

[0051] The encoding of activation value data based on RSINT format is as follows: Considering the asymmetry of activation value distribution, mainstream quantization techniques employ asymmetric quantization for activation values. Taking into account the TFLite quantization specification, the data type of the activation value zero point is consistent with the activation value itself. For quantization-aware training scenarios, quantization values ​​are defined... To learn the quantization step size S, α and β need to be set to the minimum and maximum values ​​of the RSINT format data range, x zp The activation value is zero point, and x int Both are in RSINT format. For post-training quantization scenarios, the quantization step size is calculated first. Then calculate the zero point. Finally, the quantized value is obtained. For a quantized model, as long as the original format and the RSINT format have the same data range width, the original format can be equivalently converted to the RSINT format by subtracting a data range offset value from both the activation quantization value and its zeros.

[0052] (1)

[0053] (2)

[0054] (3)

[0055] in, This is the data range offset value from CBR to RSINT.

[0056] (4)

[0057] (5)

[0058] (6)

[0059] in, This is the data range offset value from UBR to RSINT.

[0060] (7)

[0061] (8)

[0062] (9)

[0063] Since the data range of RSINT includes SBR, the above △=0. As for... Whether it is 0 or not, it can be equivalently converted to Since the data range of RSINT is different from that of SBR, formulas (8) and (9) are not bidirectionally equivalent.

[0064] Through the above formula conversion, for activation values, regardless of whether the original format is CBR, UBR, or SBR, they can all be equivalently converted to RSINT format. The advantage of this is that it can unify the multiplication type requirements of activation values.

[0065] In this embodiment, the input data bit width m is selected as 8 bits, and the bit slice bit width k is selected as 4 bits.

[0066] The basic principle of the RSINT data format is as follows:

[0067] Suppose that the m-bit two's complement integer I is defined as

[0068] (10)

[0069] in, It represents the i-th digit of the integer I.

[0070] Rewrite formula (10) to represent the bit slice structure in groups of 4 bits, and obtain

[0071] (11)

[0072] Define a 4-bit signed number The bit slice representing I[t, t+3] is calculated as follows:

[0073] (12)

[0074] The original integer I can then be represented by a bit slice as follows:

[0075] (13)

[0076] when and hour, This indicates that the highest-order bit slice has overflowed, and the binary two's complement integer I cannot be converted into the equivalent RSINT.

[0077] like Figure 3The diagram illustrates the INT8 and RSINT8 data formats and their equivalent conversion examples. The data range for INT8 is [-128, 127], while the data range for RSINT8 is [-136, 119], equivalent to a data range offset by 8 along the negative axis. The dotted areas in the diagram represent overflow regions. The RSINT8 range of [-136, -129] corresponds to the INT8 range of [120, 127], indicating that INT8 cannot be converted to the equivalent RSINT8, while other regions of INT8 can be converted to the equivalent RSINT8. Although the RSINT8 range of [-128, -120] has equivalent INT8 values, to maintain symmetry in the RSINT8 data range, the [-128, -120] range is discarded, resulting in the restricted RSINT8 format with a data range of [-119, 119], which is the valid region in the diagram.

[0078] Two examples of INT8 to RSINT8 equivalent conversion: -2 and -10. For -2, bit slice 1 receives the sign bit 1 from bit slice 0, updating bit slice 1 from 4'b1111 to 4'b0000, meaning the higher-order bit slice is 0, which is beneficial for bit slice-level sparsity mining. For -10, the sign bit of bit slice 0 is 0, so bit slice 1 remains unchanged. It can be seen that when n=1, k=4, (n+1)*k=8 bits, updating bit slice 1 only needs to consider the sign bit of bit slice 0. When n is greater than 1, updating higher-order bit slices requires considering the sign bit of lower-order bit slices and the carry from the adder. For example, the adder in bit slice 1 corresponding to -2 has a carry of 1, which will be propagated upwards when a higher-order bit slice exists.

[0079] like Figure 4 As shown, the bit-slice processing architecture of the present invention supports neural network inference in SBR and RSINT formats. The upper part corresponds to activation in RSINT8 format and weight in restricted RSINT8 format, while the lower part corresponds to activation and weight in SBR format.

[0080] For operator #0, the data loading module is responsible for the online conversion of INT8 to RSINT8 or INT7 to SBR format, similar to the activation padding function. After conversion, the data is written to the local input memory. The intermediate activations all use RSINT8 or SBR format. This part of the function is mainly implemented by the online conversion module in the vector calculation module. On the one hand, this avoids the data loading module from performing additional data conversion, and on the other hand, it allows subsequent operators to load data bit by bit, rather than forcibly loading all bit slices at the same time.

[0081] For the last operator #n, the output activation uses INT8 or INT7 format, implemented by the online conversion module in the vector calculation module. The addition of the online conversion unit enables the bit-slice processing architecture of this invention to support multiple formats such as SBR and RSINT8, and to support different activation and weight formats.

[0082] Through the above embodiments, the bit-slice data format, encoding method, and hardware architecture proposed in this invention constitute a tightly coupled and collaborative whole. It not only solves the problem of limited weight data range in existing bit-slice processing architectures, simplifies and unifies the bit-slice multiplier and bit-slice storage format, but also relaxes the constraints on activation and weight quantization algorithms, effectively improving the performance of neural network models for bit-slice processing. Therefore, the method and architecture of this invention effectively balance hardware overhead, hardware flexibility, and model accuracy.

[0083] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any way. Although the implementation process of the present invention has been described in detail above, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A data encoding method suitable for bit-slicing processing in neural networks, characterized in that: During the data loading and vector computation processes of the neural network, the bit slice data format is converted online to obtain SBR and RSINT formats; The activation value data of the network layer is encoded using the RSINT format; For the weight data of the network layer, a restricted RSINT format is set, which is called the restricted RSINT format, and the data is encoded using the restricted RSINT format; The restricted RSINT format limits the data range to [-2]. （n+1）k-1 +1+△,2 （n+1）k-1 [-1-△], (n+1) is the number of bit slices obtained by the data through RSINT format, each bit slice has a width of k, and △ is the data range offset value; SBR is a signed bit slice format; RSINT is a range offset integer format, which is implemented by adding the sign bit or carry bit of the lower-order bit slice to the next higher-order bit slice.

2. The method as described in claim 1, characterized in that: In the online conversion process, the input is binary two's complement data. After initial bit slice generation and bit slice update, the final output is a bit slice in CBR, SBR or RSINT format. CBR format is the traditional bit slice format.

3. The method as described in claim 2, characterized in that: Binary two's complement data is initially bit-sliced ​​into CBR format, and when the conversion control signal is low, the final output is a bit slice in CBR format; binary two's complement data is initially bit-sliced ​​into SBR format, and when the conversion control signal is high, the final output is a bit slice in SBR format; binary two's complement data is initially bit-sliced ​​into CBR format, and when the conversion control signal is high, the final output is a bit slice in RSINT format.

4. The method as described in claim 3, characterized in that: During the initial bit slice generation stage, for CBR format generation, the input data can be directly truncated by k bits; for SBR format generation, the highest-order bit slice is directly truncated by k bits, and subsequent lower-order bit slices are directly truncated by (k-1) bits, with the bit slice sign position being the sign bit of the input data, and k being the bit width of the bit slice.

5. The method as described in claim 3, characterized in that: During the bit slice update phase, the internal logic uses an adder, an AND gate, and an OR gate. The OR gate performs a logical OR operation on the sign bit and carry of the previous input bit slice. When the conversion control signal is high, the result of the logical OR is used to update the current input bit slice. The output of the adder is the updated bit slice, which includes one carry. The lowest-order bit slice is output directly without going through the adder, and its carry value is 0. The final output result only contains bit slice information and does not contain carry information.

6. The method as described in claim 1, characterized in that: The encoding via the restricted RSINT format specifically refers to: In the quantization-aware training scenario, the quantization step size is learned by defining quantization values, wherein floating quantization values ​​are converted and rounded to the nearest integer and restricted to a threshold range; the threshold range is between the minimum and maximum values ​​of the restricted RSINT format data range. In post-training quantization scenarios, the quantization step size is calculated first, and then the quantized value is calculated. In a quantized model, the data is directly extracted to the range of restricted RSINT format, or the quantization step size is adjusted based on the maximum value of the restricted RSINT and CBR format data ranges.

7. The method as described in claim 1, characterized in that: The encoding via RSINT format specifically refers to: In the quantization-aware training scenario, the quantization step size is learned by defining quantization values, wherein floating quantization values ​​are rounded to the nearest integer and restricted to a threshold range, which is between the minimum and maximum values ​​of the RSINT format data range; In post-training quantization scenarios, the quantization step size is calculated first, then the activation zeros are calculated, and finally the quantized value is obtained. In a quantized model, when the data range width of the original format and the RSINT format are the same, the original format can be equivalently converted to the RSINT format by subtracting a data range offset value from the activation quantization value and its zeros.

8. An apparatus employing the data encoding method according to any one of claims 1 to 7, characterized in that, include: The system includes a control module, a data loading module, a matrix calculation module, a vector calculation module, a data storage module, and multiple local storage devices. The control module receives configuration information from an external host and, after decoding the information, sends control signals to the data loading module, matrix calculation module, vector calculation module, and data storage module. The data loading module and data storage module are used for data transmission between local memory and external storage, and the data is transmitted in blocks; the data loading module has a built-in online conversion unit that supports CBR to SBR and RSINT formats for preprocessing the activation values ​​of the first operator input of the neural network; the CBR format is a traditional bit slice format; The matrix calculation module is used to perform computationally intensive operations and includes at least one 2D processing unit array. The array performs multiplication operations in the form of outer product or inner product, and the bit slices are iteratively accumulated between the bit slices in the time dimension. The vector calculation module has a built-in online conversion unit that supports CBR to SBR and RSINT formats. This online conversion unit is used to convert the results of bias, quantization and activation function operations performed by the vector calculation module from CBR format to SBR and RSINT formats and provide them to the matrix calculation module.

Images