Binarized neural network and multiplication optimization method thereof
By binarizing and quantizing the linear layer weights and activation values, combining linear operations and inverse quantization, multiplication operations in binarized neural networks are optimized, and the problems of resource waste and power consumption are solved, and more efficient network inference is achieved.
Patent Information
- Application Number
- CN202510298989.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-13
- Publication Date
- 2025-06-10
AI Technical Summary
The multiplication operation optimization of existing binary neural networks at the full connection layer is incomplete, resulting in waste of resources and increased power consumption.
By binarizing the linear layer weights, the input activation value is quantized, and the quantized activation value is linearly calculated. Finally, the binarized weights and the quantized activation value are inversely quantized to obtain the original accuracy output activation value.
It effectively reduces the number of residual multipliers in the binary neural network, reduces power consumption, and improves network inference speed and efficiency.
Smart Images
Figure CN120124694A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical fields of computer data processing and deep learning, and particularly relates to a binary neural network and a multiplication optimization method thereof. Background Art
[0002] In recent years, with the rapid development of artificial intelligence technology, deep neural networks (DNNs) have achieved remarkable application effects in the fields of computer vision, natural language processing, intelligent speech, etc. However, with the continuous expansion of the network scale, the computational complexity and resource consumption of the model have also increased significantly. Especially when deployed on embedded hardware platforms (such as FPGAs), the computational resource requirements, storage space, power consumption, etc. of the network have become important bottleneck problems.
[0003] In order to reduce hardware resource consumption and accelerate model inference, binary neural networks (BNNs) have emerged. BNNs limit the weights and activation values in the network to binary values (-1, +1), thus significantly reducing the multiplication operations required in the calculation and lowering the demand for hardware resources.
[0004] However, although existing BNNs have significantly reduced multiplication operations at the network level such as the convolutional layer, some operations in the network, such as the fully connected layer (linear layer), still use traditional multiplication operations and are not fully optimized, resulting in resource waste and increased power consumption. Summary of the Invention
[0005] The purpose of the present invention is to provide a binary neural network and a multiplication optimization method thereof, so as to at least solve the problem that the multiplication operation optimization of the existing binary neural network is incomplete, resulting in resource waste and increased power consumption.
[0006] To solve the above technical problems, the present invention provides a multiplication optimization method for a binary neural network, including: Performing binary processing on the linear layer weights to obtain binary weights; Quantifying the input activation values of the linear layer; Performing a linear operation on the quantified input activation values to obtain output quantified activation values; Using the binary weights and the quantified input activation values to dequantify the output quantified activation values to obtain output activation values with the original precision.
[0007] Optionally, in the multiplication optimization method of the binary neural network, the method for performing binary processing on the linear layer weights to obtain binary weights includes: Binary quantifying the linear layer weights to 1 bit:
[0008] Among them, represents the input linear layer weight, represents the binary weight, represents the binary function.
[0009] Optionally, in the multiplication optimization method of the binary neural network, the method for quantifying the input activation value of the linear layer includes: By means of symmetric quantization, the input activation value is quantized to n bits to scale the range of the quantized input activation value to :
[0010]
[0011]
[0012] where x represents the input activation value, represents the maximum value among the absolute values of all input activation values, represents the input activation value matrix; represents the rounding process of the clipping function; represents the quantized input activation value; n represents an integer from 1 to 64; represents the quantization function.
[0013] Optionally, in the multiplication optimization method of the binary neural network, the method for quantifying the input activation value of the linear layer further includes: By subtracting the minimum value among all input activation values from the input activation value to scale the range of the quantized input activation value to :
[0014]
[0015] where, represents the minimum value among all input activation values, represents the input activation value matrix.
[0016] Optionally, in the multiplication optimization method of the binary neural network, the linear layer weight is a 32-bit full-precision floating-point number; the input activation value is a 32-bit full-precision floating-point number; the value of n ranges from 1 to 64.
[0017] Optionally, in the multiplication optimization method of the binary neural network, the linear operation on the quantized input activation value to obtain the output quantized activation value includes:
[0018]
[0019]
[0020] Among them, k and m respectively represent the width and height of the dimension of the binary weight matrix. represents the binary weight. represents the mean value of the L1 norm of the original weight. represents the quantized input activation value. represents the maximum value among the absolute values of all input activation values. represents the output quantized activation value; n represents an integer from 1 to 64.
[0021] Optionally, in the multiplication optimization method of the binary neural network, the output quantized activation value is dequantized by using the binary weight and the quantized input activation value to obtain the output activation value with the original precision, including:
[0022] Among them, represents the rounding operation. represents the output activation value. represents the dequantization function.
[0023] To solve the above technical problems, the present invention also provides a binary neural network for implementing the multiplication optimization method of the binary neural network described in any one of the above, and the binary neural network includes a linear layer, and the linear layer includes: A weight binary quantization unit for binary quantizing the linear layer weight to obtain a binary weight. A quantization unit for quantizing the input activation value of the linear layer to obtain a quantized input activation value. A linear operation unit for performing a linear operation on the quantized input activation value to obtain an output quantized activation value. A dequantization unit for dequantizing the output quantized activation value by using the binary weight and the quantized input activation value to obtain an output activation value with the original precision.
[0024] Optionally, in the binary neural network, the binary neural network is used in a model for classification tasks, and a linear layer is provided at the end of the binary neural network as an output layer.
[0025] Optionally, in the binary neural network, the binary neural network is deployed based on an embedded hardware platform.
[0026] The binary neural network and its multiplication optimization method provided by the present invention include: binarizing the weights of the linear layer to obtain binarized weights; quantifying the input activation values of the linear layer; performing a linear operation on the quantized input activation values to obtain output quantized activation values; and dequantizing the output quantized activation values using the binarized weights and the quantized input activation values to obtain output activation values with the original precision. By performing multiplication optimization on the weights and activation values of the linear layer, the number of multiplication calculations in quantization and dequantization in the linear layer is effectively reduced, so that not only can the number of residual multipliers in the binary neural network be effectively reduced and the power consumption be reduced, but also the network inference speed and efficiency can be effectively improved, solving the problem of incomplete multiplication operation optimization in the existing binary neural network, resulting in resource waste and increased power consumption. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 is a flowchart of the multiplication optimization method of the binary neural network provided in this embodiment; Figure 2 is a schematic structural diagram of a linear layer in the binary neural network provided in this embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0028] The binary neural network and its multiplication optimization method proposed by the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the accompanying drawings are all in a very simplified form and use non-precise scales, only for the purpose of facilitating and clearly assisting in explaining the purpose of the embodiments of the present invention. In addition, the structures shown in the accompanying drawings are often part of the actual structures. In particular, the accompanying drawings need to show different focuses and sometimes use different scales.
[0029] It should be noted that the "first", "second", etc. in the description, claims and drawings of the present invention are used to distinguish similar objects, so as to describe the embodiments of the present invention, rather than to describe a specific order or sequence. It should be understood that such structures can be interchanged under appropriate circumstances. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.
[0030] For the linear layer of the existing conventional binary neural network, when the input activation value of 32-bit full-precision floating-point number (FP32) is quantized into n-bit parameters under the weight of 32-bit full-precision floating-point number, to compress 1024 channels to 10 classification channels, it is necessary to go through 1024×w×h times of quantization (32-bit and n-bit multiplication) and dequantization (32-bit and n-bit multiplication) processes, and 1024×w×h×10 times of n-bit and n-bit multiplication operations. The amount of multiplication operations is large and more multiplier resources are required.
[0031] To reduce the multiplication operations in the binary neural network, especially in the fully connected layer (linear layer), this embodiment provides a multiplication optimization method for the binary neural network, as Figure 1 shown, including: S1, perform binary processing on the linear layer weight to obtain a binary weight; S2, quantize the input activation value of the linear layer; S3, perform a linear operation on the quantized input activation value to obtain an output quantized activation value; S4, use the binary weight and the quantized input activation value to dequantize the output quantized activation value to obtain an output activation value with the original precision.
[0032] The multiplication optimization method for the binary neural network provided by this embodiment effectively reduces the number of multiplication calculations in quantization and dequantization in the linear layer by performing multiplication optimization processing on the weight and activation value of the linear layer. Thus, it can not only effectively reduce the number of residual multipliers and power consumption in the binary neural network, but also effectively improve the network inference speed and efficiency, and solves the problem of incomplete multiplication operation optimization in the existing binary neural network, resulting in resource waste and increased power consumption.
[0033] Using the multiplication optimization method for the binary neural network provided by this embodiment, for the input activation value of 32-bit full-precision floating-point number (FP32) quantized into n-bit parameters under the weight of 32-bit full-precision floating-point number in the above example, only 1024×w×h times of quantization (32-bit and n-bit multiplication) and dequantization (n-bit and n-bit multiplication) processes are required. In this way, while ensuring the calculation accuracy of the linear layer, the number of multiplication operations is reduced, that is, the demand for multipliers in the linear layer is reduced, not only reducing resource occupation and waste, but also reducing power consumption.
[0034] It should be noted that in actual applications, the implementation order of step S1 and step S2 can be swapped or synchronized. The technical solutions after the order adjustment without violating the gist of the present invention should also fall within the protection scope of this application.
[0035] Specifically, in this embodiment, step S1, the method for binarizing the weights of the linear layer to obtain the binarized weights includes: Binarize the weights of the linear layer to 1 bit:
[0036] Among them, represents the input weights of the linear layer, represents the binarized weights, represents the binarization function.
[0037] By binarizing the weights of the linear layer, the weights are converted from 32-bit (32-bit full-precision floating-point numbers) to 1-bit binarized weights, which can effectively improve the data processing efficiency of the linear layer, reduce the computational complexity, and reduce power consumption.
[0038] Furthermore, in this embodiment, step S2, the method for quantifying the input activation values of the linear layer includes: By symmetrically quantifying absmax, the input activation values are quantized to n bits to scale the range of the quantized input activation values to :
[0039]
[0040]
[0041] Among them, x represents the input activation value, represents the maximum value among the absolute values of all input activation values, represents the input activation value matrix; round represents the rounding process of the clipping function; represents the quantized input activation value; n represents an integer from 1 to 64; represents the quantization function.
[0042] In practical applications, if the input activation value is 32-bit (32-bit full-precision floating-point number), when quantized to n bits, the value of n is an integer from 1 to 64. Preferably, the value of n can be 1, 2, 4, 8, 16, 32. In a specific embodiment, when n is 8, .
[0043] Preferably, in this embodiment, step S2, the method for quantifying the input activation values of the linear layer includes: By subtracting the minimum value among all input activation values from the input activation values, the range of the quantized input activation values is scaled to , so that all values are negative:
[0044]
[0045] Among them, represents the minimum value among all input activation values, represents the input activation value matrix.
[0046] In this way, by quantifying the input activation values of the linear layer, the number of multiplication operations in the linear operation can be reduced while ensuring the accuracy of the subsequent linear operation.
[0047] Furthermore, in this embodiment, in step S3, linear operations are performed on the quantized input activation values to obtain output quantized activation values, including:
[0048]
[0049] Among them, k and m respectively represent the width and height of the dimension of the binary weight matrix, which are related to the number of channels and the convolution kernel size; represents the binary weight, represents the mean value of the L1 norm of the original weight; represents the quantized input activation value, represents the maximum value among the absolute values of all input activation values, represents the output quantized activation value; n represents an integer from 1 to 64.
[0050] Specifically, in the linear operation, assuming that the elements in the weight w and the elements in the input activation value x are independently distributed, the variance of the output activation value y can be estimated as:
[0051] If the above standard quantization method is used, the variance , which is beneficial to the stability of the binary neural network training.
[0052] And, the parameter can add sparsity to the output activation value during the quantization and dequantization processes, thus ensuring the expression ability of the binary neural network. During the quantization process, the weights of the binary neural network are discrete values. By multiplying with the parameter , some scale information of the weights can be helped to be restored, making the output of the binary neural network smoother, thereby effectively reducing the information loss caused by quantization. In addition, the parameter The quantization effect can also be adjusted to make the weight distribution after quantization closer to the distribution of the original weights. This adjustment can improve the accuracy of the quantized binary neural network during inference, especially in the case of low bit rates. In this way, the computational accuracy of the binary network can be guaranteed.
[0053] Further, in this embodiment, in step S4, the output quantized activation value is dequantized using the binary weight and the quantized input activation value to obtain the output activation value with the original accuracy, including:
[0054] Among them, represents the rounding operation, represents the output activation value, represents the dequantization function.
[0055] Specifically, in this embodiment, since the weights are binary processed, there is no need for multiplication operations, that is, the multipliers for related calculations are saved. Then, through dequantization, the parameters and are used for rescaling to restore the accuracy of dequantization to the original accuracy, thus completing the entire process of the input activation value from input to output in the linear layer.
[0056] This embodiment also provides a binary neural network for implementing the multiplication optimization method of the binary neural network as described above. The binary neural network includes a linear layer, as Figure 2 shown, and the linear layer includes: A weight binary unit for binary processing the weights of the linear layer to obtain binary weights; A quantization unit for quantizing the input activation value of the linear layer to obtain the quantized input activation value; A linear operation unit for performing a linear operation on the quantized input activation value to obtain the output quantized activation value; A dequantization unit for dequantizing the output quantized activation value using the binary weight and the quantized input activation value to obtain the output activation value with the original accuracy.
[0057] Specifically, in practical applications, the binary neural network can be used in a model for classification tasks, and a linear layer is provided at the end of the binary neural network as the output layer.
[0058] In a classification task, the last layer is usually a linear layer that directly outputs the probability values corresponding to the categories. For the linear layer of existing conventional binary neural networks, the input activation values of 32-bit full-precision floating-point numbers (FP32) are quantized into n-bit parameters under the weights of 32-bit full-precision floating-point numbers. To compress 1024 channels to 10 classification channels, it is necessary to go through 1024×w×h quantization processes (32-bit and n-bit multiplications) and dequantization processes (32-bit and n-bit multiplications), as well as 1024×w×h×10 n-bit and n-bit multiplication operations. The amount of multiplication operations is large, and more multiplier resources are required. However, for the linear layer of the binary neural network provided in this embodiment, only 1024×w×h quantization processes (32-bit and n-bit multiplications) and dequantization processes (n-bit and n-bit multiplications) are needed. In this way, while ensuring the calculation accuracy of the linear layer, the number of multiplication operations is reduced, that is, the demand for multipliers in the linear layer is reduced, which not only reduces resource occupancy and waste but also reduces power consumption.
[0059] Moreover, in practical applications, the binary neural network is deployed based on an embedded hardware platform, such as hardware platforms like FPGA and SoC.
[0060] The binary neural network and its multiplication optimization method provided in this embodiment reduce the resource occupancy in the linear layer by optimizing the calculation method of the fully connected layer (linear layer), thereby significantly reducing the consumption of hardware resources. The binary neural network and its multiplication optimization method provided in this embodiment can effectively reduce the number of residual multipliers in the binary network, reduce the chip area and power consumption on embedded hardware platforms such as FPGA, and improve the network inference speed and efficiency. The binary neural network and its multiplication optimization method provided in this embodiment can significantly improve the utilization rate and energy efficiency of hardware resources while ensuring the calculation accuracy, providing a more effective solution for the application of binary neural networks on embedded hardware platforms. The binary neural network and its multiplication optimization method provided in this embodiment significantly reduce the storage and calculation requirements of the model while maintaining the same accuracy as traditional deep learning models, and are suitable for deployment on edge devices.
[0061] It should be noted that the various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. In addition, the different parts between the embodiments can also be combined and used, and the present invention does not limit this.
[0062] The binary neural network and its multiplication optimization method provided by the present invention include: performing binary processing on the weights of the linear layer to obtain binary weights; quantifying the input activation values of the linear layer; performing a linear operation on the quantified input activation values to obtain output quantified activation values; and dequantifying the output quantified activation values by using the binary weights and the quantified input activation values to obtain output activation values with the original precision. By performing multiplication optimization processing on the weights and activation values of the linear layer, the number of multiplication calculations in quantization and dequantization in the linear layer is effectively reduced, so that not only can the number of residual multipliers in the binary neural network be effectively reduced and the power consumption be reduced, but also the network inference speed and efficiency can be effectively improved, solving the problem of incomplete multiplication operation optimization in the existing binary neural network, resulting in resource waste and increased power consumption.
[0063] The above description is only a description of the preferred embodiments of the present invention, and does not limit the scope of the present invention in any way. Any changes and modifications made by those of ordinary skill in the art of the present invention according to the above disclosure shall fall within the scope of protection of the claims.
Claims
1. A multiplication optimization method for a binarized neural network, characterized in that: include: Binarize the linear layer weights to obtain binary weights; Quantize the input activation values of the linear layer; Perform linear operations on the quantized input activation values to obtain output quantized activation values; The output quantized activation value is dequantized using the binarized weights and the quantized input activation value to obtain the output activation value of the original accuracy.
2. The multiplication optimization method of a binarized neural network according to claim 1, characterized in that: The method of binarizing the linear layer weights to obtain the binarized weights includes: The linear layer weights are binary quantized to 1 bit: in, represents the input linear layer weights, represents the binary weight, Represents a binarization function.
3. The multiplication optimization method of a binarized neural network according to claim 1, characterized in that: The method for quantizing the input activation value of the linear layer includes: The input activation value is quantized into n bits by symmetric quantization to scale the range of the quantized input activation value to : Among them, x represents the input activation value, represents the maximum absolute value of all input activation values, represents the input activation value matrix; Represents the rounding process of the clipping function; Represents the quantized input activation value; n represents an integer from 1 to 64; Represents a quantization function.
4. The multiplication optimization method of a binarized neural network according to claim 3, characterized in that: The method for quantizing the input activation value of the linear layer also includes: The quantized input activation value is scaled to the range of : in, represents the minimum value among all input activation values, Represents the input activation value matrix.
5. The multiplication optimization method of a binarized neural network according to claim 3, characterized in that: The linear layer weight is a 32-bit full-precision floating point number; the input activation value is a 32-bit full-precision floating point number; and the value of n is an integer from 1 to 64.
6. The multiplication optimization method of a binarized neural network according to claim 1, characterized in that: The performing a linear operation on the quantized input activation value to obtain an output quantized activation value includes: Among them, k and m represent the width and height of the binary weight matrix dimension respectively. represents the binary weight, Represents the L1 norm mean of the original weights; represents the quantized input activation value, represents the maximum absolute value of all input activation values, Represents the output quantized activation value; n represents an integer from 1 to 64.
7. The multiplication optimization method of a binarized neural network according to claim 6, characterized in that: The method of dequantizing the output quantized activation value by using the binarized weight and the quantized input activation value to obtain the output activation value of the original precision includes: in, Indicates rounding operation. represents the output activation value, Represents the inverse quantization function.
8. A binarized neural network, used to implement the multiplication optimization method of the binarized neural network according to any one of claims 1 to 7, characterized in that: The binarized neural network includes a linear layer, and the linear layer includes: A weight binarization unit is used to binarize the linear layer weights to obtain binarized weights; A quantization unit, used to quantize the input activation value of the linear layer to obtain a quantized input activation value; A linear operation unit, used to perform a linear operation on the quantized input activation value to obtain an output quantized activation value; The dequantization unit is used to dequantize the output quantized activation value by using the binarized weight and the quantized input activation value to obtain the output activation value of the original precision.
9. The binarized neural network according to claim 8, characterized in that: The binarized neural network is used in a model of a classification task, and a linear layer is provided at the end of the binarized neural network as an output layer.
10. The binarized neural network according to claim 8, characterized in that: The binarized neural network is deployed based on an embedded hardware platform.