An asymmetric channel expansion and contraction technology applied to a binary convolution layer

Abstract

This patent relates to an asymmetric channel expansion and contraction technique (AESBNN) applied to binary convolutional layers, belonging to the field of binary neural network optimization technology in deep neural networks. This design aims to solve the problem that existing symmetric channel expansion and contraction techniques (ESBNN) cannot balance computational complexity and model accuracy according to the needs of different scenarios due to their fixed expansion / contraction ratio. The core solution is to introduce an asymmetric ratio r to first expand the number of input channels by a factor of t to increase the dynamic range of the binary convolution output. Then, the number of output channels of the convolution kernel is reduced to 1 / (t*r) of the original number to adjust computational complexity. Finally, through channel duplication and splicing layers and batch normalization (BN) layers, the final output channels are restored to be compatible with subsequent network layers. This method provides a flexible accuracy-efficiency balance strategy: when r > 1, computational complexity is prioritized to be reduced; when r < 1, the model's representational ability is prioritized to be improved. Experiments show that this technology can achieve better performance (such as PSNR) than basic binary convolutional units in tasks such as image super-resolution with lower computational cost (such as halving the computational cost when r=2), and can be flexibly integrated into various deep neural networks as a plug-and-play general module.

Description

Technical Field

[0001] This invention relates to the optimized design of binary neural networks (BNNs) in deep neural network (DNN) technology, specifically an asymmetric channel expansion and contraction technique applied to binary convolutional layers. Background Technology

[0002] While DNNs have made groundbreaking progress in several fields, their enormous computational and memory overhead poses a significant challenge to real-time deployment on mobile or embedded platforms (such as robotics and autonomous driving). To alleviate the computational bottleneck of models, model compression techniques have emerged, mainly including quantization, pruning, knowledge distillation, and compact model design. Among these, BNNs, as the ultimate form of quantization, quantize weights and activation values ​​to a single bit, making them one of the most promising solutions for addressing deployment challenges in resource-constrained environments.

[0003] Binary Neural Networks (BNNs) can achieve up to 32 times the memory compression and 64 times the computational speedup on dedicated processors. Despite these advantages in memory, speed, and energy consumption, binary neural networks suffer from significant accuracy degradation in tasks such as image super-resolution. To bridge the gap between binary and real-valued neural networks, previous research has focused on optimizing forward inference and backpropagation algorithms.

[0004] In the computation of binary convolution, since the weights and feature maps are binarized to +1 or -1, the convolution operation can be achieved through efficient XNOR and Population Count operations. At this point, the computation result of a single convolution kernel at a single spatial location is essentially the difference between the number of activation values ​​that are completely identical or completely opposite to those within the local window of the feature map. Therefore, its output value is restricted to a finite set of discrete integers, i.e., [-CIN * k², CIN * k²] (where CIN is the number of input channels and k is the kernel size). However, the output value of full-precision convolution is a continuous real number, possessing an infinite dynamic range and a finer representational capability. Binary convolution compresses rich full-precision information into an extremely limited discrete set; therefore, binary convolution is always limited by its finite representational power.

[0005] In their paper "Expanding-and-Shrinking Binary Neural Networks," Xulong Shi et al. explored the factors influencing the set size of binary convolution output values ​​and proposed a simple yet effective channel expansion and shrinking operation based on their analysis. This operation expands the input channels and shrinks the convolution kernel. This operation improves the representational power of output features in each individual convolutional layer while maintaining the original computational complexity and number of parameters. The paper refers to the binary neural network enhanced by the channel expansion and shrinking operation as the Expanding-and-Shrinking Binary Neural Network (ESBNN).

[0006] Implementing the channel expansion and contraction operation described in the aforementioned paper within a typical binary convolutional layer first involves expanding the input channels from CIN to CIN*t (t∈Z+), where t is the channel scaling factor. The output value after binary convolution is constrained to an expanded discrete integer range, i.e., [-CIN * k² * t, CIN * k² * t]. Then, the kernel dimension is shrunk from COUT to COUT / t to maintain constant computational complexity. However, shrinking the output channels from COUT to COUT / t weakens the representational power of the next layer (i.e., the number of input channels in the next layer is also reduced) and may be incompatible with other architectural units (e.g., matching the number of feature channels is required in residual connections). To compensate for this shrinkage, the paper designs a channel duplication module, which includes a channel duplication concatenation layer and a batch normalization layer. After binary convolution, the channel duplication module creates t² copies of the feature map and concatenates them along the channel dimension, thus expanding the output channels. The final output channel count of the current layer becomes COUT*t, effectively expanding the input channels of the next layer.

[0007] In ESBNN, both channel expansion and contraction operations use the same channel scaling factor t (t∈Z+), ensuring that the expansion and contraction factors are equal. This guarantees that the computational complexity of binary convolution remains unchanged. We call an ESBNN with equal expansion and contraction factors symmetric. The paper lacks a discussion on the information representation capability and computational complexity of binary convolution when the expansion and contraction factors are unequal, i.e., when the ESBNN is asymmetric. Summary of the Invention

[0008] The technical problem this invention aims to solve is the lack of flexibility in existing symmetric channel expansion and contraction techniques. Existing ESBNN techniques use a fixed expansion-contraction ratio, which, while improving binary convolution performance, fails to balance computational complexity with model accuracy, thus limiting their application in various scenarios.

[0009] To address the aforementioned technical problems, this invention proposes an asymmetric expanding-and-shrinking binary neural network (AESBNN) technique for binary convolutional layers. Its core innovation lies in adding an asymmetric ratio (denoted as r, used to control the size of the binary convolutional kernel) to the channel scaling factor t, allowing the expanding-and-shrinking ratio to be flexibly set according to actual needs, rather than being constantly equal to 1.

[0010] Based on the above concept, the technical solution of this invention is as follows: During the forward propagation process, the input channels of the binary convolutional layer are first expanded to t times their original size through a channel replication module. Then, the number of output channels is adjusted to 1 / (t*r) of the original size by adjusting the output dimension of the convolutional kernel. Finally, the output channels are adjusted to match the subsequent network layers through the channel replication module (by default, t²*r copies are made and spliced ​​along the channels). By independently and asymmetrically configuring t and r (i.e., r≠1), precise control over the representational capability and computational complexity of the binary convolutional layer is achieved.

[0011] The beneficial effects of this invention include: providing a flexible accuracy-efficiency balance strategy: when the asymmetry ratio r > 1, the system prioritizes computational efficiency, significantly reducing computational complexity at the cost of minimal accuracy loss. When the asymmetry ratio r < 1, the system prioritizes model performance, further enhancing the representational capability of binary convolution by appropriately increasing computational overhead, thereby achieving model accuracy superior to symmetric designs. Furthermore, this technology, as a general-purpose module, can be integrated into various deep neural networks based on binary convolution (such as image recognition and image super-resolution), improving the flexibility of network design. Attached Figure Description

[0012] Figure 1 This is used to illustrate the computation process of BBCU, where B is the batch size of the feature map (this parameter is only related to training), CIN is the number of input feature map channels, COUT is the number of output feature map channels (in BBCU, CIN and COUT are equal), and H and W are the height and width of the feature map, respectively.

[0013] Figure 2 This is used to illustrate the calculation process of ESBBCU, where t is the channel scaling factor.

[0014] Figure 3 This is used to illustrate the calculation process of AESBBCU, where r is the asymmetric ratio.

[0015] Figure 4 Used to illustrate the network structure of FSRCNN.

[0016] Figure 5 This is used to illustrate the computation process of the Mapping layer after replacing the standard convolutional layer with BBCU, where m is the number of cascaded convolutional layers in the Mapping.

[0017] Figure 6 This is used to illustrate the computation process of the Mapping layer after replacing the standard convolutional layer with ESBCU.

[0018] Figure 7 This is used to illustrate the computation process of the Mapping layer after replacing the standard convolutional layer with AESBCU.

[0019] Figure 8 This is used to illustrate the PSNR values ​​of four models—FSRCNN, FSRCNN (BBCU), FSRCNN (ESBCU), and FSRCNN (AESBCU)—on the Set5 test set at different magnifications. Detailed Implementation

[0020] To further clarify the technical solution and advantages of the present invention, the present invention will be described in more detail below with reference to the accompanying drawings and specific embodiments.

[0021] In their paper "BASIC BINARY CONVOLUTION UNIT FORBINARIZED IMAGE RESTORATION NETWORK" published at ICLR 2023, Bin Xia et al. explored the properties of three key components of BNNs, including residual connections, batch normalization, and activation functions. Based on this analysis, they designed a powerful Basic Binary Convolution Unit (BBCU). Figure 1 As shown, we selected BBCU as the baseline model. By introducing channel expansion and contraction techniques into BBCU, we constructed a model as follows: Figure 2 The symmetric version of the Expanding-and-Shrinking Binary Convolution Unit (ESBCU) shown is similar to... Figure 3 The asymmetric expanding-and-shrinking binary convolution unit (AESBCU) proposed in this invention is shown. The following experiments aim to compare and analyze the performance differences among BBCU, ESBCU, and AESBCU.

[0022] Taking the Fast Super-Resolution Convolutional Neural Network (FSRCNN) proposed by Chao Dong et al. in their ECCV 2016 paper "Accelerating the Super-Resolution Convolutional Neural Network" as an example, the FSRCNN network structure is as follows: Figure 4 As shown, the network has a mapping layer containing four standard convolutional layers configured as (k=3, CIN=12, COUT=12), which can be directly replaced by BBCU, ESBCU, and AESBCU.

[0023] To verify the effectiveness of this invention, the mapping layer of the FSRCNN network is binarized in this embodiment. We replace the standard convolutional layer with the base module (BBCU), the symmetric expansion-shrinking module (ESBCU), and the asymmetric expansion-shrinking module (AESBCU) proposed in this invention. All modifications follow the same input-output specification: accepting a feature map of size (B, 12, H, W) from the output of the shrinking layer, and finally outputting a feature map of size (B, 12, H, W), ensuring compatibility with subsequent expanding layers in the network.

[0024] Option 1 is the baseline method, such as Figure 5 As shown, four cascaded BBCU modules are used, each configured as (k=3, CIN=12, COUT=12).

[0025] Option 2 uses a symmetrical ESBCU module, such as Figure 6 As shown, each module is configured as (k=3, CIN=12, COUT=12, rep=t), where the channel scaling factor t can take values ​​of 2, 3, 4, or 6. This scheme first expands the number of input channels to 12*t, then performs binary convolution. Due to its asymmetric ratio of 1 (i.e., the output channels are restored to COUT=12*t through a copy operation), its computational complexity remains consistent with the BBCU baseline. The output feature map is obtained by truncating the first 12 channels of the output feature map of the last ESBCU module.

[0026] Option 3 utilizes the AESBCU module proposed in this invention, such as... Figure 7As shown, the configuration is (k=3, CIN=12, COUT=12, rep=t, rate=r), where t takes values ​​within the same range as above, and an asymmetric ratio r (which can be 2 or 4) is introduced. The key difference is that the shrinkage of the convolutional kernel is controlled by both the scaling factor t and the asymmetric ratio r, making the number of feature map channels after binary convolution COUT / t / r. Therefore, the computational complexity of this binary convolutional layer is no longer constant but significantly reduced, approximately 1 / r of BBCU or ESBCU. For example, when r=2, the computational cost is reduced to half, thus effectively reducing model complexity with acceptable accuracy loss. The output feature map is obtained by extracting the first 12 channels of the output feature map from the last AESBCU module.

[0027] The constructed FSRCNN (ESBCU), FSRCNN (ESBCU), FSRCNN (BBCU), and full-precision FSRCNN networks were trained under the same conditions, and the results are as follows: Figure 8 As shown in the figure, the analysis reveals that after binarization of the mapping layer, the PSNR of the model generally decreases compared to the full-precision FSRCNN, but both ESBCU and AESBCU show better performance than the basic BBCU. In the symmetric structure, when the channel scaling factor t=3, ESBCU achieves the best PSNR, leading BBCU by 0.19dB, 0.12dB, and 0.19dB in the ×2 to ×4 super-resolution tasks, respectively, indicating that a moderate channel expansion / contraction factor effectively improves the representational ability of binary convolution.

[0028] The asymmetric AESBCU structure proposed in this invention exhibits unique advantages. When the expansion factor t=3 and the asymmetry ratio r=2 are set, AESBCU achieves considerable performance improvement while maintaining a lightweight structure. Its PSNR is superior to BBCU by 0.08dB, equal to it, and ahead by 0.09dB in super-resolution tasks ranging from ×2 to ×4, respectively. Although its accuracy is slightly lower than the optimally configured ESBCU (by 0.10–0.12dB), the key point is that AESBCU only requires about half the computational complexity (1 / r, or 1 / 2, of ESBCU and BBCU) to achieve similar model accuracy.

[0029] In summary, the asymmetric channel expansion and contraction technique proposed in this invention for binary convolutional layers achieves superior model performance compared to BBCU while sacrificing acceptable accuracy. It significantly reduces the computational cost of binary convolution and can be flexibly integrated into various deep neural networks as a general binary convolutional layer optimization technique.

[0030] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. An asymmetric channel expansion and contraction method applied to binary convolutional layers, characterized in that, Includes the following steps: Input channel expansion: The number of input channels of the binary convolutional layer is expanded from CIN to CIN*t, where t (t∈Z) + ) is the channel scaling factor; Binary convolution step: Perform a convolution operation on the expanded input feature map using a binary convolution kernel, wherein the number of output channels of the binary convolution kernel is shrunk to COUT / (t*r), where COUT is the original number of output channels, and r (r∈R) + The output channel adjustment steps are as follows: Channel duplication and batch normalization are performed on the feature map after binary convolution to adjust the number of output channels to match the subsequent network layers. The channel duplication operation includes creating the feature map using the t... 2 *r copies are created and stitched together along the channel dimension.

2. The method according to claim 1, characterized in that, The asymmetry ratio r is greater than 1, which is used to reduce the computational complexity of the binary convolutional layer. The asymmetry ratio r is less than 1, which is used to improve the representational ability of the binary convolutional layer.

Images