End-to-end binocular stereo matching network with minimal computation based on full binary convolution
Through the fully binary convolution and binary upsampling module, a lightweight end-to-end binocular stereo matching network is designed, which solves the problem of large computing in the existing technology and realizes the resource-constrained equipment. Efficient deployment and good error performance on SceneFlow and KITTI datasets.
Patent Information
- Application Number
- CN202210408581.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-19
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2042-04-19
AI Technical Summary
The existing deep learning binocular stereo matching scheme is computationally large and difficult to effectively deploy on devices with limited computing resources. Especially in applications such as autonomous driving and augmented reality, a lightweight binocular that can greatly reduce the number of calculations is needed. Stereo matching scheme.
Using a lightweight end-to-end binocular stereo matching network based on full binary convolution, the calculation amount is reduced through binary operation and binary convolution module, and the feature extraction and coding method are optimized. Cost-quantity aggregation, a dedicated information protection binary upsampling module and binary encoding method are designed to replace all convolution operations as binary operations.
It significantly reduces the overhead of the model, and can achieve more than 10 times of savings compared to the full-precision method. At the same time, the errors on the SceneFlow and KITTI datasets perform well, meeting the requirements of algorithm accuracy and resource limitations.
Smart Images

Figure CN114819074B_ABST
Abstract
Description
Technical Field
[0001] The invention proposes an end-to-end binocular stereo matching network with very little computational effort based on full binary convolution, which is used for depth estimation tasks and belongs to the field of computer vision stereo matching. Background Art
[0002] In complex computer vision tasks such as autonomous driving, augmented reality, and robot navigation, correctly obtaining the depth information of the image is the key to the technology, and binocular stereo matching has always been the focus of research due to its high accuracy and low cost. At present, many binocular stereo matching solutions using deep learning have greatly improved the accuracy of data processing, but at the same time, these solutions have more and more parameters, more and more computing requirements, and deeper network structures. Applications that often deploy depth estimation usually run on mobile devices and embedded platforms, including drones, vehicles, smartphones, etc. These devices with limited computing resources often have requirements for low power consumption and small memory usage. Therefore, a lightweight binocular stereo matching solution that can greatly reduce the number of calculations is needed. Summary of the invention
[0003] Purpose of the invention: In order to solve the problem of excessive computational complexity of existing stereo matching solutions, a lightweight end-to-end binocular stereo matching network based on full binary convolution is proposed.
[0004] Technical solution:
[0005] A binary convolution module dedicated to stereo matching includes two types: binary 2D convolution and binary 3D convolution. The construction method of the convolution module includes the following steps:
[0006] When the input and output feature maps are 3D data:
[0007] Step A1: Binarization of the input feature map: The input feature map is processed by the RSign2D sign function and binarized to -1 / +1 after adding the corresponding bias values in the channel dimension.
[0008] Step A2: Binarization of weights: First, find the mean of the weights in the channel dimension as the channel scaling factor, and then use the Sign function to binarize the weights to -1 / +1;
[0009] Step A3: Binary convolution operation: convolve the binary input feature map obtained in step A1 with the binary weight obtained in step A2, and then multiply it by the channel scaling factor obtained in step A2 in the channel dimension to obtain the output of the binary convolution;
[0010] Step A4: Perform 2D batch normalization on the binary convolution output of step A3;
[0011] Step A5: Activate the output of step A4 using the RPRelu2D activation function, adjust the distribution of the output through the learnable bias activation function, and obtain the final output of the binary 2D convolution module.
[0012] When the input and output feature maps are 4-dimensional data:
[0013] Step B1: Binarization of the input feature map: The input feature map is processed by the RSign3D sign function and binarized to -1 / +1 after adding the corresponding bias values in the channel dimension.
[0014] Step B2: Binarization of weights: First, find the mean of the weights in the channel dimension as the channel scaling factor, and then use the Sign function to binarize the weights to -1 / +1;
[0015] Step B3: Binary convolution operation: perform a convolution operation on the binary input feature map obtained in step B1 and the binary weight obtained in step B2, and then multiply the channel scaling factor obtained in step B2 in the channel dimension to obtain the output of the binary convolution;
[0016] Step B4: Perform 3D batch normalization on the binary convolution output of step B3;
[0017] Step B5: Activate the output of step B4 using the RPRelu3D activation function, adjust the distribution of the output through the learnable bias activation function, and obtain the final output of the binary 3D convolution module.
[0018] An information protection binary upsampling module dedicated to stereo matching includes two types: binary 2D upsampling and binary 3D upsampling. The upsampling method includes the following steps:
[0019] When the input feature map and output feature map are 3D data:
[0020] Step C1: Use bilinear interpolation to increase the input feature map resolution by 1 times, and select pixel corner alignment as the alignment method;
[0021] Step C2: The interpolation result of step C1 is operated by the binary 2D convolution module to obtain a tensor with the same number of channels as the input feature map.
[0022] Step C3: Concatenate the result of step C2 with the input feature map to effectively combine the shallow and deep features of the network, and at the same time use more feature channels to alleviate the problem of feature homogeneity.
[0023] Step C4: The cascade result of step C3 is processed again using the binary 2D convolution module to convert the output channel into a set value in the network model.
[0024] When the input feature map and output feature map are 4-dimensional data:
[0025] Step D1: Use bilinear interpolation to increase the input feature map resolution by 1 times, and select pixel corner alignment as the alignment method;
[0026] Step D2: The interpolation result of step D1 is operated using the binary 3D convolution module to obtain a tensor with the same number of channels as the input feature map.
[0027] Step D3: Add the input feature map as the residual path to the result obtained in step B2 to supplement the information loss.
[0028] A binary encoding method for an input layer, the binary encoding method comprising the following steps:
[0029] Step E1: Encode the input activation value (0-255) using 8-bit binary coding, thereby effectively expanding the number of channels in the input layer and improving the representation ability of the input layer after binarization;
[0030] Step E2: Binarization operation on weights: Use the Sign function to binarize the weights to -1 / +1;
[0031] Step E3: Binary convolution operation: perform a convolution operation on the binary coded input obtained in step E1 and the binary weight obtained in step E2;
[0032] Step E4: Perform weighted summation on each channel of the binary convolution output results of step E3, and the weighting factor of the i-th channel corresponds to 2 i .
[0033] Step E5: performing 2D batch normalization processing on the weighted summation result of step E4;
[0034] Step E6: Activate the output of step E5 using the RPRelu activation function, adjust the distribution of the output through the learnable biased activation function, and obtain the final output of the binary encoding method.
[0035] A lightweight end-to-end binocular stereo matching network based on full binary convolution, the input is a left image and a right image taken by a binocular camera, and the stereo matching network execution steps include:
[0036] Step 1: Use the binary encoding method to complete the binarization of the input convolutional layer;
[0037] Step 2: Using the twin feature extraction network constructed by the binary 2D convolution module and the binary 2D upsampling module, extract the feature structures of the left and right images to obtain the corresponding left and right feature maps;
[0038] Step 3: Construct the cost volume of the left and right feature maps obtained in step 2 according to the group correlation;
[0039] Step 4: using the cost volume aggregation network constructed by the binary 3D convolution module and the binary 3D upsampling module, the cost volume obtained in step 3 is aggregated to obtain an initial disparity;
[0040] Step 5: Using the disparity optimization network constructed by the binary 2D convolution module and the binary 2D upsampling module, perform fine disparity correction on the initial disparity obtained in step 4 to obtain the final output.
[0041] Furthermore, the step 2 is specifically as follows: after the original input image passes through 6 convolution layers with a convolution step size of 2, it will be quickly downsampled to 1 / 64 of the original size; then, it will be restored to 1 / 8 of the original size through 3 binary 2D upsampling modules; in this process, all feature maps with a size of 1 / 8 of the original size will be cascaded together to form a 384-channel feature map to construct the cost volume.
[0042] Furthermore, the step 4 is specifically as follows: the cost volume aggregation network is a U-net-like structure, the cost volume first obtains the initial aggregation features at low resolution through 6 layers of binary 3D convolution, then is upsampled and restored through 3 binary 3D upsampling modules, and finally is weighted disparity regression is performed through the SoftMax function to obtain the initial disparity estimate.
[0043] Furthermore, the step 5 is specifically as follows: at a resolution scale of 1 / 2 of the original input image, the right input image is fused with the preliminary disparity to generate a new left input image. The difference between the original left input image and the newly generated input image is then calculated, and the disparity information to be compensated is obtained by using this difference through 7 layers of binary 2D convolution and 4 binary 2D upsampling modules.
[0044] Beneficial effects: The binary upsampling module of the present invention effectively improves the feature quality and avoids the phenomenon of sparse effective information caused by filling in 0 during the binary deconvolution process. In addition, by reasonably designing the binary encoding method of the input layer, the problem of the input layer relying on floating-point convolution is solved while maintaining accuracy. Based on the above module design, all convolution operations in PBCStereo can be replaced by binary operations, which greatly reduces the cost of model calculation. Compared with the full-precision method, it can save more than 10 times the amount of calculation. At the same time, the endpoint error on the public dataset SceneFlow is 1.84, the three-pixel error percentage on KITTI 2012 is 4.46%, and the three-pixel error percentage on KITTI 2015 is 4.73%, which meets the requirements of algorithm accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1 Schematic diagram of binary convolution module:
[0046] Among them, C, H, and W represent the number of channels, height, and width of the input feature map, respectively, and D represents the disparity dimension value. Figure 1 (a) is a schematic diagram of the binary 2D convolution module corresponding to the input feature map of 3D data. Figure 1 (b) is a schematic diagram of a binary 3D convolution module corresponding to a 4-dimensional input feature map;
[0047] Figure 2 This is a schematic diagram of the binary upsampling module:
[0048] Among them, C, H, and W represent the number of channels, height, and width of the input feature map respectively, D represents the disparity dimension value, and the subscript N+1 means that the network has completed downsampling N+1 times. The dimensional conversion of the feature map is indicated in the figure. Figure 2 (a) is a schematic diagram of the binary 2D upsampling module corresponding to the input feature map of 3D data. Figure 2 (b) is a schematic diagram of a binary 3D upsampling module corresponding to a 4-dimensional input feature map;
[0049] Figure 3 It is a binary encoding method for the input layer;
[0050] Figure 4 This is a schematic diagram of the PBCStereo binary feature extraction network;
[0051] Figure 5 This is a schematic diagram of the PBCStereo binary cost aggregation network;
[0052] Figure 6 Schematic diagram of the PBCStereo binary disparity optimization network. DETAILED DESCRIPTION
[0053] The present invention will be further explained below in conjunction with the accompanying drawings.
[0054] A binary 2D convolution module dedicated to stereo matching, such as Figure 1 As shown in (a), when the input and output feature maps are 3D data:
[0055] Step A1: Binarization of the input feature map: The input feature map is processed by the RSign2D sign function and binarized to -1 / +1 after adding the corresponding bias values in the channel dimension.
[0056] Step A2: Binarization of weights: First, find the mean of the weights in the channel dimension as the channel scaling factor, and then use the Sign function to binarize the weights to -1 / +1;
[0057] Step A3: Binary convolution operation: convolve the binary input feature map obtained in step A1 with the binary weight obtained in step A2, and then multiply it by the channel scaling factor obtained in step A2 in the channel dimension to obtain the output of the binary convolution;
[0058] Step A4: Perform 2D batch normalization on the binary convolution output of step A3;
[0059] Step A5: Activate the output of step A4 using the RPRelu2D activation function, adjust the distribution of the output through the learnable bias activation function, and obtain the final output of the binary 2D convolution module.
[0060] like Figure 1 As shown in (b), when the input and output feature maps are 4-dimensional data:
[0061] Step B1: Binarization of the input feature map: The input feature map is processed by the RSign3D sign function and binarized to -1 / +1 after adding the corresponding bias values in the channel dimension.
[0062] Step B2: Binarization of weights: First, find the mean of the weights in the channel dimension as the channel scaling factor, and then use the Sign function to binarize the weights to -1 / +1;
[0063] Step B3: Binary convolution operation: perform a convolution operation on the binary input feature map obtained in step B1 and the binary weight obtained in step B2, and then multiply the channel scaling factor obtained in step B2 in the channel dimension to obtain the output of the binary convolution;
[0064] Step B4: Perform 3D batch normalization on the binary convolution output of step B3;
[0065] Step B5: Activate the output of step B4 using the RPRelu3D activation function, adjust the distribution of the output through the learnable bias activation function, and obtain the final output of the binary 3D convolution module.
[0066] An information-preserving binary upsampling module dedicated to stereo matching, such as Figure 2 As shown in (a), when the input feature map and the output feature map are 3D data:
[0067] Step C1: Use bilinear interpolation to increase the input feature map resolution by 1 times, and select pixel corner alignment as the alignment method;
[0068] Step C2: The interpolation result of step C1 is operated by the binary 2D convolution module to obtain a tensor with the same number of channels as the input feature map.
[0069] Step C3: Concatenate the result of step C2 with the input feature map to effectively combine the shallow and deep features of the network, and at the same time use more feature channels to alleviate the problem of feature homogeneity.
[0070] Step C4: The cascade result of step C3 is processed again using the binary 2D convolution module to convert the output channel into a set value in the network model.
[0071] like Figure 2 As shown in (b), when the input feature map and the output feature map are 4-dimensional data:
[0072] Step D1: Use bilinear interpolation to increase the input feature map resolution by 1 times, and select pixel corner alignment as the alignment method;
[0073] Step D2: The interpolation result of step D1 is operated using the binary 3D convolution module to obtain a tensor with the same number of channels as the input feature map.
[0074] Step D3: Add the input feature map as the residual path to the result obtained in step D2 to supplement the information loss.
[0075] like Figure 3 As shown, a binary encoding method for an input layer, the binary encoding method includes:
[0076] Step E1: Encode the input activation value (0-255) using 8-bit binary coding, thereby effectively expanding the number of channels in the input layer and improving the representation ability of the input layer after binarization;
[0077] Step E2: Binarization operation on weights: Use the Sign function to binarize the weights to -1 / +1;
[0078] Step E3: Binary convolution operation: perform a convolution operation on the binary coded input obtained in step E1 and the binary weight obtained in step E2;
[0079] Step E4: Perform weighted summation on each channel of the binary convolution output results of step E3, and the weighting factor of the i-th channel corresponds to 2 i .
[0080] Step E5: performing 2D batch normalization processing on the weighted summation result of step E4;
[0081] Step E6: Activate the output of step E5 using the RPRelu activation function, adjust the distribution of the output through the learnable biased activation function, and obtain the final output of the binary encoding method.
[0082] A lightweight end-to-end binocular stereo matching network based on full binary convolution, the input is a left image and a right image taken by a binocular camera, and the stereo matching network execution steps include:
[0083] Step 1: Adoption Figure 3 The binary encoding method completes the binarization of the input convolutional layer;
[0084] Step 2: Adoption Figure 1 The binary convolution module and Figure 2 The twin feature extraction network constructed by the binary upsampling module. The feature extraction network structure is as follows Figure 4 As shown in the figure, the feature structures of the left and right images are extracted. After the original input image passes through 6 convolution layers with a convolution step of 2, it will be quickly downsampled to 1 / 64 of the original size; then, it will be restored to 1 / 8 of the original size through 3 binary 2D upsampling modules; in this process, all feature maps with a size of 1 / 8 of the original size will be cascaded together to form a 384-channel feature map to construct the cost volume;
[0085] Step 3: Construct the cost volume of the left and right feature maps obtained in step 2 according to the group correlation;
[0086] Step 4: Adoption Figure 1 The binary convolution module and Figure 2 The cost volume aggregation network constructed by the binary upsampling module. The cost volume aggregation network structure is as follows Figure 5 As shown in the figure, it is a U-net-like structure. The cost volume first passes through 6 layers of binary 3D convolution to obtain the initial aggregation features at low resolution, then passes through 3 binary 3D upsampling modules to upsample and restore, and finally passes through the SoftMax function to perform weighted disparity regression to obtain the initial disparity estimate.
[0087] Step 5: Adoption Figure 1 The binary convolution module and Figure 2 The disparity optimization network constructed by the binary upsampling module. The disparity optimization network structure is as follows Figure 6 As shown in Figure 1, the right input image is fused with the preliminary disparity at a resolution scale of 1 / 2 of the original input image to generate a new left input image. The difference between the original left input image and the newly generated input image is then calculated, and the disparity information that needs to be compensated is obtained through 7 layers of binary 2D convolution and 4 binary 2D upsampling modules. The initial disparity obtained in step 4 is finely corrected to obtain the final output.
[0088] The present invention has been publicly tested on SceneFlow and KITTI datasets, and the test results are compared with DispNet, StereoNet, MABNet_tiny, SGM-Net, StereoBit and other solutions as shown in Table 1. From Table 1, it can be found that PBCStereo achieves considerable performance at a very low computational cost.
[0089] Table 1 Comparison of stereo matching network performance evaluation based on SceneFlow and KITTI datasets
[0090]
[0091] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. An end-to-end binocular stereo matching network with minimal computational effort based on full binary convolution, characterized in that: The input of the stereo matching network is the left and right images taken by the binocular camera. The convolution process is all binary convolution. The execution steps include the following steps: Step 1: Use the binary encoding method to complete the binarization of the input convolution layer; Step 2: Use the twin feature extraction network constructed by the binary 2D convolution module and the binary 2D upsampling module to extract the feature structures of the left and right images and obtain the corresponding left and right feature maps; Step 3: Construct the cost volume of the left and right feature maps obtained in step 2 according to the group correlation; Step 4: A cost volume aggregation network constructed by a binary 3D convolution module and a binary 3D upsampling module is used to aggregate the cost volume obtained in step 3 to obtain the initial disparity; Step 5: A disparity optimization network constructed by a binary 2D convolution module and a binary 2D upsampling module is used to perform fine disparity correction on the initial disparity obtained in step 4 to obtain the final output; The input and output feature maps are 3D data. The construction method of the binary 2D convolution module in step 2 includes the following steps: Step A1: Binarization of the input feature map: The input feature map is processed by the RSign2D sign function and binarized to -1 / +1 after adding the corresponding bias values in the channel dimension. Step A2: Binarization of weights: First, find the mean of the weights in the channel dimension as the channel scaling factor, and then use the Sign function to binarize the weights to -1 / +1; Step A3: Binary convolution operation: convolve the binary input feature map obtained in step A1 with the binary weight obtained in step A2, and then multiply it by the channel scaling factor obtained in step A2 in the channel dimension to obtain the output of the binary convolution; Step A4: Perform 2D batch normalization on the binary convolution output of step A3; Step A5: Activate the output of step A4 using the RPRelu2D activation function, adjust the distribution of the output through the learnable bias activation function, and obtain the final output of the binary 2D convolution module; The input and output feature maps are 3D data. The upsampling method based on the binary 2D upsampling module in step 2 includes the following steps: Step B1: Use bilinear interpolation to increase the input feature map resolution by 1 times, and select pixel corner alignment as the alignment method; Step B2: The interpolation result of step B1 is operated by the binary 2D convolution module to obtain a tensor with the same number of channels as the input feature map; Step B3: Concatenate the result of step B2 with the input feature map, so as to effectively combine the shallow features and deep features of the network, and use more feature channels to alleviate the problem of feature homogeneity; Step B4: the cascade result of step B3 is processed again using the binary 2D convolution module to convert the output channel into a set value in the network model; The step 4 is specifically as follows: the cost volume aggregation network is a U-net-like structure, the cost volume first obtains the initial aggregation features at low resolution through 6 layers of binary 3D convolution, then is upsampled and restored through 3 binary 3D upsampling modules, and finally is weighted disparity regression is performed through the SoftMax function to obtain the initial disparity estimation; Specifically, step 5 comprises: fusing the right input image with the preliminary disparity at a resolution scale of 1 / 2 of the original input image to generate a new left input image, and then calculating the difference between the original left input image and the newly generated input image, and using this difference to obtain the disparity information that needs to be compensated through 7 layers of binary 2D convolution and 4 binary 2D upsampling modules.
2. The end-to-end binocular stereo matching network with minimal computational effort based on full binary convolution according to claim 1, characterized in that: The binary encoding method in step 1 includes the following steps: Step 1.1: Encode the input activation value using 8-bit binary coding, thereby effectively expanding the number of channels in the input layer and improving the representation ability of the input layer after binarization; Step 1.2: Use the Sign function to binarize the weights to -1 / +1; Step 1.3: Perform a convolution operation on the binary coded input obtained in step 1.1 and the binary weight obtained in step 1.2; Step 1.4: Perform weighted summation on each channel of the binary convolution output results of step 1.3, and the weighting factor of the i-th channel corresponds to 2 i ; Step 1.5: Perform 2D batch normalization on the weighted summation result of step 1.4; Step 1.6: Use the RPRelu activation function to activate the output of step 1.5, adjust the distribution of the output through the learnable bias activation function, and obtain the final output of the binary encoding method.
3. The end-to-end binocular stereo matching network with minimal computational effort based on full binary convolution according to claim 1, characterized in that: The input and output feature maps are 4-dimensional data. The construction method of the binary 3D convolution module in step 4 includes the following steps: Step C1: Binarization of the input feature map: The input feature map is processed by the RSign3D sign function and binarized to -1 / +1 after adding the corresponding bias values in the channel dimension. Step C2: Binarization of weights: First, find the mean of the weights in the channel dimension as the channel scaling factor, and then use the Sign function to binarize the weights to -1 / +1; Step C3: Binary convolution operation: convolve the binary input feature map obtained in step C1 with the binary weight obtained in step C2, and then multiply it by the channel scaling factor obtained in step B2 in the channel dimension to obtain the output of the binary convolution; Step C4: Perform 3D batch normalization on the binary convolution output of step C3; Step C5: Activate the output of step C4 using the RPRelu3D activation function, adjust the distribution of the output through the learnable bias activation function, and obtain the final output of the binary 3D convolution module.
4. The end-to-end binocular stereo matching network with minimal computational effort based on full binary convolution according to claim 1, characterized in that: The input and output feature maps are 4-dimensional data. The upsampling method based on the binary 3D upsampling module in step 4 includes the following steps: Step D1: Use bilinear interpolation to increase the input feature map resolution by 1 times, and select pixel corner alignment as the alignment method; Step D2: The interpolation result of step D1 is operated by the binary 3D convolution module to obtain a tensor with the same number of channels as the input feature map; Step D3: Add the input feature map as the residual path to the result obtained in step D2 to supplement the information loss.
5. The end-to-end binocular stereo matching network with minimal computational effort based on full binary convolution according to claim 1, characterized in that: Specifically, step 2 is as follows: after the original input image passes through 6 convolution layers with a convolution step of 2, it will be quickly downsampled to 1 / 64 of the original size; then, it will be restored to 1 / 8 of the original size through 3 binary 2D upsampling modules; in this process, all feature maps with a size of 1 / 8 of the original size will be cascaded together to form a 384-channel feature map to construct the cost volume.
Citation Information
Patent Citations
Binocular stereo vision matching method based on neural network and operation framework thereof
CN110738241A
Multi-branch adjustable bottleneck convolution module and end-to-end stereo matching network
CN111968168A
Binocular matching method and apparatus, device and storage medium
US20210042954A1
