Anti-noise stereo matching method based on dynamic convolution and noise volume construction

Abstract

The application discloses an anti-noise stereo matching method based on dynamic convolution and noise volume construction, combines a denoising task and stereo matching, and extracts features of left and right images containing noise in parallel by using a noise feature extraction network combined with a U-Net structure of dynamic convolution and a feature extraction module of an original stereo matching network; noise features obtained by using the denoising network are used to construct a noise volume, and an original disparity cost volume is reserved; a 3*3 convolution layer and a layer of ReLU activation function are used to fuse the noise volume and the original disparity cost volume; and a fusion cost volume obtained by fusing the noise volume and the original disparity cost volume is used to perform a cost aggregation process to obtain a disparity map. Compared with an ordinary anti-noise method of binocular stereo matching, the application can improve the anti-interference ability of the stereo matching network in a noise environment, and can significantly improve the accuracy and stability of disparity prediction in the noise environment.

Description

Technical Field

[0001] This invention relates to the field of stereo matching, noise resistance methods, and deep learning technology, specifically to a noise-resistant stereo matching method based on dynamic convolution and noise volume construction. Background Technology

[0002] Stereo matching is a core concept in binocular stereo vision. It refers to calculating the depth information of an object based on the disparity between pixels in an image (i.e., the pixel offset of corresponding points in the left and right images), thereby enabling the perception of the environment and distance measurement.

[0003] Integrating deep learning (DL) technology into binocular stereo matching algorithms enables autonomous image data parsing. Convolutional Neural Networks (CNNs) are designed in a way similar to human visual processing, using convolutional kernels to operate on each image pixel individually, thereby capturing local image features. Similarly, neurons in the human visual cortex perceive image features by receiving stimuli from local regions. Due to the excellent image processing capabilities, good nonlinearity, low error rate, and high robustness of CNNs, CNN-based binocular stereo matching algorithms have gradually become the mainstream research direction in binocular stereo matching.

[0004] In 2015, Zbontar et al. first proposed using CNNs to improve stereo matching. CNNs can learn more expressive and robust feature representations from input images, enabling stereo matching algorithms to better handle complex situations such as texture discontinuities, occlusion, and lighting variations. In 2016, Mayer et al.'s pioneering work introduced DispNetC, the first end-to-end stereo matching network based on CNNs, which takes left and right image pairs as input and outputs the corresponding disparity maps. In 2018, Chang et al. introduced a pyramid pooling module based on GC-Net, proposing the stereo matching network PSMNet. In 2021, Shen et al. designed CFNet, which uses concatenation and group correlation to generate multiple combined cost volumes at specific scales. Furthermore, the CFNet method employs variance-based uncertainty estimation to reduce the disparity search space at higher resolutions and optimizes disparity from coarse to fine, alleviating the problem of disparity imbalance. In 2024, ACVNet proposed using attentional connectivity, combining attention weights from relevance cues to increase matching-related information.

[0005] Although deep learning has become a new research trend in binocular stereo matching in recent years, and many deep learning-based stereo matching models have been proposed, image noise is inevitably introduced into real-world scenarios due to the left and right eye images acquired by binocular sensors. This noise may be caused by factors such as excessively high sensor operating temperatures over time, insufficient scene brightness, inherent sensor defects, or signal interference during image transmission. Noisy input images will inevitably affect the accuracy of stereo matching.

[0006] However, existing researchers have focused more on improving the prediction accuracy of the stereo matching network structure itself, neglecting the interference and impact of noise on stereo matching networks in real-world applications such as autonomous driving. In autonomous driving scenarios, binocular sensors introduce various noises, including: dark current noise generated by the current inside the sensor, thermal noise caused by the sensor's temperature rise, readout noise introduced by electronic components during image reading, and low-light noise generated in low-light environments. These noises can lead to inaccurate feature point detection and matching, thus affecting the accuracy of disparity calculation. In severe cases, significant noise can even cause the stereo matching results to deteriorate considerably.

[0007] While some methods have improved noise resistance in stereo matching, many have failed to consider the reasonable allocation of computational resources between the noise feature extraction network and stereo matching. This results in the denoising network consuming too much computational resources in the early stages, affecting subsequent matching performance. Furthermore, some methods neglect the construction of the noise volume, leading to the inclusion of complex noise information in the cost volume.

[0008] For example, patent 2021101627621 discloses an image fusion denoising method based on U-Net and pyramid modules. This scheme uses the U-Net structure to design the stereo matching structure and the FFDNet denoising network to denoise before stereo matching. However, using FFDNet for denoising in the early stage consumes a lot of computational resources and cannot guarantee the effective execution of subsequent matching steps. Another example is patent 2023115219383, which discloses a stereo matching method based on an attention mechanism and a noise-suppressed GRU. This scheme places the noise suppression step after constructing the cost volume, that is, it uses a GRU network with added noise suppression to optimize and output the high-channel cost volume separately, ignoring the impact of noise on the construction cost volume, which can easily cause left and right disparity matching errors. Summary of the Invention

[0009] Objective: This invention aims to address the interference and impact of noise on stereo matching networks in practical applications, providing a noise-resistant stereo matching method based on dynamic convolution and noise volume construction. This invention combines denoising and stereo matching into a single network, employing a lightweight U-Net structure based on dynamic convolution to extract noise information. This effectively eliminates noise without imposing computational burden on subsequent matching steps. In areas with significant noise interference, noise volume is used to acquire noise features and eliminate noise. In areas with no or minimal noise, the existing cost volume is used to aggregate and predict disparity. By combining these two cost volumes, the stereo matching task under noise interference can be achieved.

[0010] Technical solution: The present invention provides a noise-resistant stereo matching method based on dynamic convolution and noise volume construction, comprising the following steps:

[0011] Step 1: Input the original left and right noise images into the noise feature extraction network simultaneously. The noise feature extraction network is based on the U-Net structure and combined with dynamic convolutional layers. When extracting noise features during the stereo matching process, the noise feature extraction network shares weights between the left and right noise images.

[0012] Step 2: Construct a noise volume using the noise features obtained in Step 1 according to the group correlation method. The construction method is as follows: First, divide the noise features into N groups. If the number of channels of the noise features is C, then the number of channels in each group is C / N. Perform pixel-by-pixel correlation calculation on the features of each group. For each pixel, calculate the correlation between each group of noise features in the left image and the right image, and generate a group correlation cost volume. Combine the correlation results of all groups to form the final noise volume (the noise volume obtained in this invention contains noise information and depth information).

[0013] Step 3: For the original left and right noisy images, first use the EfficientNetV2-M network model to extract multi-scale features, and then construct the disparity cost volume;

[0014] Step 4: First, concatenate the noise volume obtained in Step 2 and the disparity cost volume obtained in Step 3 along the channel dimension. Then, add a 3×3 convolutional layer and a ReLU activation function layer sequentially for fusion to obtain the fused cost volume. In areas with high noise interference, noise features are obtained from the noise volume in the fused cost volume to eliminate noise. In areas with no noise or low noise, the original disparity cost volume in the fused cost volume can be used for aggregation and disparity prediction. Since the fused cost volume contains rich noise and disparity information, it can effectively perform stereo matching tasks in noisy environments.

[0015] Step 5: Perform cost aggregation on the fusion cost volume obtained in Step 4 to obtain the initial disparity map; construct the cascaded cost volume and combine it with the initial disparity map to refine the disparity in a coarse-to-fine manner to obtain the refined final disparity map.

[0016] Furthermore, the dynamic convolutional layer of the noise feature extraction network contains multiple convolutional kernels. By assigning different weights to different convolutional kernels, the attention mechanism is used to aggregate the convolutional kernels in a non-linear manner, thereby improving the network's representational ability.

[0017] The sensor is set as follows:

[0018] y = g(W T x+b)

[0019] In the above formula, W is the weight matrix, b is the bias, and g(·) represents the activation function;

[0020] The dynamic perceptron aggregates K linear functions. Let it be defined as shown in the following formula:

[0021]

[0022]

[0023] In the above formula, the aggregate weight and bias Sharing the same attention, π k It is the k-th linear function Attention weights; aggregation model It is a non-linear model. The dynamic convolutional layer of this invention is more flexible and can capture richer and more complex feature representations.

[0024] Furthermore, in step 5, during the process of obtaining the initial disparity map, the cost self-recombination (CSR) strategy is used to calculate the offset of each pixel in the original left image, adjust the cost distribution in the aggregated cost volume, and then use disparity regression to obtain the initial disparity map.

[0025] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0026] 1. This invention incorporates dynamic convolutional layers into a lightweight feature extraction network based on the U-Net structure, which can extract rich noise features and provide a foundation for subsequent steps.

[0027] 2. This invention utilizes the depth cues contained in the noise volume to combine the denoising task with stereo matching, which is beneficial for stereo matching in noisy scenes.

[0028] 3. The present invention combines the noise volume and the original disparity cost volume to obtain the fused cost volume, which contains noise information and disparity information. Therefore, the proposed network model can effectively perform stereo matching tasks in noisy environments. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the overall network structure of the present invention;

[0030] Figure 2 This is a schematic diagram of the dynamic convolutional layer structure of the present invention;

[0031] Figure 3 This is a diagram illustrating the fusion volume construction process of the present invention;

[0032] Figure 4 This is a schematic diagram of the stereo matching network training process of the present invention;

[0033] Figure 5 In this embodiment of the invention, GwcNet combines three denoising extraction networks to address the noise variance σ. 2 A parallax image with a resolution of 30.

[0034] Figure 6 In this embodiment of the invention, EGR-Stereo combines three denoising extraction networks to address the noise variance σ. 2 A parallax image with a resolution of 30. Detailed Implementation

[0035] The technical solution of the present invention will be described in detail below, but the scope of protection of the present invention is not limited to the embodiments described.

[0036] like Figure 1 As shown, the noise-resistant stereo matching method based on dynamic convolution and noise volume construction of the present invention includes the following steps:

[0037] Step 1: Input the original left and right noise images (using images related to the autonomous driving scene) into the noise feature extraction network at the same time. The noise feature extraction network is based on the U-Net structure and combined with dynamic convolutional layers. When extracting noise features in the stereo matching process, the noise feature extraction network shares weights between the left and right noise images.

[0038] Step 2: Construct a noise volume using the noise features obtained in Step 1 according to the group correlation method. The construction method is as follows: First, divide the noise features into N groups. If the number of channels of the noise features is C, then the number of channels in each group is C / N. Perform pixel-by-pixel correlation calculation on the features of each group. For each pixel, calculate the correlation between each group of noise features in the left image and the right image, generate a group correlation cost volume, and combine the correlation results of all groups to form the final noise volume.

[0039] Step 3: For the original left and right noisy images, first use the EfficientNetV2-M network model to extract multi-scale features, and then construct the disparity cost volume; for example, the disparity cost volume can be obtained by using the existing CFNet stereo matching network correlation method.

[0040] Step 4: First, concatenate the noise volume obtained in Step 2 and the disparity cost volume obtained in Step 3 along the channel dimension. Then, sequentially add a 3×3 convolutional layer and a ReLU activation function layer for fusion, finally obtaining the fused cost volume, as shown below. Figure 3 As shown;

[0041] Step 5: Perform cost aggregation on the fusion cost volume obtained in Step 4 to obtain the initial disparity map; construct the cascaded cost volume and combine it with the initial disparity map to refine the disparity in a coarse-to-fine manner to obtain the refined final disparity map.

[0042] This invention uses the EGR-Stereo stereo matching network as a prototype, combining denoising and stereo matching. It utilizes a noise feature extraction network with a U-Net structure incorporating dynamic convolutions and the feature extraction module of the original stereo matching network to extract features from noisy left / right images in parallel. The noise features obtained from the denoising feature network are used to construct a noise volume while preserving the original disparity cost volume. Then, convolutional layers and the ReLU activation function are used to fuse the noise volume and the original disparity cost volume. Finally, the fused cost volume is combined with a cost aggregation process to obtain the final disparity map. Compared to the traditional EGR-Stereo binocular stereo matching network, this invention improves the stereo matching network's robustness in noisy environments, significantly enhancing the accuracy and stability of disparity prediction in noisy conditions.

[0043] like Figure 2 As shown, the dynamic convolutional layer of the noise feature extraction network in this embodiment has multiple convolutional kernels. By assigning different weights to different convolutional kernels, the network's representational ability is improved by using an attention mechanism to aggregate the convolutional kernels in a non-linear manner.

[0044] The sensor is set as follows:

[0045] y = g(W T x+b)

[0046] In the above formula, W is the weight matrix, b is the bias, and g(·) represents the activation function;

[0047] The dynamic perceptron aggregates K linear functions. Let it be defined as shown in the following formula:

[0048]

[0049] In the above formula, the aggregate weight and bias Sharing the same attention, π k It is the k-th linear function Attention weights; aggregation model It is a non-linear model. Therefore, compared to a static perceptron, a dynamic perceptron is more flexible and can capture richer and more complex feature representations. Meanwhile, as... Figure 2 As shown, the dynamic convolutional layer uses K convolutional kernels and assigns weights {π} to each kernel. k The results are aggregated and then output using batch normalization and activation functions.

[0050] The dynamic convolution proposed here is a novel operator design, namely the dynamic convolution operator, which can improve the feature representation capability of the algorithm without increasing the number of floating point operations per second (FLOPs).

[0051] In step 5 above, during the process of obtaining the initial disparity map, the cost self-reconstruction (CSR) strategy is used to calculate the offset of each pixel in the original left image, adjust the cost distribution in the aggregated cost volume, and then use disparity regression to obtain the initial disparity map. This step is an improvement made by EGR-Stereo to CFNet. By generating the offset of each pixel through the cost self-reconstruction strategy, the disparity is redistributed, which can alleviate the problem of disparity smoothing.

[0052] Example

[0053] To verify the performance of this invention, this embodiment applies the technical solution of this invention to a specific dataset for verification. The dataset used is the SceneFlow and KITTI datasets.

[0054] The SceneFlow dataset contains 35,454 pairs of stereo images for training and 4,370 pairs for testing. Each image pair is provided with a dense and detailed real-world disparity map and camera parameter information. All images are 960×540 resolution. A subset consists of three scenes: FlyingThings3D, a scene with randomly typed objects, including numerous floating objects with rich detail; Driving, a street scene captured simulating car driving; and Monkaa, a scene depicting monkeys in a deep forest environment, involving closer targets.

[0055] The KITTI dataset consists of two parts: KITTI 2012 and KITTI 2015. Both datasets provide real-world images captured in driving scenarios. KITTI 2012 includes 194 training and 195 test image pairs, while KITTI 2015 includes 200 training and 200 test image pairs. In both datasets, LiDAR sensors provide sparse, true disparity values ​​for the training images.

[0056] Will Figure 1 The network model of this invention runs on the PyTorch deep learning framework and is trained using two NVIDIA A6000 GPUs. First, the model is pre-trained on a clean and noise-free SceneFlow dataset. After the network initially converges, training is performed on the σ-series dataset. 2 =10, σ 2 =30, σ 2 =60, σ 2 For four different levels of noise interference (e.g., 100), the KITTI dataset with corresponding levels of Gaussian noise was used to train the noise-resistant stereo matching network.

[0057] Then as Figure 4 As shown, the weights of the trained AE+EGR-Stereo network model are saved. For each level of noise interference, the KITTI 2015 test set with the corresponding noise level interference is used as the input for forward propagation. The performance of the AN-EGR-Stereo noise-resistant stereo matching network model of this invention is evaluated using metrics such as endpoint error (EPE) and D1-all.

[0058] Table 1 shows the D1-all results of the three denoising networks (GwcNet + GwcNet, GwcNet + DnCNN, and GwcNet + SwinIR) under different levels of noise interference. It should be noted that the noise volume was not included in this set of experiments.

[0059] Table 1

[0060] Specific methods <![CDATA[σ 2 =10]]> <![CDATA[σ 2 =30]]> <![CDATA[σ 2 =60]]> <![CDATA[σ 2 =100]]> DnCNN+GwcNet 3.16 6.52 9.33 19.14 SwinIR+GwcNet 3.09 7.69 15.91 31.63 This invention + GwcNet 3.04 3.91 4.25 10.63

[0061] As shown in Table 1, the results are represented by the D1-all index, which is the percentage (%) of pixels that exceed 3 pixels or exceed 5% of the true disparity. The smaller the index, the higher the accuracy of the disparity map, which means that the current network has better noise resistance.

[0062] This invention (a denoising method based on dynamic convolution) is significantly better than the other two, in σ 2With noise levels of 100, the error is 8.51% lower than the method combined with DnCNN and 21.0% lower than the method combined with SwinIR. Figure 5 As shown, the denoising method based on dynamic convolution proposed in this invention produces the fewest artifacts in the sky area at the top of the image, followed by the stereo matching method combined with DnCNN. Furthermore, the stereo matching method combining DnCNN and SwinIR both exhibited incorrect matching in the grass area at the bottom right corner, while the denoising method based on dynamic convolution designed in this invention performed stably in this area.

[0063] Table 2 shows the D1-all results of the +EGR-Stereo, DnCNN, and SwinIR denoising networks under different levels of noise interference. It should be noted that the noise volume was not included in this set of experiments.

[0064] Table 2

[0065] Specific methods <![CDATA[σ 2 =10]]> <![CDATA[σ 2 =30]]> <![CDATA[σ 2 =60]]> <![CDATA[σ 2 =100]]> DnCNN+EGR-Stereo 2.55 6.15 7.95 17.60 SwinIR+EGR-Stereo 2.41 7.49 14.37 28.95 This invention +EGR-Stereo 2.53 3.23 3.87 9.45

[0066] As shown in Table 2, the SwinIR-based method performs best when the signal-to-noise ratio is high and the noise is weak. However, as the signal-to-noise ratio gradually decreases, the error of this method increases. In contrast, the dynamic convolution-based denoising method proposed in this invention exhibits the most stable performance, and even when the noise variance reaches σ... 2 When the value is 100, it still has a significant advantage in noise immunity compared to the other two methods, and the error is 19.50% lower than that of the SwinIR-based method.

[0067] like Figure 6 As shown, the disparity map based on the EGR-Stereo framework performs better than GwcNet overall. This indicates that the stereo matching network combined with the dynamic convolution-based denoising method proposed in this invention can more effectively perform disparity prediction tasks when facing strong noise interference, and the denoising method of this invention has strong versatility. GwcNet and EGR-Stereo were selected as disparity prediction networks, and D1-all results under different levels of noise interference were obtained by adding a noise extraction module based on dynamic convolution, as shown in Table 3.

[0068] Table 3

[0069]

[0070] Taking EGR-Stereo as an example of a disparity prediction network, this embodiment constructs a noise volume within σ... 2 Under noise interference level 30, the D1-all error index was reduced by 0.25% compared to when no noise volume was constructed. Furthermore, when the noise variance reached σ... 2=100, this indicator decreased by 1.22%, therefore, the advantage of constructing noise volume becomes more obvious as noise gradually increases. It can be seen that the noise feature extraction network and noise volume construction module based on dynamic convolution proposed in this invention can improve the anti-interference ability of the stereo matching network in noisy environments, and can significantly improve the accuracy and stability of disparity prediction in noisy environments.

Claims

1. A noise-resistant stereo matching method based on dynamic convolution and noise volume construction, characterized in that, Includes the following steps: Step 1: Input the original left and right noise images into the noise feature extraction network simultaneously. The noise feature extraction network is based on the U-Net structure and combined with dynamic convolutional layers. When extracting noise features during stereo matching, the noise feature extraction network shares weights between the left and right noise images. Step 2: Construct a noise volume using the noise features obtained in Step 1 according to the group correlation method. The construction method is as follows: First, divide the noise features into N groups. If the number of channels of the noise features is C, then the number of channels in each group is C / N. Perform pixel-by-pixel correlation calculation on the features of each group. For each pixel, calculate the correlation between each group of noise features in the left image and the right image, generate a group correlation cost volume, and combine the correlation results of all groups to form the final noise volume. Step 3: For the original left and right noisy images, first use the EfficientNetV2-M network model to extract multi-scale features, and then construct the disparity cost volume; Step 4: First, concatenate the noise volume obtained in Step 2 and the disparity cost volume obtained in Step 3 along the channel dimension. Then, add a 3×3 convolutional layer and a ReLU activation function layer in sequence for fusion to finally obtain the fused cost volume. Step 5: Perform cost aggregation on the fusion cost volume obtained in Step 4 to obtain the initial disparity map; construct the cascaded cost volume and combine it with the initial disparity map to refine the disparity in a coarse-to-fine manner to obtain the refined final disparity map.

2. The noise-resistant stereo matching method based on dynamic convolution and noise volume construction according to claim 1, characterized in that, The noise feature extraction network has multiple convolutional kernels in its dynamic convolutional layer. By assigning different weights to different convolutional kernels, the attention mechanism is used to aggregate the convolutional kernels in a non-linear manner. The sensor is set as follows: y=g(W T x+b) In the above formula, W is the weight matrix, b is the bias, and g(·) represents the activation function; The dynamic perceptron aggregates K linear functions. Let it be defined as shown in the following formula: In the above formula, the aggregate weight and bias Sharing the same attention, π k It is the k-th linear function Attention weights; Aggregation Model It is a nonlinear model.

3. The noise-resistant stereo matching method based on dynamic convolution and noise volume construction according to claim 1, characterized in that, In step 5, during the process of obtaining the initial disparity map, the cost self-recombination (CSR) strategy is used to calculate the offset of each pixel in the original left image, adjust the cost distribution in the aggregated cost volume, and then use disparity regression to obtain the initial disparity map.

Images