Deep stereo matching method based on PSMNet optimized feature extraction

By splitting the convolutional layer and BN layer in PSMNet, combining the hysteresis attention mechanism and ASPP module, the feature extraction is optimized, and the problem of poor matching results in a specific region is solved, achieving more efficient feature extraction and Real-time improvement.

CN118570492BActive Publication Date: 2024-11-22CHANGCHUN INST OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411000882.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2024-11-22
Estimated Expiration
2044-07-25

AI Technical Summary

Technical Problem

The existing depth stereo matching method results in poor stereo matching results and poor real-time real-time depth estimation in the case of weak texture areas, repeated texture areas, reflection areas and occlusion areas.

Method used

Based on PSMNet optimized feature extraction method, the three layers of the convolution, normalization and linear correction unit ReLU are split into three basic convolution layers, and combined with the convbn layer, adding hysteresis attention mechanism and ASPP module to perform multiple Scale feature extraction reduces feature information loss and improves feature extraction effect in the region of interest.

Benefits of technology

It significantly improves the accuracy and effect of feature extraction, enhances the extraction ability of information at different scales, and improves feature extraction performance without excessively increasing the complexity of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118570492B_ABST
    Figure CN118570492B_ABST
Patent Text Reader

Abstract

The present invention belongs to the field of computer vision technology, and specifically relates to a deep stereo matching method based on PSMNet optimized feature extraction, wherein the three layers of convolution, normalization and linear correction unit ReLU in the original PSMNet spliced ​​by firstconv in the feature extraction class are separated respectively, and the convolution layer and BN layer of each part are combined by defining a convbn layer; the defined convbn layer performs convolution normalization processing on the initial image, and adds a linear correction unit ReLU to the initial image after convolution normalization processing, and extracts the first part of the deep image features of the initial image; based on the hysteresis attention mechanism algorithm, the first part of the deep image features are further extracted to obtain the second part of the deep image features; and the second part of the deep image features are subjected to multi-scale feature extraction through the ASPP module. The loss of feature information when being transmitted between layers is reduced, and the ability to extract information of different scales is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a deep stereo matching method based on PSMNet optimized feature extraction. Background Art

[0002] Depth estimation of stereo images is crucial for computer vision applications, including autonomous driving of vehicles, 3D model reconstruction, and object detection and recognition, which aims to find corresponding points in the left and right views of stereo images. However, there are still many challenges in depth estimation, such as weak texture areas, repeated texture areas, reflection areas, occlusion areas, etc., which will lead to poor stereo matching results.

[0003] Stereo matching basically includes four steps: cost calculation, cost aggregation, disparity calculation, and disparity refinement. Traditional algorithms have low accuracy but fast speed. Learning-based methods have gradually improved the accuracy of feature extraction, but require a lot of training time. At present, more and more scholars are paying attention to deep learning methods. Zbontar and LeCun use convolutional neural networks to extract features from stereo images. Compared with traditional manual feature extraction methods, the basic convolution layer can effectively improve the efficiency of feature extraction and has high robustness. However, this method still uses traditional pipelines in other steps, resulting in poor real-time performance. Mayer et al. first proposed an end-to-end stereo matching network called DispNet, which integrates all the steps of traditional algorithms into one network. It can directly obtain disparity maps by sending stereo image pairs to the network input with good performance. Chang et al. proposed a stereo matching network PSMNet, which uses a residual neural network as the backbone of the feature extraction part.

[0004] Therefore, how to improve the existing deep stereo matching method of stereo images to improve the accuracy of feature extraction and enhance the feature extraction effect is a technical problem that needs to be solved urgently. Summary of the invention

[0005] The purpose of the present invention is to provide a deep stereo matching method based on PSMNet optimized feature extraction, so as to improve the existing deep stereo matching method of stereo images, so as to improve the accuracy of feature extraction and enhance the feature extraction effect.

[0006] In order to achieve the above technical effects, the technical solution adopted by the present invention is as follows:

[0007] The deep stereo matching method based on PSMNet optimized feature extraction includes the following steps:

[0008] S1: Split the original PSMNet composed of convolution, normalization and linear correction unit ReLU in the feature extraction class into three basic convolution layers, and combine the convolution layer and BN layer of each part by defining the convbn layer;

[0009] S2: Perform convolution normalization on the initial image according to the defined convbn layer, and add a linear correction unit ReLU to the initial image after convolution normalization to extract the first part of the deep image features of the initial image;

[0010] S3: further extracting the first part of the depth image features based on the hysteresis attention mechanism algorithm to obtain the second part of the depth image features;

[0011] S4: Perform multi-scale feature extraction on the second part of the depth image features through the ASPP module, and output the final image features.

[0012] Preferably, step S1 also includes setting parameters for defining a convbn layer, and the set parameters include in_planes, out_planes, kernel_size, stride, pad, and dilation, wherein in_planes corresponds to the input image size, out_planes corresponds to the output image size, kernel_size corresponds to the convolution kernel size, stride corresponds to the convolution kernel step size, pad corresponds to the padding size around the image, and dilation corresponds to the convolution expansion rate.

[0013] Preferably, step S2 includes the following process:

[0014] S21: According to the constructed three-layer basic convolution layer, including the first layer of basic convolution layer, the second layer of basic convolution layer and the third layer of basic convolution layer, the feature extraction result obtained by interleaving the first layer of basic convolution layer and the second layer of basic convolution layer is added to the features extracted by the third layer of basic convolution layer;

[0015] S22: Construct the Runge-Kutta structure through the process of step S1. The specific formula is:

[0016] ;

[0017] in, It is the feature result output after the image passes through the first basic convolution layer. for The feature result of the second basic convolutional layer is added with 1.5 times the difference between the previous and next feature results.

[0018] Preferably, step S3 includes the following specific process:

[0019] S31: Construct the ResNet basic block. The basic block in the original PSMNet has four layers. The defined convbn layer builds two convolutional layers and one downsampling layer to reduce the dimension of the feature image.

[0020] S32: A CBAM attention mechanism is added to the second layer and the fourth layer of the ResNet basic block, wherein the CBAM attention mechanism includes a channel attention mechanism and a spatial attention mechanism, and the first part of the deep image features is input into the channel attention mechanism module;

[0021] S33: pass the output features into the spatial attention mechanism module;

[0022] S34: The CBAM attention mechanism is integrated with the hysteresis structure and connected after the deep image features in the first part;

[0023] S35: After adding the CBAM attention mechanism to the second and fourth ResNet basic blocks and fusing the hysteresis structure, the concat function and two additional convolutional layers are used to further refine the image feature parts that the CBAM attention mechanism pays more attention to.

[0024] Preferably, in step S32, the specific formula for inputting the first part of the deep image features into the channel attention mechanism module is as follows:

[0025] M c (F) = σ (MLP(AvgPool(F))+MLP(MaxPool(F)));

[0026] Among them, M c is the channel attention weight matrix, σ is the sigmoid activation function, the MLP module compresses and then expands the channel, AvgPool is the average pooling operation, MaxPool is the maximum pooling operation, and F is given an intermediate feature map as input.

[0027] Preferably, the specific formula for transferring the output features into the spatial attention mechanism module in step S33 is as follows:

[0028] ;

[0029] Among them, M s is the spatial attention weight matrix, A 7×7 convolution operation is selected, σ is the sigmoid activation function, AvgPool is the average pooling operation, MaxPool is the maximum pooling operation, and F is given an intermediate feature map as input;

[0030] In step S34, the CBAM attention mechanism is integrated with the hysteresis structure and connected after the first part of the deep image features. The formula of the hysteresis structure in the network is as follows:

[0031] ;

[0032] Among them, S n ,n=0,1...,N-1 represents the output of the nth convolutional block, For each convolution operation defined, Input is the input dimension, and α is a defined hyperparameter used to adjust the size of the hysteresis structure transmitting features in the reverse direction. A n-1 is the attention mask, learned via back-propagation in a self-supervised manner, S n and S n-1 It is the image result before and after output.

[0033] Preferably, step S4 includes the following specific process:

[0034] S41: Build the basic ASPP module, match the convbn layer with the ReLU linear activation unit. Since the ASPP module is a parallel structure, the dilation parameter in convbn is adjusted to its own size.

[0035] S42: construct a five-layer ASPP module, inherit the image features obtained in step S3, and set the expansion rates to 1, 6, 12, 18, and 24 respectively;

[0036] S43: Use a layer of concat splicing and lastconv convolution operation to fuse information features of different sizes together, and output the final image features through a layer of convolution.

[0037] The beneficial effects of the present invention include:

[0038] The deep stereo matching method based on PSMNet optimized feature extraction provided by the present invention comprises the following steps: firstconv in the feature extraction class is spliced ​​into three layers, namely, convolution, normalization and linear correction unit ReLU, and the convolution layer and BN layer of each part are combined by defining a convbn layer; the defined convbn layer performs convolution normalization processing on the initial image, and a linear correction unit ReLU is added to the initial image after the convolution normalization processing to extract the first part of deep image features of the initial image; the first part of deep image features are further extracted based on a hysteresis attention mechanism algorithm to obtain the second part of deep image features; and the second part of deep image features are subjected to multi-scale feature extraction through an ASPP module.

[0039] The Runge-Kutta structure is added to feature extraction to effectively reduce the loss of feature information when it is transferred between layers. The hysteresis attention mechanism module effectively increases the feature extraction effect of the region of interest. The ability to extract information at different scales is improved by adding the ASPP module. Compared with the basic model PSMNet, the model of this application enhances the effect of feature extraction and achieves a significant improvement in feature extraction performance without increasing the complexity of the model too much. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is the basis of the original PSMNet model and the description of the initial first three convolutional layers;

[0041] Figure 2 The overall network architecture adds hysteresis attention structure, Runge-Kutta structure and ASPP module on the basis of PSMNet;

[0042] Figure 3 Detailed diagram of a set of Runge-Kutta modules added to the first three base convolutional layers in the network architecture;

[0043] Figure 4 Build a schematic diagram for the modules of ASPP in the network architecture;

[0044] Figure 5 Schematic diagram of image comparison results of the output disparity map of the network of the present invention and the original model, (a) (b) are two grayscale images of the original image for reference, (c) (d) are disparity maps generated by the original model, and (e) (f) are disparity maps generated by the network of the present invention. DETAILED DESCRIPTION

[0045] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application usually described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0046] The present application will present various aspects, embodiments or features around a system that may include multiple devices, components, modules, etc. It should be understood and appreciated that each system may include additional devices, components, modules, etc., and / or may not include all of the devices, components, modules, etc. discussed in conjunction with the figures. In addition, combinations of these schemes may also be used.

[0047] In addition, in the embodiments of the present application, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present concepts in a specific way.

[0048] The following is combined with Figure 1~Figure 5 The present invention is further described in detail:

[0049] See attached Figure 1-Figure 5 As shown, the deep stereo matching method based on PSMNet optimized feature extraction includes the following steps:

[0050] S1: Split the original PSMNet composed of convolution, normalization and linear correction unit ReLU in the feature extraction class into three basic convolution layers, and combine the convolution layer and BN layer of each part by defining the convbn layer;

[0051] S2: Perform convolution normalization on the initial image according to the defined convbn layer, and add a linear correction unit ReLU to the initial image after convolution normalization to extract the first part of the deep image features of the initial image;

[0052] S3: further extracting the first part of the depth image features based on the hysteresis attention mechanism algorithm to obtain the second part of the depth image features;

[0053] S4: Perform multi-scale feature extraction on the second part of the depth image features through the ASPP module, and output the final image features.

[0054] In the first three basic convolutional layers of the PSMNet network, the Runge-Kutta structure is used. First, since the defined firstconv includes the first three basic convolutional layers, the three basic convolutional layers in the original PSMNet should be separated separately, and the convolutional layers and BN layers of each part are combined by defining the convbn layer. Step S1 also includes setting parameters for defining the convbn layer. The set parameters include in_planes, out_planes, kernel_size, stride, pad, and dilation, where in_planes corresponds to the input image size, out_planes corresponds to the output image size, kernel_size corresponds to the convolution kernel size, stride corresponds to the convolution kernel step size, pad corresponds to the padding size around the image, and dilation corresponds to the convolution expansion rate. The specific settings of the three basic convolutional layers are: the input dimension of the first basic convolutional layer is 3, the output dimension is 32, the convolution kernel size is 3, the stride is 2, the padding is 1, and the expansion rate is 1. The second first basic convolutional layer and the third first basic convolutional layer only change the input dimension to 32 and the stride to 1.

[0055] See also Figure 2 , step S2 includes the following process:

[0056] S21: According to the constructed three-layer basic convolution layer, including the first layer of basic convolution layer, the second layer of basic convolution layer and the third layer of basic convolution layer, the feature extraction result obtained by interleaving the first layer of basic convolution layer and the second layer of basic convolution layer is added to the features extracted by the third layer of basic convolution layer;

[0057] S22: Construct the Runge-Kutta structure through the process of step S1. The specific formula is:

[0058] ;

[0059] in, It is the feature result output after the image passes through the first basic convolution layer. for The feature result after the second basic convolution layer is 1.5 times the difference between the previous and next feature results.

[0060] The specific formula for adding the feature extraction result obtained by interlacing the first and second basic convolutional layers to the features extracted by the third basic convolutional layer in step S21 is as follows:

[0061] .

[0062] See also Figure 3 After building three basic convolutional layers, the formula Corresponding to Y on the graphi-2 part, Corresponding to Y on the graph i-2 -Y i-1 , which is ΔY i-2 part, Corresponding to Y on the graph i-3 part, because The output result of the algorithm adds the feature extraction results interspersed from the first two layers, thereby reducing the feature loss between layers.

[0063] Step S3 includes the following specific processes:

[0064] S31: Construct ResNet basic blocks. The basic blocks in the original PSMNet have four layers in total. The defined convbn layer builds two convolutional layers combined with a downsampling layer to reduce the dimension of the feature image and prevent overfitting.

[0065] S32: A CBAM attention mechanism is added to the second layer and the fourth layer of the ResNet basic block, wherein the CBAM attention mechanism includes a channel attention mechanism and a spatial attention mechanism, and the first part of the deep image features is input into the channel attention mechanism module;

[0066] S33: pass the output features into the spatial attention mechanism module;

[0067] S34: The CBAM attention mechanism is integrated with the hysteresis structure and connected after the deep image features in the first part;

[0068] S35: After adding the CBAM attention mechanism to the second and fourth layers of ResNet basic blocks and fusing the hysteresis structure, the image feature parts that the CBAM attention mechanism pays more attention to are further refined by using the concat function splicing and two additional convolution layers. The input and output dimensions of the first layer CBAM module are 64×64, and the input and output dimensions of the second layer CBAM module are 128×128. In addition, the present invention defines that the experimental effect is the best when the hyperparameter is 0.1.

[0069] The specific formula for inputting the first part of the deep image features into the channel attention mechanism module in step S32 is as follows:

[0070] M c (F) = σ (MLP(AvgPool(F))+MLP(MaxPool(F)));

[0071] Among them, M cis the channel attention weight matrix, σ is the sigmoid activation function, the MLP module compresses and then expands the channel, AvgPool is the average pooling operation, MaxPool is the maximum pooling operation, and F is given an intermediate feature map as input.

[0072] The specific formula for passing the output features into the spatial attention mechanism module in step S33 is as follows:

[0073] ;

[0074] Among them, M s is the spatial attention weight matrix, A 7×7 convolution operation is selected, σ is the sigmoid activation function, AvgPool is the average pooling operation, MaxPool is the maximum pooling operation, and F is given an intermediate feature map as input.

[0075] In step S34, the CBAM attention mechanism is integrated with the hysteresis structure and connected after the first part of the deep image features. The formula of the hysteresis structure in the network is as follows:

[0076] ;

[0077] Among them, S n ,n=0,1...,N-1 represents the output of the nth convolutional block, For each convolution operation defined, Input is the input dimension, and α is a defined hyperparameter used to adjust the size of the hysteresis structure transmitting features in the reverse direction. A n-1 is the attention mask, learned via back-propagation in a self-supervised manner, S n and S n-1 It is the image result before and after output.

[0078] Step S4 includes the following specific processes:

[0079] S41: Build the basic ASPP module, and match the convbn layer with the ReLU linear activation unit. Since the ASPP module is a parallel structure, the dilation parameter in convbn is adjusted to the respective size to increase the size of different receptive fields and increase the ability to extract different information;

[0080] S42: construct a five-layer ASPP module, inherit the image features obtained in step S3, and set the expansion rates to 1, 6, 12, 18, and 24 respectively;

[0081] S43: Use a layer of concat splicing and lastconv convolution operation to fuse information features of different sizes together, and output the final image features through a layer of convolution.

[0082] In summary, the deep stereo matching method based on PSMNet optimized feature extraction provided by the present invention is to separate the three layers of convolution, normalization and linear correction unit ReLU from the original PSMNet spliced ​​by firstconv in the feature extraction class, and combine the convolution layer and BN layer of each part by defining a convbn layer; the defined convbn layer performs convolution normalization processing on the initial image, adds a linear correction unit ReLU to the initial image after convolution normalization processing, and extracts the first part of the deep image features of the initial image; based on the hysteresis attention mechanism algorithm, the first part of the deep image features are further extracted to obtain the second part of the deep image features; the second part of the deep image features are multi-scale feature extraction through the ASPP module. The Runge-Kutta structure is added to the feature extraction to effectively reduce the loss of feature information when it is transmitted between layers. Through the hysteresis attention mechanism module, the feature extraction effect of the region of interest is effectively increased. The ability to extract information at different scales is improved by adding the ASPP module. Compared with the basic model PSMNet, the model of the present application enhances the effect of feature extraction, and achieves a significant improvement in feature extraction performance without increasing the complexity of the model too much.

[0083] The above-mentioned embodiments only express the specific implementation method of the present application, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the protection scope of the present application. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the technical solution concept of the present application, and these all belong to the protection scope of the present application.

Claims

1. Deep stereo matching method based on PSMNet optimized feature extraction, It is characterized in that The following steps are involved: S1: Split the original PSMNet composed of convolution, normalization and linear correction unit ReLU in the feature extraction class into three basic convolution layers, and combine the convolution layer and BN layer of each part by defining the convbn layer; S2: Perform convolution normalization on the initial image according to the defined convbn layer, and add a linear correction unit ReLU to the initial image after convolution normalization to extract the first part of the deep image features of the initial image; S3: further extracting the first part of the depth image features based on the hysteresis attention mechanism algorithm to obtain the second part of the depth image features; S4: performing multi-scale feature extraction on the second part of the depth image features through the ASPP module, and outputting the final image features; Wherein, step S3 includes the following specific process: S31: Construct the ResNet basic block. The basic block in the original PSMNet has four layers. The defined convbn layer builds two convolutional layers and one downsampling layer to reduce the dimension of the feature image. S32: A CBAM attention mechanism is added to the second layer and the fourth layer of the ResNet basic block, wherein the CBAM attention mechanism includes a channel attention mechanism and a spatial attention mechanism, and the first part of the deep image features is input into the channel attention mechanism module; S33: pass the output features into the spatial attention mechanism module; S34: The CBAM attention mechanism is integrated with the hysteresis structure and connected after the deep image features in the first part; S35: After adding the CBAM attention mechanism to the second and fourth layers of ResNet basic blocks and fusing the hysteresis structure, the concat function and two additional convolutional layers are used to further refine the image feature parts that the CBAM attention mechanism pays more attention to. In step S34, the CBAM attention mechanism is integrated with the hysteresis structure and connected after the first part of the deep image features. The formula of the hysteresis structure in the network is as follows: ; Among them, Sn, n = 0, 1..., N-1 represents the output of the nth convolutional block, For each convolution operation defined, Input is the input dimension, α is the defined hyperparameter used to adjust the size of the feature transmission of the hysteresis structure in the reverse direction, An-1 is the attention mask, which is learned through back propagation in a self-supervised manner, and Sn and Sn-1 are the image results of the front and back outputs.

2. The deep stereo matching method based on PSMNet optimized feature extraction according to claim 1, It is characterized in that Step S1 also includes setting parameters for defining the convbn layer, and the set parameters include in_planes, out_planes, kernel_size, stride, pad, and dilation, where in_planes corresponds to the input image size, out_planes corresponds to the output image size, kernel_size corresponds to the convolution kernel size, stride corresponds to the convolution kernel step size, pad corresponds to the padding size around the image, and dilation corresponds to the convolution expansion rate.

3. The deep stereo matching method based on PSMNet optimized feature extraction according to claim 1, It is characterized in that Step S2 includes the following process: S21: According to the constructed three-layer basic convolution layer, including the first layer of basic convolution layer, the second layer of basic convolution layer and the third layer of basic convolution layer, the feature extraction result obtained by interleaving the first layer of basic convolution layer and the second layer of basic convolution layer is added to the features extracted by the third layer of basic convolution layer; S22: Construct the Runge-Kutta structure through the process of step S1. The specific formula is: ; in, It is the feature result output after the image passes through the first basic convolution layer. for The feature result of the second basic convolutional layer is added with 1.5 times the difference between the previous and next feature results.

4. The deep stereo matching method based on PSMNet optimized feature extraction according to claim 1, It is characterized in that The specific formula for inputting the first part of the deep image features into the channel attention mechanism module in step S32 is as follows: M c (F) = σ (MLP(AvgPool(F))+MLP(MaxPool(F))); Among them, M c is the channel attention weight matrix, σ is the sigmoid activation function, the MLP module compresses and then expands the channel, AvgPool is the average pooling operation, MaxPool is the maximum pooling operation, and F is given an intermediate feature map as input.

5. The deep stereo matching method based on PSMNet optimized feature extraction according to claim 1, It is characterized in that The specific formula for passing the output features into the spatial attention mechanism module in step S33 is as follows: ; Among them, M s is the spatial attention weight matrix, A 7×7 convolution operation is selected, σ is the sigmoid activation function, AvgPool is the average pooling operation, MaxPool is the maximum pooling operation, and F is given an intermediate feature map as input.

6. The deep stereo matching method based on PSMNet optimized feature extraction according to claim 1, It is characterized in that Step S4 includes the following specific processes: S41: Build the basic ASPP module, match the convbn layer with the ReLU linear activation unit. Since the ASPP module is a parallel structure, the dilation parameter in convbn is adjusted to its own size. S42: construct a five-layer ASPP module, inherit the image features obtained in step S3, and set the expansion rates to 1, 6, 12, 18, and 24 respectively; S43: Use a layer of concat splicing and lastconv convolution operation to fuse information features of different sizes together, and output the final image features through a layer of convolution.