Stereo matching method based on lightweight context anchor point attention
By employing a lightweight context anchor attention-based stereo matching method, combined with depthwise separable convolution and multi-scale dilated convolution, the problems of high computational complexity and low accuracy in weakly textured regions in UAV stereo matching are solved, achieving efficient and accurate stereo matching results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-26
- Publication Date
- 2026-04-10
AI Technical Summary
Existing stereo matching technologies suffer from high computational complexity on mobile carriers such as drones and low matching accuracy in areas with weak texture, making it difficult to meet the requirements for real-time performance and accuracy.
A lightweight stereo matching method with context anchor attention is adopted. Feature extraction and fusion are performed through depthwise separable convolution and context anchor attention mechanism. Combined with multi-scale dilated convolution aggregation module, the computational burden is reduced and the matching accuracy of weak texture regions is improved.
It achieves efficient matching on devices with limited computing resources, improves the matching accuracy of weak texture areas, and meets the real-time and accuracy requirements of UAV stereo vision.
Smart Images
Figure CN121837855A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision technology, specifically relating to a stereo matching method based on lightweight context anchor attention. Background Technology
[0002] In stereo vision applications on mobile platforms such as drones, existing stereo matching technologies face a dual technical bottleneck. Firstly, traditional stereo matching networks typically employ deep convolutional neural network architectures, resulting in high computational complexity and a massive number of parameters. For example, the classic PSMNet has millions of parameters. When running on devices with limited computing resources, such as drones, it suffers from slow speed, excessive power consumption, and even deployment issues, failing to meet real-time requirements. Secondly, in stereo matching tasks in weakly textured regions, traditional methods rely on local image feature matching. When faced with scenes lacking obvious texture features, such as skies, water surfaces, or white walls, the lack of unique visual features makes it difficult for the algorithm to accurately establish pixel-level correspondences, leading to increased matching errors and even mismatches, severely impacting the accuracy of 3D reconstruction and depth estimation. Summary of the Invention
[0003] This invention provides a stereo matching method based on lightweight context anchor attention, which has low computational burden while ensuring matching accuracy requirements.
[0004] To achieve the above technical objectives, the present invention adopts the following technical solution:
[0005] A stereo matching method based on lightweight context anchor attention includes:
[0006] Both left and right images were acquired, and features were extracted using the lightweight feature extraction module.
[0007] For the extracted left and right image features, a hybrid cost body is constructed by concatenating them by channel based on the grouped correlation cost body and the feature concatenation cost body;
[0008] A multi-scale dilated convolution aggregation module is used to perform layer-by-layer downsampling and upsampling operations on the hybrid cost volume to obtain the cost aggregate of the image;
[0009] Disparity regression is performed on the cost aggregate to obtain the disparity at each location.
[0010] Furthermore, the lightweight feature extraction module employs depthwise separable convolution to encode features of the input image in the first stage, residual block groups for processing in the second stage, and context anchor attention mechanism for fusing channel and spatial dimensions in the third stage, ultimately outputting image features.
[0011] Furthermore, the lightweight feature extraction module processes the feature map obtained in the first stage using four residual block groups in the second stage; each residual block in the residual block group adopts an improved BasicBlock; the improved BasicBlock uses two depthwise separable convolutional layers connected in series, and maintains gradient flow through skip connections, as shown below:
[0012]
[0013] In the formula, The input features for the residual block are... These represent the first and second depthwise separable convolutions, respectively. Represents the ReLU activation function. This represents the output characteristics of the residual block.
[0014] Furthermore, the third stage uses a context anchor attention mechanism to fuse channel and spatial dimensions, including:
[0015] First, the feature map output from the second stage is used as input to the context anchor attention mechanism. Channel attention and spatial attention are then applied to process the input feature map, as follows:
[0016]
[0017]
[0018] In the formula, This represents the channel attention weight matrix. Represents the spatial attention weight matrix. The input feature map is the context anchor attention mechanism. This indicates the average pooling operation. It is the weight matrix of the multilayer perceptron. Represents the ReLU activation function. This represents the sigmoid activation function. [;] represents the max pooling operation, and [;] represents the channel concatenation operation. express Convolution operation;
[0019] Then, the output feature maps of channel attention and spatial attention are integrated with the input feature map and output as follows:
[0020]
[0021] In the formula, This represents element-wise multiplication. This represents the output feature map after optimization by the context anchor attention mechanism.
[0022] Furthermore, the method for constructing the group-related cost body is as follows:
[0023] Divide the input feature maps of C channels into G groups on average, and calculate the correlation between the left and right image features in each group:
[0024]
[0025] In the formula, Indicates the first Correlation feature map of left and right images within a group. This represents the coordinates of a pixel on the feature map. This represents the disparity value at that point. Indicates a group index. It is the number of channels in each group. Indicates the channel index. and These represent the features of the left and right images, respectively.
[0026] The correlation feature maps of each group are simultaneously subjected to diffusion convolution with diffusion rates of R=1, 2, and 3, and then concatenated to obtain the grouped cost body.
[0027] Furthermore, the method for constructing the cascaded cost body is as follows: the left and right feature maps of each input channel are concatenated along the channel dimension, as expressed by:
[0028]
[0029] In the formula, Indicates the cascaded cost body; This means retrieving all elements in that dimension. This represents a connection operation at the channel dimension.
[0030] Furthermore, the multi-scale dilated convolution aggregation module includes:
[0031] First, the input cost volume is downsampled by scales of 1 / 2, 1 / 4, and 1 / 8 respectively through three parallel 3D convolutional layers:
[0032]
[0033] in, Indicates a mixed cost body. Indicates the input Step size is 3D convolution operations, Cost volumes obtained from separate downsampling;
[0034] Then, in each scale branch, feature extraction is performed using two consecutive 3D convolutional layers:
[0035]
[0036]
[0037]
[0038] In the formula This represents a 3D convolution operation. Represents the ReLU activation function. This indicates a batch normalization operation. , , These represent cost volume features at scales of 1 / 2, 1 / 4, and 1 / 8, respectively.
[0039] Next, a bottom-up feature fusion strategy is adopted:
[0040]
[0041] in, This indicates a 3D transpose convolution operation. It is a 1×1×1 convolution used to adjust the number of channels. Features obtained through progressive upsampling and integration.
[0042] Ultimately, the fused features Resolution of upsampled to the original input mixing cost volume
[0043]
[0044] In the formula, This represents the 3D interpolation upsampling function. Indicates the scaling factor. This represents the aggregated cost body that is ultimately output by the multi-scale dilated convolution aggregation module.
[0045] Furthermore, when employing a bottom-up feature fusion strategy, a residual block is added after each fusion operation:
[0046]
[0047] in, This refers to the input features of the residual block, which are... or , This represents a 3D convolution operation. Represents the ReLU activation function. This refers to the output characteristics of the residual block.
[0048] Furthermore, the 3D convolutional layers in the multi-scale dilated convolution aggregation module all employ dilated convolution with a diffusion rate corresponding to the scale.
[0049] Furthermore, the disparity regression of the cost aggregate is specifically performed as follows:
[0050] First, the cost aggregate obtained from the multi-scale dilated convolution aggregation module is converted into the following probability distribution:
[0051]
[0052] in, It is a location parallax value The corresponding cost, It is the disparity value at that location. The probability, Indicates the maximum disparity value. Represents an exponential function;
[0053] Then, the expected disparity value is calculated by weighted summation:
[0054]
[0055] in Indicates position The final output is the continuous disparity value.
[0056] Beneficial effects
[0057] This invention presents a lightweight stereo matching method based on contextual anchor attention. It decomposes traditional convolution into depthwise and pointwise convolutions using depthwise separable convolutions to reduce computational load. Combined with a contextual anchor attention mechanism, it filters key feature channels and integrates environmental information to achieve lightweight feature extraction and fusion. Simultaneously, a multi-scale dilated convolution aggregation module is designed, utilizing 3D convolutions with different dilation rates to expand the receptive field, enhance the cost volume feature representation capability, and improve the matching accuracy of weakly textured regions. Therefore, this invention reduces the model's computational burden through lightweight design, supports deployment on edge devices, and enhances the matching capability of weakly textured regions through multi-scale fusion and environmental information integration, balancing lightweight design with accuracy requirements to meet the requirements of binocular stereo matching tasks. Attached Figure Description
[0058] Figure 1 This is a network structure diagram of the stereo matching method based on lightweight context anchor attention in an embodiment of this application.
[0059] Figure 2 This is a structural diagram of the quantization feature extraction module described in the embodiments of this application.
[0060] Figure 3 This is a structural diagram of the cost body construction module in an embodiment of this application.
[0061] Figure 4This is a structural diagram of the multi-scale dilated convolution aggregation module described in the embodiments of this application. Detailed Implementation
[0062] The embodiments of the present invention will be described in detail below. These embodiments are based on the technical solutions of the present invention and provide detailed implementation methods and specific operation processes to further explain the technical solutions of the present invention.
[0063] This embodiment provides a stereo matching method based on lightweight context anchor attention, referencing... Figure 1 As shown, it includes:
[0064] Step 1: Obtain the left and right images, and extract features from both using the lightweight feature extraction module. For example... Figure 2 As shown.
[0065] The first stage of the lightweight feature extraction module uses depthwise separable convolution to encode features in the input image, reducing the computational cost of the convolution extraction layer by separating intra-channel convolution and cross-channel feature convolution.
[0066] Depthwise separable convolution decomposes the traditional convolution operation into two steps: depthwise convolution and pointwise convolution. For the input graph X, the mathematical expressions for depthwise convolution and pointwise convolution are as follows:
[0067] 1) Depthwise convolution (independent convolution for each channel):
[0068]
[0069] 2) Pointwise convolution (cross-channel information fusion):
[0070]
[0071] in, This represents the output feature map of the depthwise convolution operation. This represents the output feature map of pointwise convolution. This represents the c-th channel of the input image. This represents the spatial coordinates of the current pixel in the image. This represents the offset of the convolution kernel relative to the current pixel. yes Convolution kernel, yes The weights of the convolution kernel, It is the number of input channels for a depthwise separable convolution.
[0072] The lightweight feature extraction module uses four residual block groups (Layer 1~4) to process the feature map obtained in the first stage in the second stage.
[0073] Each residual block in the residual block group employs an improved BasicBlock; the improved BasicBlock uses two 3×3 depthwise separable convolutional layers cascaded together, and maintains gradient flow through skip connections, as shown below:
[0074]
[0075] In the formula, The input features for the residual block are... These represent the first and second depthwise separable convolutions, respectively. Represents the ReLU activation function. This represents the output characteristics of the residual block.
[0076] After the first stage of convolution, the spatial resolution of the feature map is reduced to half of the original input, which is achieved by using a stride of 2 in the first convolution layer. The first residual block in Layer 2 further downsamples the feature map to 1 / 4 of its original size using a convolution with a stride of 2.
[0077] Layers 1, 2, 3, and 4 are sequentially connected in a series. Finally, the output features of Layers 2, 3, and 4 are aligned to spatial resolution through upsampling and subsampling, and then concatenated along the channel dimension to produce the output. Provide the third-stage context anchor attention mechanism:
[0078]
[0079] Subsequently, to enhance the network's ability to perceive contextual information, the third stage of the lightweight feature extraction module employs a contextual anchor attention mechanism. Stable features are captured by anchor branches, and global statistical information for each channel is obtained by compressing the spatial dimension through global average pooling. A two-layer MLP is used for nonlinear transformation to learn the interrelationships between channels, thereby learning to identify the feature channels most helpful for matching. Contextual branches are used to integrate environmental information, and both average pooling and max pooling are used to obtain different statistical features. These two pooling features are then concatenated and integrated to learn a broader range of environmental dependencies, providing rich and minimally redundant feature information for subsequent matching without incurring excessive computational burden. Finally, the fusion of channels and spatial dimensions is completed, outputting the final extracted image features.
[0080] First, the feature map output from the second stage is used as input to the context anchor attention mechanism. Channel attention and spatial attention are then applied to process the input feature map, as follows:
[0081]
[0082]
[0083] In the formula, This represents the channel attention weight matrix. Represents the spatial attention weight matrix. The input feature map is the context anchor attention mechanism. This indicates the average pooling operation. It is the weight matrix of the multilayer perceptron. Represents the ReLU activation function. This represents the sigmoid activation function. [;] represents the max pooling operation, and [;] represents the channel concatenation operation. express Convolution operation;
[0084] Then, the output feature maps of channel attention and spatial attention are integrated with the input feature map and output as follows:
[0085]
[0086] In the formula, This represents element-wise multiplication. This represents the output feature map after optimization by the context anchor attention mechanism.
[0087] In step 1 of this embodiment, the feature processing scheme maintains the powerful feature extraction capability of the deep learning model, while balancing computational efficiency and feature representation capability through depthwise separable convolution and anchor attention. This allows the network to adaptively focus on important features, providing a reliable matching basis for subsequent steps.
[0088] Step 2: Based on the grouped correlation cost volume and the feature concatenation cost volume, construct a hybrid cost volume by concatenating the extracted left and right image features by channel.
[0089] In this embodiment, the cost body construction adopts a hybrid cost body construction scheme that combines group correlation cost body and feature connectivity cost body, and the structure is as follows: Figure 3 As shown in the figure. This construction scheme uses group correlation for efficient similarity measurement with low computational complexity. It preserves the original feature information through feature concatenation, which helps to handle complex matching situations.
[0090] (1) Constructing the grouped correlation cost body. The construction of the grouped correlation cost body involves dividing the feature channels into multiple groups and calculating the correlation independently within each group. This can reduce computational complexity and provide richer matching information.
[0091] First, the feature maps of the input C channels are divided into G groups (G=40 in this embodiment), each group containing Each channel. Calculate the correlation of features between the left and right images in each group:
[0092]
[0093] In the formula, Indicates the first Correlation feature map of left and right images within a group. This represents the coordinates of a pixel on the feature map. This represents the disparity value at that point. Indicates a group index. It is the number of channels in each group. Indicates the channel index. and These represent the features of the left and right images, respectively.
[0094] Then, the correlation feature maps of each group are simultaneously subjected to diffusion convolution with diffusion rates of R=1, 2, and 3, and then concatenated to obtain the grouped cost body.
[0095] (2) Constructing the cascaded cost body. The feature concatenation cost body construction connects the features of the left and right images along the channel dimension, preserving the complete information of the original features. The expression is:
[0096]
[0097] In the formula, Indicates the cascaded cost body; This means retrieving all elements in that dimension. This represents a connection operation at the channel dimension.
[0098] Then, the cost bodies of each independent group are concatenated along the channel dimension to integrate the group-related cost bodies of all groups, and then connected with the cascaded cost bodies along the channel dimension to form the final hybrid cost body. .
[0099] Step 3: Using a multi-scale dilated convolution aggregation module, perform layer-by-layer downsampling and upsampling operations on the mixed cost volume to obtain the cost aggregate of the image.
[0100] After lightweighting the network, some accuracy loss is inevitable, especially in weakly textured regions. Because these regions have little difference in color and texture from surrounding pixels, accurate matching is often difficult. However, stereo matching tasks require maintaining a certain level of accuracy to achieve their goals. Using dilated convolutions to capture contextual information at multiple scales and incorporating pixels with significant color and texture features into the computation can provide more discriminative information for weakly textured regions, improving their matching accuracy. Multi-scale feature aggregation can adapt to scale changes in the scene, enhancing the expressive power of features. Dilated convolutions expand the receptive field while maintaining a constant number of parameters, preventing excessive computational burden on the module. Therefore, this step designs a multi-scale dilated convolution aggregation module that aggregates 3D cost volume features containing different scales using 3D convolutions with different dilation rates. This expands the receptive field, enhances the expressive power of cost volume features, and improves the network's matching accuracy for weakly textured regions without imposing an excessive burden. Figure 4 As shown.
[0101] The multi-scale dilated convolution aggregation module receives a 4D tensor from the cost volume construction stage, with a size of [missing information]. Where H and W are the height and width of the original image, D is the maximum disparity value (192), and 32 is the number of feature channels. To make full use of multi-scale information, the network is designed with three processing branches at different resolutions, which aggregate at 1 / 2, 1 / 4, and 1 / 8 cost volume scales, respectively.
[0102] Step 3.1: The input cost volume is downsampled by three parallel 3D convolutional layers at scales of 1 / 2, 1 / 4, and 1 / 8, respectively.
[0103]
[0104] in, Indicates a mixed cost body. Indicates the input Step size is 3D convolution operations, These are the cost volumes obtained from downsampling.
[0105] Step 3.2: In each scale branch, feature extraction is performed using two consecutive 3D convolutional layers:
[0106]
[0107]
[0108]
[0109] In the formula, This represents a 3D convolution operation. Represents the ReLU activation function. This indicates a batch normalization operation. , , These represent cost volume features at scales of 1 / 2, 1 / 4, and 1 / 8, respectively.
[0110] To further improve the cost aggregation effect, step 3.2 employs dilated convolution to expand the receptive field for 3D convolution operations at different scales. The mathematical expression for 3D dilated convolution is:
[0111]
[0112] in It refers to the location eigenvalues, These are spatial coordinates. These are parallax coordinates. Indicates input features, The weight parameters represent the 3D convolution kernel. It is an index inside the convolution kernel. It is the expansion rate. In this embodiment, different scale branches use different expansion rates (1, 2, 4) to match the cost bodies of different scales.
[0113] Step 3.3 employs a bottom-up feature fusion strategy. First, the lowest resolution 1 / 8 cost volume information is processed, then progressively upsampled and fused with the high-resolution cost volume information. The upsampling process is implemented using 3D transposed convolution:
[0114]
[0115] in, This indicates a 3D transpose convolution operation. It is a 1×1×1 convolution used to adjust the number of channels. Features obtained through progressive upsampling and integration.
[0116] To enhance the feature fusion effect, when using a bottom-up feature fusion strategy, a residual block is added after each fusion operation:
[0117]
[0118] in, This refers to the input features of the residual block, which are... or , This represents a 3D convolution operation. Represents the ReLU activation function. This refers to the output characteristics of the residual block.
[0119] Step 3.4, merge the features The resolution of upsampling to the original input mixing cost volume ( ):
[0120]
[0121] In the formula, This represents the 3D interpolation upsampling function. Indicates the scaling factor. This represents the aggregated cost body that is ultimately output by the multi-scale dilated convolution aggregation module.
[0122] Step 4: Perform disparity regression on the cost aggregate to obtain the disparity at each location.
[0123] After the cost aggregation module, a cost aggregate for the left image is obtained. Applying the softmax function to the mixed cost aggregate along the disparity dimension transforms the cost aggregate obtained by the multi-scale dilated convolution aggregation module into the following probability distribution:
[0124]
[0125] in, It is a location parallax value The corresponding cost, It is the disparity value at that location. The probability, Indicates the maximum disparity value. Represents an exponential function;
[0126] Then, the expected disparity value is calculated by weighted summation:
[0127]
[0128] in, Indicates position The final output is the continuous disparity value.
[0129] The above embodiments are preferred embodiments of this application. Those skilled in the art can make various changes or improvements based on them. Without departing from the overall concept of this application, these changes or improvements should fall within the scope of protection claimed in this application.
Claims
1. A stereo matching method based on lightweight context-anchor attention, characterized in that, The application relates to a stereo matching method based on lightweight feature extraction module, multi-scale cost aggregation module and context-anchor attention mechanism. The method comprises the following steps: Obtaining left and right images, and extracting features from the images by using a lightweight feature extraction module; Based on the grouped correlation cost volume and the feature-level concatenated cost volume, the features of the left and right images are spliced in the channel to construct a mixed cost volume; The mixed cost volume is subjected to layer-by-layer down-sampling and up-sampling operations by using a multi-scale dilated convolution aggregation module to obtain a cost aggregation volume of the image; 2. The stereo matching method of claim 1, wherein, The cost aggregation volume is subjected to disparity regression to obtain the disparity of each position.
3. The stereo matching method of claim 2, wherein, The lightweight feature extraction module comprises the following steps: ； In the formula, is the input feature of the residual block, respectively represent the first and second depth separable convolution, represents the ReLU activation function, represents the output feature of the residual block.
4. The stereo matching method of claim 1, wherein, In the first stage, deep separable convolution is used to encode the input image; In the second stage, residual block groups are used to process the feature map obtained in the first stage; ； ； wherein, denotes a channel attention weight matrix, denotes a spatial attention weight matrix, is an input feature map of the context anchor attention mechanism, denotes an average pooling operation, is a weight matrix of a multi-layer perceptron, denotes a ReLU activation function, denotes a sigmoid activation function, denotes a max pooling operation, [;] denotes a channel concatenation operation, denotes a convolution operation; In the third stage, a context-anchor attention mechanism is used to fuse the channel and spatial dimensions. ； In the formula, represents element-wise multiplication, represents the output feature map after the context anchor attention mechanism optimization.
5. The stereo matching method of claim 1, wherein, In the second stage, the feature map output in the second stage is used as the input of the context-anchor attention mechanism, and the input feature map is processed by using channel attention and spatial attention, respectively, and expressed as follows: Then, the output feature maps of the channel attention and the spatial attention are integrated with the input feature map to output, and expressed as follows: ； In the formula, represents the first correlation feature map of the left and right images in the group, represents the coordinates of the pixel point on the feature map, represents the disparity value of the point, represents the group index, is the number of channels of each group, represents the channel index, and respectively represent the left and right image features; The construction method of the grouped correlation cost volume is as follows:
6. The stereo matching method of claim 1, wherein, The input feature map of C channels is evenly divided into G groups, and the correlation of the left and right image features of each group is calculated: ； wherein represents a concatenation cost tensor; represents taking all elements of this dimension, represents a concatenation operation over the channel dimension.
7. The stereo matching method of claim 1, wherein, The correlation feature map of each group is subjected to diffusion convolution with a diffusion rate of R=1, 2 and 3, respectively, and then spliced to obtain a grouped cost volume. The construction method of the concatenated cost volume is as follows: ； wherein, denotes a mixed cost volume, denotes a cost volume for the input is subjected to a 3D convolution operation with a stride of is down-sampled respectively to obtain a cost volume; The left and right feature maps of each channel are connected in the channel dimension, and the expression is as follows: ； ； ； wherein denotes a 3D convolution operation, denotes a ReLU activation function, denotes a batch normalization operation, , , denote cost volume features at 1 / 2, 1 / 4 and 1 / 8 scales, respectively. The multi-scale dilated convolution aggregation module comprises the following steps: ； wherein, denotes a 3D transpose convolution operation, is a 1x1x1 convolution for adjusting the number of channels, is a step-up sampling integrated feature; Finally, the fused features are upsampled to the resolution of the original input mixture cost volume ； In the formula, denotes a 3D interpolation up-sampling function, denotes a scaling factor, denotes an aggregated cost volume which is the final output of the multi-scale dilated convolution aggregation module.
8. The stereo matching method of claim 7, wherein, Firstly, the input cost volume is subjected to down-sampling processing with scales of 1 / 2, 1 / 4 and 1 / 8 by using three parallel 3D convolution layers: ； wherein, denotes the input feature of the residual block, or , denotes a 3D convolution operation, denotes a ReLU activation function, denotes the output feature of the residual block.
9. The stereo matching method of claim 7, wherein, Then, in each scale branch, two consecutive 3D convolution layers are used for feature extraction:
10. The stereo matching method of claim 1, wherein, Next, a bottom-up feature fusion strategy is adopted: When the bottom-up feature fusion strategy is adopted, a residual block is added after each fusion operation: ； wherein is the position the parallax value the corresponding generation value, is the probability of the position parallax value , denotes the maximum parallax value, denotes the exponential function; The 3D convolution layers in the multi-scale dilated convolution aggregation module are subjected to dilated convolution with a diffusion rate corresponding to the scale. The disparity regression of the cost aggregation volume is as follows: Firstly, the cost aggregation volume obtained by the multi-scale dilated convolution aggregation module is converted into the following probability distribution: Then, the expected disparity value is calculated by using the weighted summation method: ； wherein representing a position final output continuous parallax values.