A lightweight binocular stereo matching method based on multi-scale feature extraction

Abstract

The embodiments of this specification provide a lightweight binocular stereo matching method based on multi-scale feature extraction. The feature extractor uses block convolution and residual blocks for downsampling to retain more effective feature information and generate high-precision feature maps. The semantic guidance module retains the original image information and performs effective disparity correction. A lightweight cost aggregation network with a lightweight structure is used to perform cost aggregation at a fast inference speed. The TensorRT optimizer is used to accelerate the inference of the network layer and is deployed in real time on the resource-constrained NVIDIA Jetson Nano device while maintaining high accuracy.

Description

Technical Field

[0001] This document relates to the field of binocular stereo matching technology, and in particular to a lightweight binocular stereo matching method based on multi-scale feature extraction. Background Art

[0002] With the rapid development of science and technology, artificial intelligence has gradually become a mainstream development across all industries. Binocular stereo matching has been extensively applied in various fields, such as virtual reality, autonomous driving, drones, and intelligent manufacturing, and has great commercial and practical value. Binocular stereo matching is a technology that uses binocular cameras to image the same scene to obtain three-dimensional geometric information of objects. It enables cameras that only have two-dimensional planar information to have stereoscopic vision perception similar to that of the human eye. The purpose of the stereo matching algorithm is to find the positions of corresponding points in the left and right images captured by the binocular cameras, thereby obtaining a disparity map. Further calculations on the disparity map generate a depth map. This generally requires steps such as feature extraction, cost volume construction, cost aggregation, and disparity regression.

[0003] In recent years, binocular stereo matching algorithms based on deep convolutional neural networks have made significant progress in accuracy. However, these high-precision algorithms typically require high computational cost, high power consumption, and high latency to implement, making their real-time deployment on resource-constrained embedded platforms difficult. It is difficult to generate high-precision disparity maps in real time on resource-constrained embedded platforms. Therefore, in order to obtain high-precision disparity estimation results in real time on resource-constrained embedded platforms, it is necessary to develop an efficient, low-power, and lightweight binocular stereo matching algorithm that can be processed on resource-constrained embedded platforms. Summary of the Invention

[0004] One or more embodiments of this specification provide a lightweight binocular stereo matching method based on multi-scale feature extraction, comprising the following steps:

[0005] S1. A feature extractor based on block convolution extracts features from the input image, generating feature maps with a resolution of 1 / 4 and 1 / 16 of the original input image.

[0006] S2. Pass the left input image through a maximum pooling layer and three residual layers to obtain a 1 / 4 resolution feature map and construct a semantic guidance module;

[0007] S3. Use the 1 / 16 resolution feature map extracted by the feature extractor to construct a group distance cost volume, and regularize the group distance cost volume through a lightweight cost aggregation network to obtain an aggregated matching cost volume;

[0008] S4. Generate a coarse estimated disparity map at 1 / 16 resolution from the aggregated matching cost volume through disparity regression. Upsample the coarse estimated disparity map at 1 / 16 resolution to a disparity map at 1 / 4 resolution through bilinear interpolation. This map is then concatenated with the 1 / 4 resolution feature map output by the semantic guidance module and upsampled to full size through bilinear interpolation to obtain the first-stage disparity map.

[0009] S5. Upsampling the coarse estimated disparity map of 1 / 16 resolution in the first stage to a 1 / 4 resolution disparity map by bilinear interpolation, and concatenating it with the 1 / 4 resolution feature map obtained by the semantic guidance module to obtain a disparity-corrected 1 / 4 resolution disparity map. Dynamically offsetting the disparity-corrected 1 / 4 resolution disparity map and the left image features to construct a group-related cost volume. The cost volume is passed through a cost aggregation network and disparity regression to obtain a residual map. The residual map is added to the disparity-corrected 1 / 4 resolution disparity map in the first stage to generate a 1 / 4 resolution precisely estimated disparity map, which is then upsampled to full size by bilinear interpolation to obtain a second-stage disparity map.

[0010] S6. Use the Adam optimizer to optimize the loss function model; use TensorRT to accelerate the inference of the optimized model.

[0011] One or more embodiments of this specification provide an electronic device, including:

[0012] processor; and,

[0013] A memory is arranged to store computer-executable instructions, which, when executed, enable the processor to implement the steps of the above-mentioned lightweight binocular stereo matching method based on multi-scale feature extraction.

[0014] One or more embodiments of this specification provide a storage medium for storing computer-executable instructions, which, when executed, implement the steps of the above-mentioned lightweight binocular stereo matching method based on multi-scale feature extraction.

[0015] The beneficial effects of the present invention are as follows:

[0016] This paper proposes a lightweight binocular stereo matching method based on multi-scale feature extraction. The feature extractor uses block convolution and residual blocks for downsampling, which is conducive to retaining more effective feature information and generating high-precision feature maps. The semantic guidance module can better preserve the original image information and perform effective disparity correction. The provided lightweight cost aggregation network has a lightweight structure and can perform cost aggregation at a fast inference speed. The TensorRT optimizer is used to accelerate the inference of the network layer, which can be deployed in real time on resource-constrained NVIDIA Jetson Nano devices while maintaining high accuracy.

[0017] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 A flowchart of a lightweight binocular stereo matching method based on multi-scale feature extraction provided in one or more embodiments of this specification;

[0020] Figure 2 A schematic diagram of the overall framework of a lightweight binocular stereo matching method based on multi-scale feature extraction provided in one or more embodiments of this specification;

[0021] Figure 3 A schematic diagram of a feature extraction process using a block convolution-based feature extractor for a lightweight binocular stereo matching method based on multi-scale feature extraction provided in one or more embodiments of this specification;

[0022] Figure 4 A schematic diagram of a cost aggregation network structure of a lightweight binocular stereo matching method based on multi-scale feature extraction provided in one or more embodiments of this specification;

[0023] Figure 5 A schematic diagram of the structure of an electronic device provided in one or more embodiments of this specification. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below in conjunction with the drawings in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this document.

[0025] Method Example

[0026] According to an embodiment of the present invention, a lightweight binocular stereo matching method based on multi-scale feature extraction is provided. Figure 1 A flowchart of a lightweight binocular stereo matching method based on multi-scale feature extraction provided in one or more embodiments of this specification is provided. Figure 2 A schematic diagram of the overall framework of a lightweight binocular stereo matching method based on multi-scale feature extraction provided in one or more embodiments of this specification is shown in FIG. Figure 1 and Figure 2 As shown, the lightweight binocular stereo matching method based on multi-scale feature extraction according to an embodiment of the present invention specifically includes:

[0027] S1. The feature extractor based on block convolution extracts features from the input image and obtains feature maps with a resolution of 1 / 4 and 1 / 16 of the original input image.

[0028] The specific process of extracting features from input images by the feature extractor based on block convolution includes:

[0029] The feature extractor first passes through a 4×4 tile convolution layer with a stride of 4 and reduces the size of the feature map to 1 / 4 the resolution of the original input image;

[0030] Then, the size is continuously reduced to 1 / 8 and 1 / 16 resolution relative to the original input image through three consecutive residual blocks and two 2×2 block convolutions, while the size of the feature channel is continuously increased. The residual block contains a 3×3 depth-wise separable convolution with a stride of 1 and two 1×1 point convolutions.

[0031] in,

[0032] The steps of the first 4×4 block convolutional layer with a stride of 4 include:

[0033] Use 4×4 2D convolution with a stride of 4 for dicing;

[0034] Use leak ReLu to optimize the learning ability of the network model;

[0035] Use Batch Normalization to improve the convergence performance of the network model.

[0036] The steps of the 2×2 block convolution in the second and third layers include:

[0037] Use 2×2 two-dimensional convolution with a stride of 2 for dicing;

[0038] Use leak ReLu to optimize the learning ability of the network model;

[0039] Use Batch Normalization to improve the convergence performance of the network model.

[0040] The steps of the three residual blocks include:

[0041] Filtering is done using a 3×3 depthwise separable convolution with a stride of 1;

[0042] Use LayerNorm to optimize the learning ability of the network model;

[0043] Filtering is done using a 1×1 point convolution with a stride of 1;

[0044] Use GELU to improve the convergence performance of the network model;

[0045] Use 1×1 point convolution with a stride of 1 to adjust the number of output channels.

[0046] Finally, after the original input image is downsampled to 1 / 4, 1 / 8 and 1 / 16 resolutions by the feature extractor, a deeper feature extraction is performed on the 1 / 4 and 1 / 16 resolution features to obtain a 1 / 4 resolution feature map and a 1 / 16 resolution feature map, as shown in FIG. Figure 3 shown.

[0047] S2. Pass the left input image through the maximum pooling layer and three residual layers to obtain a 1 / 4 resolution feature map and construct a semantic guidance module.

[0048] Specifically:

[0049] First, the left input image is used to obtain a 1 / 4 resolution feature map through the maximum pooling layer, and then the number of input channels is adjusted through a 3×3 ordinary 2D convolution with a stride of 1;

[0050] Then, it is filtered through three residual blocks, each of which contains two 3×3 ordinary 2D convolutions with a stride of 1;

[0051] Finally, the number of output channels is adjusted by a 3×3 ordinary two-dimensional convolution with a stride of 1, wherein the number of output channels is the same as the number of channels of the coarse estimated disparity map with a resolution of 1 / 16 that is upsampled to a disparity map with a resolution of 1 / 4 by bilinear interpolation.

[0052] S3. Use the 1 / 16 resolution feature map extracted by the feature extractor to construct a group distance cost volume, and regularize the group distance cost volume through a lightweight cost aggregation network to obtain an aggregated matching cost volume.

[0053] The lightweight cost aggregation network is used to regularize the group distance cost volume, and the aggregated matching cost volume is obtained as follows:

[0054] Constructing a distance-related grouping cost volume using the 1 / 4 and 1 / 16 feature maps extracted by the feature extractor, including: constructing a grouping distance cost volume using the 1 / 16 feature map in a first-stage disparity estimation; and constructing a grouping related cost volume using the 1 / 4 feature map in a second-stage disparity estimation;

[0055] The specific calculation method of the group distance cost volume and the group related cost volume is as follows:

[0056]

[0057]

[0058] Among them, ||·-·||1 and <·,·> respectively represent the calculation of the L1 distance and inner product between two features, C gwd and C gwc They represent the group distance cost value and group correlation cost value of the pixel similarity in the input image, d represents the disparity, x and y represent the feature vectors, g represents the number of the feature group, and f l and f r Represent the left image features and the right image features respectively;

[0059] The distance-correlation grouping cost volume calculates the distance and correlation of feature groups at all disparity levels, and regularizes the cost volume through a lightweight cost aggregation network to obtain the aggregated matching cost volume.

[0060] The lightweight cost aggregation network consists of two three-dimensional convolutional layers with a convolution kernel size of 5×5×5 and two residual blocks; the dimension of the cost volume is increased by the first 5×5×5 three-dimensional convolutional layer, which is conducive to learning more contextual features; the cost volume is further optimized by the two residual blocks, each of which is composed of a three-dimensional convolutional layer with a convolution kernel size of 5×5×5, two three-dimensional convolutional layers with a convolution kernel size of 1×1×1 and a skip connection structure; finally, the dimension of the cost volume is reduced by the second 5×5×5 three-dimensional convolutional layer to obtain the final refined cost volume.

[0061] Each 5×5×5 three-dimensional convolution and residual block uses Batch Normalization and ReLu activation functions. Each residual block contains a 5×5×5 three-dimensional depth-separable convolution and two 1×1×1 three-dimensional point convolutions.

[0062] S4. The aggregated matching cost volume generates a coarse estimated disparity map with a resolution of 1 / 16 through disparity regression. The coarse estimated disparity map with a resolution of 1 / 16 is upsampled to a disparity map with a resolution of 1 / 4 through bilinear interpolation. After being spliced ​​with the feature map with a resolution of 1 / 4 output by the semantic guidance module, it is upsampled to the full size through bilinear interpolation to obtain the first-stage disparity map.

[0063] The specific implementation methods of disparity regression and first-stage disparity map are as follows:

[0064] After obtaining the aggregated matching cost volume, the matching cost volume is regressed using the soft argmin operation to obtain a coarse estimated disparity map with a resolution of 1 / 16. This is then upsampled to a disparity map with a resolution of 1 / 4 by bilinear interpolation. This disparity map with a resolution of 1 / 4 is then concatenated with the feature map with a resolution of 1 / 4 output by the semantic guidance module, and then upsampled to full size by bilinear interpolation to obtain the first-stage disparity map. The specific formula for the disparity regression is as follows:

[0065]

[0066] Where M represents the maximum disparity, d represents the disparity, and C i (d') and C i (d) denote the cost values ​​of disparity levels d' and d in the current pixel i. Specifically, the soft-argmin operation is performed by taking C i (d') and C i The negative value of (d) converts the cost volume into a probability volume. The soft-argmin weighted summation of all disparity values ​​can restore the minimum disparity of the cost matching and improve the convergence speed of the model.

[0067] S5. The 1 / 16 resolution coarse estimated disparity map in the first stage is upsampled to a 1 / 4 resolution disparity map through bilinear interpolation, and is spliced ​​with the 1 / 4 resolution feature map passed through the semantic guidance module to obtain a disparity-corrected 1 / 4 resolution disparity map. The disparity-corrected 1 / 4 resolution disparity map and the left image features are dynamically offset to construct a group-related cost volume. The cost volume obtains a residual map through a cost aggregation network and disparity regression. The residual map is added to the 1 / 4 resolution disparity map after the disparity correction in the first stage to generate a 1 / 4 resolution precisely estimated disparity map, which is upsampled to full size through bilinear interpolation to obtain the second-stage disparity map.

[0068] The method of step S5 is specifically as follows:

[0069] The 1 / 16 resolution coarse estimated disparity map of the first stage is upsampled to a 1 / 4 resolution disparity map through bilinear interpolation, and then concatenated with the 1 / 4 resolution feature map of the semantic guidance module to obtain a disparity-corrected 1 / 4 resolution disparity map. The features of the right image of the second stage are distorted.

[0070] The distance-correlation grouping cost volume is constructed using the 1 / 4 resolution left image features extracted by the feature extractor and the distorted right image features;

[0071] A residual map is obtained through the cost aggregation network and disparity regression, which is added to the disparity map enlarged in the first stage to correct the disparity, generating a 1 / 4 resolution fine disparity map, which is then upsampled to full size through bilinear interpolation to obtain the second stage disparity map.

[0072] S6. Use the Adam optimizer to optimize the loss function model and use TensorRT to accelerate the inference of the optimized model.

[0073] After obtaining the disparity map, in order to improve the convergence speed of the model, this embodiment uses the Adam optimizer to optimize the objective function, where betas1 = 0.9 and betas2 = 0.999. The loss weight coefficients of the first and second phases of joint training are different. The weight coefficient of the first phase is λ1 = 0.5, and the weight coefficient of the second phase is λ2 = 0.7. The specific formula of the loss function is as follows:

[0074]

[0075]

[0076]

[0077] Where x is the input variable of the loss function, d is the disparity estimate, is the ground truth disparity at the same pixel, N is the number of candidate disparities, is the loss function of the first stage, is the loss function of the second stage, and Lsum represents the final loss function of the model.

[0078] Specifically, the method of using TensorRT to accelerate reasoning of the optimized model is as follows:

[0079] This example pre-trains the model 40 times on the SceneFlow dataset, then uses the pre-training results to perform fine-tuning training 800 times on the KITTI 2012 and KITTI 2015 datasets. Finally, the fine-tuned model is accelerated for inference using the TensorRT optimizer on the NVIDIA Jetson Nano embedded device, and the optimized model is deployed on the Jetson Nano device.

[0080] The beneficial effects of the present invention are as follows:

[0081] This paper proposes a lightweight binocular stereo matching method based on multi-scale feature extraction. The feature extractor uses block convolution and residual blocks for downsampling, which is conducive to retaining more effective feature information and generating high-precision feature maps. The semantic guidance module can better preserve the original image information and perform effective disparity correction. The provided lightweight cost aggregation network has a lightweight structure and can perform cost aggregation at a fast inference speed. The TensorRT optimizer is used to accelerate the inference of the network layer, which can be deployed in real time on resource-constrained NVIDIA Jetson Nano devices while maintaining high accuracy.

[0082] Device Example 1

[0083] An embodiment of the present invention provides an electronic device, such as Figure 5 As shown, it includes: a memory 50, a processor 52, and a computer program stored in the memory 50 and executable on the processor 25. When the computer program is executed by the processor 52, the following method steps are implemented:

[0084] S1. A feature extractor based on block convolution extracts features from the input image, generating feature maps with a resolution of 1 / 4 and 1 / 16 of the original input image.

[0085] S2. Pass the left input image through a maximum pooling layer and three residual layers to obtain a 1 / 4 resolution feature map and construct a semantic guidance module;

[0086] S3. Use the 1 / 16 resolution feature map extracted by the feature extractor to construct a group distance cost volume, and regularize the group distance cost volume through a lightweight cost aggregation network to obtain an aggregated matching cost volume;

[0087] S4. Generate a coarse estimated disparity map at 1 / 16 resolution from the aggregated matching cost volume through disparity regression. Upsample the coarse estimated disparity map at 1 / 16 resolution to a disparity map at 1 / 4 resolution through bilinear interpolation. This map is then concatenated with the 1 / 4 resolution feature map output by the semantic guidance module and upsampled to full size through bilinear interpolation to obtain the first-stage disparity map.

[0088] S5. Upsampling the coarse estimated disparity map of 1 / 16 resolution in the first stage to a 1 / 4 resolution disparity map by bilinear interpolation, and concatenating it with the 1 / 4 resolution feature map obtained by the semantic guidance module to obtain a disparity-corrected 1 / 4 resolution disparity map. Dynamically offsetting the disparity-corrected 1 / 4 resolution disparity map and the left image features to construct a group-related cost volume. The cost volume is passed through a cost aggregation network and disparity regression to obtain a residual map. The residual map is added to the disparity-corrected 1 / 4 resolution disparity map in the first stage to generate a 1 / 4 resolution precisely estimated disparity map, which is then upsampled to full size by bilinear interpolation to obtain a second-stage disparity map.

[0089] S6. Use the Adam optimizer to optimize the loss function model; use TensorRT to accelerate inference of the optimized model.

[0090] Device Example 2

[0091] An embodiment of the present invention provides a computer-readable storage medium having stored thereon a program for implementing information transmission. When the program is executed by the processor 52, the following method steps are implemented:

[0092] S1. A feature extractor based on block convolution extracts features from the input image, generating feature maps with a resolution of 1 / 4 and 1 / 16 of the original input image.

[0093] S2. Pass the left input image through a maximum pooling layer and three residual layers to obtain a 1 / 4 resolution feature map and construct a semantic guidance module;

[0094] S3. Use the 1 / 16 resolution feature map extracted by the feature extractor to construct a group distance cost volume, and regularize the group distance cost volume through a lightweight cost aggregation network to obtain an aggregated matching cost volume;

[0095] S4. Generate a coarse estimated disparity map at 1 / 16 resolution from the aggregated matching cost volume through disparity regression. Upsample the coarse estimated disparity map at 1 / 16 resolution to a disparity map at 1 / 4 resolution through bilinear interpolation. This map is then concatenated with the 1 / 4 resolution feature map output by the semantic guidance module and upsampled to full size through bilinear interpolation to obtain the first-stage disparity map.

[0096] S5. Upsampling the coarse estimated disparity map of 1 / 16 resolution in the first stage to a 1 / 4 resolution disparity map by bilinear interpolation, and concatenating it with the 1 / 4 resolution feature map obtained by the semantic guidance module to obtain a disparity-corrected 1 / 4 resolution disparity map. Dynamically offsetting the disparity-corrected 1 / 4 resolution disparity map and the left image features to construct a group-related cost volume. The cost volume is passed through a cost aggregation network and disparity regression to obtain a residual map. The residual map is added to the disparity-corrected 1 / 4 resolution disparity map in the first stage to generate a 1 / 4 resolution precisely estimated disparity map, which is then upsampled to full size by bilinear interpolation to obtain a second-stage disparity map.

[0097] S6. Use the Adam optimizer to optimize the loss function model; use TensorRT to accelerate the inference of the optimized model.

[0098] The computer-readable storage medium in this embodiment includes, but is not limited to, ROM, RAM, magnetic disk, or optical disk.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A lightweight binocular stereo matching method based on multi-scale feature extraction, characterized in that: The method comprises the following steps: S1. extracting features from an input image using a feature extractor based on block convolution to obtain feature maps with a resolution of 1 / 4 and 1 / 16 of the original input image; S2. Pass the left input image through a maximum pooling layer and three residual layers to obtain a 1 / 4 resolution feature map and construct a semantic guidance module. The specific method is as follows: First, the left input image is used to obtain a 1 / 4 resolution feature map through the maximum pooling layer, and then the number of input channels is adjusted through a 3×3 ordinary 2D convolution with a stride of 1; Then, it is filtered through three residual blocks, each of which contains two 3×3 ordinary 2D convolutions with a stride of 1; Finally, adjusting the number of output channels by a 3×3 ordinary two-dimensional convolution with a stride of 1, wherein the number of output channels is the same as the number of channels of the disparity map obtained by upsampling the 1 / 16 resolution coarse estimated disparity map to 1 / 4 resolution through bilinear interpolation; S3. Use the extracted 1 / 16 resolution feature map to construct a group distance cost volume, and regularize the group distance cost volume through a lightweight cost aggregation network to obtain an aggregated matching cost volume; The lightweight cost aggregation network consists of two 3D convolutional layers with a convolution kernel size of 5×5×5 and two residual blocks; the dimensionality of the cost volume is increased by the first 5×5×5 3D convolutional layer; the cost volume is further optimized by the two residual blocks, each of which is composed of a 3D convolutional layer with a convolution kernel size of 5×5×5, two 3D convolutional layers with a convolution kernel size of 1×1×1, and a skip connection structure; finally, the dimensionality of the cost volume is reduced by the second 5×5×5 3D convolutional layer to obtain the final refined cost volume; S4. Generate a coarse estimated disparity map at 1 / 16 resolution from the aggregated matching cost volume through disparity regression. Upsample the coarse estimated disparity map at 1 / 16 resolution to a disparity map at 1 / 4 resolution through bilinear interpolation. This map is then concatenated with the 1 / 4 resolution feature map output by the semantic guidance module and upsampled to full size through bilinear interpolation to obtain the first-stage disparity map. S5. Upsampling the coarse estimated disparity map of 1 / 16 resolution in the first stage to a 1 / 4 resolution disparity map by bilinear interpolation, and concatenating it with the 1 / 4 resolution feature map obtained by the semantic guidance module to obtain a disparity-corrected 1 / 4 resolution disparity map. Dynamically offsetting the disparity-corrected 1 / 4 resolution disparity map and the left image features to construct a group-related cost volume. The cost volume is passed through a cost aggregation network and disparity regression to obtain a residual map. The residual map is added to the disparity-corrected 1 / 4 resolution disparity map in the first stage to generate a 1 / 4 resolution precisely estimated disparity map, which is then upsampled to full size by bilinear interpolation to obtain a second-stage disparity map. S6. Use the Adam optimizer to optimize the loss function model; use TensorRT to accelerate the inference of the optimized model.

2. The method according to claim 1, characterized in that The specific method of extracting features from the input image by the feature extractor based on block convolution in step S1 is as follows: The feature extractor first passes through a 4×4 tile convolution layer with a stride of 4 and reduces the size of the feature map to 1 / 4 the resolution of the original input image; Then, the size is continuously reduced to 1 / 8 and 1 / 16 resolution relative to the original input image through three consecutive residual blocks and two 2×2 block convolutions, while the size of the feature channel is continuously increased. The residual block contains a 3×3 depth-wise separable convolution with a stride of 1 and two 1×1 point convolutions. After the original input image is downsampled to 1 / 4, 1 / 8 and 1 / 16 resolutions step by step by the feature extractor, a deeper feature extraction is performed on the 1 / 4 and 1 / 16 resolution features to obtain a 1 / 4 resolution feature map and a 1 / 16 resolution feature map.

3. A method according to claim 2, characterized in that In step S3, the cost volume is regularized by the lightweight cost aggregation network, and the aggregated matching cost volume is obtained as follows: Constructing a group distance cost volume using the 1 / 4 and 1 / 16 feature maps extracted by the feature extractor, including: constructing a group distance cost volume using the 1 / 16 feature map in a first-stage disparity estimation; and constructing a group correlation cost volume using the 1 / 4 feature map in a second-stage disparity estimation; The group distance cost volume and group related cost volume are as follows: ； ； in and Respectively represent the calculation of the L1 distance and inner product between two features, C gwd and C gwc They represent the group distance cost value and group correlation cost value of the pixel similarity in the input image, d represents the disparity, x and y represent the feature vectors, g represents the number of the feature group, and f l and f r Represent the left image features and the right image features respectively; The distance-correlation grouping cost volume calculates the distance and correlation of feature groups at all disparity levels, and regularizes the cost volume through a lightweight cost aggregation network to obtain the aggregated matching cost volume.

4. The method according to claim 3, characterized in that The method for implementing the disparity regression and the first-stage disparity map in step S4 is specifically as follows: After obtaining the aggregated matching cost volume, the matching cost volume is regressed using the soft argmin operation to obtain a coarse estimated disparity map with a resolution of 1 / 16. This is then upsampled to a disparity map with a resolution of 1 / 4 by bilinear interpolation. This disparity map with a resolution of 1 / 4 is then concatenated with the feature map with a resolution of 1 / 4 output by the semantic guidance module, and then upsampled to full size by bilinear interpolation to obtain the first-stage disparity map. The specific formula for the disparity regression is as follows: ； Where M represents the maximum disparity, d represents the disparity, and C i (d') and C i (d) represent the cost values ​​of disparity levels d′ and d in the current pixel i, respectively.

5. The method according to claim 4, characterized in that The method of step S5 is specifically as follows: The 1 / 16 resolution coarse estimated disparity map of the first stage is upsampled to a 1 / 4 resolution disparity map through bilinear interpolation, and then concatenated with the 1 / 4 resolution feature map of the semantic guidance module to obtain a disparity-corrected 1 / 4 resolution disparity map. The features of the right image of the second stage are distorted. The 1 / 4 resolution left image features extracted by the feature extractor and the distorted right image features are used to construct the group-related cost volume; A residual map is obtained through the cost aggregation network and disparity regression, which is added to the disparity map enlarged in the first stage to correct the disparity, generating a 1 / 4 resolution fine disparity map, which is then upsampled to full size through bilinear interpolation to obtain the second stage disparity map.

6. The method according to claim 1, characterized in that After obtaining the disparity map, the Adam optimizer is used to optimize the loss function. The loss weight coefficients of the first and second phases of joint training are different. The weight coefficient of the first phase is =0.5, the weight coefficient of the second stage is =0.7, the specific formula of the loss function is as follows: ； ； ； Where x is the input variable, d is the disparity estimate, is the ground truth disparity at the same pixel, N is the number of candidate disparities, and are the loss functions of the first and second stages respectively.

7. An electronic device, characterized in that: include: processor; as well as, A memory arranged to store computer-executable instructions, wherein when the computer-executable instructions are executed, the processor implements the steps of the lightweight binocular stereo matching method based on multi-scale feature extraction according to any one of claims 1 to 6.

8. A storage medium, characterized in that: Used to store computer-executable instructions, which, when executed, implement the steps of the lightweight binocular stereo matching method based on multi-scale feature extraction as described in any one of claims 1 to 6.

Images