A Two-Stage Real-Time Binocular Depth Estimation Method and Device Based on Group Hybridization

Abstract

This specification provides a two-stage real-time binocular depth estimation method and apparatus based on grouped hybridization. The method includes: extracting features from the original input image to obtain feature maps at 1 / 4 and 1 / 8 resolution relative to the original input image; constructing a grouped distance cost using the 1 / 8 resolution feature map to obtain an aggregated matching cost, and further obtaining a first-stage disparity map; enlarging the coarsely estimated disparity map at 1 / 8 resolution to a 1 / 4 resolution disparity map, constructing a grouped correlation cost, generating a finely estimated disparity map at 1 / 4 resolution, and further obtaining a second-stage disparity map; and using the Adam optimizer to optimize the loss function based on the first-stage disparity map and the second-stage disparity map to obtain an optimized model, and using the TensorRT optimizer to accelerate inference of the network layers of the optimized model.

Description

Technical Field

[0001] This document relates to the field of computer technology, and in particular to a two-stage real-time binocular depth estimation method, apparatus, electronic device and storage medium based on grouped hybridization. Background Technology

[0002] Binocular depth estimation algorithms are widely used in fields such as robot navigation, augmented reality, smart cities, and autonomous driving. Therefore, accurate and fast binocular depth estimation algorithms are of great significance for resource-constrained embedded platforms. In recent years, with the continuous innovation of deep convolutional neural networks, binocular depth estimation algorithms based on deep convolutional networks have achieved significant improvements in accuracy. However, current high-precision binocular depth estimation algorithms typically suffer from high computational cost, high power consumption, and high latency, making it difficult to deploy existing algorithms in real-time on resource-constrained embedded platforms.

[0003] The main steps of a binocular depth estimation algorithm include feature extraction, cost estimation, cost aggregation, and disparity regression. Feature extraction, cost estimation, and cost aggregation play a decisive role in the network's accuracy and inference speed. For feature extraction, existing methods primarily use the U-Net network to extract features from stereo input images. Specifically, this network is a symmetric feature encoding and decoding architecture that can simultaneously output feature maps of different sizes. However, U-Net often loses some important feature information during the encoding process. For cost estimation, existing methods mainly use full distance, full correlation, and group correlation cost estimation to calculate the matching cost. Specifically, full distance generates a single-channel distance map for each disparity level, and full correlation generates a single-channel correlation map for each disparity level. Since full distance and full correlation only generate one single-channel distance map and correlation map, a lot of feature information is lost. Group correlation divides the left and right features into several groups and then calculates the correlation map for each group, which can obtain multiple cost matching schemes. Finally, these matching schemes are merged into a single group correlation. Although group correlation retains more feature information, it is still difficult to provide a good similarity measurement quickly. For the cost aggregation step, existing methods mainly employ a stacked hourglass model combined with intermediate supervision to adjust the matching cost. Specifically, this method uses an encoder-decoder architecture, combining intermediate supervision to perform repeated fine-to-coarse and then coarse-to-fine processing. However, since the stacked hourglass model consists of many three-dimensional convolutional layers, the computational complexity is relatively high, thus failing to meet the requirements for real-time deployment on resource-constrained embedded devices.

[0004] To reduce the computational complexity of the cost aggregation step, existing methods employ a coarse-to-fine progressive refinement strategy for depth estimation. Specifically, this method first constructs a matching cost using low-size feature maps to obtain a coarsely estimated disparity map. This disparity map is then upsampled using bilinear interpolation, and a smaller disparity offset is used at a higher size to correct the previous coarse disparity estimate. This method significantly reduces the computational complexity of the cost aggregation step; however, using low-size feature maps to construct the cost makes it difficult to obtain high-precision disparity estimation results. Furthermore, to further improve the disparity estimation results, existing methods use a multi-stage coarse-to-fine strategy. However, as the number of operation levels increases, the computation time of this method increases significantly. In summary, existing algorithms still struggle to generate high-precision disparity maps in real-time on resource-constrained embedded platforms. Therefore, the current problem to be solved is how to obtain high-precision disparity estimation results in real-time on resource-constrained embedded platforms. Summary of the Invention

[0005] The purpose of this invention is to provide a two-stage real-time binocular depth estimation method, apparatus, electronic device, and storage medium based on grouped hybridization, in order to solve the aforementioned problems in the prior art.

[0006] This invention provides a two-stage real-time stereo depth estimation method based on grouped hybridization, comprising:

[0007] Feature extraction of the original input image is performed using a feature extractor based on block convolution, resulting in feature maps with resolutions of 1 / 4 and 1 / 8 relative to the original input image.

[0008] The group distance cost is constructed using a feature map with a resolution of 1 / 8. The group distance cost is then regularized using a lightweight cost aggregation network to obtain the aggregated matching cost. The aggregated matching cost is then used to generate a coarsely estimated disparity map with a resolution of 1 / 8 through disparity regression. Finally, it is upsampled to full size using bilinear interpolation to obtain the first-stage disparity map.

[0009] The coarsely estimated disparity map at 1 / 8 resolution is enlarged into a disparity map at 1 / 4 resolution. Based on the disparity map at 1 / 4 resolution and the features of the left image, a grouped correlation cost is constructed using dynamic offsets. The grouped correlation cost is then processed through a cost aggregation network and disparity regression to obtain a residual map. This residual map is added to the enlarged disparity map at 1 / 4 resolution to generate a finely estimated disparity map at 1 / 4 resolution. Finally, the map is upsampled to full size using bilinear interpolation to obtain the second-stage disparity map.

[0010] Based on the first-stage disparity map and the second-stage disparity map, the Adam optimizer is used to optimize the loss function to obtain the optimized model. The TensorRT optimizer is then used to accelerate the inference of the network layers of the optimized model.

[0011] This invention provides a two-stage real-time binocular depth estimation device based on grouped hybridization, comprising:

[0012] The extraction module is used to extract features from the original input image using a feature extractor based on block convolution, and obtain feature maps with resolutions of 1 / 4 and 1 / 8 relative to the original input image.

[0013] The first-stage disparity map module is used to construct a group distance cost using a 1 / 8 resolution feature map. The group distance cost is regularized by a lightweight cost aggregation network to obtain an aggregated matching cost. The aggregated matching cost is then used to generate a coarsely estimated disparity map at 1 / 8 resolution through disparity regression. Finally, it is upsampled to full size through bilinear interpolation to obtain the first-stage disparity map.

[0014] The second-stage disparity map module is used to enlarge the coarsely estimated disparity map at 1 / 8 resolution into a 1 / 4 resolution disparity map. Based on the 1 / 4 resolution disparity map and the features of the left image, a dynamic offset is used to construct a grouped correlation cost. The grouped correlation cost is then processed through a cost aggregation network and disparity regression to obtain a residual map. The residual map is added to the enlarged 1 / 4 resolution disparity map to generate a finely estimated 1 / 4 resolution disparity map. Finally, the map is upsampled to full size using bilinear interpolation to obtain the second-stage disparity map.

[0015] The optimization inference module is used to optimize the loss function based on the first-stage disparity map and the second-stage disparity map, obtain the optimized model, and use the TensorRT optimizer to accelerate the inference of the network layers of the optimized model.

[0016] This invention also provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the above-described two-stage real-time binocular depth estimation method based on grouping and mixing.

[0017] This invention also provides a computer-readable storage medium storing an information transmission implementation program, which, when executed by a processor, implements the steps of the above-described two-stage real-time binocular depth estimation method based on grouping and mixing.

[0018] Compared with the prior art, the embodiments of the present invention have at least the following beneficial effects:

[0019] (1) In the feature extraction step, this embodiment of the invention proposes a feature extractor based on block convolution, which uses continuous block convolution for downsampling. Compared with the prior art, this invention uses block convolution for downsampling, which can retain more feature information.

[0020] (2) In the cost quantity construction step, this embodiment of the invention proposes a two-stage method of distance-correlation grouping hybrid method to construct the cost quantity. The distance-correlation grouping cost quantity generates a multi-channel distance map and a correlation map for each disparity level. Compared with the prior art, this strategy can integrate more feature channel information and provide better similarity measurement.

[0021] (3) In the cost aggregation step, this embodiment of the invention proposes a lightweight three-dimensional cost aggregation network, which uses only four 5×5×5 three-dimensional convolutional layers in each stage to optimize the distance-correlation grouping cost. Compared with the prior art, the computational cost of cost aggregation in this invention is relatively small, and the 5×5×5 three-dimensional convolution has a larger receptive field, which improves the accuracy of identifying textureless regions.

[0022] (4) The embodiments of the present invention employ a two-stage coarse-to-fine processing method for disparity estimation. Compared with the prior art, this processing method can significantly reduce the computational complexity of the cost aggregation step and significantly improve the inference speed of the network.

[0023] (5) This embodiment of the invention uses the TensorRT optimizer to accelerate inference at the network layer. Compared with the prior art, this embodiment of the invention can deploy the KITTI 2012 and KITTI 2015 datasets in real time on resource-constrained NVIDIA Jetson Nano devices while maintaining high accuracy, which is of great significance in fields such as robot navigation, augmented reality, smart cities and autonomous driving. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in one or more embodiments of this specification or in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart of a two-stage real-time binocular depth estimation method based on grouping and mixing, according to an embodiment of the present invention.

[0026] Figure 2 This is an overall framework diagram of the two-stage real-time binocular depth estimation method based on distance-correlation grouping hybrid according to an embodiment of the present invention;

[0027] Figure 3 This is a schematic diagram of a feature extractor based on block convolution extracting features at 1 / 8 and 1 / 4 resolution according to an embodiment of the present invention;

[0028] Figure 4 This is a schematic diagram of the specific structure of the distance-related grouping cost divided into 32 groups according to an embodiment of the present invention;

[0029] Figure 5 This is a schematic diagram of the specific structure of the lightweight cost aggregation network in this embodiment of the invention, which is divided into four 5×5×5 three-dimensional convolutional layers in each stage.

[0030] Figure 6 This is a schematic diagram of a two-stage real-time binocular depth estimation device based on grouping and mixing according to an embodiment of the present invention;

[0031] Figure 7 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0032] The purpose of this invention is to overcome the defects and shortcomings of existing technologies and provide a two-stage real-time binocular depth estimation method based on grouped hybridization. Unlike the aforementioned technologies, which suffer from problems such as easy loss of effective feature information during feature extraction, inability to quickly provide good similarity measurement when constructing cost quantities, difficulty in efficiently regularizing cost quantities during cost aggregation, and difficulty in generating high-precision disparity maps in real time on resource-constrained embedded platforms, this method, through the provided block-convolutional feature extractor, distance-correlation grouped cost quantities, lightweight cost aggregation network, and TensorRT optimization model, can obtain high-precision disparity estimation results in real time on resource-constrained embedded platforms.

[0033] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.

[0034] Method Implementation Examples

[0035] According to embodiments of the present invention, a two-stage real-time binocular depth estimation method based on grouped hybridization is provided. Figure 1 This is a flowchart of a two-stage real-time binocular depth estimation method based on grouping and mixing, according to an embodiment of the present invention. Figure 1As shown, the two-stage real-time binocular depth estimation method based on grouping and mixing according to an embodiment of the present invention specifically includes:

[0036] Step 101: Feature extraction is performed on the original input image using a block-based convolution feature extractor to obtain feature maps at 1 / 4 and 1 / 8 resolution relative to the original input image; specifically including:

[0037] Using a feature extractor based on block convolution, the original input image is first downsampled to 1 / 2, 1 / 4, and 1 / 8 resolutions using block convolution. Then, deep feature extraction is performed on the features at 1 / 4 and 1 / 8 resolutions to obtain feature maps at 1 / 4 and 1 / 8 resolutions relative to the original input image.

[0038] Step 102: Construct a group distance cost using 1 / 8 resolution feature maps. Regularize the group distance cost using a lightweight cost aggregation network to obtain the aggregated matching cost. Generate a coarsely estimated disparity map at 1 / 8 resolution using disparity regression, and then upsample it to full size using bilinear interpolation to obtain the first-stage disparity map. Specifically, constructing the group distance cost using 1 / 8 resolution feature maps includes:

[0039] According to Formula 1, the grouping distance cost is constructed using feature maps at 1 / 8 resolution:

[0040]

[0041] Where ||·-·||1 represents calculating the L1 distance between two features, C gwd The group distance cost represents the pixel similarity among pixels in the input image, d represents disparity, x and y represent feature vectors, g represents the feature group number, and f represents the feature group number. l and f r These represent the features of the left and right images, respectively.

[0042] The lightweight cost aggregation network includes four 3D convolutional layers with a kernel size of 5×5×5. The first 5×5×5 3D convolution is used to increase the dimension of cost, the second and third 5×5×5 3D convolutions are used to optimize cost, and the fourth 5×5×5 3D convolutional layer is used to reduce the dimension of cost.

[0043] The aggregated matching cost is used to generate a coarsely estimated disparity map at 1 / 8 resolution through disparity regression, and then upsampled to full size through bilinear interpolation to obtain the first-stage disparity map, which specifically includes:

[0044] Based on Equation 3, disparity regression is performed using the optimized matching cost amount from the cost aggregation network to obtain a coarsely estimated disparity map at 1 / 8 resolution. This map is then upsampled to full size using bilinear interpolation to obtain the first-stage disparity map.

[0045]

[0046] Where M represents the maximum disparity, and the maximum disparity M in the first stage is 24, d represents the disparity, and C i (d') and C i (d) represents the cost of disparity levels d' and d in the current pixel i, respectively.

[0047] Step 103: Enlarge the coarsely estimated disparity map at 1 / 8 resolution to a 1 / 4 resolution disparity map. Construct a grouped correlation cost based on the 1 / 4 resolution disparity map and the features of the left image using dynamic offsets. Obtain a residual map from the grouped correlation cost using a cost aggregation network and disparity regression. Add the residual map to the enlarged 1 / 4 resolution disparity map to generate a finely estimated 1 / 4 resolution disparity map. Upsample the map to full size using bilinear interpolation to obtain the second-stage disparity map. Specifically, constructing the grouped correlation cost based on the 1 / 4 resolution disparity map and the features of the left image includes:

[0048] Based on Formula 2, the grouping correlation cost is constructed by dynamically offsetting the 1 / 4 resolution disparity map and the features of the left image:

[0049]

[0050] Where <·, ·> denotes calculating the inner product between two features, C gwc The group correlation cost represents the pixel similarity in the input image, where d represents disparity, x and y represent feature vectors, g represents the feature group number, and f represents the group correlation cost. l and f r These represent the features of the left and right images, respectively.

[0051] Step 104: Based on the first-stage disparity map and the second-stage disparity map, the Adam optimizer is used to optimize the loss function to obtain the optimized model. The TensorRT optimizer is then used to accelerate inference in the network layers of the optimized model. Specifically:

[0052] The loss function shown in Equation 4-6 is optimized using the Adam optimizer; where...

[0053]

[0054]

[0055]

[0056] Where x is the input variable and d is the disparity estimate. Here, N represents the true ground disparity at the same pixel, and N is the number of candidate disparities. and These are the loss functions for the first and second stages, respectively. The loss weight coefficients for the joint training of the first and second stages are different. The weight coefficient for the first stage is λ1 = 0.5, and the weight coefficient for the second stage is λ2 = 0.7.

[0057] The technical solutions of the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0058] This invention discloses a two-stage real-time binocular depth estimation method based on grouped hybridization, such as... Figure 2 As shown, the specific steps include:

[0059] Step a: Use a feature extractor based on block convolution to extract features from the input image to obtain feature maps at 1 / 4 and 1 / 8 resolution relative to the original input image;

[0060] Step b: Use the 1 / 8 resolution feature maps extracted by the feature extractor to construct the group distance cost;

[0061] Step c: The cost is regularized by a lightweight cost aggregation network to obtain the aggregated matching cost.

[0062] Step d: The aggregated matching cost is used to generate a coarsely estimated disparity map at 1 / 8 resolution through disparity regression, and then upsampled to full size through bilinear interpolation to obtain the first-stage disparity map;

[0063] Step e: The coarse disparity map estimated at 1 / 8 resolution in the first stage is magnified into a disparity map at 1 / 4 resolution. This disparity map and the features of the left image are dynamically offset to construct a grouped correlation cost. This cost is used to obtain a residual map through a cost aggregation network and disparity regression. This residual map is added to the disparity map magnified in the first stage to generate a fine disparity map estimated at 1 / 4 resolution. It is then upsampled to full size through bilinear interpolation to obtain the second-stage disparity map.

[0064] Step f: Optimize the model using the Adam optimizer to optimize the loss function;

[0065] Step g: Use TensorRT to accelerate inference of the optimized model.

[0066] Specifically, the implementation method of the block convolutional feature extractor in step a above is as follows:

[0067] The feature extractor first convolves the feature map using two 3×3 convolutional layers with a stride of 1. Then, it continuously reduces the size of the feature map to 1 / 2, 1 / 4, and 1 / 8 resolution relative to the original input image through three consecutive block convolutions and six regular convolutions, while continuously increasing the size of the feature channels, as shown in Table 1. The block convolution steps include: first, using a 2×2 two-dimensional convolution with a stride of 2 for block segmentation; second, using Leak ReLU to optimize the learning ability of the network model; and third, using Batch Normalization to improve the convergence performance of the network model. After progressively downsampling the original input image to 1 / 2, 1 / 4, and 1 / 8 resolution, the feature extractor performs deeper feature extraction on the 1 / 4 and 1 / 8 resolution features, resulting in 1 / 4 and 1 / 8 resolution feature maps, as shown in Table 1. Figure 3 As shown in the figure. The feature maps corresponding to 1 / 4 and 1 / 8 resolution features have 4c and 8c channels, respectively, where c represents a hyperparameter of the feature extractor.

[0068] Table 1 Feature extractor based on block convolution

[0069]

[0070] Specifically, the distance-related grouping cost in step b is implemented as follows:

[0071] The distance-correlation grouping cost provided in this embodiment of the invention consists of two parts: grouped distance cost and grouped correlation cost. The basic idea of ​​distance-correlation grouping is: in the first stage, the distance cost at each disparity level is calculated. In the second stage, the correlation cost at each disparity level is calculated. In each stage, the features of the left and right images are first divided into several groups, and then the distance map (first stage) or correlation map (second stage) is calculated for each group. This yields multiple distance or correlation cost matching schemes. Finally, these matching schemes are merged into a single grouped distance cost or grouped correlation cost, such as... Figure 4 As shown. Group distance cost C gwd Grouping-related cost C gwc The specific formulas are as follows:

[0072]

[0073]

[0074] Where ||·-·||1 and <·,·> represent calculating the L1 distance and inner product between two features, respectively, d represents disparity, x and y represent two input feature vectors, g represents the feature group number, and f l f represents the feature of the left figure. r The right figure represents a feature.

[0075] Specifically, the lightweight cost aggregation network implementation method in step c is as follows:

[0076] The distance-relevance grouping cost may be affected by unstable factors such as weak texture, occlusion, and blurry matching in the input image. To mitigate this issue, this invention employs a lightweight cost aggregation network to regularize the cost, resulting in an aggregated matching cost, such as... Figure 5 As shown, this aggregation network consists of four 5×5×5 3D convolutional layers. The first 5×5×5 3D convolutional layer is used to increase the dimensionality of the cost vector, which is beneficial for learning more contextual features. The second and third 5×5×5 3D convolutional layers are used to further optimize the cost vector. The fourth 5×5×5 3D convolutional layer is used to reduce the dimensionality of the cost vector to obtain the final refined cost vector. Each 5×5×5 3D convolution uses Batch Normalization and ReLU activation functions.

[0077] Specifically, the disparity regression and the first-stage disparity map implementation method in step d are as follows:

[0078] After obtaining the aggregated matching cost, this embodiment of the invention uses a soft argmin operation to perform disparity regression on this cost, obtaining a coarsely estimated disparity map at 1 / 8 resolution. This is then upsampled to full size using bilinear interpolation to obtain the first-stage disparity map. The specific formula for disparity regression is as follows:

[0079]

[0080] Where d represents disparity, M represents maximum disparity, and the maximum disparity in the first stage is M = 24. Ci(d') and Ci(d) represent the cost of disparity levels d' and d in the current pixel i, respectively. Specifically, the soft-argmin operation takes C... i (d') and C i The negative value of (d) transforms the cost quantity into a probability quantity. The soft-argmin weighted sum of all disparity values ​​can recover the minimum disparity of cost matching and improve the convergence speed of the model.

[0081] Specifically, the residual prediction and the second-stage disparity map implementation method in step e are as follows:

[0082] To reduce the maximum disparity between two pixels in the input stereo image, this patent limits the disparity range to between -2 and 2 by predicting the residual. Specifically, firstly, the coarsely estimated disparity map at 1 / 8 resolution in the first stage is upsampled to a 1 / 4 resolution disparity map, and the features of the right image in the second stage are distorted. Then, the distance-correlation grouping cost is constructed using the 1 / 4 resolution left image features extracted by the feature extractor and the distorted right image features. Finally, a residual map is obtained through a cost aggregation network and disparity regression. This residual map is added to the enlarged disparity map in the first stage to correct the disparity, generating a fine 1 / 4 resolution disparity map. Finally, it is upsampled to full size through bilinear interpolation to obtain the second-stage disparity map.

[0083] Specifically, the loss function for step f is implemented as follows:

[0084] After obtaining the disparity map, to improve the model's convergence speed, this embodiment of the invention uses the Adam optimizer to optimize the objective function, where betas1 = 0.9 and betas2 = 0.999. The loss weight coefficients for the first and second stages of joint training are different: the weight coefficient for the first stage is λ1 = 0.5, and the weight coefficient for the second stage is λ2 = 0.7. The specific formula for the loss function is as follows:

[0085]

[0086]

[0087]

[0088] Where x is the input variable of the loss function, and d is the disparity estimate. Here, N represents the true ground disparity at the same pixel, and N is the number of candidate disparities. The loss function for the first stage, Lsum represents the loss function for the second stage, and Lsum represents the final loss function of the model.

[0089] Specifically, the TensorRT inference acceleration implementation method for step g is as follows:

[0090] This invention pre-trains the model 40 times on the SceneFlow dataset, and then uses the pre-trained results to fine-tune the model 800 times on the KITTI 2012 and KITTI 2015 datasets. Finally, the fine-tuned model is accelerated for inference using the TensorRT optimizer on an NVIDIA Jetson Nano embedded device, and the optimized model is deployed to the Jetson Nano device.

[0091] In summary, this invention proposes a real-time binocular depth estimation method based on a two-stage distance-correlation grouping hybrid approach, enabling real-time and efficient disparity estimation. The feature extractor provided by this invention uses block convolution for downsampling, which helps retain more effective feature information and generate high-precision feature maps. The distance-correlation grouping method used in this invention to construct the cost vector provides excellent similarity measurement. The lightweight cost aggregation network provided by this invention has a compact structure, employing only four 5×5×5 3D convolutional layers in each stage, thus enabling fast inference speed for cost aggregation. Furthermore, existing techniques typically cannot generate high-precision disparity maps in real-time on resource-constrained embedded devices. In contrast, this invention uses the TensorRT optimizer to accelerate network inference, enabling real-time deployment on resource-constrained NVIDIA Jetson Nano devices while maintaining high accuracy.

[0092] Device Example 1

[0093] According to embodiments of the present invention, a two-stage real-time binocular depth estimation device based on grouped hybridization is provided. Figure 6 This is a schematic diagram of a two-stage real-time binocular depth estimation device based on grouped hybridization according to an embodiment of the present invention, as shown below. Figure 6 As shown, the two-stage real-time binocular depth estimation device based on grouping and mixing according to an embodiment of the present invention specifically includes:

[0094] The extraction module 60 is used to extract features from the original input image using a feature extractor based on block convolution, and obtain feature maps with resolutions of 1 / 4 and 1 / 8 relative to the original input image.

[0095] The first-stage disparity map module 62 is used to construct a group distance cost using a feature map with a resolution of 1 / 8. The group distance cost is regularized by a lightweight cost aggregation network to obtain an aggregated matching cost. The aggregated matching cost is used to generate a coarsely estimated disparity map with a resolution of 1 / 8 through disparity regression. The map is then upsampled to full size through bilinear interpolation to obtain the first-stage disparity map.

[0096] The second-stage disparity map module 64 is used to enlarge the coarsely estimated disparity map at 1 / 8 resolution into a 1 / 4 resolution disparity map. Based on the 1 / 4 resolution disparity map and the features of the left image, a grouping correlation cost is constructed by dynamic offset. The grouping correlation cost is used to obtain a residual map through a cost aggregation network and disparity regression. The residual map is added to the enlarged 1 / 4 resolution disparity map to generate a finely estimated 1 / 4 resolution disparity map. The map is then upsampled to full size through bilinear interpolation to obtain the second-stage disparity map.

[0097] The optimization inference module 66 is used to optimize the loss function based on the first-stage disparity map and the second-stage disparity map, obtain the optimized model, and use the TensorRT optimizer to accelerate the inference of the network layers of the optimized model.

[0098] Compared with the prior art, the embodiments of the present invention have at least the following beneficial effects:

[0099] (1) In the feature extraction step, this embodiment of the invention proposes a feature extractor based on block convolution, which uses continuous block convolution for downsampling. Compared with the prior art, this invention uses block convolution for downsampling, which can retain more feature information.

[0100] (2) In the cost quantity construction step, this embodiment of the invention proposes a two-stage method of distance-correlation grouping hybrid method to construct the cost quantity. The distance-correlation grouping cost quantity generates a multi-channel distance map and a correlation map for each disparity level. Compared with the prior art, this strategy can integrate more feature channel information and provide better similarity measurement.

[0101] (3) In the cost aggregation step, this embodiment of the invention proposes a lightweight three-dimensional cost aggregation network, which uses only four 5×5×5 three-dimensional convolutional layers in each stage to optimize the distance-correlation grouping cost. Compared with the prior art, the computational cost of cost aggregation in this invention is relatively small, and the 5×5×5 three-dimensional convolution has a larger receptive field, which improves the accuracy of identifying textureless regions.

[0102] (4) The embodiments of the present invention employ a two-stage coarse-to-fine processing method for disparity estimation. Compared with the prior art, this processing method can significantly reduce the computational complexity of the cost aggregation step and significantly improve the inference speed of the network.

[0103] (5) This embodiment of the invention uses the TensorRT optimizer to accelerate inference at the network layer. Compared with the prior art, this embodiment of the invention can deploy the KITTI 2012 and KITTI 2015 datasets in real time on resource-constrained NVIDIA Jetson Nano devices while maintaining high accuracy, which is of great significance in fields such as robot navigation, augmented reality, smart cities and autonomous driving.

[0104] The embodiments of the present invention are device embodiments corresponding to the above method embodiments. The specific operation of each module can be understood with reference to the description of the method embodiments, and will not be repeated here.

[0105] Device Example 2

[0106] This invention provides an electronic device, such as... Figure 7As shown, it includes: a memory 70, a processor 72, and a computer program stored in the memory 70 and executable on the processor 72, wherein the computer program, when executed by the processor 72, performs the steps as described in the method embodiment.

[0107] Device Example 3

[0108] This invention provides a computer-readable storage medium storing an information transmission implementation program, which, when executed by a processor 72, performs the steps described in the method embodiment.

[0109] The computer-readable storage media described in this embodiment include, but are not limited to, ROM, RAM, disk, or optical disk.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A two-stage real-time binocular depth estimation method based on packet mixing, characterized in that, Comprise: feature extraction is performed on the original input image by using a patch convolution-based feature extractor, and feature maps of 1 / 4 and 1 / 8 resolutions relative to the original input image are obtained; using the 1 / 8 resolution feature map to construct a grouped distance cost volume, and using a lightweight cost aggregation network to regularize the grouped distance cost volume to obtain an aggregated matching cost volume, and using the aggregated matching cost volume to generate a 1 / 8 resolution coarse estimated disparity map through disparity regression, and then up-sampling to full size through bilinear interpolation to obtain a first stage disparity map; wherein using the 1 / 8 resolution feature map to construct a grouped distance cost volume specifically comprises: According to formula 1, the 1 / 8 resolution feature map is used to construct a grouped distance cost volume: where ||·-·||1 denotes the L1 distance between two features, C gwd denotes the group distance value of the pixel similarity in the input picture, d denotes the disparity, x and y denote the feature vectors, g denotes the number of feature groups, f l and f r denote the left image features and the right image features, respectively; The 1 / 8 resolution coarse estimated disparity map is enlarged to a 1 / 4 resolution disparity map, and a grouped correlation cost volume is constructed according to the 1 / 4 resolution disparity map and the left image features, and a residual map is obtained by using the cost aggregation network and the disparity regression, and the residual map is added to the enlarged 1 / 4 resolution disparity map to generate a 1 / 4 resolution fine estimated disparity map, and then up-sampling to full size through bilinear interpolation to obtain a second stage disparity map; wherein constructing a grouped correlation cost volume according to the 1 / 4 resolution disparity map and the left image features specifically comprises: According to formula 2, a grouped correlation cost volume is constructed according to the 1 / 4 resolution disparity map and the left image features: where <·, ·> denotes the inner product between two features, C gwc denotes the group-wise cost value of the pixel similarity in the input picture, d denotes the disparity, x and y denote the feature vectors, g denotes the number of feature groups, f l and f r denote the left and right image features, respectively; Based on the first stage disparity map and the second stage disparity map, the loss function is optimized using the Adam optimizer to obtain an optimized model, and the network layer of the optimized model is inferred using the TensorRT optimizer to speed up the inference.

2. The method of claim 1, wherein, The feature extraction is performed on the original input image by using a patch convolution-based feature extractor, and feature maps of 1 / 4 and 1 / 8 resolutions relative to the original input image are obtained specifically comprising: Using a patch convolution-based feature extractor, first using patch convolution to down-sample the original input image to 1 / 2 resolution, 1 / 4 resolution and 1 / 8 resolution, and then performing deep degree feature extraction on the 1 / 4 resolution and 1 / 8 resolution features to obtain feature maps of 1 / 4 and 1 / 8 resolutions relative to the original input image.

3. The method of claim 1, wherein, The lightweight cost aggregation network comprises four three-dimensional convolution layers with a kernel size of 5x5x5, wherein the first 5x5x5 three-dimensional convolution is used to increase the dimension of the cost volume, the second and third 5x5x5 three-dimensional convolutions are used to optimize the cost volume, and the fourth 5x5x5 three-dimensional convolution layer is used to reduce the dimension of the cost volume.

4. The method of claim 1, wherein, The aggregated matching cost volume is generated through disparity regression to obtain a 1 / 8 resolution coarse estimated disparity map, and then up-sampling to full size through bilinear interpolation to obtain a first stage disparity map specifically comprising: Based on formula 3, the matching cost volume optimized by the cost aggregation network is used for disparity regression to obtain a 1 / 8 resolution coarse estimated disparity map, and then up-sampling to full size through bilinear interpolation to obtain a first stage disparity map: where M represents the maximum disparity, the maximum disparity M of the first stage is 24, d represents the disparity, C i (d') and C i (d) represent the cost value of the disparity level d' and d in the current pixel i, respectively.

5. The method of claim 1, wherein, The model optimization of the loss function is performed using an Adam optimizer based on the first-stage disparity map and the second-stage disparity map, and specifically includes: The loss function shown in formulas 4-6 is optimized using an Adam optimizer; wherein, wherein x is an input variable, d is a disparity estimation value, is a ground truth disparity at the same pixel, N is a number of candidate disparities, and are loss functions of the first stage and the second stage respectively, the loss weight coefficients of the first stage and the second stage are different, the weight coefficient of the first stage is λ1=0.5, and the weight coefficient of the second stage is λ2=0.

7.

6. A two-stage real-time binocular depth estimation apparatus based on packet mixing, characterized by, The method comprises the following steps: The extraction module is configured to extract features of the original input image using a patch convolution-based feature extractor to obtain feature maps with 1 / 4 and 1 / 8 resolutions relative to the original input image; The first-stage disparity map module is configured to construct a grouped distance cost volume using the feature map with the 1 / 8 resolution, regularize the grouped distance cost volume through a lightweight cost aggregation network, obtain an aggregated matching cost volume, generate a coarse estimated disparity map with the 1 / 8 resolution through disparity regression from the aggregated matching cost volume, and upsample the coarse estimated disparity map to full size through bilinear interpolation to obtain the first-stage disparity map; specifically, the first-stage disparity map module is configured to construct a grouped distance cost volume using the feature map with the 1 / 8 resolution according to formula 1: where ||·-·||1 denotes the L1 distance between two features, C gwd denotes the group distance value of the pixel similarity in the input picture, d denotes the disparity, x and y denote the feature vectors, g denotes the number of feature groups, f l and f r denote the left image features and the right image features, respectively; The second-stage disparity map module is configured to enlarge the coarse estimated disparity map with the 1 / 8 resolution to a 1 / 4 resolution disparity map, construct a grouped correlation cost volume based on the 1 / 4 resolution disparity map and left image features, obtain a residual map through a cost aggregation network and disparity regression from the grouped correlation cost volume, add the residual map to the enlarged 1 / 4 resolution disparity map to generate a fine estimated disparity map with the 1 / 4 resolution, and upsample the fine estimated disparity map to full size through bilinear interpolation to obtain the second-stage disparity map; specifically, the second-stage disparity map module is configured to construct a grouped correlation cost volume based on the 1 / 4 resolution disparity map and left image features according to formula 2: where <·, ·> denotes the inner product between two features, C gwc denotes the group-wise cost value of the pixel similarity in the input picture, d denotes the disparity, x and y denote the feature vectors, g denotes the number of feature groups, f l and f r denote the left and right image features, respectively; The optimization inference module is configured to perform model optimization of the loss function using an Adam optimizer based on the first-stage disparity map and the second-stage disparity map to obtain an optimized model, and perform inference acceleration on network layers of the optimized model using a TensorRT optimizer.

7. An electronic device, comprising: The computer program stored on the memory and executable on the processor implements the steps of the two-stage real-time binocular depth estimation method based on grouping mixing according to any one of claims 1 to 5 when executed by the processor. The computer readable storage medium stores an implementation program of information transmission, and the program implements the steps of the two-stage real-time binocular depth estimation method based on grouping mixing according to any one of claims 1 to 5 when executed by the processor.

8. A computer-readable storage medium, characterized in that, ​

Images