Lightweight binocular stereo matching method based on step-by-step long-range capture and detail restoration
The lightweight binocular stereo matching method addresses the challenge of balancing real-time performance and accuracy by employing a novel model with long-range capture and detail restoration techniques, achieving efficient processing and accurate depth estimation in diverse environments.
Patent Information
- Application Number
- CN202411345689.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-26
- Publication Date
- 2025-01-03
- Estimated Expiration
- 2044-09-26
AI Technical Summary
Traditional binocular stereo matching algorithms struggle to balance real-time performance and accuracy, particularly in capturing long-range dependencies and restoring details, especially in complex environments.
A lightweight binocular stereo matching method utilizing a lightweight feature extractor, cascade and group cost volume construction, dense scale perception fusion, and long-range perception fusion modules, optimized with smooth L1 loss and multi-stage training, to capture long-range relationships and restore details.
The method achieves real-time performance of 0.03s per image pair with 2.1M parameters, effectively capturing long-range dependencies and restoring details, demonstrating good generalization across natural and medical scene images.
Smart Images
Figure CN118864562B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a lightweight binocular stereo matching method based on step-by-step long-range capture and detail restoration. Background Art
[0002] Binocular stereo matching technology is an important depth perception technology in 3D vision methods. This technology recovers depth information from a pair of disparity maps obtained by a pair of binocular cameras and obtains a dense depth map. Dense and accurate depth estimation in the depth map is crucial to effectively solve higher-level tasks in computer vision, such as autonomous driving, 3D reconstruction, surgical perception, etc. Depth maps can be obtained using active devices or calculated from camera images. The former is limited by the environment and the performance of the device itself, and often only sparse depth maps can be obtained, which cannot meet some tasks with high performance requirements. The latter allows more information to be obtained in more complex environments with the help of algorithms, which is more in line with realistic scenes. Stereo matching algorithms based on camera images have a long history of development. Traditional algorithms cannot achieve relatively good results in textureless areas, repeated patterns, and thin structures. In recent years, with the development of deep learning, stereo matching algorithms based on this have achieved better performance in various complex scenarios by learning strong representations with the help of deep neural networks.
[0003] In practical application scenarios, balancing the real-time performance and accuracy of binocular stereo matching networks is a difficult problem. The mainstream method adopts a convolutional neural network based on a cost volume structure. However, due to the inherent local inductive bias of convolutional neural networks, the network usually cannot capture long-range dependencies well and focuses more on processing local information. Based on this, some scholars propose to use a low-resolution cost volume to capture long-range dependencies, and then use the corresponding resolution cost volume to restore the corresponding details, or use an hourglass-shaped cost volume aggregation module to extract long-range relationships through multiple downsampling and convolution. These methods often ignore the rich information between cost volumes, and often fail to capture long-range relationships under the constraints of real-time performance. The hourglass-shaped cost aggregation module is often unable to handle long-range relationships well due to the limitation of convolution. Binocular stereo matching usually requires the restoration of dense depth information in the field of view. More accurate capture of long-range relationships and recovery of details can often achieve better depth capture effects. Therefore, in order to effectively solve the above problems, a lightweight binocular stereo matching method based on step-by-step long-range capture and detail recovery is proposed. Summary of the invention
[0004] In view of the above technical problems, the present invention provides a lightweight binocular stereo matching method based on step-by-step long-range capture and detail restoration.
[0005] The technical solution adopted by the present invention to solve the technical problem is:
[0006] A lightweight binocular stereo matching method based on step-by-step long-range capture and detail restoration, the method comprising the following steps:
[0007] S100: Build a lightweight binocular stereo matching model, which includes a lightweight feature extractor, a cascade and group cost volume construction module, a dense scale perception fusion module, and a long-range perception fusion module. The model is optimized using smooth L1 loss and a multi-stage training method.
[0008] S200: Obtain a disparity map as an input image through a binocular camera, extract features of the input view through a lightweight feature extractor, and obtain a set of features with different resolutions;
[0009] S300: The cascade and group cost volume construction module is used to construct cascade cost volumes containing semantic information and group cost volumes containing similarity information of different scales according to the extracted features with different resolutions;
[0010] S400: the dense scale-aware fusion module fuses and upsamples group cost volumes of different scales in an iterative manner to obtain a new group cost volume that fuses multi-scale information and a cost volume that is updated after upsampling;
[0011] S500: The long-range perception fusion module uses the group cost volume containing similarity information to constrain the cascade cost volume containing semantic information to obtain new fused cost volume features. The fused cost volume is aggregated using an hourglass-shaped convolution module, and the above is sampled and fused with the corresponding new group cost volume to obtain the final cost volume.
[0012] S600: Using sparse disparity regression on the acquired final cost volume, and using an upsampling method to restore the disparity map of the final resolution to obtain a final result.
[0013] Preferably, the lightweight feature extractor includes a main feature extractor and a secondary feature extractor, wherein the main feature extractor is an encoder-decoder structure, the encoder is a residual network MobileNet-V3 pre-trained by ImageNet, and the secondary feature extractor includes two groups of convolutional neural networks. S200 includes:
[0014] The acquired input image is input into the encoder. At each different resolution in the encoder, it undergoes a convolution with a kernel of (1,4,4) and a step size of (1,2,2). It is then concatenated with the feature map of the corresponding resolution in the decoder and undergoes a convolution with a kernel of 3 and a step size of 1. The feature map of the corresponding resolution of the main feature extractor is output.
[0015] Use sub-feature extractors to learn sampling functions;
[0016] For the feature maps obtained by the primary feature extractor and the secondary feature extractor, a cascade operation is used at the corresponding resolutions to obtain a set of feature maps with different final resolutions.
[0017] Preferably, S300 includes:
[0018] S310: for the feature maps of the left and right views of the feature maps of each resolution output by the feature extractor, use two layers of convolution with a convolution kernel of 3 and a step size of 1 to perform feature compression on the feature maps of the left and right views, respectively compressing them to 1 / 2 of the feature dimension;
[0019] S320: Obtain a cascaded cost volume by cascading the compressed feature maps of the left and right views , the group cost volume is obtained by finding the group similarity of the feature maps of the original left and right views , specifically:
[0020] ;
[0021] in Represents a cascade operation, Represents the feature map features of the corresponding compressed left view, Represents the feature map features of the corresponding compressed right view, Respectively represent the horizontal and vertical coordinates in the image coordinate system, represents the disparity dimension, in the group cost volume, The feature vector representing the grouped left view, The feature vector representing the grouped right view, Indicates the number of groups. Indicates the number of channels of the feature vector;
[0022] S330: For the cascaded cost volume, a 3D convolution with a convolution kernel of 3 and a step size of 1 is used to compress the cascaded cost volume to align the corresponding group cost volume.
[0023] Preferably, S400 includes:
[0024] For each group cost volume of each resolution, it is cascaded with the group cost volume of a larger resolution to obtain a new group cost volume, specifically:
[0025] ;
[0026] in, represents the new group cost body, represents the cascade function, D represents the downsampling function, and i represents the resolution of the i-th layer cost volume;
[0027] Remove the resolution of 1 / 16 , the 1 / 16 resolution Upsampled and then compared with 1 / 8 resolution Connect them to get the new cost volume at 1 / 8 resolution, specifically:
[0028] ;
[0029] Wherein, U represents linear upsampling processing.
[0030] Preferably, the fused cost volume in S500 is aggregated using an hourglass-shaped convolution module, and the above sampling is fused with the corresponding new group cost volume to obtain the final cost volume, including:
[0031] An hourglass-shaped convolutional network is used to aggregate the new fused cost volume features. The resulting cost volume aggregates the information of each resolution. Then, the cost volume is processed using the softmax function in the disparity dimension. The resulting new cost volume is multiplied with the cascaded cost volume as a constraint coefficient to obtain the final cost volume at the resolution. The same operation is used at each resolution, specifically:
[0032] ;
[0033] Hourglass represents an hourglass-shaped convolutional neural network. The hourglass-shaped convolutional neural network includes three specific models: HourglassSimple, HourglassSimple2, and Hourglass. represents the final cost volume of the i-th layer.
[0034] Preferably, for the cost volume at a resolution of 1 / 32, the structure of HourglassSimple is used: for a given cost volume, it is first subjected to a layer of 3D convolution with a Leaky ReLU activation function, with a convolution kernel of 3 and a step size of 1, and then subjected to two layers of 3D convolution with a Leaky ReLU, with a step size of 3 and a dilation scale of 2, to obtain an intermediate cost volume; at the same time, the original given cost volume is subjected to a layer of 3D convolution with a convolution kernel of 1 and a step size of 1, and then added to the obtained intermediate cost volume, and then subjected to Leaky ReLU and a layer of 3D convolution with a convolution kernel of 3 and a step size of 1, to finally obtain the corresponding cost volume;
[0035] use Represents a 3D convolution with a convolution kernel of m, a step size of n, and a dilation coefficient of o. LReLU is used to represent the Leaky ReLU function. The above process can be expressed as:
[0036] ;
[0037] ;
[0038] ;
[0039] Among them, V represents the original given cost volume, represents the intermediate cost body, Represents the final cost volume of the corresponding resolution.
[0040] Preferably, for the cost volume at 1 / 8 resolution, the structure of HourglassSimple2 is used, in which convolution and deconvolution with downsampling function are used to restore the corresponding information: for a given cost volume, it is first processed by a layer of 3D convolution with a Leaky ReLU activation function, a convolution kernel of 3, and a step size of 2, and then a 3D convolution with a Leaky ReLU activation function, a convolution kernel of 1, and a step size of 1 and a 3D deconvolution with a convolution kernel of 3 and a step size of 2 to obtain an intermediate cost volume; at the same time, the original given cost volume is processed by a layer of 3D convolution with a convolution kernel of 1 and a step size of 1, and then added to the obtained intermediate cost volume, and then processed by Leaky ReLU to obtain the final cost volume;
[0041] use It represents 3D deconvolution with convolution kernel m and step length n. The specific process is as follows:
[0042] ;
[0043] .
[0044] Preferably, for the cost volume at 1 / 4 resolution, use the Hourglass aggregation module: for a given cost volume, first pass a layer of 3D convolution with a Leaky ReLU activation function, a convolution kernel of 3, and a step size of 2, and then pass a 3D convolution with a Leaky ReLU activation function, a convolution kernel of 3, and a step size of 1, and then obtain the first intermediate cost volume ; The first intermediate cost body continues to pass through the 3D convolution with Leaky ReLU activation function, convolution kernel 3, step size 2, 3D convolution with LeakyReLU activation function, convolution kernel 3, step size 1, and 3D deconvolution with convolution kernel 3 and step size 2. The resulting cost body is the same as the first intermediate cost body after the 3D convolution with Leaky ReLU activation function, convolution kernel 1, step size 1. After addition, the second intermediate cost volume is obtained by continuing to pass through the Leaky ReLU activation function and the 3D deconvolution with a convolution kernel of 3 and a step size of 2. It is added to the original given cost volume processed by 3D convolution with a convolution kernel of 1 and a step size of 1, and finally passes through a layer of Leaky ReLU activation function. The specific process is as follows:
[0045] ;
[0046] ;
[0047] ;
[0048] in, represents the first intermediate cost body, Represents the second intermediate cost volume.
[0049] Preferably, in S600, an upsampling method is used to restore the disparity map of the final resolution, specifically:
[0050] ;
[0051] in is the relationship vector after softmax processing, and They represent the restored disparity information and the original disparity information respectively, S represents the step size, p and q represent the coordinates of the restored image respectively, and a and b represent the index of an adjacent area.
[0052] Preferably, the preset loss function is a smooth L1 loss function, specifically:
[0053] ;
[0054] in, represents the smoothed L1 loss, Represents the prediction result, represents the true value;
[0055] The multi-stage training method optimization model includes three training stages. The first stage supervises the final result and the final cost volume result. The second stage supervises the cost volume output after aggregation at each resolution stage on this basis. The last stage continues to perform deep supervision on the cost volume output after LRM fusion at each resolution stage based on the second stage. The specific deep supervision process is as follows:
[0056] ;
[0057] ;
[0058] in represents a high-dimensional cost volume, represents the hyperparameters of the calculation, represents a downsampling function, represents the disparity regression function, represents the compression process of the matrix cost body, Represents the parameters of coefficient parallax regression, Top represents taking maximum value, t represents the index number, and r represents the rth supervision.
[0059] The above lightweight binocular stereo matching method based on step-by-step long-range capture and detail restoration, the proposed model is an ultra-lightweight model, with only 2.1M parameters, and the real-time performance of this model can reach 0.03s to process each pair of images. In addition, this model can achieve good generalization performance, that is, it has good effects on both natural scene images and medical scene images. BRIEF DESCRIPTION OF THE DRAWINGS
[0060] Figure 1 It is a flow chart of a lightweight binocular stereo matching method based on step-by-step long-range capture and detail restoration in one embodiment of the present invention;
[0061] Figure 2 A schematic diagram of a multi-stage training method in one embodiment of the present invention;
[0062] Figure 3 Three hourglass-shaped convolutional network structures in one embodiment of the present invention; among them, (a) is the Hourglass structure, (b) is the HourglassSimple structure, and (c) is the HourglassSimple2 structure;
[0063] Figure 4 This is a schematic diagram of the structure of a dense scale perception fusion module DSF in one embodiment of the present invention;
[0064] Figure 5 This is a specific schematic diagram of the long-range perception fusion module LRM fusion process in one embodiment of the present invention;
[0065] Figure 6 Schematic diagram of the effect of the present invention on processing street scenes and laparoscopic scenes, wherein (a) is a left view of a city street scene, (b) is a left view of a laparoscopic scene, (c) is a parallax prediction of a city street scene, (d) is a parallax prediction of a laparoscopic scene, (e) is a prediction error of a city street scene, and (f) is a prediction error of a laparoscopic scene. DETAILED DESCRIPTION
[0066] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings.
[0067] In one embodiment, Figure 1As shown, a lightweight binocular stereo matching method based on step-by-step long-range capture and detail recovery comprises the following steps:
[0068] S100: Build a lightweight binocular stereo matching model, which includes a lightweight feature extractor, a cascade and group cost volume construction module, a dense scale perception fusion module, and a long-range perception fusion module. The model is optimized using smooth L1 loss and a multi-stage training method.
[0069] In one embodiment, the preset loss function is a smooth L1 loss function, specifically:
[0070] ;
[0071] in, represents the smoothed L1 loss, Represents the prediction result, Represents the true value.
[0072] The multi-stage training process used is as follows Figure 2 As shown, the multi-stage training method optimization model includes three training stages. The first stage supervises the final result and the final cost body result, which is marked as 1. The second stage supervises the cost body of the aggregated output of each resolution stage on this basis, which are marked as 1 and 2. The last stage continues to perform deep supervision on the cost body of the LRM fusion output of each resolution stage on the basis of the second stage, and the corresponding ones are marked as 1, 2 and 3 that need to be supervised.
[0073] To simplify the operation, the intermediate cost body is not processed to generate the intermediate depth map, but unified as the structure of the cost body, so the depth supervision cannot be used directly. For a given cost body, the group dimension is usually greater than 1, so an additional convolution is required for compression. The specific depth supervision process is as follows:
[0074] ;
[0075] ;
[0076] in represents a high-dimensional cost volume, represents the hyperparameters of the calculation, represents a downsampling function, represents the disparity regression function, represents the compression process of the matrix cost body, Represents the parameters of coefficient parallax regression, Top represents taking maximum value, t represents the index number, and r represents the rth supervision.
[0077] S200: A disparity map is obtained as an input image through a binocular camera, and features of the input view are extracted through a lightweight feature extractor to obtain a set of features with different resolutions.
[0078] In one embodiment, the lightweight feature extractor includes a main feature extractor and a secondary feature extractor, wherein the main feature extractor is an encoder-decoder structure, the encoder is a residual network MobileNet-V3 pre-trained by ImageNet, and the secondary feature extractor includes two groups of convolutional neural networks. S200 includes:
[0079] The acquired input image is input into the encoder. At each different resolution in the encoder, a convolution with a kernel of (1,4,4) and a step size of (1,2,2) is performed to double the original resolution. Then, it is concatenated with the feature map of the corresponding resolution in the decoder and a convolution with a kernel of 3 and a step size of 1 is performed to output the feature map of the corresponding resolution of the main feature extractor.
[0080] Since the SPX-based upsampling method is adopted, the sampling function is learned using a sub-feature extractor;
[0081] For the feature maps obtained by the primary feature extractor and the secondary feature extractor, a cascade operation is used at the corresponding resolution to obtain a set of feature maps with different final resolutions. The feature channels of the feature maps at the final resolutions of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 are 90, 90, 96, and 96, respectively.
[0082] S300: The cascade and group cost volume construction module is used to construct cascade cost volumes containing semantic information and group cost volumes containing similarity information of different scales according to the extracted features of different resolutions.
[0083] In one embodiment, S300 includes:
[0084] S310: for the feature maps of the left and right views of the feature maps of each resolution output by the feature extractor, use two layers of convolution with a convolution kernel of 3 and a step size of 1 to perform feature compression on the feature maps of the left and right views, respectively compressing them to 1 / 2 of the feature dimension;
[0085] S320: Obtain a cascaded cost volume by cascading the compressed feature maps of the left and right views , the group cost volume is obtained by finding the group similarity of the feature maps of the original left and right views , specifically:
[0086] ;
[0087] ;
[0088] in Represents a cascade operation, Represents the feature map features of the corresponding compressed left view, Represents the feature map features of the corresponding compressed right view, Respectively represent the horizontal and vertical coordinates in the image coordinate system, represents the disparity dimension, in the group cost volume, The feature vector representing the grouped left view, The feature vector representing the grouped right view, Indicates the number of groups. Indicates the number of channels of the feature vector;
[0089] S330: For the cascaded cost volume, a 3D convolution with a convolution kernel of 3 and a step size of 1 is used to compress the cascaded cost volume to align the corresponding group cost volume.
[0090] S400: The dense scale-aware fusion module fuses and upsamples group cost volumes of different scales in an iterative manner to obtain a new group cost volume that fuses multi-scale information and an updated cost volume after upsampling.
[0091] Specifically, the global perception ability of the model at each scale is enhanced by directly obtaining the constrained cost volume that fuses multi-scale information in an iterative manner, and the model's processing of details is enhanced by restoring detail information in the upsampling stage.
[0092] In one embodiment, S400 includes:
[0093] For each group cost volume of each resolution, it is cascaded with the group cost volume of a larger resolution to better obtain the long-range relationship between the cost volumes of each resolution and obtain a new group cost volume, specifically:
[0094] ;
[0095] in, represents the new group cost body, represents the cascade function, D represents the downsampling function, and i represents the resolution of the i-th layer cost volume;
[0096] In order to improve the real-time performance of the model, but also save each resolution information, remove the resolution of 1 / 16 , but at the resolution of 1 / 32, due to the relationship of formula (6), part of the information at the original resolution of 1 / 16 is preserved. At the same time, for the resolution of 1 / 8 , the 1 / 16 resolution Upsampled and then compared with 1 / 8 resolution Connect them to get the new cost volume at 1 / 8 resolution, specifically:
[0097] ;
[0098] Wherein, U represents linear upsampling processing.
[0099] On the basis of capturing long-range dependencies and taking into account the restoration of details, the final processed cost volume for each resolution is , sample the above and combine with the corresponding Fusion, in this process , a cost volume with long-range information is used to fuse it.
[0100] S500: The long-range perception fusion module uses the group cost volume containing similarity information to constrain the cascade cost volume containing semantic information to obtain new fused cost volume features. The fused cost volume is aggregated using an hourglass-shaped convolution module, and the above is sampled and fused with the corresponding new group cost volume to obtain the final cost volume.
[0101] In one embodiment, the fused cost volume in S500 is aggregated using an hourglass-shaped convolution module, and the upsampled cost volume is fused with the corresponding new group cost volume to obtain a final cost volume, including:
[0102] An hourglass-shaped convolutional network is used to aggregate the new fused cost volume features. The resulting cost volume aggregates the information of each resolution. Then, the cost volume is processed using the softmax function in the disparity dimension. The resulting new cost volume is multiplied with the cascaded cost volume as a constraint coefficient to obtain the final cost volume at the resolution. The same operation is used at each resolution, specifically:
[0103] ;
[0104] Hourglass represents an hourglass-shaped convolutional neural network, which includes three specific models: Figure 3 (a)-(c) are Hourglass, HourglassSimple and HourglassSimple2 respectively. represents the final cost volume of the i-th layer.
[0105] In one embodiment, for the cost volume at a resolution of 1 / 32, the structure of HourglassSimple is used: for a given cost volume, it first undergoes a layer of 3D convolution with a Leaky ReLU activation function, with a convolution kernel of 3 and a step size of 1, and then undergoes two layers of 3D convolution with a Leaky ReLU with a step size of 3 and a dilation scale of 2 to obtain an intermediate cost volume; at the same time, the original given cost volume undergoes a layer of 3D convolution with a convolution kernel of 1 and a step size of 1, and then is added to the obtained intermediate cost volume, and then undergoes Leaky ReLU and a layer of 3D convolution with a convolution kernel of 3 and a step size of 1 to finally obtain the corresponding cost volume;
[0106] use Represents a 3D convolution with a convolution kernel of m, a step size of n, and a dilation coefficient of o. LReLU is used to represent the Leaky ReLU function. The above process can be expressed as:
[0107] ;
[0108] ;
[0109] ;
[0110] Among them, V represents the original given cost volume, represents the intermediate cost body, Represents the final cost volume of the corresponding resolution.
[0111] In one embodiment, for the cost volume at 1 / 8 resolution, the structure of HourglassSimple2 is used. In HourglassSimple2, convolution and deconvolution with downsampling functions are used to restore corresponding information: for a given cost volume, it is first processed by a layer of 3D convolution with a Leaky ReLU activation function, a convolution kernel of 3, and a step size of 2, and then a 3D convolution with a Leaky ReLU activation function, a convolution kernel of 1, and a step size of 1 and a 3D deconvolution with a convolution kernel of 3 and a step size of 2 are performed to obtain an intermediate cost volume; at the same time, the original given cost volume is processed by a layer of 3D convolution with a convolution kernel of 1 and a step size of 1, and then added to the obtained intermediate cost volume, and then processed by Leaky ReLU to obtain the final cost volume;
[0112] use It represents 3D deconvolution with convolution kernel m and step length n. The specific process is as follows:
[0113] ;
[0114] .
[0115] In one embodiment, for the cost volume at 1 / 4 resolution, the Hourglass aggregation module is used: for a given cost volume, it is first processed by a layer of 3D convolution with a Leaky ReLU activation function, a convolution kernel of 3, and a stride of 2, and then by a 3D convolution with a Leaky ReLU activation function, a convolution kernel of 3, and a stride of 1. At this time, the first intermediate cost volume is obtained. ; The first intermediate cost body continues to pass through the 3D convolution with Leaky ReLU activation function, convolution kernel 3, step size 2, 3D convolution with Leaky ReLU activation function, convolution kernel 3, step size 1, and 3D deconvolution with convolution kernel 3 and step size 2. The resulting cost body is the same as the first intermediate cost body after the 3D convolution with Leaky ReLU activation function, convolution kernel 1, step size 1. After addition, the second intermediate cost volume is obtained by continuing to pass through the Leaky ReLU activation function and the 3D deconvolution with a convolution kernel of 3 and a step size of 2. It is added to the original given cost volume processed by 3D convolution with a convolution kernel of 1 and a step size of 1, and finally passes through a layer of Leaky ReLU activation function. The specific process is as follows:
[0116] ;
[0117] ;
[0118] ;
[0119] in, represents the first intermediate cost body, Represents the second intermediate cost volume.
[0120] For the fused cost body, the present invention uses the same hourglass-shaped convolutional neural network module for aggregation, which is specifically set to use the HourglassSimple structure at a resolution of 1 / 32, the HourglassSimple2 structure at a resolution of 1 / 8, and the Hourglass structure at a resolution of 1 / 4.
[0121] S600: Using sparse disparity regression on the acquired final cost volume, and using an upsampling method to restore the disparity map of the final resolution to obtain a final result.
[0122] For the learned cost volume at 1 / 4 resolution, i.e., the final cost volume, the present invention uses the sparse regression represented by formula (3) to obtain the corresponding disparity map, and uses the SPX-based upsampling method to obtain the disparity map of the original resolution.
[0123] In one embodiment, in S600, an additional set of feature extractors is used to learn the connectivity between pixels, and then a corresponding relationship vector is obtained, and an upsampling method is used to restore the disparity map of the final resolution, specifically:
[0124] ;
[0125] in is the relationship vector after softmax processing, and They represent the restored disparity information and the original disparity information respectively, S represents the step size, p and q represent the coordinates of the restored image respectively, and a and b represent the index of an adjacent area.
[0126] Compared with the prior art, the present invention has the following technical effects:
[0127] The model proposed in this invention is an ultra-lightweight model with only 2.1M parameters. At the same time, the real-time performance of this model can reach 0.03s to process each pair of images. In addition, this model can achieve good generalization performance, that is, it has good effects on both natural scene images and medical scene images. Figure 6 shown. Figure 6 On the left is the processing effect in the urban scene, and on the right is the processing effect of the medical laparoscope image, where (a) is the left view of the urban street scene, (b) is the left view of the laparoscope scene, (c) is the disparity prediction of the urban street scene, (d) is the disparity prediction of the laparoscope scene, (e) is the prediction error of the urban street scene, and (f) is the prediction error of the laparoscope scene.
[0128] The above is a detailed introduction to the lightweight binocular stereo matching method based on step-by-step long-range capture and detail recovery provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the core idea of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.
Claims
1. A lightweight binocular stereo matching method based on step-by-step long-range capture and detail restoration, characterized in that: The method comprises the following steps: S100: Build a lightweight binocular stereo matching model, which includes a lightweight feature extractor, a cascade and group cost volume construction module, a dense scale perception fusion module, and a long-range perception fusion module. The model is optimized using smooth L1 loss and a multi-stage training method. S200: Obtain a disparity map as an input image through a binocular camera, extract features of the input view through a lightweight feature extractor, and obtain a set of features with different resolutions; S300: The cascade and group cost volume construction module is used to construct cascade cost volumes containing semantic information and group cost volumes containing similarity information of different scales according to the extracted features with different resolutions; S400: The dense scale-aware fusion module fuses and upsamples group cost volumes of different scales in an iterative manner to obtain a new group cost volume that fuses multi-scale information and a group cost volume that is updated after upsampling and contains similarity information; S400 includes: The group cost volume of each resolution is cascaded with the group cost volume with a larger resolution to obtain a new group cost volume, specifically: ； in, represents the new group cost body, represents the group cost body, represents the cascade function, D represents the downsampling function, and i represents the resolution of the i-th layer cost volume; Remove the resolution of 1 / 16 , the 1 / 16 resolution Upsampled and then compared with 1 / 8 resolution Connect them to get the new group cost volume at 1 / 8 resolution, specifically: ； Wherein, U represents linear upsampling processing; S500: The long-range perception fusion module uses the group cost volume containing similarity information updated after upsampling to constrain the cascade cost volume containing semantic information to obtain new fused cost volume features. The fused cost volume is aggregated using an hourglass-shaped convolution module, and the aggregated cost volume is upsampled and fused with the corresponding new group cost volume to obtain the final cost volume. S600: Using sparse disparity regression on the acquired final cost volume, and using an upsampling method to restore the disparity map of the final resolution to obtain a final result.
2. The method according to claim 1, characterized in that The lightweight feature extractor includes a main feature extractor and a secondary feature extractor. The main feature extractor is an encoder-decoder structure, the encoder is the residual network MobileNet-V3 pre-trained on ImageNet, and the secondary feature extractor includes two sets of convolutional neural networks. S200 includes: The acquired input image is input into the encoder. At each different resolution in the encoder, it undergoes a convolution with a kernel of (1,4,4) and a step size of (1,2,2). It is then concatenated with the feature map of the corresponding resolution in the decoder and undergoes a convolution with a kernel of 3 and a step size of 1. The feature map of the corresponding resolution of the main feature extractor is output. Use sub-feature extractors to learn sampling functions; For the feature maps obtained by the primary feature extractor and the secondary feature extractor, a cascade operation is used at the corresponding resolutions to obtain a set of feature maps with different final resolutions.
3. The method according to claim 2, characterized in that S300 includes: S310: for the feature maps of the left and right views of the feature maps of each resolution output by the feature extractor, use two layers of convolution with a convolution kernel of 3 and a step size of 1 to perform feature compression on the feature maps of the left and right views, respectively compressing them to 1 / 2 of the feature dimension; S320: Obtain a cascaded cost volume by cascading the compressed feature maps of the left and right views , the group cost volume is obtained by finding the group similarity of the feature maps of the original left and right views , specifically: ； ； in Represents a cascade operation, Represents the feature map features of the corresponding compressed left view, Represents the feature map features of the corresponding compressed right view, Respectively represent the horizontal and vertical coordinates in the image coordinate system, represents the disparity dimension, in the group cost volume, The feature vector representing the grouped left view, The feature vector representing the grouped right view, Indicates the number of groups. Indicates the number of channels of the feature vector; S330: For the cascaded cost volume, a 3D convolution with a convolution kernel of 3 and a step size of 1 is used to compress the cascaded cost volume to align the corresponding group cost volume.
4. The method according to claim 3, characterized in that The fused cost body in S500 is aggregated using an hourglass-shaped convolution module, and the aggregated cost body is upsampled and fused with the corresponding new group cost body to obtain the final cost body, including: An hourglass-shaped convolutional network is used to aggregate the new fused cost volume features. The resulting cost volume aggregates the information of each resolution. Then, the cost volume is processed using the softmax function in the disparity dimension. The resulting new cost volume is multiplied with the cascaded cost volume as a constraint coefficient to obtain the final cost volume at the resolution. The same operation is used at each resolution, specifically: ； Hourglass represents an hourglass-shaped convolutional neural network. The hourglass-shaped convolutional neural network includes three specific models: HourglassSimple, HourglassSimple2, and Hourglass. represents the final cost volume of the i-th layer.
5. The method according to claim 4, characterized in that For the cost volume at a resolution of 1 / 32, the structure of HourglassSimple is used: for a given cost volume, it first undergoes a layer of 3D convolution with a Leaky ReLU activation function, with a kernel of 3 and a step size of 1, and then undergoes two layers of 3D convolution with a Leaky ReLU with a step size of 3 and a dilation scale of 2 to obtain an intermediate cost volume; at the same time, the original given cost volume undergoes a layer of 3D convolution with a kernel of 1 and a step size of 1, and then is added to the obtained intermediate cost volume, and then passes through a Leaky ReLU and a layer of 3D convolution with a kernel of 3 and a step size of 1 to finally obtain the corresponding cost volume; use Represents a 3D convolution with a convolution kernel of m, a step size of n, and a dilation coefficient of o. LReLU is used to represent the Leaky ReLU function. The above process can be expressed as: ； ； ； Among them, V represents the original given cost volume, represents the intermediate cost body, Represents the final cost volume of the corresponding resolution.
6. The method according to claim 5, characterized in that For the cost volume at 1 / 8 resolution, the structure of HourglassSimple2 is used. In HourglassSimple2, convolution and deconvolution with downsampling function are used to restore the corresponding information: for a given cost volume, it is first processed by a layer of 3D convolution with Leaky ReLU activation function, kernel 3, and stride 2, and then a 3D convolution with Leaky ReLU activation function, kernel 1, stride 1 and a 3D deconvolution with kernel 3 and stride 2 are performed to obtain the intermediate cost volume; at the same time, the original given cost volume is processed by a layer of 3D convolution with kernel 1 and stride 1, and then added to the obtained intermediate cost volume, and then processed by Leaky ReLU to obtain the final cost volume; use It represents 3D deconvolution with convolution kernel m and step length n. The specific process is as follows: ； 。 7. The method according to claim 6, characterized in that For the cost volume at 1 / 4 resolution, use the Hourglass aggregation module: for a given cost volume, first pass it through a layer of 3D convolution with a Leaky ReLU activation function, a convolution kernel of 3, and a stride of 2, and then pass it through a 3D convolution with a Leaky ReLU activation function, a convolution kernel of 3, and a stride of 1, and then get the first intermediate cost volume ; The first intermediate cost body continues to pass through the 3D convolution with Leaky ReLU activation function, convolution kernel 3, step size 2, 3D convolution with Leaky ReLU activation function, convolution kernel 3, step size 1, and 3D deconvolution with convolution kernel 3 and step size 2. The resulting cost body is the same as the first intermediate cost body after the 3D convolution with Leaky ReLU activation function, convolution kernel 1, step size 1. After addition, the second intermediate cost volume is obtained by continuing to pass the Leaky ReLU activation function and the 3D deconvolution with a convolution kernel of 3 and a step size of 2. It is added to the original given cost volume processed by 3D convolution with a convolution kernel of 1 and a step size of 1, and finally passes through a layer of Leaky ReLU activation function. The specific process is as follows: ； ； ； in, represents the first intermediate cost body, Represents the second intermediate cost volume.
8. The method according to claim 7, characterized in that In S600, an upsampling method is used to restore the disparity map of the final resolution, specifically: ； in is the relationship vector after softmax processing, and They represent the restored disparity information and the original disparity information respectively, S represents the step size, p and q represent the coordinates of the restored image respectively, and a and b represent the index of an adjacent area.
9. The method according to claim 8, characterized in that The default loss function is the smooth L1 loss function, specifically: ； in, represents the smoothed L1 loss, Represents the prediction result, represents the true value; The multi-stage training method optimization model includes three training stages. The first stage supervises the final result and the final cost volume result. The second stage supervises the cost volume output after aggregation at each resolution stage on this basis. The last stage continues to perform deep supervision on the cost volume output after LRM fusion at each resolution stage based on the second stage. The specific deep supervision process is as follows: ； ； in represents a high-dimensional cost volume, represents the hyperparameters of the calculation, represents a downsampling function, represents the disparity regression function, represents the compression process of the matrix cost body, Represents the parameters of coefficient parallax regression, Top represents taking maximum value, t represents the index number, and r represents the rth supervision.