Method and system for stereo matching from monocular images with feature consistency

Abstract

The application discloses a method and system for stereo matching from monocular images by means of feature consistency. The method comprises the following steps: firstly, generating a sharpened disparity map from a single real image; then using these imperfect disparity maps and the single real image to generate a stereo image pair for training of a stereo matching model; finally, replacing the feature extraction layer of the stereo matching model with a teacher-student model, and imposing a binocular contrast feature constraint on the output of the model. Compared with the prior art, the application adopts a data synthesis method to generate a large amount of stereo image pair data set to solve the problem of insufficient training data for the stereo matching task; the feature extraction structure of the stereo matching model is modified to improve the domain generalization ability of the network. The application combines data enhancement with a stereo matching model with a contrast learning structure, so that the accuracy of the whole system is significantly improved.

Description

Technical Field

[0001] This invention relates to the field of binocular depth estimation, and specifically to a stereo matching method and system based on deep learning. Background Technology

[0002] Stereo matching, also known as disparity estimation or binocular depth estimation, takes as input a pair of epipolar-corrected left and right images captured simultaneously. l and I r Its output is a disparity map d, composed of the disparity values ​​of each pixel in a reference image (usually the left image). Disparity is the pixel-level difference between the positions of corresponding points in the left and right images of a 3D scene. If the disparity map d can be obtained, it can be calculated using the formula... Obtain the depth map z, where f is the focal length of the stereo camera.

[0003] Deep learning-based stereo matching methods are a way to acquire depth information, overcoming the limitations of sparse LiDAR data and the unsuitability of depth cameras for outdoor scenes. However, these methods all require a large number of left and right views and their corresponding disparity map labels. While left and right views can be easily collected in the real world using binocular cameras, disparity map labels typically require LiDAR scanners, which is a costly solution. Therefore, creating a synthetic dataset with perfect disparity information has become a pressing technical problem.

[0004] While synthetic datasets can largely address the issue of insufficient datasets in the stereo matching domain, there are generally significant domain style differences between synthetic and real datasets. If the stereo matching network cannot adapt well to these domain style differences, it will inevitably lead to a decrease in model accuracy. Therefore, the domain adaptation problem in the stereo matching domain has become an urgent technical challenge. Summary of the Invention

[0005] To address at least one of the existing problems—how to synthesize stereo matching binocular training datasets to solve the problem of insufficient training data for stereo matching tasks, and how to improve stereo matching models to enhance their domain generalization ability—this invention provides a method, system, apparatus, and medium for stereo matching from monocular images by leveraging feature consistency.

[0006] To achieve the objective of this invention, this invention provides a method for stereo matching from monocular images using feature consistency, comprising the following steps:

[0007] Generate corresponding binocular images and disparity maps based on monocular images;

[0008] The feature extraction layer of the stereo matching model is transformed into a teacher-student model, and a binocular contrast feature loss is applied. L f To contrast the loss, (u,v) represents the number of pixels in the image, M u,v The mask represents the unoccluded area, and C represents the pixel coordinate space;

[0009] The binocular images and disparity maps are fed into the modified stereo matching model for training. During training, the left and right images obtained in the previous steps, along with their corresponding disparity maps, are fed into the modified stereo matching model. This allows the feature consistency training network to improve the accuracy of binocular depth estimation.

[0010] The single image to be tested is input into the modified stereo matching model obtained through training to obtain a disparity map and a depth map.

[0011] When making predictions, the system can input only a single image and output a disparity map and depth map from the binocular camera.

[0012] Furthermore, the step of generating a corresponding binocular image and disparity map based on the monocular image includes:

[0013] To acquire a monocular image, such as using an RGB camera to acquire image information, the image is set as the left image in the binocular image;

[0014] The monocular image is fed into a pre-trained monocular depth estimation network to generate a depth map.

[0015] Based on the camera parameters, the depth map is converted into a disparity map;

[0016] The obtained disparity map is sharpened to obtain a clearer disparity map;

[0017] Based on the obtained disparity map, the monocular image is processed by forward warping and background filling to obtain another monocular image in the binocular image. As previously set, the image is the left image in the binocular image, and the image obtained here is the right image.

[0018] Furthermore, through Convert the depth map Z to a disparity map Z max is the maximum value of the depth map, and s is the random sampling scale factor.

[0019] Furthermore, in the improved stereo matching model:

[0020] A contrastive learning mechanism is introduced: the left feature extraction layer of the stereo matching model is used as the backbone network of the teacher model, and the right feature extraction layer is used as the backbone network of the student model. The network structure of the student model and the teacher model is the same, only the network parameters are different. In this framework, the source domain and target domain data are trained together end-to-end.

[0021] Source and target data are obtained from the feature maps encoded by the teacher and student models: paired source domain data, i.e., query features of the left view, are obtained using labeled disparity D. Key features of the right view Where (u,v) represents a pixel in an image of size H×W, and d represents the disparity;

[0022] Momentum-based dynamic dictionary: The dictionary has a capacity of K, and its data are all from the most recent target domain data, exhibiting a first-in, first-out (FIFO) characteristic. In each minimum batch, the latest N target domain data are stored, and the oldest N data are deleted. During training, the weight parameters W of the student model are adjusted. k Update:

[0023] W k ←W k +(1-α)W q ,

[0024] Where α∈(0,1) is the momentum coefficient, W q Indicates the teacher model parameters;

[0025] Remove occluded regions: Estimate the confidence map from the parallax space, and pixels with low confidence will be identified as occluded regions.

[0026] Furthermore, the mask M for the unobstructed area u,v for

[0027]

[0028] In the formula, R u,v This is the reprojection error.

[0029] Furthermore, when training the modified stereo matching model, a supervised learning approach is used to minimize the weighted sum of disparity loss and binocular contrast loss:

[0030] L = L disp +λ bcf L bcf

[0031] Where, λ bcf It is the balance coefficient, L disp It is a smooth L1 loss.

[0032] Furthermore, L disp The loss function is:

[0033]

[0034]

[0035] Where N is the number of pixels, d is the disparity map label, and It is the predicted disparity, where x represents This represents the classic smoothed L1 loss.

[0036] The present invention also provides a stereo matching system for monocular images by means of feature consistency, comprising the following modules:

[0037] The stereo training data synthesis module is used to generate corresponding binocular images and disparity maps based on monocular images;

[0038] The model transformation module is used to transform the feature extraction layer of the stereo matching model into a teacher-student model and apply a binocular contrast feature loss. L f To contrast the loss, (u,v) represents the number of pixels in the image, M u,v The mask represents the unoccluded area, and C represents the pixel coordinate space;

[0039] The model training module feeds the binocular images and disparity maps into the modified stereo matching model to train the modified stereo matching model.

[0040] The model prediction module takes the single image to be tested and inputs it into the modified stereo matching model obtained through training to obtain a disparity map and a depth map.

[0041] The present invention also provides a system apparatus for stereo matching from monocular images by means of feature consistency, comprising:

[0042] At least one processor;

[0043] At least one memory for storing at least one program;

[0044] When the at least one program is executed by the at least one processor, the at least one processor performs the aforementioned method.

[0045] The present invention also provides a storage medium storing a processor-executable program, which, when executed by a processor, is used to perform the aforementioned method.

[0046] Compared with the prior art, the present invention can achieve at least the following beneficial effects:

[0047] (1) The method for synthesizing stereo training datasets proposed in this invention can greatly reduce the dependence on disparity map labels and their corresponding stereo pairs. This invention can convert any monocular image dataset into a binocular image dataset, making it possible to train stereo networks using existing large monocular image datasets.

[0048] (2) The improved stereo matching model of the present invention can improve the domain generalization performance of the stereo matching model and improve the robustness of the stereo matching results.

[0049] (3) The present invention can significantly improve the accuracy and domain generalization performance of end-to-end stereo matching models. Attached Figure Description

[0050] Figure 1 This is a framework diagram of the stereo matching model before its improvement in this embodiment of the invention;

[0051] Figure 2 This invention provides an overall flowchart of a method for stereo matching from monocular images using feature consistency as an embodiment of the present invention.

[0052] Figure 3 This is a flowchart illustrating a training method for stereo matching from monocular images using feature consistency, as provided in an embodiment of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] Please see Figure 1 The present invention provides a method for stereo matching from monocular images by means of feature consistency, comprising the following steps:

[0055] Step 1: Use an RGB camera to acquire image information and set the image as the left image in the binocular image.

[0056] Step 2: Feed the left image into a pre-trained monocular depth estimation network to generate a depth map.

[0057] Step 3: Convert the depth map into a disparity map based on the camera parameters.

[0058] Step 4: Sharpen the obtained disparity map to get a clearer disparity map.

[0059] Step 5: Based on the obtained disparity map, perform forward warping and background filling on the left image to obtain the right image in the binocular image.

[0060] Step 6: Transform the feature extraction layer of the stereo matching model into a teacher-student model and apply binocular contrast feature loss.

[0061] Step 7: Train the improved stereo matching model. During model training, the left and right images and their corresponding disparity maps obtained in the above steps are fed into the modified stereo matching model. This allows the feature consistency training network to improve the accuracy of binocular depth estimation.

[0062] Step 8: Use the improved stereo matching model obtained from training for prediction. During model prediction, only a single image can be input into the system, and the final output can be the disparity map and depth map from the stereo camera.

[0063] To address the issue of insufficient training datasets for deep stereo matching models, the data synthesis method of this invention generates corresponding binocular images and disparity map labels based on existing monocular images. Specifically, it includes the following steps:

[0064] Collect monocular images and use them as input for a stereo training data synthesis method;

[0065] Use a monocular depth estimation network to predict the depth map of a monocular image;

[0066] Given the focal length and pose relationship of a stereo camera, through Convert the depth map Z to a disparity map Z max is the maximum value of the depth map, and s is the random sampling scale factor to ensure that the generated disparity is within a reasonable range. At the same time, disparity sharpening is performed to improve the initial disparity prediction effect.

[0067] The corresponding stereo image pairs are synthesized from the new viewpoint using forward warping and background padding.

[0068] Preferably, the stereo data synthesis method is a fully automated process that can generate a stereo training dataset from an unstructured monocular image set.

[0069] Preferably, the improved stereo matching model is as follows:

[0070] ① Comparative learning framework for stereo matching models

[0071] This invention introduces a contrastive learning mechanism into the framework of a stereo matching model. Specifically, the left feature extraction layer of the stereo matching model is used as the backbone network of the teacher model, and the right feature extraction layer is used as the backbone network of the student model. The network structures of the student and teacher models are the same, only the network parameters differ. In this framework, data from both the source and target domains are trained end-to-end.

[0072] ② Obtain source domain data and target domain data

[0073] Source and target data are obtained from the feature maps encoded by the teacher and student models. Paired source domain data, i.e., the query features of the left view, can be obtained using labeled disparity D. Key features of the right view Where (u,v) represents a pixel in an image of size H×W, and d represents the disparity. Theoretically, HW-1 target domain data can be obtained from paired features, but this would significantly increase computational costs. To reduce computational costs, in some embodiments of the present invention, from N target domain data are randomly selected from a 50×50 local window, where 0 < N ≤ 2500.

[0074] ③ Momentum Comparison Dynamic Dictionary

[0075] The dictionary has a capacity of K (K is much larger than N), and its data comes from the most recent target domain data. It functions like a queue, exhibiting a first-in, first-out (FIFO) characteristic. In each minimum batch, the N most recent target domain data are stored, while the N oldest data are deleted. During training, the weight parameters W of the student model are adjusted. k Update:

[0076] W k ←W k +(1-α)W q (1)

[0077] Where α∈(0,1) is the momentum coefficient. W q This represents the teacher model parameters and is updated through backpropagation.

[0078] ④ Remove obstructed areas

[0079] Binocular stereo matching is widely plagued by occlusion issues, which can cause mismatches in the source domain data. Therefore, occluded pixels need to be removed. This invention directly estimates a confidence map from the disparity space and uses it to detect incorrect disparity estimation results. Pixels with low confidence are identified as occluded regions. This invention uses a method for generating the confidence map and a left-right consistency check. Reprojection error R u,vThe difference in label disparity values ​​between paired pixel locations in a stereo image is calculated and can be used as a standard for checking the validity of the match. The mask M represents the unoccluded region. u,v Defined as:

[0080]

[0081] Where δ is a threshold, and in some embodiments of the present invention, δ = 0.3 when training a synthetic dataset and δ = 3 when training a real dataset.

[0082] Binocular contrast feature loss L bcf Defined as contrast loss L f Weighted average value in pixel coordinate space C:

[0083]

[0084] Preferably, the training method is as follows:

[0085] Using supervised learning training, minimize the weighted sum of disparity loss and binocular contrast loss:

[0086] L = L disp +λ bcf L bcf (4)

[0087] Where λ bcf It is the balance coefficient, L disp It is a smoothed L1 loss. Compared to L2 loss, the smoothed L1 loss is widely used for bounding box regression in object detection due to its robustness and low sensitivity to outliers. disp The loss function is defined as:

[0088]

[0089]

[0090] Where N is the number of pixels, and d is the disparity map label. It is the predicted disparity, where x represents This represents the classic smoothed L1 loss.

[0091] During model training, the left and right images and their corresponding disparity maps obtained from the above steps are fed into the modified stereo matching model. By using feature consistency training network, the accuracy of binocular depth estimation can be improved.

[0092] The improved stereo matching model method, through an improved learning framework and binocular contrast feature loss, can effectively promote cross-domain consistency of binocular features and improve the generalization performance of the stereo matching model.

[0093] When making predictions, the system can input only a single image and output a disparity map and depth map from the binocular camera.

[0094] The method provided in the foregoing embodiments of the present invention combines data augmentation with a stereo matching model with a contrastive learning structure, which significantly improves the accuracy of the entire system. It can be applied to most existing stereo matching models and achieves end-to-end training, thereby improving the accuracy of stereo matching.

[0095] In some embodiments of the present invention, a stereo matching system for monocular images by means of feature consistency is provided, characterized in that, for implementing the method provided in the foregoing embodiments, it includes the following modules:

[0096] The stereo training data synthesis module is used to generate corresponding binocular images and disparity maps based on monocular images;

[0097] The model transformation module is used to transform the feature extraction layer of the stereo matching model into a teacher-student model and apply a binocular contrast feature loss. L f To contrast the loss, (u,v) represents the number of pixels in the image, M u,v The mask represents the unoccluded area, and C represents the pixel coordinate space;

[0098] The model training module feeds the binocular images and disparity maps into the modified stereo matching model to train the modified stereo matching model.

[0099] The model prediction module takes the single image to be tested and inputs it into the modified stereo matching model obtained through training to obtain a disparity map and a depth map.

[0100] In some embodiments of the present invention, a stereo matching system apparatus for performing stereo matching from monocular images by means of feature consistency is also provided, comprising:

[0101] At least one processor;

[0102] At least one memory for storing at least one program;

[0103] When the at least one program is executed by the at least one processor, the at least one processor implements the method provided in the foregoing embodiments.

[0104] In some embodiments of the present invention, a storage medium is also provided, wherein a processor-executable program is stored, which, when executed by a processor, is used to perform the methods provided in the foregoing embodiments.

[0105] The present invention provides a stereo matching system, apparatus and medium for performing stereo matching from monocular images by means of feature consistency, which has the same beneficial effects as the methods described above.

[0106] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for stereo matching from monocular images using feature consistency, characterized in that, Includes the following steps: Generate corresponding binocular images and disparity maps based on monocular images; specifically including: Acquire monocular images; The monocular image is fed into a pre-trained monocular depth estimation network to generate a depth map. Based on the camera parameters, the depth map is converted into a disparity map; The obtained disparity map is sharpened to obtain a clearer disparity map; Based on the obtained disparity map, the monocular image is processed by forward warping and background filling to obtain another monocular image in the binocular image; pass Convert the depth map Z to a disparity map ,in is the maximum value of the depth map, and s is the random sampling scale factor; The feature extraction layer of the stereo matching model is transformed into a teacher-student model, and a binocular contrast feature loss is applied. , To contrast the loss, (u,v) represents the number of pixels in the image. The mask represents the unoccluded area, and C represents the pixel coordinate space; The binocular images and disparity maps are fed into the modified stereo matching model to train the modified stereo matching model. The single image to be tested is input into the modified stereo matching model obtained through training to obtain a disparity map and a depth map. In the modified stereo matching model: A contrastive learning mechanism is introduced: the left feature extraction layer of the stereo matching model is used as the backbone network of the teacher model, and the right feature extraction layer is used as the backbone network of the student model. The network structure of the student model and the teacher model is the same, only the network parameters are different. In this framework, the source domain and target domain data are trained together end-to-end. Source and target data are obtained from the feature maps encoded by the teacher and student models: paired source domain data, i.e., query features of the left view, are obtained using labeled disparity D. Key features of the right view , where (u,v) represents a pixel in an image of size H×W, and d represents the disparity; Momentum-based dynamic dictionary: The dictionary has a capacity of K, and its data are all from the most recent target domain data, exhibiting a first-in, first-out (FIFO) characteristic. In each minimum batch, the latest N target domain data are stored, and the oldest N data are deleted. During training, the weight parameters of the student model are adjusted. Update: Where α∈(0,1) is the momentum coefficient. Indicates the parameters of the teacher model; Remove occluded regions: Estimate the confidence map from the parallax space, and pixels with low confidence will be identified as occluded regions.

2. The method for stereo matching from monocular images using feature consistency according to claim 1, characterized in that, Mask of unobstructed areas for In the formula, This is the reprojection error.

3. A method for stereo matching from monocular images using feature consistency according to any one of claims 1-2, characterized in that, When training the modified stereo matching model, a supervised learning approach is used to minimize the weighted sum of disparity loss and binocular contrast loss. in, It is the balance coefficient. It is a smooth L1 loss.

4. The method for stereo matching from monocular images using feature consistency as described in claim 3, characterized in that, The loss function is: Where N is the number of pixels, d is the disparity map label, and It is the predicted disparity, where x represents , This represents smoothed L1 loss.

5. A stereo matching system for monocular images using feature consistency, characterized in that, A method for implementing any one of claims 1-4 includes the following modules: The stereo training data synthesis module is used to generate corresponding binocular images and disparity maps based on monocular images; The model transformation module is used to transform the feature extraction layer of the stereo matching model into a teacher-student model and apply a binocular contrast feature loss. , To contrast the loss, (u,v) represents the number of pixels in the image. The mask represents the unoccluded area, and C represents the pixel coordinate space; The model training module feeds the binocular images and disparity maps into the modified stereo matching model to train the modified stereo matching model. The model prediction module takes the single image to be tested and inputs it into the modified stereo matching model obtained through training to obtain a disparity map and a depth map.

6. A stereo matching system apparatus for performing stereo matching from monocular images by means of feature consistency, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor performs the method according to any one of claims 1-4.

7. A storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to perform the method described in any one of claims 1-4.

Images