A Domain Generalization Stereo Matching Method Based on Broad-spectrum Features
By introducing broad-spectrum features and feature adapters into the stereo matching network, the generalization problem of the stereo matching network in different scenarios is solved, and high-quality disparity map acquisition in real scenarios under virtual data training is realized.
Patent Information
- Application Number
- CN202210297315.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-24
- Publication Date
- 2025-08-01
- Estimated Expiration
- 2042-03-24
AI Technical Summary
The existing stereo matching network has poor generalization in different scenarios, resulting in insufficient accuracy of depth acquisition, especially in occlusion areas and weak texture areas.
The domain generalization stereo matching method based on broad-spectrum features is adopted to improve the robustness of the feature by using pre-trained features of large-scale data sets and constructing feature adapters, filtering out irrelevant information, combining cosine similarity and cost aggregation.
Under virtual data training, high-quality disparity maps can be obtained in multiple real scenarios, which improves the robustness and accuracy of depth acquisition.
Smart Images
Figure CN114612701B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the fields of computer vision, scene understanding, and stereo vision, and particularly relates to a domain generalization stereo matching method based on broad-spectrum features. Background Art
[0002] Scene depth acquisition is an important fundamental problem in the field of computer vision. It can accurately perceive the geometric structure of a scene from an image and is the basis for many upstream applications, with wide applications in fields such as robotics and navigation, such as 3D reconstruction, Simultaneous Localization and Mapping (SLAM), and Augmented Reality (AR). As one of the important means of obtaining depth information, stereo matching technology has a research history of several decades. Obtaining depth using stereo matching algorithms has the advantages of low cost and convenient deployment. Existing methods are mainly divided into traditional stereo matching methods and stereo matching methods based on neural network learning. Due to problems such as occluded regions, illumination changes, and a large number of weakly textured regions under different perspectives, the accuracy of depth values obtained by traditional stereo matching methods is relatively low.
[0003] In recent years, due to the booming development of deep learning, some stereo matching methods based on deep convolutional networks have received wide attention. It utilizes the powerful parsing ability of deep learning for scenes, constructs discriminative feature expressions through network learning, and at the same time the network itself learns how to perform cost aggregation, that is, performs some disparity propagation within the image, so as to effectively process the disparity values of points in difficult-to-match regions such as occlusions and weakly textured regions. Despite the powerful learning ability of deep convolutional neural networks, training a network usually requires a large amount of data. For a pixel-level task like stereo matching, obtaining annotations of real scenes is time-consuming and laborious; at the same time, the trained network is also easily troubled by overfitting, and models trained in a certain scene often cannot perform well in other scenes with large image variations.
[0004] Recently, domain generalization stereo matching networks have become a new research topic, that is, constructing stereo matching networks that can obtain high-precision disparity maps in different scenes. Domain generalization models only need to be trained with easily obtained virtual data and can be deployed and used in real scenes, with extremely high application value. Summary of the Invention
[0005] In order to solve the problem of poor generalization in existing supervised stereo matching networks, the present invention proposes a domain generalization stereo matching method based on broad-spectrum features, thereby improving the robustness and accuracy of scene depth acquisition.
[0006] To achieve the above object, the technical solution of the present invention is as follows:
[0007] In the first aspect of the embodiments of the present invention, a domain generalization stereo matching method based on broad-spectrum features is proposed, which specifically includes the following steps:
[0008] S1, select a fully supervised binocular stereo dataset as dataset D0;
[0009] S2, select several groups of pictures from dataset D0 to train a stereo matching network; in this network, first extract the features of the left and right images, then construct a cost volume using cosine similarity, and finally output the final disparity map through cost aggregation;
[0010] S3, replace the features of the trained stereo matching network with broad-spectrum features;
[0011] S4, construct a feature adapter to transform the broad-spectrum features, and use the ground truth of the disparity in dataset D0 to supervise and train the cost aggregation module and the feature adapter of the stereo matching network.
[0012] Preferably, input the left and right views obtained by a binocular stereo camera for testing. Use the stereo matching network trained in step S4 to extract the features of the left and right images through broad-spectrum feature extraction. Subsequently, the feature passes through the feature adapter to filter out the information related to stereo matching. Next, construct a cost volume using cosine similarity, and then obtain the final high-quality disparity map through cost aggregation.
[0013] Preferably, the dataset D0 includes N0 groups of left and right images obtained by a binocular stereo camera, and the disparity map corresponding to one of the images.
[0014] Preferably, the dataset D0 is a virtual dataset or a real dataset.
[0015] Preferably, step S2 is specifically to select several groups of pictures from dataset D0 to construct multiple batches of samples. In each batch of samples, it is best to include no less than 4 groups of pictures. Among them, the network first extracts the features of the left and right images; then calculates the cosine similarity between the left image features and the corresponding right image features under different disparity conditions to construct a four-dimensional cost volume; next, the cost aggregation module in the stereo matching network further filters the initial cost volume, and finally obtains the disparity of each point by weighted averaging the candidate values of the custom disparity.
[0016] Preferably, the broad-spectrum features come from a model pre-trained on a large-scale dataset. When obtaining features on this pre-trained model, keep the resolution of the broad-spectrum features consistent with the features of the stereo matching network trained in step S2.
[0017] Preferably, the feature adapter is a shallow network with no more than 10 convolutional layers.
[0018] Preferably, both the stereo matching network and the feature adapter are trained using ground truth disparity for supervision, and the specific measurement methods include but are not limited to L1 norm, Euclidean distance, etc.
[0019] In a second aspect of the embodiments of the present invention, an electronic device is proposed, including a memory and a processor, wherein the memory is coupled to the processor; wherein, the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned domain generalization stereo matching method based on broad-spectrum features.
[0020] In a third aspect of the embodiments of the present invention, a computer-readable storage medium is proposed, on which a computer program is stored, wherein when the program is executed by a processor, the above-mentioned domain generalization stereo matching method based on broad-spectrum features is implemented.
[0021] Advantages of the present invention: The stereo matching method based on broad-spectrum features proposed by the present invention can obtain a disparity map with high quality in various real-world scene tests under the condition of only using virtual data for training. At the same time, due to the large-scale use of pre-trained models in the field of computer vision, it is very convenient to obtain broad-spectrum features, making this method have high application value. Description of the Drawings
[0022] Figure 1 is a flowchart of the operation of a conventional stereo matching network;
[0023] Figure 2 is a flowchart of constructing a domain generalization stereo matching network proposed by the present invention;
[0024] Figure 3 is an effect diagram of the model tested on various real-world scene datasets in the embodiments of the present invention;
[0025] Figure 4 is a schematic diagram of a domain generalization stereo matching device based on broad-spectrum features provided in the embodiments of the present invention. Detailed Embodiments
[0026] In order to make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be further described in detail below with reference to specific embodiments.
[0027] In the present invention, the training dataset D0 can use the publicly available SceneFlow dataset (which can be downloaded and obtained at the website https: / / lmb.informatik.uni-freiburg.de / resources / datasets / SceneFlowDatasets.en.html).
[0028] In the present invention, the constructed conventional stereo matching network can be any stereo matching network based on 3D convolution, such as the PSMNet proposed in "Pyramid Stereo Matching Network". The adopted broad-spectrum features can be the features of any pre-trained model on a large-scale dataset. In this embodiment, the features of the VGG model pre-trained for classification tasks on ImageNet are adopted.
[0029] The following embodiments are implemented on the premise of the technical solution of the present invention, and detailed implementation manners and specific operation processes are given. However, the protection scope of the present invention is not limited to the following embodiments.
[0030] The present invention proposes a domain generalization stereo matching method based on broad-spectrum features, which specifically includes the following steps:
[0031] S1, select a fully supervised binocular stereo dataset as the training dataset D0.
[0032] The dataset D0 includes the left and right images obtained by N0 groups of binocular stereo cameras, and the disparity map corresponding to one of the images. The dataset D0 can be a virtual dataset generated by computer software or a real collected dataset.
[0033] S2, select several groups of pictures from the dataset D0 to train the stereo matching network; in this network, first extract the features of the left and right images, then construct a cost volume using cosine similarity, and finally output the final disparity map through cost aggregation. Specifically:
[0034] The step S2 is specifically to select several groups of pictures from the dataset D0 to construct multiple batches of samples. In each batch of samples, it is preferably to include no less than 4 groups of pictures. Among them, the feature extraction module in the stereo matching network extracts the features of the left and right images to obtain the features of the stereo matching network; then calculate the cosine similarity between the left image features and the corresponding right image features under different disparity conditions to construct a four-dimensional cost volume; next, the cost aggregation module in the stereo matching network further filters the initial cost volume, and finally obtains the disparity of each point by weighted averaging the candidate values of the custom disparity.
[0035] S3, replace the features of the stereo matching network trained in step S2 with broad-spectrum features.
[0036] The broad-spectrum features come from a model pre-trained on a large-scale dataset (such as ImageNet, COCO dataset). When obtaining features on this pre-trained model, keep the feature resolution of the broad-spectrum features consistent with that of the conventional stereo matching network (that is, the stereo matching network trained in step S2). The broad-spectrum features can construct a relatively general feature expression when facing image data in different domains.
[0037] S4. Construct a feature adapter to transform the broad-spectrum feature and filter out information irrelevant to the stereo matching task; use the ground truth of the disparity in dataset D0 to supervise the training of the cost aggregation module and the feature adapter of the stereo matching network.
[0038] The feature adapter is a shallow network with no more than 10 convolutional layers to prevent model overfitting.
[0039] Transform the broad-spectrum feature through the feature adapter. The specific process is as follows: input the broad-spectrum feature into the feature adapter, and the output is the transformed broad-spectrum feature, that is, the information irrelevant to the stereo matching task in the broad-spectrum feature is filtered out.
[0040] Both the stereo matching network and the feature adapter are trained with supervision provided by the ground truth of the disparity in dataset D0. The specific measurement methods include but are not limited to L1 norm, Euclidean distance, etc.
[0041] S5. Input the left and right views obtained by the binocular stereo camera for testing. Use the stereo matching network trained in step S4 to extract robust features of the left and right images through the broad-spectrum feature. Subsequently, the feature passes through the feature adapter to filter out information related to stereo matching. Next, construct a cost volume through cosine similarity, and then obtain the final high-quality disparity map through cost aggregation.
[0042] Embodiment 1
[0043] In this embodiment, the training dataset D0 is obtained from the SceneFlow dataset. The dataset D0 contains the ground truth of the disparity of binocular images and is the dataset for the first part of supervised training.
[0044] 1. Use dataset D0 to train the conventional existing stereo matching network as Figure 1 shown. Among them, the cost volume of the network is constructed using cosine similarity.
[0045] (1.1) Randomly shuffle dataset D0 to construct multiple batches of samples. In each sample batch, it is preferably to include no less than 4 groups of pictures. In this embodiment, each sample batch contains 8 groups of pictures. Each group of pictures includes the calibrated left and right pictures from the binocular camera, and the ground truth disparity map corresponding to the left picture. In this embodiment, first process the original data, including cropping the original data, and randomly cropping the pictures into 512*256 image patches. According to Figure 1 the flowchart shown, input the left and right image patches into the stereo matching network to obtain the output disparity map D p , and construct a loss by supervising this output. In this embodiment, the loss term is calculated using Euclidean distance:
[0046]
[0047] Among them, D t is the ground truth disparity map in the dataset.
[0048] (1.2) According to Figure 2 the flowchart shown, replace the feature extraction module of the conventional stereo matching network with the features of the pre-trained model on a large-scale dataset. Because this feature has broad-spectrum characteristics and can obtain relatively robust feature expressions when different domain images are input. When replacing, select the part with the same output resolution as the feature resolution of the conventional stereo matching network to prevent resolution changes from interfering with other modules of the network.
[0049] (1.3) According to Figure 2 the flowchart shown, in addition to the broad-spectrum features, the embodiment of the present invention additionally constructs a feature adapter. In this example, since the broad-spectrum features are trained for classification tasks and are relatively low-level, the information contained may not be fully applicable to stereo matching. In order to make the features more discriminative and construct a more accurate cost volume, the embodiment of the present invention inputs the broad-spectrum features into a feature adapter. The feature adapter consists of 5 convolutional layers, which ensures a certain expression ability while keeping the number of parameters small and reducing the risk of overfitting. When training the feature adapter, in this embodiment, the disparity map output by the entire network is still supervised, but the parameters of the broad-spectrum features and the cost aggregation module are kept fixed, and only the feature adapter is updated.
[0050] (1.4) After completing the replacement of the feature part, this embodiment retrains the cost aggregation module to make it more adaptable to the robust cost volume constructed by the migrated broad-spectrum features. The training of this part is still achieved by supervising the output disparity map, and at the same time, the parameters of the broad-spectrum features and the feature adapter are kept fixed.
[0051] 2. In the test stage, input the calibrated left and right images obtained by the binocular camera in the real scene, and obtain a high-precision disparity map through the constructed domain generalization stereo matching network for subsequent tasks such as depth acquisition.
[0052] In this embodiment, the model trained on the virtual dataset SceneFlow is tested on four real-scene datasets of KITTI 2015, KITTI 2012, Middlebury, and ETH3D. In addition to PSMNet, GANet is additionally used as the basic model for evaluation.
[0053] The qualitative results are as Figure 3 shown, Figure 3 where (a) in Figure 3 is the left image in the binocular image, Figure 3Among them, (c) is the prediction result based on the PSMNet model, Figure 3 Among them, (d) is the prediction result based on the GANet model. Although the models are trained on the virtual dataset SceneFlow, they all present high-quality disparity maps during testing. In particular, for object edges and weakly textured regions including roads, the models in this example perform well and have extremely high application value.
[0054] The quantitative evaluation results are shown in Table 1. As shown in the results in Table 1, replacing the broad-spectrum features, adding a feature adapter, and retraining the cost aggregation module can improve the performance of the models on both the KITTI 2015 and Middlebury datasets. For the ETH3D dataset, replacing the broad-spectrum features cannot bring an improvement in accuracy because the stereo images of the ETH3D data are grayscale images, and the classification model used to obtain the broad-spectrum features is trained on the ImageNet dataset, which does not include grayscale images. Therefore, even the broad-spectrum features cannot guarantee complete domain invariance. Nevertheless, the broad-spectrum features are robust enough for a large number of application scenarios, as demonstrated by the test results of KITTI 2015 and Middlebury.
[0055] Table 1: Test result table
[0056]
[0057]
[0058] Corresponding to the foregoing embodiments of the domain generalization stereo matching method based on broad-spectrum features, the present invention also provides embodiments of a domain generalization stereo matching device based on broad-spectrum features.
[0059] See Figure 4 , an embodiment of a domain generalization stereo matching device provided by an embodiment of the present invention includes one or more processors for implementing the domain generalization stereo matching method based on broad-spectrum features in the foregoing embodiments.
[0060] Embodiments of the domain generalization stereo matching device based on broad-spectrum features of the present invention can be applied to any device with data processing capabilities, and the any device with data processing capabilities can be a device or apparatus such as a computer. Embodiments of the device can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a logically meaningful device, it is formed by the processor of any device with data processing capabilities where it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for running. From the hardware level, such as Figure 4As shown in the figure, it is a hardware structure diagram of any device with data processing capabilities where the domain generalization stereo matching device based on broad-spectrum features of the present invention is located. In addition to Figure 4 the shown processor, memory, network interface, and non-volatile memory, in the embodiments, any device with data processing capabilities where the device is located may usually include other hardware according to the actual functions of the device with data processing capabilities, which will not be elaborated herein.
[0061] For the specific implementation process of the functions and roles of each unit in the above device, please refer to the implementation process of the corresponding steps in the above method, which will not be elaborated herein.
[0062] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial descriptions of the method embodiments. The device embodiments described above are only illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of the present invention. Those of ordinary skill in the art can understand and implement it without creative efforts.
[0063] The embodiments of the present invention also provide a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, the domain generalization stereo matching method based on broad-spectrum features in the above embodiments is implemented.
[0064] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium may also be any device with data processing capabilities, such as a plug-in hard disk, a Smart Media Card (SMC), an SD card, a Flash Card, etc. equipped on the device. Further, the computer-readable storage medium may also include both an internal storage unit of any device with data processing capabilities and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the device with data processing capabilities, and may also be used to temporarily store the data that has been output or will be output.
[0065] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.
Claims
1. A domain generalization stereo matching method based on broad-spectrum features, characterized in that, Specifically, it includes the following steps: S1. Select a fully supervised binocular stereo dataset as dataset D0; S2. Select several groups of pictures from dataset D0 to train a stereo matching network; in this network, first extract the features of the left and right pictures, then construct a cost volume using cosine similarity, and finally output the final disparity map through cost aggregation; S3. Replace the features of the trained stereo matching network with broad-spectrum features; The broad-spectrum features come from a model pre-trained on a large-scale dataset. When obtaining features from this pre-trained model, keep the resolution of the broad-spectrum features consistent with the features of the stereo matching network obtained in step S2; S4. Construct a feature adapter to transform the broad-spectrum features, and use the ground truth of the disparity in dataset D0 to supervise and train the cost aggregation module and the feature adapter of the stereo matching network.
2. The domain generalization stereo matching method based on broad-spectrum features according to claim 1, wherein Input the left and right views obtained by a binocular stereo camera for testing. Use the stereo matching network trained in step S4 to extract the features of the left and right pictures through broad-spectrum feature extraction. Then, the features pass through the feature adapter to filter out the information related to stereo matching, construct a cost volume using cosine similarity, and finally obtain the final high-quality disparity map through cost aggregation.
3. The domain generalization stereo matching method based on broad-spectrum features according to claim 1, characterized in that Dataset D0 contains N0 groups of left and right pictures obtained by a binocular stereo camera, and the disparity map corresponding to one of the pictures.
4. The domain generalization stereo matching method based on broad-spectrum features according to claim 1, characterized in that Dataset D0 is a virtual dataset.
5. The domain generalization stereo matching method based on broad-spectrum features according to claim 1, wherein Specifically, step S2 is to select several groups of pictures from dataset D0 to construct multiple batches of samples, and each batch of samples contains no less than 4 groups of pictures; among them, the network first extracts the features of the left and right pictures; then calculates the cosine similarity between the features of the left picture and the corresponding features of the right picture under different disparities to construct a four-dimensional cost volume; next, the cost aggregation module in the stereo matching network further filters the initial cost volume, and finally obtains the disparity of each point by weighted averaging the candidate values of the custom disparity.
6. The domain generalization stereo matching method based on broad-spectrum features according to claim 1, wherein The feature adapter is a shallow network with no more than 10 convolutional layers.
7. The domain generalization stereo matching method based on broad-spectrum features according to claim 1, wherein Both the stereo matching network and the feature adapter are trained using the ground truth of the disparity to provide supervision, and the specific measurement methods include L1 norm or Euclidean distance.
8. An electronic device, comprising a memory and a processor, wherein, The memory is coupled to the processor; wherein, the memory is used to store program data, and the processor is used to execute the program data to implement the domain generalization stereo matching method based on broad-spectrum features described in any one of claims 1-7 above.
9. A computer-readable storage medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the domain generalization stereo matching method based on broad-spectrum features described in any one of claims 1-7.