Binocular depth estimation data set cross-domain migration method

Abstract

The invention discloses a binocular depth estimation data set cross-domain migration method, which adjusts parallax distribution of a pre-training data set by measuring data distribution difference, and comprises the following steps: calculating a Pearson's correlation coefficient between an actual application scene data set and the pre-training data set; calculating a parallax deviation range of the pre-training data set according to the Pearson's correlation coefficient; randomly generating a parallax offset according to the parallax offset range, cutting left and right images of the pre-training data set, and adjusting a parallax true value at the same time; the model only pays attention to a specified parallax range through mask operation, and the model is finely adjusted on a pre-training data set for adjusting parallax distribution. According to the method, the parallax distribution of the pre-training data set is adjusted and is fitted with the parallax distribution of an actual application scene, so that the generalization of the model is improved, and a better reasoning effect can still be kept in a new scene.

Description

Technical Field

[0001] The present invention relates to the field of binocular depth estimation, and in particular to a method for cross-domain migration of a binocular depth estimation data set. Background Art

[0002] Stereo depth estimation is an important task in the field of computer vision, which aims to estimate the depth information of different points in the scene by using images acquired by a stereo camera. It can be used in many applications such as 3D reconstruction, object detection and tracking, indoor navigation, etc.

[0003] The development of stereo depth estimation has gone through several stages. The earliest methods are based on geometric principles, such as disparity estimation. By calculating the offset between pixels between two cameras, the depth information of the object can be inferred. However, these methods are sensitive to problems such as lighting changes, texture loss, and edge blur, and they are not effective for complex scenes and areas with lack of texture.

[0004] With the rise of deep learning, methods based on convolutional neural networks (CNNs) have gradually become mainstream. These methods learn the mapping from input images to depth images by training a deep neural network. Typical methods include end-to-end depth estimation using convolutional neural networks, unsupervised depth estimation using autoencoders, and supervision using optical flow estimation and disparity maps.

[0005] However, there are still some challenges and shortcomings in binocular depth estimation. After searching, it was found that the Chinese patent application with publication number CN111179325A provides a binocular depth estimation method and device. This method only provides a method for obtaining a disparity map in a pre-training scenario, and does not consider the generalization problem of the network in actual application scenarios. In the existing depth estimation dataset, due to the limitations of shooting conditions and scenes, there may be an imbalance in the number of samples in different depth ranges. This may cause the network to perform poorly in certain depth ranges, and the trained model cannot accurately estimate the depth in new scenes or perspectives.

[0006] In order to improve the generalization of the depth estimation network, a feasible method is to change the disparity distribution of the pre-training dataset to adjust it to a distribution similar to that of the actual application scenario. This allows the network to be exposed to more data similar to the actual scene during training, better covering the depth range and distribution in the actual application scenario, thereby improving the performance of the network in actual applications. In addition, changing the disparity distribution of the dataset can increase the diversity of the data and improve the network's adaptability to various complex scenarios. Summary of the invention

[0007] In order to overcome the problem of poor generalization performance of the binocular depth estimation network in actual application scenarios, the present invention provides a method for cross-domain migration of a binocular depth estimation dataset. By analyzing the similarity between a pre-trained dataset and a dataset of an actual application scenario, the disparity distribution of the pre-trained dataset is adjusted, the reasoning effect of the network in the actual application scenario is improved, and the generalization ability of the stereo matching network is improved.

[0008] To achieve the above object, the present invention discloses a method for cross-domain migration of a binocular depth estimation dataset, the steps comprising: Step S10: construct an actual application scenario dataset by sampling, calculate the Pearson correlation coefficient between the pre-training dataset A and the actual application scenario dataset B, and quantitatively measure the correlation between the two datasets: Step S101: Randomly sample from the disparity map in the pre-training dataset A and the actual application scenario dataset B, sample a number of pixels in equal proportion from the randomly selected samples to construct a row vector space, and calculate the average vector of the pre-training dataset A and the actual application scenario dataset B respectively. and ;

[0009] Where N is the number of randomly sampled samples in the data set. is the row vector obtained by sampling the disparity map in the pre-training dataset A in equal proportion, is the row vector obtained by sampling the disparity map in the actual application scenario dataset B in equal proportion; Step S102: Calculate the difference matrix of the pre-training dataset A and the actual application scenario dataset B respectively and ;

[0010] Step S103: Calculate the Pearson correlation coefficient based on the mean vector and the difference matrix ;

[0011] Step S20: Calculate the parallax offset range based on the Pearson correlation coefficient and ;

[0012] Among them, K is the number of pixels sampled in equal proportion; Step S30: randomly obtain an offset according to the parallax offset range ,if Less than 0, the right image is shifted to the left based on random cropping pixels, and the true disparity value is subtracted from the corresponding offset; if Greater than 0, the right image is shifted to the right based on random cropping pixels, and the true disparity value plus the corresponding offset; Step S40: Calculate the disparity range of the actual application scenario, construct a mask, and use the mask to make the model focus only on the specified disparity range; the model is trained on a pre-training data set with adjusted disparity distribution, and fine-tuned in the actual application scenario.

[0013]

[0014] Where f is the focal length of the binocular camera, b is the baseline distance of the binocular camera, , It is the minimum and maximum value of the distance from the camera in the actual application scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 The figure is a flowchart of the steps of the method for cross-domain migration of data sets of the present invention.

[0016] Figure 2 A schematic diagram of adjusting disparity distribution for the method of cross-domain migration of data sets of the present invention. DETAILED DESCRIPTION

[0017] The following describes the implementation of the present invention through specific examples and in conjunction with the accompanying drawings. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific examples, and the details in this specification can also be modified and changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention.

[0018] Figure 1 This is a flow chart of the steps of a method for cross-domain migration of a binocular depth estimation dataset according to the present invention. Figure 1 As shown in the figure, the present invention is a dataset migration method, which uses the SceneFlow public dataset (https: / / lmb.informatik.unifreiburg.de) as the pre-training dataset, the KITTI public dataset (https: / / www.cvlibs.net / datasets / kitti) as the actual application scenario dataset, and AAnet (https: / / github.com / haofeixu / aanet) as the binocular depth estimation network to describe the specific implementation steps of the dataset cross-domain migration method: Step S10: Calculate the Pearson correlation coefficient between the SceneFlow dataset and the KITTI dataset to quantitatively measure the correlation between the two datasets: Step S101: Randomly extract 500 samples from the disparity maps in the SceneFlow dataset and the KITTI dataset, sample 1000 pixels in the randomly extracted sample disparity maps in equal proportion to construct a row vector space, and calculate the average vector of the SceneFlow dataset and the KITTI dataset respectively. and ;

[0019] in is the row vector obtained by sampling the disparity map in the SceneFlow dataset in equal proportion. is the row vector obtained by sampling the disparity map in the KITTI dataset in equal proportion; Step S102: Calculate the difference matrix of SceneFlow dataset and KITTI dataset respectively and ;

[0020] Step S103: Calculate the Pearson correlation coefficient based on the mean vector and the difference matrix ;

[0021] Step S20: Calculate the parallax offset range based on the Pearson correlation coefficient and ;

[0022]

[0023] Step S30: randomly obtain an offset according to the parallax offset range ,if Less than 0, the right image is shifted to the left based on random cropping pixels, and the true disparity value is subtracted from the corresponding offset; if Greater than 0, the right image is shifted to the right based on random cropping pixels, and the true disparity value plus the corresponding offset; Step S40: Calculate the parallax range of the actual application scene, construct a mask, and use the mask to make the model focus only on the specified parallax range;

[0024] Where f is the focal length of the binocular camera, b is the baseline distance of the binocular camera, , is the minimum and maximum value of the distance from the camera in the actual application scene. The semi-supervised stereo matching network AAnet is used for pre-training on the SceneFlow dataset with adjusted disparity distribution. The training parameters are shown in Table 1: Table 1: AAnet training parameters Maximum parallax 128 Optimizer Adam Learning rate schedule MultiStepLR Learning Rate 0.001 Weight decay 0.0001 Number of training rounds 256 The parallax offset calculated by the above steps is between 0 and 40. After the model is pre-trained on SceneFlow, it is fine-tuned and evaluated on the KITTI dataset. The evaluation results are shown in Table 2. Table 2: Adjusted disparity distribution evaluation results Whether to adjust the parallax distribution Average pixel error 1 pixel error 2 pixel error 3 pixel error no 0.620 0.094 0.043 0.029 yes 0.570 0.090 0.042 0.026 It can be seen that after adjusting the disparity distribution of the SceneFlow dataset, fine-tuning and inference on the KITTI dataset increased the average pixel error from 0.620 to 0.570, and the 1, 2, and 3 pixel errors were all improved to a certain extent, thereby verifying that the present invention can improve the reasoning effect of the network in actual application scenarios by adjusting the disparity distribution of the pre-training dataset, and improve the generalization ability of the stereo matching network.

Claims

1. A method for cross-domain migration of a binocular depth estimation dataset, characterized in that: The following steps are involved: S10: constructing an actual application scenario dataset through sampling, calculating the average vector and difference matrix of the actual application scenario dataset and the pre-training dataset respectively, and then calculating the Pearson correlation coefficient between the two datasets; S20: Calculate the disparity offset range of the pre-training dataset according to the Pearson correlation coefficient; S30: randomly obtaining an offset according to the parallax offset range, and then offsetting the right image based on random cropping, and adjusting the parallax true value according to the offset; S40: Calculate the disparity range of the actual application scenario, construct a mask, and use the mask to make the model focus only on the specified disparity range; the model is trained on a pre-training data set with adjusted disparity distribution.

2. A method for measuring the relevance of a data set according to claim 1, characterized in that: By calculating the average vector and difference matrix of the two data sets, and then calculating the Pearson correlation coefficient, the correlation between the data sets can be quantitatively measured.

3. The method for modifying the disparity distribution of a data set according to claim 1, characterized in that: On the basis of random cropping, a random translation transformation is added to change the disparity distribution of the data set by changing the true disparity value.

Images