Monocular self-supervised depth estimation method based on moving object mask, electronic device and storage medium

Abstract

The application discloses a monocular self-supervised depth estimation method based on a moving object mask, which can greatly improve the accuracy of monocular depth estimation in a dynamic scene and avoid the ill-posed problem caused by image reconstruction based on object motion compensation, and comprises the following steps: 1, given a target frame image and a neighboring frame image, calculating a neighboring frame pose estimation, a single-frame depth estimation, a target frame feature map and a neighboring frame feature map; 2, reconstructing the target frame feature map and constructing a time sequence feature; 3, calculating a multi-frame depth estimation of an original image resolution; 4, constructing a moving object mask based on the multi-frame depth estimation and the single-frame depth estimation; training the single-frame depth estimation by using the moving object mask; training the multi-frame depth estimation by using an original target frame reconstruction loss; and 5, realizing joint training of a depth estimation network and a camera pose estimation network through a reconstruction loss based on the single-frame depth estimation after the mask and a reconstruction loss based on the multi-frame depth estimation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of depth recovery from motion images, in particular to a monocular self-supervised depth estimation method based on a moving object mask, an electronic device and a storage medium. BACKGROUND

[0002] With the development of deep learning related technologies, a series of monocular image based depth estimation methods have been produced. Among them, the self-supervised depth estimation method based on monocular video is to use the photometric reconstruction error of the target frame image as the supervision signal, which eliminates the dependence of the depth estimation neural network on the depth true value label, and improves the application of the monocular depth estimation method. Monocular depth estimation method can be further divided into depth estimation based on single frame image as input, and depth estimation based on multiple frame image as input.

[0003] The self-supervised signal used in the self-supervised monocular depth estimation method is based on the static scene assumption, and the reconstruction of the target frame image is realized by image depth estimation and camera motion estimation. The influence of object motion on the reconstruction of the target frame image is usually ignored, which leads to the fact that the reconstruction of the moving object region in the image brings false supervision signal to the training of the depth estimation network, thereby reducing the accuracy of such monocular depth estimation method in the scene containing a large number of moving objects.

[0004] The existing monocular depth estimation method completely ignoring dynamic objects will produce extremely inaccurate depth estimation results for moving objects. In order to cope with the influence of moving objects, some methods estimate the motion of objects in the scene through neural network, so as to make up for the defects of static scene assumption. However, the method based on object motion compensation is an ill-posed problem because it combines camera motion estimation and object motion estimation to realize the reconstruction of target frame image. The image reconstruction loss after object motion compensation is difficult to bring accurate self-supervised signal to depth estimation. In order to make up for this defect, it is usually necessary to use sparsity loss for object depth estimation, or further introduce additional semantic segmentation label to distinguish potential moving objects. However, the ill-posedness of image reconstruction makes the sparsity loss very sensitive to parameters, which limits the wide application of the method based on motion compensation in different scenes. SUMMARY

[0005] The purpose of the present application is to provide a monocular self-supervised depth estimation method based on a moving object mask, an electronic device and a storage medium, which solves the problems existing in the prior art and can greatly improve the accuracy of monocular depth estimation in dynamic scenes, while avoiding the ill-posed problem brought by image reconstruction based on object motion compensation.

[0006] In order to achieve the above purpose, one of the solutions of the present application is:

[0007] A monocular self-supervised depth estimation method based on a moving object mask includes the following steps:

[0008] Step 1: Given the target frame image Adjacent frame images The pose estimation of adjacent frames is obtained through the camera pose network. The target frame image is obtained by using a UNet-type convolutional network as the depth estimation network. Corresponding single-frame depth estimation The first two convolutional layers of the depth estimation network are used to process the target frame image. Adjacent frame images Extract texture features to obtain the target frame feature map. Adjacent frame feature maps ;

[0009] Step 2: Depth estimation based on a single frame Neighboring frame pose estimation The calculation formula for reconstructing the feature map of the target frame is as follows:

[0010] ;

[0011] in, This represents the reconstructed feature map of the target frame. Indicates the use of the first Reconstruct a hypothetical depth plane; This represents the bilinear interpolation method; This represents a reconstruction mapping based on depth estimation and camera pose. Represents the camera intrinsic parameter matrix; Indicates depth estimation based on a single frame The imaginary depth plane is calculated using the following formula:

[0012] ;

[0013] in, The number of target frame feature maps to be reconstructed, in this invention ;

[0014] Based on the obtained Constructing temporal features from the reconstructed target frame feature map The calculation formula is as follows:

[0015] ;

[0016] in, Indicates the channel Norm; This indicates splicing along the channel dimension;

[0017] Step 3: Combine single-frame depth estimation Feature map Using a translation window transformer Convolutional layers and the sigmoid activation function yield multi-frame depth estimation at 1 / 4 the resolution of the original image. Single-frame depth estimation Feature map Convex upsampling weights are obtained through convolutional layers. Multi-frame depth estimation at 1 / 4 original image resolution Convex upsampling is performed to obtain multi-frame depth estimates at the original image resolution. ;

[0018] Step 4: Depth estimation based on multiple frames Compared with single-frame depth estimation Constructing a mask for moving objects The calculation formula is as follows:

[0019] ;

[0020] in, Indicates masking moving objects The value is truncated to Between; Mask of moving objects It is worth it The closer the values ​​of the soft masks are to each other, the better. The area corresponding to it is more likely to be a moving object;

[0021] Using moving objects as masks Reduce the weights of the target frame reconstruction loss for moving object regions to train single-frame depth estimation. The calculation formula is as follows:

[0022] ;

[0023] ;

[0024] in, Indicates depth estimation based on a single frame The reconstruction losses; Indicates depth estimation based on a single frame Pose estimation of adjacent frames The target frame image reconstruction result is calculated using the following formula: ; Indicates the photometric error of the image; Indicates the structural similarity of images;

[0025] Multi-frame depth estimation is trained using the reconstruction loss from the original target frame. The calculation formula is as follows:

[0026] ;

[0027] wherein, denotes the reconstruction loss based on multi-frame depth estimation ; denotes the target frame image reconstruction result based on multi-frame depth estimation and adjacent frame pose estimation ;

[0028] Step 5, joint training of the depth estimation network and the camera pose estimation network is implemented through the reconstruction loss based on single-frame depth estimation after the mask, the reconstruction loss based on multi-frame depth estimation , and the calculation formula is:

[0029] , denotes the final reconstruction loss used for training.

[0030] In the step 1, the UNet type convolutional network takes HRNet as the encoder backbone.

[0031] In the steps 1 to 3, the target frame feature map , the adjacent frame feature map , the time sequence feature and the multi-frame depth estimation are all 1 / 4 of the original image resolution.

[0032] The second solution of the application is:

[0033] An electronic device comprises a processor, a memory and an application program; the application program is stored in the memory and is configured to execute the monocular self-supervised depth estimation method based on the motion object mask by the processor.

[0034] The third solution of the application is:

[0035] A computer readable storage medium, the storage medium stores a computer program; the computer program is executed in the computer, and the computer program makes the computer execute the monocular self-supervised depth estimation method based on the motion object mask.

[0036] After the above technical solution is adopted, the application has the following technical effects:

[0037] This invention addresses the inaccuracies of multi-frame input depth estimation in moving object regions by leveraging the significant inaccuracies of multi-frame depth estimation compared to single-frame depth estimation. It constructs a moving object mask by combining the depth estimation results from a single frame input. Furthermore, the moving object mask reduces the training weights of the reconstruction loss at moving object locations, minimizing the impact of motion on depth estimation accuracy. This invention uses unconstrained target frame reconstruction loss to train multi-frame depth estimation and the moving object mask to constrain the training of single-frame depth estimation, thereby improving the accuracy of single-frame depth estimation in moving object regions. This further addresses the inconsistency between single-frame and multi-frame depth estimation at moving object locations and enhances the effectiveness of the corresponding moving object mask. Attached Figure Description

[0038] Figure 1 This is a framework diagram of a specific embodiment of the present invention. Detailed Implementation

[0039] To further explain the technical solution of the present invention, the present invention will be described in detail below through specific embodiments.

[0040] refer to Figure 1 As shown, this invention discloses a monocular self-supervised depth estimation method based on a moving object mask, comprising the following steps:

[0041] Step 1: Given the target frame image Adjacent frame images The pose estimation of adjacent frames is obtained through the camera pose network. The target frame image is obtained by using a UNet-type convolutional network as the depth estimation network. Corresponding single-frame depth estimation The first two convolutional layers of the depth estimation network are used to process the target frame image. Adjacent frame images Extract texture features to obtain the target frame feature map. Adjacent frame feature maps ;

[0042] Step 2: Depth estimation based on a single frame Neighboring frame pose estimation The calculation formula for reconstructing the feature map of the target frame is as follows:

[0043] ;

[0044] in, This represents the reconstructed feature map of the target frame. Indicates the use of the first Reconstruct a hypothetical depth plane; This represents the bilinear interpolation method; This represents a reconstruction mapping based on depth estimation and camera pose. Represents the camera intrinsic parameter matrix; Indicates depth estimation based on a single frame The imaginary depth plane is calculated using the following formula:

[0045] ;

[0046] in, The number of target frame feature maps to be reconstructed, in this invention ;

[0047] Based on the obtained Constructing temporal features from the reconstructed target frame feature map The calculation formula is as follows:

[0048] ;

[0049] in, Indicates the channel Norm; This indicates splicing along the channel dimension;

[0050] Step 3: Combine single-frame depth estimation Feature map Using the Swing-Transformer Convolutional layers and the sigmoid activation function yield multi-frame depth estimation at 1 / 4 the resolution of the original image. Single-frame depth estimation Feature map Convex upsampling weights are obtained through convolutional layers. Multi-frame depth estimation at 1 / 4 original image resolution Convex upsampling is performed to obtain multi-frame depth estimates at the original image resolution. ;

[0051] Step 4: Depth estimation based on multiple frames Compared with single-frame depth estimation Constructing a mask for moving objects The calculation formula is as follows:

[0052] ;

[0053] in, Indicates masking moving objects The value is truncated to Between; Mask of moving objects It is worth it The closer the values ​​of the soft masks are to each other, the better. the corresponding region is more likely to be a moving object;

[0054] using a moving object mask reducing the weight of the moving object region target frame reconstruction loss to train single-frame depth estimation , the calculation formula of which is:

[0055] ;

[0056] ;

[0057] wherein, represents the reconstruction loss based on single-frame depth estimation ; represents the target frame image reconstruction result based on single-frame depth estimation and adjacent frame pose estimation , the calculation formula of which is ; represents the photometric error of the image; represents the structural similarity of the image;

[0058] training multi-frame depth estimation through the original target frame reconstruction loss , the calculation formula of which is:

[0059] ;

[0060] wherein, represents the reconstruction loss based on multi-frame depth estimation ; represents the target frame image reconstruction result based on multi-frame depth estimation and adjacent frame pose estimation ;

[0061] Step 5, joint training of the depth estimation network and the camera pose estimation network through the reconstruction loss based on single-frame depth estimation after the mask, the reconstruction loss based on multi-frame depth estimation , the calculation formula of which is:

[0062] , represents the final reconstruction loss used for training.

[0063] Through the above scheme, the application uses the characteristics that the multi-frame depth estimation has significant inaccuracy in the moving object area compared with the single-frame depth estimation to combine the single-frame input depth estimation result to construct the moving object mask; and reduces the training weight of the reconstruction loss of the image moving object through the moving object mask, and reduces the influence of the moving object on the depth estimation accuracy; the application uses the unconstrained target frame reconstruction loss for training the multi-frame depth estimation, uses the moving object mask for constraining the training of the single-frame depth estimation, improves the accuracy of the single-frame depth estimation in the moving object area, and further improves the inconsistency of the single-frame depth estimation and the multi-frame depth estimation in the moving object area, and the effectiveness of the corresponding moving object mask.

[0064] The following shows a specific embodiment of the application.

[0065] In the above step 1, the UNet type convolutional network takes HRNet as the encoder backbone.

[0066] In the above steps 1 to 3, the target frame feature map , the adjacent frame feature map , the time sequence feature and the multi-frame depth estimation are all 1 / 4 of the original image resolution.

[0067] The application also discloses an electronic device, which comprises a processor, a memory and an application program; the application program is stored in the memory and is configured to execute the above method by the processor.

[0068] The application also discloses a computer readable storage medium, which stores a computer program; the computer program makes the computer execute the above method when executed in the computer. The storage medium includes various storage program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0069] The above embodiments and drawings do not limit the product form and style of the application, and any appropriate changes or modifications made by ordinary skilled persons in the art shall be considered as not departing from the patent scope of the application.

Claims

1. A monocular self-supervised depth estimation method based on a moving object mask, characterized in that... Includes the following steps: Step 1: Given the target frame image Adjacent frame images The pose estimation of adjacent frames is obtained through the camera pose network. The target frame image is obtained by using a UNet-type convolutional network as the depth estimation network. Corresponding single-frame depth estimation The first two convolutional layers of the depth estimation network are used to process the target frame image. Adjacent frame images Extract texture features to obtain the target frame feature map. Adjacent frame feature maps ; Step 2: Depth estimation based on a single frame Neighboring frame pose estimation The calculation formula for reconstructing the feature map of the target frame is as follows: ； in, This represents the reconstructed feature map of the target frame. Indicates the use of the first Reconstruct a hypothetical depth plane; This represents the bilinear interpolation method; This represents a reconstruction mapping based on depth estimation and camera pose. Represents the camera intrinsic parameter matrix; Indicates depth estimation based on a single frame The imaginary depth plane is calculated using the following formula: ； in, This indicates the number of feature maps in the reconstructed target frame. ; Based on the obtained Constructing temporal features from the reconstructed target frame feature map The calculation formula is as follows: ； in, Indicates the channel Norm; This indicates splicing along the channel dimension; Step 3: Combine single-frame depth estimation Feature map Using a translation window transformer Convolutional layers and the sigmoid activation function yield multi-frame depth estimation at 1 / 4 the resolution of the original image. Single-frame depth estimation Feature map Convex upsampling weights are obtained through convolutional layers. Multi-frame depth estimation at 1 / 4 original image resolution Convex upsampling is performed to obtain multi-frame depth estimates at the original image resolution. ; Step 4: Depth estimation based on multiple frames Compared with single-frame depth estimation Constructing a mask for moving objects The calculation formula is as follows: ； in, Indicates masking moving objects The value is truncated to Between; Mask of moving objects It is worth it The closer the values ​​of the soft masks are to each other, the better. The area corresponding to it is more likely to be a moving object; Using moving objects as masks Reduce the weights of the target frame reconstruction loss for moving object regions to train single-frame depth estimation. The calculation formula is as follows: ； ； in, Indicates depth estimation based on a single frame The reconstruction losses; Indicates depth estimation based on a single frame Pose estimation of adjacent frames The target frame image reconstruction result is calculated using the following formula: ; Indicates the photometric error of the image; Indicates the structural similarity of images; Multi-frame depth estimation is trained using the reconstruction loss from the original target frame. The calculation formula is as follows: ； in, Indicates depth estimation based on multiple frames The reconstruction losses; Indicates depth estimation based on multiple frames Pose estimation of adjacent frames The target frame image reconstruction result; Step 5: Single-frame depth estimation after masking Reconstruction loss, based on multi-frame depth estimation The reconstruction loss enables joint training of the depth estimation network and the camera pose estimation network, and its calculation formula is as follows: , This represents the reconstruction loss ultimately used for training.

2. The monocular self-supervised depth estimation method based on moving object mask as described in claim 1, characterized in that: In step 1, the UNet-type convolutional network uses HRNet as the encoder backbone.

3. The monocular self-supervised depth estimation method based on moving object masks as described in claim 1, characterized in that: In steps 1 to 3, the target frame feature map Adjacent frame feature maps Time series characteristics and multi-frame depth estimation All are 1 / 4 of the original image resolution.

4. An electronic device, characterized in that... It includes a processor, memory, and an application program; the application program is stored in the memory and configured to be executed by the processor as described in any one of claims 1 to 3, using a monocular self-supervised depth estimation method based on a moving object mask.

5. A computer-readable storage medium, characterized in that... The storage medium stores a computer program; when the computer program is executed in the computer, it causes the computer to perform the monocular self-supervised depth estimation method based on a moving object mask as described in any one of claims 1 to 3.

Images