Monocular self-supervision depth estimation method based on moving object mask, electronic equipment and storage medium

Abstract

The invention discloses a monocular self-supervision depth estimation method based on a moving object mask, which can greatly improve the accuracy of monocular depth estimation in a dynamic scene and avoid the problem of discomfort caused by image reconstruction based on object motion compensation, and comprises the following steps of: 1, giving a target frame image and an adjacent frame image; calculating adjacent frame pose estimation, single frame depth estimation, a target frame feature map and an adjacent frame feature map; 2, reconstructing a target frame feature map and constructing time sequence features; 3, calculating multi-frame depth estimation of the original image resolution; 4, constructing a moving object mask based on multi-frame depth estimation and single-frame depth estimation; training single-frame depth estimation by using a moving object mask; training multi-frame depth estimation through original target frame reconstruction loss; and 5, realizing joint training of the depth estimation network and the camera pose estimation network through the masked reconstruction loss based on the single-frame depth estimation and the reconstruction loss based on the multi-frame depth estimation.

Description

Technical Field

[0001] The present invention relates to the technical field of depth recovery from moving images, and in particular to a monocular self-supervised depth estimation method based on a moving object mask, an electronic device and a storage medium. Background Art

[0002] With the development of deep learning related technologies, a series of depth estimation methods based on monocular images have emerged. Among them, the self-supervised depth estimation method based on monocular video uses the photometric reconstruction error of the target frame image as a supervisory signal, eliminating the dependence of the depth estimation neural network on the true depth label, and improving the application of the monocular depth estimation method. The monocular depth estimation method can be further divided into depth estimation based on a single frame image as input and depth estimation based on multiple frames as input.

[0003] The self-supervised signal used in the self-supervised monocular depth estimation method is based on the static scene assumption. The target frame image is reconstructed by image depth estimation and camera's own motion estimation. The influence of object motion on the reconstruction of the target frame image is usually ignored, resulting in the reconstruction of the moving object area in the image bringing erroneous supervisory signals to the training of the depth estimation network, thereby reducing the accuracy of such monocular depth estimation methods in scenes containing a large number of moving objects.

[0004] Existing monocular depth estimation methods that completely ignore dynamic objects will produce extremely inaccurate depth estimation results for moving objects. In order to cope with the impact of moving objects, some methods use neural networks to estimate the motion of objects in the scene, thereby making up for the shortcomings of the static scene assumption. However, methods based on object motion compensation are ill-posed problems because combining the camera's own motion estimation with the object's motion estimation to achieve target frame image reconstruction. The image reconstruction loss after object motion compensation is difficult to provide accurate self-supervisory signals for depth estimation. In order to make up for this shortcoming, it is usually necessary to use sparsity loss for object depth estimation, or to further introduce additional semantic segmentation labels to distinguish potential moving objects. However, the ill-posedness of image reconstruction makes the sparsity loss very sensitive to parameters, which limits the widespread application of motion compensation methods in different scenarios. Summary of the invention

[0005] The purpose of the present invention is to provide a monocular self-supervised depth estimation method based on moving object mask, an electronic device and a storage medium, which can solve the problems existing in the prior art and can greatly improve the accuracy of monocular depth estimation in dynamic scenes, while avoiding the instability problem caused by image reconstruction based on object motion compensation.

[0006] In order to achieve the above object, one of the solutions of the present invention is: A monocular self-supervised depth estimation method based on moving object mask comprises the following steps: Step 1: Given the target frame image , adjacent frame images , the pose estimation of adjacent frames is obtained through the camera pose network , using the UNet type convolutional network as the depth estimation network to obtain the target frame image Corresponding single-frame depth estimation ; Use the first two convolutional layers of the depth estimation network to respectively , adjacent frame images Extract texture features and obtain the target frame feature map , adjacent frame feature map ; Step 2: Depth estimation based on a single frame , adjacent frame pose estimation Reconstruct the target frame feature map, and the calculation formula is: ; in, represents the reconstructed target frame feature map, Indicates the use of An imaginary depth plane is used for reconstruction; Represents the bilinear interpolation method; Represents the reconstruction map based on depth estimation and camera pose; Represents the camera intrinsic parameter matrix; Represents depth estimation based on a single frame The imaginary depth plane is calculated as: ; in, Indicates the number of reconstructed target frame feature maps. ; Based on the obtained The reconstructed target frame feature map constructs the temporal feature , and its calculation formula is: ; in, Indicates the channel norm; Indicates splicing in the channel dimension; Step 3: Combine single frame depth estimation Feature map , using a translation window transformer, The convolutional layer and Sigmoid activation function obtain multi-frame depth estimation with 1 / 4 original image resolution ; Estimate the depth of a single frame Feature map Convex upsampling weights are obtained through convolutional layers , multi-frame depth estimation with 1 / 4 original image resolution Perform convex upsampling to obtain multi-frame depth estimation of the original image resolution ; Step 4: Depth estimation based on multiple frames Depth estimation with single frame Constructing moving object masks , and its calculation formula is: ; in, Indicates that the moving object is masked The value is truncated to Between; Moving Object Mask It is worth The soft mask between The corresponding area is more likely to be a moving object; Using Moving Object Masks Reduce the weight of the target frame reconstruction loss in the moving object area to train single-frame depth estimation , and its calculation formula is: ; ; in, Represents depth estimation based on a single frame reconstruction losses; Represents depth estimation based on a single frame And adjacent frame pose estimation The target frame image reconstruction result is calculated as follows: ; Represents the photometric error of the image; Represents the structural similarity of images; Training multi-frame depth estimation via original target frame reconstruction loss , and its calculation formula is: ; in, Represents depth estimation based on multiple frames reconstruction losses; Represents depth estimation based on multiple frames And adjacent frame pose estimation The target frame image reconstruction result; Step 5: Single-frame depth estimation after masking Reconstruction loss, based on multi-frame depth estimation The reconstruction loss realizes the joint training of the depth estimation network and the camera pose estimation network, and its calculation formula is: , represents the final reconstruction loss used for training.

[0007] In step 1, the UNet type convolutional network uses HRNet as the encoder backbone.

[0008] In steps 1 to 3, the target frame feature map , adjacent frame feature map , Timing Characteristics and multi-frame depth estimation Both are 1 / 4 of the original image resolution.

[0009] The second solution of the present invention is: An electronic device comprises a processor, a memory and an application program; the application program is stored in the memory and is configured so that the processor executes the monocular self-supervised depth estimation method based on a moving object mask.

[0010] The third solution of the present invention is: A computer-readable storage medium stores a computer program; when the computer program is executed in a computer, the computer is caused to execute the monocular self-supervised depth estimation method based on a moving object mask.

[0011] After adopting the above technical solution, the present invention has the following technical effects: The present invention combines the inaccuracy of depth estimation in the moving object area by the depth estimation method of multi-frame input, utilizes the characteristic that multi-frame depth estimation has significant inaccuracy in the moving object area compared with single-frame depth estimation, and constructs a moving object mask in combination with the depth estimation result of single-frame input; and reduces the training weight of the reconstruction loss at the moving object of the image through the moving object mask, thereby reducing the influence of the moving object on the accuracy of depth estimation; the present invention uses the unconstrained target frame reconstruction loss for training multi-frame depth estimation, and uses the moving object mask to constrain the training of single-frame depth estimation, thereby improving the accuracy of single-frame depth estimation in the moving object area, thereby further improving the inconsistency between single-frame depth estimation and multi-frame depth estimation at the moving object, and the effectiveness of the corresponding moving object mask. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a framework diagram of a specific embodiment of the present invention. DETAILED DESCRIPTION

[0013] In order to further explain the technical solution of the present invention, the present invention is described in detail below through specific embodiments.

[0014] refer to Figure 1 As shown, the present invention discloses a monocular self-supervised depth estimation method based on a moving object mask, comprising the following steps: Step 1: Given the target frame image , adjacent frame images , the pose estimation of adjacent frames is obtained through the camera pose network , using the UNet type convolutional network as the depth estimation network to obtain the target frame image Corresponding single-frame depth estimation ; Use the first two convolutional layers of the depth estimation network to respectively , adjacent frame images Extract texture features and obtain the target frame feature map , adjacent frame feature map ; Step 2: Depth estimation based on a single frame , adjacent frame pose estimation Reconstruct the target frame feature map, and the calculation formula is: ; in, represents the reconstructed target frame feature map, Indicates the use of An imaginary depth plane is used for reconstruction; Represents the bilinear interpolation method; Represents the reconstruction map based on depth estimation and camera pose; Represents the camera intrinsic parameter matrix; Represents depth estimation based on a single frame The imaginary depth plane is calculated as: ; in, Indicates the number of reconstructed target frame feature maps. ; Based on the obtained The reconstructed target frame feature map constructs the temporal feature , and its calculation formula is: ; in, Indicates the channel norm; Indicates splicing in the channel dimension; Step 3: Combine single frame depth estimation Feature map , using the Swin-Transformer, The convolutional layer and Sigmoid activation function obtain multi-frame depth estimation with 1 / 4 original image resolution ; Estimate the depth of a single frame Feature map Convex upsampling weights are obtained through convolutional layers , multi-frame depth estimation with 1 / 4 original image resolution Perform convex upsampling to obtain multi-frame depth estimation of the original image resolution ; Step 4: Depth estimation based on multiple frames Depth estimation with single frame Constructing moving object masks , and its calculation formula is: ; in, Indicates that the moving object is masked The value is truncated to Between; Moving Object Mask It is worth The soft mask between The corresponding area is more likely to be a moving object; Using Moving Object Masks Reduce the weight of the target frame reconstruction loss in the moving object area to train single-frame depth estimation , and its calculation formula is: ; ; in, Represents depth estimation based on a single frame reconstruction losses; Represents depth estimation based on a single frame And adjacent frame pose estimation The target frame image reconstruction result is calculated as follows: ; Represents the photometric error of the image; Represents the structural similarity of images; Training multi-frame depth estimation via original target frame reconstruction loss , and its calculation formula is: ; in, Represents depth estimation based on multiple frames reconstruction losses; Represents depth estimation based on multiple frames And adjacent frame pose estimation The target frame image reconstruction result; Step 5: Single-frame depth estimation after masking Reconstruction loss, based on multi-frame depth estimation The reconstruction loss realizes the joint training of the depth estimation network and the camera pose estimation network, and its calculation formula is: , represents the final reconstruction loss used for training.

[0015] Through the above scheme, the present invention combines the inaccuracy of depth estimation in the moving object area by the depth estimation method of multi-frame input, utilizes the characteristic that multi-frame depth estimation has significant inaccuracy in the moving object area compared with single-frame depth estimation, and constructs a moving object mask in combination with the depth estimation result of single-frame input; and reduces the training weight of the reconstruction loss at the moving object of the image through the moving object mask, thereby reducing the influence of the moving object on the accuracy of depth estimation; the present invention uses unconstrained target frame reconstruction loss for training multi-frame depth estimation, and uses moving object mask to constrain the training of single-frame depth estimation, thereby improving the accuracy of single-frame depth estimation in the moving object area, thereby further improving the inconsistency between single-frame depth estimation and multi-frame depth estimation at the moving object, and the effectiveness of the corresponding moving object mask.

[0016] Specific embodiments of the present invention are shown below.

[0017] In step 1 above, the UNet-type convolutional network uses HRNet as the encoder backbone.

[0018] In the above steps 1 to 3, the target frame feature map , adjacent frame feature map , Timing Characteristics and multi-frame depth estimation Both are 1 / 4 of the original image resolution.

[0019] The present invention also discloses an electronic device, comprising a processor, a memory and an application program; the application program is stored in the memory and is configured to execute the above method by the processor.

[0020] The present invention also discloses a computer-readable storage medium, which stores a computer program; when the computer program is executed in a computer, the computer executes the above method. The storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0021] The above embodiments and drawings do not limit the product form and style of the present invention. Any appropriate changes or modifications made by ordinary technicians in the relevant technical field should be deemed to be within the patent scope of the present invention.

Claims

1. A monocular self-supervised depth estimation method based on moving object mask, characterized in that The following steps are involved: Step 1: Given the target frame image , adjacent frame images , the pose estimation of adjacent frames is obtained through the camera pose network , using the UNet type convolutional network as the depth estimation network to obtain the target frame image Corresponding single-frame depth estimation ; Use the first two convolutional layers of the depth estimation network to respectively , adjacent frame images Extract texture features and obtain the target frame feature map , adjacent frame feature map ; Step 2: Depth estimation based on a single frame , adjacent frame pose estimation Reconstruct the target frame feature map, and the calculation formula is: ； in, represents the reconstructed target frame feature map, Indicates the use of An imaginary depth plane is used for reconstruction; Represents the bilinear interpolation method; Represents the reconstruction map based on depth estimation and camera pose; Represents the camera intrinsic parameter matrix; Represents depth estimation based on a single frame The imaginary depth plane is calculated as: ； in, Indicates the number of reconstructed target frame feature maps. ; Based on the obtained The reconstructed target frame feature map constructs the temporal feature , and its calculation formula is: ； in, Indicates the channel norm; Indicates splicing in the channel dimension; Step 3: Combine single frame depth estimation Feature map , using a translation window transformer, The convolutional layer and Sigmoid activation function obtain multi-frame depth estimation with 1 / 4 original image resolution ; Estimate the depth of a single frame Feature map Convex upsampling weights are obtained through convolutional layers , multi-frame depth estimation with 1 / 4 original image resolution Perform convex upsampling to obtain multi-frame depth estimates of the original image resolution ; Step 4: Depth estimation based on multiple frames Depth estimation with single frame Constructing moving object masks , and its calculation formula is: ； in, Indicates that the moving object is masked The value is truncated to Between; Moving Object Mask It is worth The soft mask between The corresponding area is more likely to be a moving object; Using Moving Object Masks Reduce the weight of the target frame reconstruction loss in the moving object area to train single-frame depth estimation , and its calculation formula is: ； ； in, Represents depth estimation based on a single frame reconstruction losses; Represents depth estimation based on a single frame And adjacent frame pose estimation The target frame image reconstruction result is calculated as follows: ; Represents the photometric error of the image; Represents the structural similarity of images; Training multi-frame depth estimation via original target frame reconstruction loss , and its calculation formula is: ； in, Represents depth estimation based on multiple frames reconstruction losses; Represents depth estimation based on multiple frames And adjacent frame pose estimation The target frame image reconstruction result; Step 5: Single-frame depth estimation after masking Reconstruction loss, based on multi-frame depth estimation The reconstruction loss realizes the joint training of the depth estimation network and the camera pose estimation network, and its calculation formula is: , represents the final reconstruction loss used for training.

2. The method for monocular self-supervised depth estimation based on moving object mask according to claim 1, characterized in that: In step 1, the UNet type convolutional network uses HRNet as the encoder backbone.

3. The method for monocular self-supervised depth estimation based on moving object mask according to claim 1, characterized in that: In steps 1 to 3, the target frame feature map , adjacent frame feature map , Timing Characteristics and multi-frame depth estimation Both are 1 / 4 of the original image resolution.

4. An electronic device, characterized in that It comprises a processor, a memory and an application; the application is stored in the memory and is configured so that the processor executes the monocular self-supervised depth estimation method based on moving object mask as described in any one of claims 1 to 3.

5. A computer-readable storage medium, characterized in that The storage medium stores a computer program; when the computer program is executed in a computer, the computer executes the monocular self-supervised depth estimation method based on moving object mask as described in any one of claims 1 to 3.

Images