An underwater self-supervised monocular depth estimation method based on knowledge distillation

Abstract

The present invention discloses an underwater self-supervised monocular depth estimation method based on knowledge distillation, which belongs to the technical field of depth estimation and is used for depth estimation, including building a monocular depth estimation deep learning network and a camera attitude estimation deep learning network, training two teacher deep learning networks, inputting the output values ​​of the teacher monocular depth estimation deep learning network and the teacher camera attitude estimation deep learning network into a moving object prediction module and an underwater image enhancement module to obtain a moving object mask and an underwater clear image; inputting the input value and output value of the teacher monocular depth estimation deep learning network into a rotation and cropping module together to produce paired samples under a rotation perspective, and training two student deep learning networks. The present invention provides a more accurate monocular depth estimation result for underwater scenes, has stronger robustness in the face of images taken from different perspectives, and can provide enhanced underwater images according to a physical model.

Description

Technical Field

[0001] The invention discloses an underwater self-supervised monocular depth estimation method based on knowledge distillation, belonging to the technical field of depth estimation. Background Art

[0002] The acquisition of depth information is a prerequisite for underwater observation and operation tasks such as visual navigation, visual reconstruction, and visual measurement by underwater robots (ROVs) and autonomous underwater vehicles (AUVs). However, commonly used depth sensors such as lidar and sonar face the problems of high cost, large size, and inaccurate depth acquisition underwater. In order to save costs, the existing technology performs depth estimation by processing images. At present, underwater monocular depth estimation algorithms are mainly divided into traditional underwater monocular depth estimation methods and underwater monocular depth estimation methods based on deep learning. Traditional underwater monocular depth estimation methods are often based on the physical model of underwater imaging. These methods rely on fragile assumptions and their depth estimation reliability is not high. In addition, due to the complexity of the underwater environment, the attenuation and scattering of light in different water environments are different. A single physical model is often not applicable to a variety of underwater scenes, which leads to poor generalization of traditional methods. With the advancement of deep learning technology, thanks to its good feature learning and predictive expression capabilities, underwater monocular depth estimation research has made rapid progress, but it still faces great limitations. First, it is very difficult to obtain underwater image samples and their corresponding depth maps in a real environment. Due to the lack of training samples, the usual supervised learning method is difficult to apply to underwater monocular depth estimation. Secondly, there are a variety of moving objects in the underwater environment, represented by fish, which can easily interfere with depth estimation. In addition, images taken underwater inevitably have degradation, which increases the difficulty of inferring depth from the image. Finally, ROVs and AUVs have six degrees of freedom and move in three-dimensional space underwater, so the underwater images collected by them have a variety of shooting angles, which puts higher requirements on the robustness of depth estimation methods. Summary of the invention

[0003] The purpose of the present invention is to provide an underwater self-supervised monocular depth estimation method based on knowledge distillation to solve the problem of inaccurate underwater monocular depth estimation in the prior art.

[0004] An underwater self-supervised monocular depth estimation method based on knowledge distillation, comprising:

[0005] S1: Construct training samples from continuous underwater video frame sequences. A training sample contains underwater image target frames. and the previous frame With the next frame Source frame ;

[0006] S2: Build a monocular depth estimation deep learning network and a camera pose estimation deep learning network. Based on knowledge distillation, decompose the monocular depth estimation deep learning network into a teacher monocular depth estimation deep learning network and a student monocular depth estimation deep learning network. Based on knowledge distillation, decompose the camera pose estimation deep learning network into a teacher camera pose estimation deep learning network and a student camera pose estimation deep learning network.

[0007] S3: Train two teacher deep learning networks, input the output values ​​of the teacher monocular depth estimation deep learning network and the teacher camera posture estimation deep learning network into the moving object prediction module and the underwater image enhancement module to obtain the moving object mask and the underwater clear image; input the input value and output value of the teacher monocular depth estimation deep learning network into the rotation and cropping module together to produce paired samples under the rotation perspective;

[0008] S4: Train two student deep learning networks and use a visually degraded underwater image as the input of the monocular depth estimation deep learning network and the camera pose estimation deep learning network. After being processed by the moving object prediction module and the underwater image enhancement module, the network outputs an estimated depth map and an enhanced image with improved visual quality.

[0009] The monocular depth estimation deep learning network adopts an encoder-decoder structure, including a hole convolution module, a feature aggregation module, an upsampling module and a depth prediction output module. It inputs an underwater image and outputs a depth map corresponding to the underwater image.

[0010] The camera pose estimation deep learning network is constructed by stacking convolutional layers, activation layers, and pooling layers with a deep feature extraction structure. It extracts the deep feature expressions of each level of the input image, inputs two consecutive frames of underwater images, and outputs the camera pose changes between the two frames.

[0011] After training two teacher deep learning networks, we use the estimated depth and camera pose transformation to reconstruction , using the reconstructed With unreconstructed The pixel value difference loss and structural similarity loss constrain the two teacher deep learning networks, and feedback optimizes the model parameters of the two teacher deep learning networks.

[0012] Training two teacher deep learning networks involves Input teacher monocular depth estimation deep learning network , output The corresponding estimated depth map ;

[0013] Will and Common-input teacher camera pose estimation deep learning network , the output is arrive The estimated camera pose transformation ;

[0014] Using the reconstructed With unreconstructed The pixel value difference loss and structural similarity loss between the two teachers’ deep learning networks include recording the camera’s intrinsic parameter matrix as ,remember for For any pixel in for Any pixel in and The corresponding relationship is:

[0015] ;

[0016] After the reconstruction Recorded as ,calculate and The pixel difference loss between and structural similarity loss :

[0017] ;

[0018] ;

[0019] In the formula, and are the height and width of the image, yes The height, width and number of channels of is the total number of pixels in the image, is the center pixel of the image block, and They are exist The pixel mean and standard deviation, and They are exist The pixel mean and standard deviation, , ;

[0020] Feedback optimization of model parameters of two teacher deep learning networks including linear combinations and Get the reprojection loss , minimizing the reprojection loss Iterative update optimization and Model parameters:

[0021] ;

[0022] In the formula, .

[0023] Obtaining moving object masks includes moving object prediction modules based on and Reprojection loss for predicting moving object masks :

[0024] ; ;

[0025] In the formula, Brackets for Iverson, For image index, is a statistically based threshold, , for the third and fourth quartiles;

[0026] Obtaining clear underwater images includes an underwater image enhancement module that takes an underwater degraded image and a corresponding depth map as input and outputs a clear underwater image:

[0027] ; ;

[0028] In the formula, is an underwater degraded image, For clear images, is the forward scattering coefficient, For depth, For background light, is the illumination map of the underwater degraded image, and the background light is estimated by the mean of the darkest 0.1% pixels in the degraded image. .

[0029] Generating paired samples under rotational viewpoints involves taking the input value of the teacher monocular depth estimation deep learning network and output value , together with the input to the rotation and cropping module, the rotation and cropping module will and Rotate the same angle , and then and Perform center cropping, the height after cropping With width The relationship is:

[0030] ; ;

[0031] In the formula, and yes and Uncropped height and width.

[0032] Training two student deep learning networks involves transforming the estimated depth and camera pose using the reconstructed With unreconstructed The pixel value difference loss and structural similarity loss are used to constrain the two student deep learning networks, and the rotation distillation loss is used as an additional constraint of the two student deep learning networks to feedback and optimize the model parameters of the two student deep learning networks.

[0033] Will , After the underwater image enhancement module, the corresponding enhanced underwater image is generated and recorded as , .

[0034] Using the reconstructed With unreconstructed The pixel value difference loss and structural similarity loss constrain the two student deep learning networks, including the student monocular depth estimation deep learning network and Student Camera Pose Estimation Deep Learning Network The output of reconstruction , reconstructed Recorded as ,calculate and Between and ;

[0035] Will Multiply To filter the influence of dynamic objects, by minimizing the loss Iterative update optimization and The model parameters.

[0036] Using rotation distillation loss as an additional constraint for the two student deep learning networks includes The input is rotated , due to rotary distillation losses Constraints:

[0037] ;

[0038] In the formula, for The output, for and of, for The variance of for The variance of .

[0039] Compared with the prior art, the present invention has the following beneficial effects: the present invention provides more accurate monocular depth estimation results for underwater scenes, has stronger robustness for images taken from different perspectives, and can provide enhanced underwater images based on physical models. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flowchart of the underwater monocular depth estimation method;

[0041] Figure 2 It is a schematic diagram of the self-supervised constraints of the underwater monocular depth estimation method;

[0042] Figure 3 It is a schematic diagram of the structure of the underwater monocular depth estimation network;

[0043] Figure 4 is a test image of the embodiment;

[0044] Figure 5 The Monodepth2 model of the prior art is Figure 4 The output result of

[0045] Figure 6 The HR-Depth model of the existing technology is Figure 4 The output result of

[0046] Figure 7 The MonoViT model of the prior art is Figure 4 The output result of

[0047] Figure 8 The present invention is Figure 4 The output result is: DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention is described clearly and completely below. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] An underwater self-supervised monocular depth estimation method based on knowledge distillation, comprising:

[0050] S1: Construct training samples from continuous underwater video frame sequences. A training sample contains underwater image target frames. and the previous frame With the next frame Source frame ;

[0051] S2: Build a monocular depth estimation deep learning network and a camera pose estimation deep learning network. Based on knowledge distillation, decompose the monocular depth estimation deep learning network into a teacher monocular depth estimation deep learning network and a student monocular depth estimation deep learning network. Based on knowledge distillation, decompose the camera pose estimation deep learning network into a teacher camera pose estimation deep learning network and a student camera pose estimation deep learning network.

[0052] S3: Train two teacher deep learning networks, input the output values ​​of the teacher monocular depth estimation deep learning network and the teacher camera posture estimation deep learning network into the moving object prediction module and the underwater image enhancement module to obtain the moving object mask and the underwater clear image; input the input value and output value of the teacher monocular depth estimation deep learning network into the rotation and cropping module together to produce paired samples under the rotation perspective;

[0053] S4: Train two student deep learning networks and use a visually degraded underwater image as the input of the monocular depth estimation deep learning network and the camera pose estimation deep learning network. After being processed by the moving object prediction module and the underwater image enhancement module, the network outputs an estimated depth map and an enhanced image with improved visual quality.

[0054] The monocular depth estimation deep learning network adopts an encoder-decoder structure, including a hole convolution module, a feature aggregation module, an upsampling module and a depth prediction output module. It inputs an underwater image and outputs a depth map corresponding to the underwater image.

[0055] The camera pose estimation deep learning network is constructed by stacking convolutional layers, activation layers, and pooling layers with a deep feature extraction structure. It extracts the deep feature expressions of each level of the input image, inputs two consecutive frames of underwater images, and outputs the camera pose changes between the two frames.

[0056] After training two teacher deep learning networks, we use the estimated depth and camera pose transformation to reconstruction , using the reconstructed With unreconstructed The pixel value difference loss and structural similarity loss constrain the two teacher deep learning networks, and feedback optimizes the model parameters of the two teacher deep learning networks.

[0057] Training two teacher deep learning networks involves Input teacher monocular depth estimation deep learning network , output The corresponding estimated depth map ;

[0058] Will and Common-input teacher camera pose estimation deep learning network , the output is arrive The estimated camera pose transformation ;

[0059] Using the reconstructed With unreconstructed The pixel value difference loss and structural similarity loss between the two teachers’ deep learning networks include recording the camera’s intrinsic parameter matrix as ,remember for For any pixel in for Any pixel in and The corresponding relationship is:

[0060] ;

[0061] The rebuilt Recorded as ,calculate and The pixel difference loss between and structural similarity loss :

[0062] ;

[0063] ;

[0064] In the formula, and are the height and width of the image, yes The height, width and number of channels of is the total number of pixels in the image, is the center pixel of the image block, and They are exist The pixel mean and standard deviation, and They are exist The pixel mean and standard deviation, , ;

[0065] Feedback optimization of model parameters of two teacher deep learning networks including linear combinations and Get the reprojection loss , minimizing the reprojection loss Iterative update optimization and Model parameters:

[0066] ;

[0067] In the formula, .

[0068] Obtaining moving object masks includes moving object prediction modules based on and Reprojection loss for predicting moving object masks :

[0069] ; ;

[0070] In the formula, Brackets for Iverson, For image index, is a statistically based threshold, , for the third and fourth quartiles;

[0071] Obtaining clear underwater images includes an underwater image enhancement module that takes an underwater degraded image and a corresponding depth map as input and outputs a clear underwater image:

[0072] ; ;

[0073] In the formula, is an underwater degraded image, For clear images, is the forward scattering coefficient, For depth, For background light, is the illumination map of the underwater degraded image, and the background light is estimated by the mean of the darkest 0.1% pixels in the degraded image. .

[0074] Generating paired samples under rotational viewpoints involves taking the input value of the teacher monocular depth estimation deep learning network and output value , together with the input to the rotation and cropping module, the rotation and cropping module will and Rotate the same angle , and then and Perform center cropping, the height after cropping With width The relationship is:

[0075] ; ;

[0076] In the formula, and yes and Uncropped height and width.

[0077] Training two student deep learning networks involves transforming the estimated depth and camera pose using the reconstructed With unreconstructed The pixel value difference loss and structural similarity loss are used to constrain the two student deep learning networks, and the rotation distillation loss is used as an additional constraint of the two student deep learning networks to feedback and optimize the model parameters of the two student deep learning networks.

[0078] Will , After the underwater image enhancement module, the corresponding enhanced underwater image is generated and recorded as , .

[0079] Using the reconstructed With unreconstructed The pixel value difference loss and structural similarity loss constrain the two student deep learning networks, including the student monocular depth estimation deep learning network and Student Camera Pose Estimation Deep Learning Network The output of reconstruction , reconstructed Recorded as ,calculate and Between and ;

[0080] Will Multiply To filter the influence of dynamic objects, by minimizing the loss Iterative update optimization and The model parameters.

[0081] Using rotation distillation loss as an additional constraint for the two student deep learning networks includes The input is rotated , due to rotary distillation losses Constraints:

[0082] ;

[0083] In the formula, for The output, for and of, for The variance of for The variance of .

[0084] In the embodiment of the present invention, the flowchart of the underwater monocular depth estimation method is as follows: Figure 1 As shown in the figure, firstly, a pair of underwater images with degraded visual imaging quality is obtained as the input image, and then the trained underwater monocular deep network is used to predict the depth map corresponding to the image. If a clear underwater image corresponding to the input degraded underwater image is required, the depth map and the input image can be input into the underwater image enhancement module, and the underwater clear image can be obtained according to the underwater imaging physical model. In the underwater monocular depth estimation method, a data set is firstly made from a sequence of underwater continuous frames, and teacher model training, student model training and model deployment are performed. The schematic diagram of the self-supervised constraints of the underwater monocular depth estimation method is shown in FIG. Figure 2 As shown in the figure, the self-supervised constraint method uses the target frame depth map predicted by the monocular depth estimation network and the camera pose transformation predicted by the camera pose estimation network to reconstruct the target frame, and constrains the monocular depth estimation network and the camera pose estimation network according to the pixel difference loss between the reconstructed target frame and the target frame, that is, the reprojection loss. The structural diagram of the underwater monocular depth estimation network is shown in Figure 3 As shown in the figure, the underwater degraded image is imported into the network, and the downsampling module is used to distribute it to three different scale spaces. The depth map is predicted in each scale space through the hole convolution module, feature aggregation module and depth prediction module.

[0085] Comparison of the depth estimation results of the underwater monocular depth estimation network with other methods Figure 4 , Figure 5 , Figure 6 , Figure 7 , Figure 8 As shown, Figure 4 is the test image input to the network, Figure 5 is the output of Monodepth2, Figure 6 is the output result of HR-Depth, Figure 7 is the output of MonoViT, Figure 8is the output result of the present invention. In the output depth map, warmer or redder colors represent smaller depth values, and colder or bluer colors represent larger depth values. Monodepth2, HR-Depth and MonoViT are commonly used self-supervised monocular depth estimation networks. The depth estimation results of the present invention have more accurate results, clearer edges and more accurate background estimation than these methods.

[0086] The deep regression network structure of the present invention uses regression prediction network structures such as Alex-Net and VGG-Net, the encoder-decoder structure uses deep encoder-decoder structures such as U-Net and Dense-Net, the underwater video frame dataset uses, for example, the FLSea dataset, the convolution layer-activation layer-pooling layer uses, for example, Conv-ReLu-MaxPool, and the hole convolution uses, for example, Dilated Conv. The examples in this paragraph are all prior arts in the field, so the details are not explained.

[0087] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features may be replaced by equivalents, and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An underwater self-supervised monocular depth estimation method based on knowledge distillation, characterized in that: include: S1: Construct training samples from continuous underwater video frame sequences. A training sample contains underwater image target frames. and the previous frame With the next frame Source frame ; S2: Build a monocular depth estimation deep learning network and a camera pose estimation deep learning network. Based on knowledge distillation, decompose the monocular depth estimation deep learning network into a teacher monocular depth estimation deep learning network and a student monocular depth estimation deep learning network. Based on knowledge distillation, decompose the camera pose estimation deep learning network into a teacher camera pose estimation deep learning network and a student camera pose estimation deep learning network. S3: Train two teacher deep learning networks, input the output values ​​of the teacher monocular depth estimation deep learning network and the teacher camera posture estimation deep learning network into the moving object prediction module and the underwater image enhancement module to obtain the moving object mask and the underwater clear image; input the input value and output value of the teacher monocular depth estimation deep learning network into the rotation and cropping module together to produce paired samples under the rotation perspective; S4: Train two student deep learning networks, use a visually degraded underwater image as the input of the monocular depth estimation deep learning network and the camera pose estimation deep learning network, and output an estimated depth map and an enhanced image with improved visual quality after being processed by the moving object prediction module and the underwater image enhancement module; The monocular depth estimation deep learning network adopts an encoder-decoder structure, including a dilated convolution module, a feature aggregation module, an upsampling module, and a depth prediction output module. It inputs an underwater image and outputs a depth map corresponding to the underwater image. The camera pose estimation deep learning network is constructed by stacking convolutional layer, activation layer and pooling layer with deep feature extraction structure, extracting the deep feature expression of each level of the input image, inputting two consecutive frames of underwater images, and outputting the camera pose change between the two frames; After training two teacher deep learning networks, we use the estimated depth and camera pose transformation to reconstruction , using the reconstructed With unreconstructed The pixel value difference loss and structural similarity loss between them constrain the two teacher deep learning networks, and feedback optimizes the model parameters of the two teacher deep learning networks; Training two teacher deep learning networks involves Input teacher monocular depth estimation deep learning network , output The corresponding estimated depth map ; Will and Common-input teacher camera pose estimation deep learning network , the output is arrive The estimated camera pose transformation ; Using the reconstructed With unreconstructed The pixel value difference loss and structural similarity loss between the two teachers’ deep learning networks include recording the camera’s intrinsic parameter matrix as ,remember for For any pixel in for Any pixel in and The corresponding relationship is: ； The rebuilt Recorded as ,calculate and The pixel difference loss between and structural similarity loss : ； ； In the formula, and are the height and width of the image, yes The height, width and number of channels of is the total number of pixels in the image, is the center pixel of the image block, and They are exist The pixel mean and standard deviation, and They are exist The pixel mean and standard deviation, , ; Feedback optimization of model parameters of two teacher deep learning networks including linear combinations and Get the reprojection loss , minimizing the reprojection loss Iterative update optimization and Model parameters: ； In the formula, ; Obtaining moving object masks includes moving object prediction modules based on and Reprojection loss for predicting moving object masks : ； ； In the formula, Brackets for Iverson, For image index, is a statistically based threshold, , for the third and fourth quartiles; Obtaining clear underwater images includes an underwater image enhancement module that takes an underwater degraded image and a corresponding depth map as input and outputs a clear underwater image: ； ； In the formula, is an underwater degraded image, For clear images, is the forward scattering coefficient, For depth, For background light, is the illumination map of the underwater degraded image, and the background light is estimated by the mean of the darkest 0.1% pixels in the degraded image. .

2. The underwater self-supervised monocular depth estimation method based on knowledge distillation according to claim 1 is characterized in that: Generating paired samples under rotational viewpoints involves taking the input value of the teacher monocular depth estimation deep learning network and output value , together with the input to the rotation and cropping module, the rotation and cropping module will and Rotate the same angle , and then and Perform center cropping, the height after cropping With width The relationship is: ； ； In the formula, and yes and Uncropped height and width.

3. The underwater self-supervised monocular depth estimation method based on knowledge distillation according to claim 2 is characterized in that: Training two student deep learning networks involves transforming the estimated depth and camera pose using the reconstructed With unreconstructed The pixel value difference loss and structural similarity loss are used to constrain the two student deep learning networks, and the rotation distillation loss is used as an additional constraint of the two student deep learning networks to feedback and optimize the model parameters of the two student deep learning networks. Will , After the underwater image enhancement module, the corresponding enhanced underwater image is generated and recorded as , .

4. The underwater self-supervised monocular depth estimation method based on knowledge distillation according to claim 3 is characterized in that: Using the reconstructed With unreconstructed The pixel value difference loss and structural similarity loss constrain the two student deep learning networks, including the student monocular depth estimation deep learning network and Student Camera Pose Estimation Deep Learning Network The output of reconstruction , reconstructed Recorded as ,calculate and Between and ; Will Multiply To filter the influence of dynamic objects, by minimizing the loss Iterative update optimization and The model parameters.

5. The underwater self-supervised monocular depth estimation method based on knowledge distillation according to claim 4 is characterized in that: Using rotation distillation loss as an additional constraint for the two student deep learning networks includes The input is rotated , due to rotary distillation losses Constraints: ； In the formula, for The output, for and of, for The variance of for The variance of .

Images