An unsupervised single-view ship depth estimation method

Abstract

The present invention relates to a ship depth estimation method. In view of the defects of other depth estimation methods, an unsupervised single-view ship depth estimation method is disclosed. The present invention first builds a knowledge distillation network framework, and uses a high-performance teacher network under the framework to train a student network with a low parameter amount, improves the depth estimation performance of the student network, and obtains a low-parameter and high-performance depth estimation network; then, a ship data set is constructed through rendering software and a ship 3D model; finally, the model is trained and tested on the ship data set to obtain a network model that can perform unsupervised single-view depth estimation of the ship. The model and method of the present invention complete the ship depth estimation based on a single view for the first time, and the training process does not require real depth data and complex image annotation, and has a smaller scale model parameter amount.

Description

Technical Field

[0001] The invention relates to a ship depth estimation method, and in particular to an unsupervised single-view ship depth estimation method. Background Art

[0002] Depth estimation, as a research direction in computer vision, aims to obtain a depth image that reflects the distance between an object and a camera. The depth image can provide depth information for tasks such as 3D reconstruction and distance perception. With the development of shipping, water traffic safety has received more and more attention. Efficiently obtaining depth images of ships is of great significance for the perception of waterway traffic conditions and the guarantee of ship navigation safety.

[0003] Depth estimation based on deep learning is mainly divided into multi-view depth estimation methods and single-view depth estimation methods. Multi-view depth estimation methods, such as methods based on binary classification, temporal network depth estimation methods, and methods using the MVS algorithm to improve feature extraction capabilities and optimize depth estimation effects, are to input two images of different views, stereo match the pixels of the two images, and calculate disparity (i.e., the coordinate change of the same object in the two images). The coordinate change information is related to the relative distance of the object. The coordinate change of objects with a relatively far position is smaller than that of objects with a relatively close position. Therefore, depth information can be obtained through disparity. Single-view depth estimation methods, such as depth estimation methods using image rough estimation and fine estimation combined, depth estimation methods based on residual networks, and depth estimation methods based on dilated convolutions, are to input a single image through a heuristic algorithm, obtain depth information through the clues of the image itself, and can also be divided into supervised (all input images have corresponding real depth maps), semi-supervised (only some of the input images have corresponding real depth maps), and unsupervised training (all input images have no corresponding real depth maps) according to different supervision conditions of model training.

[0004] The research on unsupervised deep learning ship depth estimation involved in the present invention is to solve the following defects of other depth estimation methods:

[0005] (1) The multi-view depth estimation method has limitations. The multi-view depth estimation method requires pixel matching and disparity calculation for the left and right views. However, the quality of pixel matching is difficult to guarantee, and this method requires high computing power.

[0006] (2) The weight files of high-performance deep learning models are too large and have low applicability, and cannot be deployed on a variety of devices. Currently, most models with good depth estimation performance use residual network structures, which have complex network structures and large weight files, which greatly limits the applicability of the models on devices with different computing capabilities.

[0007] (3) Supervised and semi-supervised depth estimation requires real depth data, which requires a lot of data annotation. Real depth data is often required as ground truth in model training. It is very difficult to obtain real data, and some models require input data to be annotated, which increases the workload. Summary of the invention

[0008] In view of the above problems, the present invention provides an unsupervised single-view ship depth estimation method.

[0009] In order to achieve the above object, the present invention adopts the following technical solutions:

[0010] The present invention provides an unsupervised single-view ship depth estimation model, comprising: a teacher network depth estimation module, a student network depth estimation module, a Light Net network, a View Net network, an Albedo Net network, and an SD Net network;

[0011] The teacher network depth estimation module includes a ResNet network as an encoder structure, an adaptive dense feature fusion ADFF structure for multi-dimensional feature processing, and a Laplacian pyramid (LPD) structure as a decoder structure.

[0012] The student network depth estimation module includes a Ghost Net network as an encoder structure and a NNConv5 structure as a decoder structure.

[0013] Furthermore, the NNConv5 structure of the student network depth estimation module consists of a five-layer network, each layer consists of a Depthwise layer, a Pointwise layer, and an upsampling layer; the Depthwise layer and the Pointwise layer will restore the depth information layer by layer, and the upsampling layer performs interpolation operations after the Depthwise layer and the Pointwise layer to double the spatial resolution of the image.

[0014] The present invention also provides an unsupervised single-view ship depth estimation method based on the above estimation model, comprising the following steps:

[0015] (1) Establishment of an unsupervised single-view ship depth estimation model;

[0016] (2) Establishment of a knowledge distillation network framework for unsupervised single-view ship depth estimation;

[0017] (3) Construct a ship depth estimation dataset;

[0018] (4) Training of unsupervised single-view ship depth estimation model;

[0019] (5) Testing of the unsupervised single-view ship depth estimation model.

[0020] Furthermore, the unsupervised single-view ship depth estimation model includes a teacher network depth estimation module, a student network depth estimation module, a Light Net network, a View Net network, an Albedo Net network, and an SD Net network; wherein,

[0021] The process of establishing the teacher network depth estimation module is as follows: using a high-performance ResNet network as the encoder structure, using the adaptive dense feature fusion ADFF structure to process features of multiple dimensions, and finally using the Laplacian pyramid structure (LPD) as the decoder structure of the module;

[0022] The process of establishing the student network depth estimation module is as follows: using the Ghost Net network as the encoder structure, and using the NNConv5 structure with added Depthwise and Pointwise structures as the decoder structure of the depth estimation module;

[0023] The Light Net network, View Net network, Albedo Net network, and SD Net network are used to extract the illumination information, viewing angle information, albedo map, and symmetry information map of the image, respectively.

[0024] Furthermore, the specific process of establishing the unsupervised single-view ship depth estimation knowledge distillation network framework is as follows: first, a single RGB image is input into the teacher network depth estimation module with complex structure and powerful performance, and the Light Net network, View Net network, Albedo Net network, and SD Net network respectively extract the depth map, illumination information, perspective information, albedo map, and symmetry information map of the image; then, the extracted illumination information and depth map are rendered to obtain a shaded depth map; the shaded depth map and albedo map are synthesized to obtain a standard perspective map, and the reconstructed RGB image is obtained by forward rendering the standard perspective map, depth map, and perspective information; based on the input image, the reconstruction loss of the reconstructed image is constructed to regress the parameters of the teacher network model; finally, the teacher network depth estimation module is replaced by the student network depth estimation module, and while constructing the reconstruction loss, the multi-dimensional corresponding feature map loss of the teacher network depth estimation module and the student network depth estimation module when the same image input is constructed. Through the joint optimization of the two losses, the lightweight student network obtains performance close to that of the teacher network. Compared with the teacher network depth estimation module, the weight file size of the student network depth estimation module is greatly reduced.

[0025] Furthermore, the specific process of constructing the ship depth estimation dataset is: creating a 3D ship model, using KeyShot 3D realistic rendering software to set the lighting and perspective physical parameters to render the 3D model of the ship, and obtaining a realistic ship image as a ship depth estimation dataset; the ship depth estimation dataset includes a training set, a validation set, and a test set.

[0026] Furthermore, the specific process of training the unsupervised single-view ship depth estimation model is as follows: transforming the training set images in the ship depth estimation dataset to a set resolution size, and inputting the transformed single-view ship images into the network in batches according to the number of Batch Sizes, and training according to the process of establishing the unsupervised single-view ship depth estimation knowledge distillation network framework, after multiple iterations, stopping the training after the iterative Epoch, obtaining the network weight file, and obtaining a model capable of performing depth estimation on the ship.

[0027] Furthermore, the specific process of the unsupervised single-view ship depth estimation model test is: using the test set images in the ship depth estimation dataset to test the effect of the model capable of depth estimation obtained after training, and obtaining the final model that can be used for depth estimation by comparing with the validation set.

[0028] Compared with the prior art, the present invention has the following advantages:

[0029] 1. The present invention discloses an unsupervised single-view ship depth estimation model and estimation method, which can realize unsupervised depth estimation of ship targets in a single view. This method first establishes an unsupervised single-view ship depth estimation model, builds a knowledge distillation network framework based on the model, and uses a high-performance teacher network under the framework to train a student network with low parameters, improve the depth estimation performance of the student network, and obtain a low-parameter and high-performance depth estimation network; then, a ship data set is constructed through rendering software and a ship 3D model; finally, the model is trained and tested on the ship data set to obtain a network model that can perform unsupervised single-view depth estimation of the ship. This method completes the ship depth estimation based on a single view for the first time. The training process does not require real depth data and complex image annotation, and has a smaller scale model parameter amount.

[0030] 2. Compared with the teacher network depth estimation module, the weight file size of the student network depth estimation module is greatly reduced.

[0031] 3. In order to overcome the influence of real ship images on model training, the present invention creates a 3D ship model and uses rendering software to obtain a realistic image. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1This is a framework diagram of the unsupervised single-view ship depth estimation model of the present invention.

[0033] Figure 2 This is a structural diagram of the teacher network depth estimation module of the present invention.

[0034] Figure 3 This is a structural diagram of the student network depth estimation module of the present invention.

[0035] Figure 4 This is a structural diagram of the Albedo Net network, View Net network, SD Net network, and Light Net network of the present invention.

[0036] Attached Figure 5 This is the depth estimation result of the ship image of the present invention. DETAILED DESCRIPTION

[0037] The technical solution of the present invention is specifically and detailedly described below in conjunction with the embodiments of the present invention and the accompanying drawings. It should be noted that for ordinary technicians in this field, several modifications and improvements can be made without departing from the principle of the present invention, which should also be regarded as belonging to the protection scope of the present invention.

[0038] Example 1

[0039] Unsupervised single-view ship depth estimation model, the framework diagram is as follows Figure 1 As shown:

[0040] The model includes: a teacher network depth estimation module, a student network depth estimation module, a Light Net network, a View Net network, an Albedo Net network, and an SD Net network;

[0041] The teacher network depth estimation module includes a ResNet network as an encoder structure, an adaptive dense feature fusion structure for multi-dimensional feature processing, and a Laplacian pyramid structure as a decoder structure; the student network depth estimation module includes a Ghost Net network as an encoder structure and a NNConv5 structure as a decoder structure. The NNConv5 structure consists of five layers, each of which consists of a Depthwise layer, a Pointwise layer, and an upsampling layer; the Depthwise layer and the Pointwise layer will restore the depth information layer by layer, and the upsampling layer will perform interpolation operations after the Depthwise layer and the Pointwise layer to double the spatial resolution of the image.

[0042] Example 2

[0043] (1) Design of teacher network depth estimation module

[0044] The depth estimation module uses a high-performance ResNet feature extraction network encoder, an adaptive dense feature fusion (ADFF) structure, and a Laplacian pyramid structure decoder. The residual network structure of the ResNet feature extraction network encoder is as follows: Figure 2 As shown in the figure, the information obtained by splicing the output information with the input information is used as the input information of the next layer of the network to protect the integrity of the information and solve the problem of information loss and attrition to a certain extent. The encoder will output the feature maps extracted by different network layers of the encoder to the next layer, which are 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the size of the original image. The ADFF structure will perform up- and down-sampling operations on these feature maps to obtain a series of feature maps with sizes of 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 compared to the input single view, and then splice the feature maps of the same size and remove the redundant channels through convolution as the fusion signal. The Laplacian pyramid structure is used as the decoder of the teacher network depth estimation module. When the decoder gradually restores the depth map through the residual structure, the fusion signal obtained by ADFF is added To restore the high-resolution feature map, the specific calculation is: first, the minimum resolution of ADFF Perform Batch Normal standardization and use the Relu activation function to activate the preliminarily processed feature map T L ; Then, T L After the LPD processing layer, different resolution fusion signals are gradually added To restore the depth map.

[0045] (2) Design of student network depth estimation module

[0046] Student network depth estimation module (such as Figure 3 ) uses the GhostNet network as the depth estimation module encoder and the NNConv5 structure as the decoder. The encoder extracts features through multiple layers of Ghost layers. The Ghost layer first performs a convolution operation, then uses a simple linear transformation to obtain more features, and uses a residual structure to maintain the original features. The encoder transfers the feature maps output by a part of the network layer to the depth estimation module decoder. The decoder NNConv5 consists of a five-layer network, each layer consists of a Depthwise layer, a Pointwise layer, and an upsampling layer. The Depthwise layer and the Pointwise layer restore the depth information layer by layer. The upsampling layer performs interpolation operations after the Depthwise layer and the Pointwise layer to double the spatial resolution of the intermediate feature map to meet the resolution requirements of the NNConv5 residual structure.

[0047] (3) Design a knowledge distillation network framework for unsupervised single-view ship depth estimation

[0048] First, the network takes a single RGB image as input and uses the depth estimation module in step (1) to extract depth information to obtain a depth map. Figure 4 The Light Net, View Net, Albedo Net, and SD Net shown in the figure obtain illumination information, viewing angle information, albedo map, and symmetry information map, respectively. Among them, the illumination information and the depth map are rendered to obtain a shaded depth map. On the basis of assuming that the depth map and the albedo map are symmetrical relative to a fixed vertical plane, the depth map and the albedo map are flipped, and the standard viewing angle map is constructed by fusing the illumination information. The image has good symmetry under the standard viewing angle. Then, the depth map, standard viewing angle map, and viewing angle information before flipping are input into the renderer to reconstruct the image P′; the depth map, standard viewing angle map, and viewing angle information after flipping are input into the renderer to reconstruct the image P″. Finally, P′ and P″ are combined, and the symmetry information map is used to calculate the difference with the initial image through formula (1) to train the entire network.

[0049]

[0050] Among them, L R is the rendering loss; d′, d″ are the symmetric information map before flipping and the symmetric information map after flipping, respectively; W, H are the width and height of the image, respectively; P′, P″ are the reconstructed image before flipping and the reconstructed image after flipping, respectively; P is the original input image; λ1 is the weight of the rendering image loss after flipping.

[0051] Since only using rendering loss will lead to blurry reconstructed images, perceptual loss is added to the loss function, and the calculation is as shown in formula (2). Like rendering loss, perceptual loss also consists of two parts: before flipping and after flipping. First, the two images reconstructed before and after flipping and the training image are split into several layers and input into the VGG-16 network to extract features. The feature maps of each layer of the reconstructed image before and after flipping and the training image after splitting are obtained respectively. The difference between the feature maps of each layer is used as the perceptual loss. Here, the VGG-16 network does not need any training.

[0052] L r =L r1 +λ2L r2 (2)

[0053]

[0054]

[0055] Among them, L ris the perceptual loss; W, H are the width and height of the image respectively; P′, P″ are the reconstructed image before flipping and after flipping respectively; P is the training image; λ2 is the weight of the perceptual loss after flipping; V i (P″), V i (P), V i (P′) is the feature map of the i-th layer training image, reconstructed image, and flipped reconstructed image extracted by the VGG-16 network; d′ i , d″ i They are the symmetric information graph before flipping and the symmetric information graph after flipping of the i-th layer.

[0056] The total reconstruction loss is shown in formula (5), which consists of rendering loss and perceptual loss:

[0057] L=L R +λ l L r (5)

[0058] Where L is the reconstruction loss; λ l is the weight of the perceptual loss.

[0059] (4) Unsupervised training of student network under the knowledge distillation network framework

[0060] like Figure 1 As shown, the depth estimation module of the teacher network is replaced with the student network depth estimation module obtained in step (2), and the reconstruction loss and distillation loss are calculated for training. The training process for calculating the reconstruction loss is the same as step (3). When the student network depth estimation module is trained, the same RGB image is simultaneously input to the teacher network depth estimation module to obtain a series of feature maps of different dimensions of the teacher network. The distillation loss is constructed by combining a series of feature maps obtained in the student network depth estimation module, as shown in formula (6). In the process of optimizing the distillation loss function, the student network depth estimation module can obtain performance close to that of the teacher network depth estimation module.

[0061]

[0062] Among them, L di is the distillation loss; F s A set of feature maps extracted for the student network depth estimation module; F t A set of feature maps extracted for the teacher network depth estimation module; f s , f t F s and F t The corresponding feature map in ; W, H are the width and height of the feature map respectively.

[0063] (5) Building a ship dataset

[0064] By creating a 3D ship model, using KeyShot 3D realistic rendering software, setting different external environments such as lighting and angles to render realistic images of the ship model, a ship dataset consisting of 1,200 ship images was obtained. 1,080 images were used as the training set of the dataset, 120 images were used as the validation set, and 120 images were used as the test set (the validation set and the test set contain the same data).

[0065] (6) Model training on ship dataset

[0066] The entire depth estimation model is trained on the training set of the ship data set obtained in step (5). The size of the input image is uniformly set to 128*128, the batch size is set to 32, the epoch is set to 100, the learning rate is set to 0.01, and the network learning rate is adjusted by simulating the cosine annealing strategy. Through continuous iteration, the numerical change of the loss function tends to be stable, that is, the training is completed. Finally, the ship depth estimation model and the network weight file are obtained.

[0067] (7) Model testing

[0068] After the model is trained, the images in the test set are input into the trained depth estimation model, and the depth estimation module outputs the depth map as shown in the attached figure. Figure 5 It is shown that the unsupervised single-view ship depth estimation model and estimation method of the present invention can well realize the unsupervised depth estimation of the ship target under a single view.

Claims

1. An unsupervised single-view ship depth estimation model, characterized in that: The model includes: a teacher network depth estimation module, a student network depth estimation module, a Light Net network, a View Net network, an Albedo Net network, and an SD Net network; The teacher network depth estimation module includes a ResNet network as an encoder structure, an adaptive dense feature fusion structure for multi-dimensional feature processing, and a Laplacian pyramid structure as a decoder structure. The student network depth estimation module includes a Ghost Net network as an encoder structure and a NNConv5 structure as a decoder structure; The knowledge distillation network framework for unsupervised single-view ship depth estimation is established. The specific process includes: First, a single RGB image is input into the teacher network depth estimation module and the Light Net network, View Net network, Albedo Net network, and SD Net network to extract the image's depth map, lighting information, viewing angle information, albedo map, and symmetry information map respectively; Then, the extracted illumination information and depth map are rendered to obtain a shaded depth map; the shaded depth map and albedo map are synthesized to obtain a standard view map, and the standard view map, depth map, and view information are forward rendered to obtain a reconstructed RGB image; based on the input image, a reconstruction loss of the reconstructed image is constructed to regress the parameters of the teacher network model; The reconstruction loss consists of rendering loss and perceptual loss. The rendering loss is expressed as follows: Among them, L R is the rendering loss; d′, d″ are the symmetric information map before flipping and the symmetric information map after flipping respectively; W, H are the width and height of the image respectively; the depth map, standard view map and view information before flipping are input into the renderer to reconstruct the image P′; the depth map, standard view map and view information after flipping are input into the renderer to reconstruct the image P″; P is the original input image; λ1 is the weight of the rendered image loss after flipping; The perceptual loss is also composed of two parts, before and after flipping. The two images reconstructed before and after flipping and the training image are split into several layers and input into the VGG-16 network to extract features. The feature maps of each layer of the reconstructed image before and after flipping and the training image after splitting are obtained respectively. The difference between the feature maps of each layer is used as the perceptual loss. The perceptual loss is expressed as follows: L r =L r1 +λ2L r2 Among them, L r is the perceptual loss; λ2 is the weight of the perceptual loss after flipping; V i (P″), V i (P), V i (P ′ ) is the feature map of the i-th layer training image, reconstructed image, and flipped reconstructed image extracted by the VGG-16 network; d i ′,d″ i They are the symmetric information graph before flipping and the symmetric information graph after flipping of the i-th layer of the layer respectively; Finally, the depth estimation module of the teacher network is replaced with the student network depth estimation module, and the reconstruction loss and distillation loss are calculated for training. When the student network depth estimation module is trained, the same RGB image is simultaneously input into the teacher network depth estimation module to obtain a series of feature maps of different dimensions of the teacher network. The distillation loss is constructed by combining a series of feature maps obtained in the student network depth estimation module. In the process of optimizing the distillation loss function, the performance close to that of the teacher network depth estimation module is obtained. Among them, L di is the distillation loss; F s A set of feature maps extracted for the student network depth estimation module; F t A set of feature maps extracted for the teacher network depth estimation module; f s , f t F s and F t The corresponding feature map in ; W, H are the width and height of the feature map respectively.

2. The unsupervised single-view ship depth estimation model according to claim 1, characterized in that: The NNConv5 structure of the student network depth estimation module consists of a five-layer network, each layer consists of a Depthwise layer, a Pointwise layer, and an upsampling layer; the Depthwise layer and the Pointwise layer will restore the depth information layer by layer, and the upsampling layer performs interpolation operations after the Depthwise layer and the Pointwise layer to double the spatial resolution of the image.

3. An unsupervised single-view ship depth estimation method based on the estimation model according to any one of claims 1 or 2, characterized in that: The following steps are involved: Establishment of unsupervised single-view ship depth estimation model; Establishment of knowledge distillation network framework for unsupervised single-view ship depth estimation; Construct a ship depth estimation dataset; Training of unsupervised single-view ship depth estimation models; Testing of an unsupervised single-view ship depth estimation model.

4. The unsupervised single-view ship depth estimation method according to claim 3, characterized in that: The unsupervised single-view ship depth estimation model includes a teacher network depth estimation module, a student network depth estimation module, a LightNet network, a View Net network, an Albedo Net network, and an SD Net network; wherein, The process of establishing the teacher network depth estimation module is as follows: using a high-performance ResNet network as the encoder structure, using an adaptive dense feature fusion structure to process features of multiple dimensions, and finally using a Laplacian pyramid structure as the decoder structure of the module; The process of establishing the student network depth estimation module is as follows: using the Ghost Net network as the encoder structure, and using the NNConv5 structure with added Depthwise and Pointwise structures as the decoder structure of the depth estimation module; The Light Net network, View Net network, Albedo Net network, and SD Net network are used to extract the illumination information, viewing angle information, albedo map, and symmetry information map of the image, respectively.

5. The unsupervised single-view ship depth estimation method according to claim 3, characterized in that: The specific process of constructing the ship depth estimation dataset is as follows: creating a 3D ship model, using KeyShot 3D realistic rendering software to set physical parameters such as lighting and perspective to render the 3D model of the ship, and obtaining realistic ship images as the ship depth estimation dataset; the ship depth estimation dataset includes a training set, a validation set, and a test set.

6. The unsupervised single-view ship depth estimation method according to claim 3, characterized in that: The specific process of training the unsupervised single-view ship depth estimation model is as follows: transforming the training set images in the ship depth estimation dataset to a set resolution size, and inputting the transformed single-view ship images into the network in batches according to the number of Batch Sizes, and training according to the process of establishing the unsupervised single-view ship depth estimation knowledge distillation network framework, after multiple iterations, stopping the training after the iterative Epoch, obtaining the network weight file, and obtaining a model capable of performing depth estimation on the ship.

7. The unsupervised single-view ship depth estimation method according to claim 3, characterized in that: The specific process of the unsupervised single-view ship depth estimation model test is: using the test set images in the ship depth estimation dataset to test the effect of the model that can estimate the depth of the ship obtained after training, and obtaining the final model that can be used for depth estimation by comparing with the validation set.

Images