An underwater binocular depth estimation method based on unsupervised adaptive network
By adopting an unsupervised adaptive network in underwater depth estimation, combining style and feature adaptive modules, and using teacher-student network collaborative training, the problems of data limitations and inter-domain differences in underwater depth estimation are solved, achieving high Accurate underwater binocular depth estimation performance.
Patent Information
- Application Number
- CN202111589965.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-23
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2041-12-23
AI Technical Summary
The prior art has data limitations and inter-domain differences in underwater depth estimation, resulting in limited performance of underwater depth estimation. Especially under low light and geometric distortion conditions, it is difficult for existing methods to achieve high-quality depth estimation.
Using an unsupervised adaptive network method, the style adaptive submodule and feature adaptive submodule are used to combine onshore annotation data and underwater real data to build a style transformation network and a three-dimensional matching network, and use teacher-student network collaborative training. , narrow the interdomain difference between simulation data and real data, and achieve high-precision underwater binocular depth estimation.
The performance of underwater binocular depth estimation has been successfully improved, and higher ranging accuracy and better generalization capabilities have been achieved, which is better than existing methods and can generate high-quality depth maps in complex underwater environments.
Smart Images

Figure CN114299130B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of image processing and computer stereo vision, and specifically relates to an underwater binocular depth estimation method based on an unsupervised adaptive network. Background Art
[0002] With the development of computer stereo vision technology, binocular depth estimation is not only widely used in stereo vision tasks for land scenes such as autonomous driving, but also gradually plays an increasingly important role in underwater environment perception in ocean exploration, especially in close-range perception scenes such as underwater detection, positioning, and fishing. However, the depth estimation performance of real underwater data based on deep learning has encountered many bottlenecks. Specifically, due to factors such as weak lighting conditions and geometric distortion, underwater depth sensing equipment is expensive, the cost of acquiring underwater color image data with depth labels is too high, and the color change and image blur of underwater scenes cannot be avoided. Therefore, it is impractical to use sufficient high-quality stereo images to supervise the training of underwater depth estimation networks. In addition, there are great inter-domain differences in the data collected in different scenes, which makes the existing depth estimation model unable to be directly applied to underwater scenes through transfer learning mode, which limits the performance improvement of underwater depth estimation networks to a certain extent.
[0003] In order to solve the limitations of training data in underwater depth estimation, existing methods generally use image generation technology to synthesize underwater style data, and additionally train the depth estimation network based on the underwater style data and the corresponding depth map. However, most of these methods are based on monocular depth estimation mode and do not consider the highly ill-posed problem of lack of geometric constraints when predicting monocular images. Another commonly used method is based on the binocular unsupervised mode, using stereo image pairs as input and using the photometric consistency between stereo pairs to assist training. However, existing methods simply use underwater stereo data to train the existing unsupervised backbone network without considering the basic degradation characteristics of underwater data, and the ideal depth estimation performance is still difficult to achieve. Summary of the invention
[0004] In order to solve the above problems, the present invention conducts in-depth research on the depth estimation task based on stereo image pairs. The core idea of the present invention is to utilize a terrestrial stereo dataset with depth labels and a real underwater stereo dataset without labels, and under the premise of fully considering the degradation characteristics of underwater data, by learning the underwater data features, gradually training the stereo matching network, and inferring a high-precision underwater depth map. The method proposed in the present invention mainly includes a style adaptation submodule of the surface layer and a feature adaptation submodule of the semantic layer, which effectively narrows the domain differences in an unsupervised domain adaptation training method. Experimental results show that compared with other state-of-the-art underwater depth estimation methods, the present invention successfully achieves better underwater binocular depth estimation performance.
[0005] The present invention takes the underwater scene stereoscopic vision perception technology as the core problem. Aiming at the problem that the existing deep learning methods are overly dependent on high-quality training data and lack in-depth research on unsupervised underwater stereo vision technology, an underwater binocular depth estimation method based on an unsupervised adaptive network is designed. By constructing style and feature adaptive modules, the complex visual laws of real underwater scenes are excavated to achieve efficient underwater binocular depth estimation. First, in order to solve the problem of low ranging accuracy caused by the complex and changeable underwater environment, the key features of the underwater scene are extracted, and the underwater image conversion method is developed. Secondly, the disparity estimation method in the underwater low-texture environment is studied, and an unsupervised underwater stereo matching network is constructed. The domain differences between the simulation data and the real underwater images are solved through domain adaptation training, so as to achieve high-precision stereo matching of underwater images.
[0006] The specific technical solution of the present invention is:
[0007] An underwater binocular depth estimation method based on an unsupervised adaptive network includes a style adaptive submodule and a feature adaptive submodule. The style adaptive submodule generates images of different underwater styles that conform to natural scene conditions by introducing depth information into simulated underwater data, and considers the parallax consistency of the left and right images to constrain the distortion that may occur during the image generation process. The feature adaptive submodule addresses the performance bottleneck caused by the inability of simulated data to fully fit the semantic layer features of underwater images. Through the collaborative training of the teacher-student network, the domain difference between simulated data and real underwater data is narrowed, and the performance of underwater binocular depth estimation is further improved.
[0008] Specifically, in the style adaptation submodule, an unsupervised stereo image style transfer network is constructed, which aims to combine the content information such as the structure of the land image with the style features of the underwater image to synthesize the underwater style stereo image pair with depth labels. The generation of this simulation data is conducive to the subsequent training of the stereo matching network on the real underwater data. Among them, the present invention also introduces a torsion regularization method to strengthen the left-right consistency between the two views, thereby constraining the geometric consistency of the underwater style stereo image generation. However, considering that the generalization of the stereo matching network trained using only the above synthetic data for real underwater images is poor and cannot be fully applied to real underwater data, the present invention uses a feature adaptation submodule to break through the training bottleneck of the depth estimation network on real underwater data. Due to the great difference in the semantic content features of land and underwater data, directly aligning features through a global adversarial method not only cannot narrow the inter-domain differences, but is even very likely to cause negative transfer, and the performance degradation caused by the wrong semantic feature alignment on network training is far greater than the performance improvement of training using land data. Therefore, the present invention proposes a teacher-student network adaptive training method for feature layer adaptation, which gradually narrows the domain differences by forcing the teacher network and the student network to constrain the consistency of the prediction results of the target underwater data. The main implementation steps are as follows:
[0009] 1) Preparation of dataset.
[0010] The initial data is a land-labeled dataset, including land color images and corresponding depth maps used for training. In addition, a small number of underwater real color images are used for training and testing of unsupervised depth estimation networks for actual underwater environments.
[0011] 1-1) KITTI2012 (Andreas Geiger, Philip Lenz, and Raquel Urtasun, “Are we ready for autonomous driving The kitti vision benchmark suite,” CVPR, 2012.): includes color left and right image pairs and corresponding true depth maps, which are used for training style transfer networks and stereo matching networks.
[0012] 1-2) Underwater Real Dataset USD (Dana Berman, Deborah Levy, Shai Avidan, and Tali Treibitz, "Underwater single image color restoration using haze-lines and a new quantitative dataset," TPAMI, 2020.): Includes color left and right image pairs for training unsupervised domain adaptation, and the true depth map is only used for the testing process.
[0013] 2) Construction of style adaptation sub-module.
[0014] 2-1) The style transfer subnetwork is built based on AdaIN (X.Huang and S.Belongie, "Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization", ICCV, 2017.), which is composed of a cascade of encoder E and generator G. It can combine the content information of land images with the style features of underwater images to generate underwater simulation data.
[0015] The encoder E is built based on the VGG19 (K.Simonyan and Andrew Zisserman, "Very Deep Convolutional Networks for Large-Scale Image Recognition", ICLR, 2017.) structure, with land images and underwater real images as the input data of the network. The encoded feature maps are reorganized and sent to the generator G.
[0016] The generator G contains 9 layers of deconvolution operations. The feature maps from different scales in the encoder E are added to the network layer in G as supplementary semantic information through jump connections, and finally a simulated dataset of underwater style is obtained.
[0017] 2-2) Construct the loss function of the unsupervised style transfer sub-network. In order to obtain simulation data that conforms to the characteristics of real underwater scenes, the simulation data generated by the style transfer sub-network is input into the feature extraction module. The feature extraction module uses the VGG19 pre-trained model to extract high-dimensional perceptual feature maps at different network layers, including content features and style features, and construct content and style loss functions, so that the feature extraction module can extract the simulation data generated by the high-dimensional feature layer while maintaining the onshore content information. It can better learn the color style of underwater images; in addition, the torsion loss function is used to constrain the geometric consistency of the stereo image pair and restore the structural distortion that may be caused by a single image during the style transfer process.
[0018] 3) Construction of feature adaptation sub-module.
[0019] 3-1) Design a feature adaptive submodule, use simulated underwater data and real underwater data as input data of the source domain and target domain respectively, and finally obtain the depth estimation result of the real underwater data. The module adopts an integrated learning method, designs a weighted parameter optimization update method, and uses the teacher-student depth estimation network N with the same structure. t ,N s , combined with the supervised training model Ω s , strengthen the model Ω t The error correction capability of unlabeled target domain data prediction results in a more comprehensive and highly generalizable underwater binocular depth estimation model.
[0020] 3-2) Construct the loss function of the unsupervised depth estimation subnetwork. First, input the simulated left and right image pairs generated by the style transfer network, use the depth map of the land image as the supervision signal, and use the supervision loss function to continuously optimize the student stereo matching network model; at the same time, the underwater real data will be input into the student and teacher networks at the same time, and the network model will be optimized using the consistency loss of the prediction results between different networks; and based on the above supervision loss and consistency loss, the total loss function of the unsupervised depth estimation subnetwork is constructed in the feature adaptation submodule.
[0021] 4) Based on steps 2) and 3), train the style conversion and stereo matching sub-networks
[0022] First, the style transfer sub-network is trained based on perceptual loss using real data from land and water. The perceptual loss includes content loss L c and style loss L s ; Add the torsion loss function L after the training model converges w Continue training for a small number of iterations to eventually obtain labeled simulated underwater data;
[0023] Then, based on L supThe simulated underwater annotation dataset obtained by the style transfer sub-network is used to supervise the pre-training of the stereo matching student sub-network; finally, the underwater real data is simultaneously input into the student network N s and teacher network N t , an unsupervised adaptive learning method is used to update the network parameters and narrow the inter-domain differences of the depth estimation subnetwork on the semantic feature layer.
[0024] The beneficial effects of the present invention are as follows: the present invention proposes a novel underwater stereo matching network, which infers depth maps from real underwater stereo images in an unsupervised adaptive manner. The style adaptive submodule can generate underwater synthetic binocular data that meets geometric consistency constraints to solve the problem of lack of training data; and the feature adaptive submodule is implemented by using self-integration technology to minimize the large domain difference of semantic features between synthetic data and real underwater data. The above method framework is easy to construct, and the algorithm accuracy is higher than that of existing methods. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 It is the overall network framework proposed by the present invention, including the style adaptation submodule and the feature adaptation submodule. c ,L s Respectively represent the content loss and style loss function during style transfer sub-network training, L w represents the distortion loss function required for style transfer of stereo image pairs; L sup represents the style stereo matching supervised training loss function, L con represents the consistency loss function for unsupervised stereo matching training.
[0026] Figure 2It is a display of the depth map results of underwater data sets under different methods. From left to right, they are underwater input images (Inputs), depth map truth (GT), land stereo matching network Gwc-Net (Xiaoyang Guo, Kai Yang et.al. "Group-wise correlation stereo network," CVPR, 2019.), existing monocular underwater depth estimation network MU-Net (Xinchen Ye, Zheng Li et.al. "Deep joint depth estimation and color correction from monocular underwater images based on unsupervised adaptation networks," TCSVT, 2019.), existing binocular unsupervised binocular depth estimation network SU-Net (Katherine A. Skinner, Junming Zhang et.al. "Uwstereon et: Unsupervised learning for depthestimation and color correction of underwater stereo imagery," ICRA, 2019.) and the depth map prediction results of the method proposed in this invention (Ours). DETAILED DESCRIPTION
[0027] The following is a detailed description of the unsupervised adaptive network of the present invention in conjunction with the accompanying drawings:
[0028] 1) Data preprocessing
[0029] 1-1) Data preparation: Given a land RGB image {S l ,S r ,D S} and the corresponding depth map as the source domain, the underwater stereo image {T l ,T r} as the target domain. First, {S l ,S r} Input the unsupervised binocular style transfer network to generate the corresponding synthetic stereo image Then, and As the input of the teacher-student network, the domain difference between synthetic data and real data is narrowed through adaptive training. In the test phase, the student model N s Predicting depth maps from real underwater stereo images
[0030] 1-2) Split Dataset: Onshore labeled data {S l ,S r ,D S}A total of 194 pairs, all used for training in the style transfer process, to obtain labeled simulation data. In the stereo matching network, the ratio of simulation data training to verification data is 180:14. Underwater real binocular data {T l ,T r There are 57 pairs in total, all of which participate in the training and testing of the unsupervised stereo matching network, and the depth map is only used in the testing phase.
[0031] 1-3) Underwater real image correction: Use MATLAB calibration tools to perform image correction on underwater binocular data based on the camera parameters provided by the data set, weaken the distortion of the data itself, and reduce the data error of the left and right image pairs on the vertical axis.
[0032] 1-4) Underwater real data enhancement: During the training process of the stereo matching network, underwater real data is used as the target domain, and Gaussian random enhancement is used to alleviate the problem of too little data and enhance the network's ability to resist noise.
[0033] 2) Construction of style adaptation sub-module
[0034] 2-1) Style conversion sub-network construction
[0035] Unlike adversarial generation-based methods that require training a specific style transfer model for each underwater scene, this paper continues the adaptive instance normalization (AdaIN) method and combines the perceptual constraints of high-dimensional features to achieve style transfer while retaining content features. Only one network needs to be trained to perform style transfer in different underwater scenes. Figure 1 As shown, first, the land stereo image pairs are sent to the encoder E built based on the VGG19 structure to obtain the corresponding feature map {E(S l ),E(S r )}. At the same time, select an underwater stereo image as the target image (such as T l ) input encoder E, and obtain feature map E(T l ). The stylized feature map is obtained by recombining the feature maps of different domains. The formula is as follows:
[0036]
[0037] Among them, E(S l ) is the feature map of the left image on land. μ,σ are the mean and standard deviation of the image features.
[0038] Finally, the generated The generator module G contains 9 layers of deconvolution operations, each of which includes convolution conv, activation function ReLU and bilinear interpolation upsampling operations. In addition, the feature maps from different scales in the encoder E are added to the corresponding network layer in G through skip connections as supplementary semantic information, and the simulated image of underwater style is output.
[0039] 2-2) Construction of high-dimensional perception loss function
[0040] In order to preserve the scene structure and other content information of the generated image, and at the same time to extract the style features of the real underwater image, the VGG19 pre-trained model is used as the feature extraction module, and the simulated data generated by the style transfer network is used as input. High-dimensional perceptual features, including content features and style features, are extracted at different network layers to construct the content loss L c and style loss L s , respectively expressed as:
[0041]
[0042] Among them, F(·) and φ represent the high-dimensional content feature map and style vector extracted by VGG19 respectively. c ,l s Represents the number of layers of content features and style features used to calculate the loss function, where image style representation is the Gram matrix, which is used to quantitatively describe the image style information by performing cross-correlation calculation on the feature maps. ||·||2 represents the L2 norm, which is used to calculate the cross-correlation of all feature maps (including content features F(·) and style features ) and perform weighted summation to obtain L c and L s The expression of w l is the weight of the lth layer. It should be noted that the simulated left and right images generated by the above process may be distorted, destroying the geometric relationship between the two views. Therefore, the present invention introduces the torsion loss function L w To constrain the geometric consistency of underwater style stereo image generation:
[0043]
[0044] Among them, ||·||1 represents the L1 loss function (the same below), W(·) represents the twist operation, and the synthetic right figure is used And the predicted depth map D S Torsed left view of reconstructed simulation data And with the simulation data on the left Together we build L wTherefore, the final loss function L of the style transfer sub-network training is SA The statement is as follows:
[0045] L SA =α1L c +α2L s +α3L w
[0046] 3) Construction of feature adaptation submodule
[0047] 3-1) Construction of self-integrated domain adaptive depth estimation framework
[0048] Since synthetic data cannot fully inherit the characteristics of the underwater environment, it is necessary not only to be close to the target domain in the representation layer to improve the performance of underwater stereo matching, but also to further reduce the domain difference between the synthetic data domain and the real underwater domain in the feature layer. In order to make the teacher network have stronger domain generalization ability, the present invention adopts an ensemble learning method and proposes a domain adaptation method for training the depth estimation network, using simulated underwater data and real underwater data as input data of the source domain and target domain respectively, and outputting the depth estimation result of the real underwater data. It includes two teacher-student depth estimation networks N with the same structure and different optimization and update methods. t ,N s , the structure adopts Gwc-Net (Xiaoyang Guo, Kai Yang et.al. "Group-wise correlation stereo network," CVPR, 2019.), such as Figure 1 In the training process of self-ensemble domain adaptation, the exponential moving average method (EMA) is used in each iteration to update the parameters of the teacher network, that is, the parameters Ω of the student model s Multiply by a certain decay rate, participate in the parameter update of the teacher network, and strengthen the network N t The error correction capability of unlabeled target domain data prediction results in a more comprehensive and highly generalizable underwater binocular depth estimation model Ω t The expression is as follows:
[0049]
[0050] in are the network model parameters of the teacher and student networks at the i-th iteration, respectively, and λ is a weight parameter with a value of (0-1).
[0051] 3-2) Construction of loss function of depth estimation sub-network
[0052] For student networks sIn the domain training process, the simulated underwater left and right image pairs generated by the style transfer network are used for supervised training with the corresponding land image depth map to obtain the pre-trained model. The supervised loss function L sup The expression is as follows:
[0053]
[0054] in, For the simulation data in the student network N s The disparity prediction results.
[0055] In addition, in order to make the network model better adapt to underwater scenes, the randomly enhanced target domain data are input into the student network N s and teacher network N t , predicted And use the consistency loss function L con Update of constrained network parameters:
[0056]
[0057] On the other hand, since the teacher network has stronger generalization, the prediction results of the teacher network can be used as pseudo labels to supervise the training of the student network during the training process, thus realizing the self-supervised training of the student network. In summary, the training loss function L for unsupervised depth estimation is FA The expression is as follows:
[0058] L FA =β1L sup +β2L con
[0059] Among them, β1 and β2 are weight parameters.
[0060] 4) Network training and experimental results
[0061] 4-1) Network training and parameter setting
[0062] First, using real data from land and underwater, based on perceptual loss (including content loss L c and style loss L s ) to train the style transfer sub-network. After the training model converges, the torsion loss function L is added w Continue training for a small number of iterations and finally obtain labeled simulated underwater data. During the training process, the parameters are set to α1 = 5.0, α2 = 0.1, and α3 = 0.1.
[0063] Then, based on L sup The simulated underwater annotation dataset obtained by the style transfer sub-network is used to supervise the pre-training of the stereo matching sub-network. Finally, the real underwater data is simultaneously input into Ns and N t , based on L FA The network parameters are updated using self-ensemble unsupervised learning. The parameter is set to a learning rate of lr = 0.0001, and is reduced by a factor of 10 after 200 training epochs. Other parameters are set to λ = 0.99, β1 = 1, and β2 = 0.5.
[0064] 4-2) Depth Estimation Comparison Experimental Results
[0065] Figure 2 The predicted depth maps of different methods are shown, and it can be seen intuitively that the method of the present invention can obtain the visual prediction result closest to GT.
[0066] The evaluation indicators of the underwater binocular depth estimation task are EPE and D1, which represent the pixel-by-pixel average disparity error and the predicted outlier pixel ratio of the disparity prediction map, respectively. The outliers are represented by pixels with disparity errors greater than 3px or 0.05d (d is the actual disparity value). The smaller the values of EPE and D1 indicators, the higher the network performance. As shown in Table 1, compared with the existing monocular unsupervised network (MU-Net) and stereo unsupervised underwater depth estimation network (SU-Net), it can be observed that in four different underwater scenes (the subscripts in the table are K, M, N, S), the method of the present invention (Ours) has improved the network performance to varying degrees, which is better than the previous monocular or stereo underwater depth estimation methods.
[0067] Table 1 Depth estimation results of underwater binocular data using different methods.
[0068] Experimental methods <![CDATA[EPE K ]]> <![CDATA[D1 K ]]> <![CDATA[EPE M ]]> <![CDATA[D1 M ]]> <![CDATA[EPE N ]]> <![CDATA[D1 N ]]> <![CDATA[EPE S ]]> <![CDATA[D1 S ]]> MU-Net 7.0974 0.6794 9.4024 0.5588 7.9691 0.4948 14.5912 0.6460 SU-Net 12.5766 0.7780 26.9227 0.8939 19.7091 0.8981 12.6444 0.8600 Ours 6.6597 0.6932 8.1247 0.5789 6.3310 0.4589 10.1833 0.6133
Claims
1. An underwater binocular depth estimation method based on an unsupervised adaptive network, characterized in that: The steps include: 1) Prepare initial data The initial data is a land-labeled dataset, including land color images and corresponding depth maps used for training; underwater real color images are used for training and testing of unsupervised depth estimation networks for actual underwater environments; 2) Construction of style adaptation sub-module 2-1) The style transfer sub-network is built based on AdaIN and consists of a cascade of encoder E and generator G. It combines the content information of the land image with the style features of the underwater image to generate underwater simulation data; The encoder E is built based on the VGG19 structure, with land images and underwater real images as the input data of the network. The feature maps obtained after encoding are reorganized and sent to the generator G; The generator G contains 9 layers of deconvolution operations. The feature maps from different scales in the encoder E are added to the network layer of G as supplementary semantic information through skip connections, and finally a simulated dataset of underwater style is obtained. 2-2) Construct the loss function of the unsupervised style transfer sub-network; In order to obtain simulation data that conforms to the characteristics of real underwater scenes, the simulation data generated by the style transfer sub-network is input into the feature extraction module. The feature extraction module uses the VGG19 pre-trained model to extract high-dimensional perceptual feature maps at different network layers, including content features and style features, and construct content and style loss functions, so that the feature extraction module can extract the simulation data generated by the high-dimensional feature layer to better learn the color style of the underwater image on the basis of maintaining the onshore content information; Use the torsion loss function to constrain the geometric consistency of the stereo image pair and restore the structural distortion that may be caused by a single image during the style transfer process; 3) Construction of feature adaptation submodule 3-1) Design a feature adaptive submodule, use simulated underwater data and real underwater data as input data of the source domain and target domain respectively, and finally obtain the depth estimation result of the real underwater data; the module adopts an ensemble learning method, designs a weighted parameter optimization update method, and uses the teacher-student depth estimation network N with the same structure. t , N s , the structure is Gwc-Net, combined with the model Ω trained with labeled source domain data supervision s , strengthen the model Ω t The error correction capability of unlabeled target domain data prediction results in an underwater binocular depth estimation model. 3-2) Construct the loss function of the unsupervised depth estimation subnetwork; first input the simulated left and right image pairs generated by the style transfer network, use the depth map of the land image as the supervision signal, and use the supervision loss function to continuously optimize the student stereo matching network model; at the same time, the underwater real data will be input into the student and teacher networks at the same time, and the network model will be optimized using the consistency loss of the prediction results between different networks; and based on the above supervision loss and consistency loss, the total loss function of the unsupervised depth estimation subnetwork is constructed in the feature adaptation submodule; 4) Based on steps 2) and 3), train the style conversion and stereo matching sub-networks First, the style transfer sub-network is trained based on perceptual loss using real data from land and water. The perceptual loss includes content loss L c and style loss L s ; Add the torsion loss function L after the training model converges w Continue training for a small number of iterations to eventually obtain labeled simulated underwater data; Then, based on L sup The simulated underwater annotation dataset obtained by the style transfer sub-network is used to supervise the pre-training of the stereo matching student sub-network; Finally, the underwater real data is simultaneously input into the student network N s and teacher network N t , an unsupervised adaptive learning method is used to update the network parameters and narrow the inter-domain differences of the depth estimation subnetwork on the semantic feature layer.
2. The underwater binocular depth estimation method based on unsupervised adaptive network as claimed in claim 1, characterized in that: The construction of the style adaptation submodule includes the following steps: 2-1) Style conversion sub-network construction Based on the AdaIN method, combined with the perceptual constraints of high-dimensional features, style transfer is achieved while retaining content features; the left and right image pairs are sent as input to the encoder E to obtain the corresponding feature maps {E(S l ), E(S r )}; At the same time, select a real underwater color image as the target image T l Input encoder E and get feature map E(T l ); The style feature map is obtained by recombining the feature maps of different domains, and the formula is as follows: Among them, E(S l ) is the feature map of the left image on land; μ, σ are the mean and standard deviation of the image features; Finally, the generated The output is sent to the generator module G; the generator G consists of 9 layers of deconvolution operations, each of which includes convolution conv, activation function ReLU and bilinear interpolation upsampling operations; the feature maps from different scales in the encoder E are added to the corresponding network layer in G as supplementary semantic information through skip connections, and the simulated image of underwater style is output. 2-2) Construction of style conversion sub-network loss function In order to preserve the scene structure and other content information of the generated image, and at the same time to extract the style features of the real underwater image, the VGG19 pre-trained model is used as the feature extraction module to extract the high-dimensional perceptual features of the simulation data for constructing the content loss L c and style loss L s , respectively expressed as: Where F(·) and φ represent the high-dimensional content feature map and style vector extracted by VGG19 respectively; l c , l s Represents the number of layers of content features and style features used to calculate the loss function, where image style representation is the Gram matrix, and the image style information is quantitatively described as a whole by performing cross-correlation calculation on the feature maps; ||·||2 represents the L2 norm, and L is obtained by weighted summation of all feature maps. c and L s The expression of w l is the weight of the lth layer, and the feature map includes content features F(·) and style features Introducing the torsion loss function L w To constrain the geometric consistency of underwater style stereo image generation: Among them, ||·||1 represents the L1 loss function (the same below), W(·) represents the twist operation, and the synthetic right figure is used And the predicted disparity map D S Torsed left view of reconstructed simulation data And with the simulation data on the left Together we build L w ; Therefore, the final loss function L of the style transfer sub-network training SA The statement is as follows: L SA =α1L c +α2L s +α3L w 。 3. The underwater binocular depth estimation method based on unsupervised adaptive network as claimed in claim 1, characterized in that: The construction of the feature adaptation submodule includes the following steps: 3-1) Construction of self-integrated domain adaptive depth estimation framework Using ensemble learning, a domain adaptation method for training depth estimation network is proposed. The simulated underwater data and real underwater data are used as input data of source domain and target domain respectively, and the output is the depth estimation result of real underwater data. It includes two teacher-student depth estimation networks N with the same structure but different optimization and updating methods. t , N s In the training process of self-ensemble domain adaptation, the exponential sliding average method is used to update the parameters of the teacher network in each iteration, that is, the parameters Ω of the student model s Multiply by the decay rate, participate in the parameter update of the teacher network, and strengthen the network N t The error correction capability of unlabeled target domain data prediction is used to obtain the underwater binocular depth estimation model Ω t ; The expression is as follows: in are the network model parameters of the teacher and student networks at the i-th iteration, respectively, and λ is a weight parameter with a value of (0-1); 3-2) Construction of loss function of depth estimation sub-network For student networks s In the domain training process, the simulated underwater left and right images generated by the style transfer network are used to supervise the training with the corresponding land image depth map to obtain the pre-trained model; the supervision loss function L sup The expression is as follows: in, For the simulation data in the student network N s The disparity prediction result of In order to make the network model better adapt to underwater scenes, the randomly enhanced target domain data are input into the student network N s and teacher network N t , predicted And use the consistency loss function L con Update of constrained network parameters: Since the teacher network has stronger generalization, the prediction results of the teacher network can be used as pseudo labels to supervise the training of the student network during the training process, thus realizing self-supervised training of the student network. Then, the training loss function L for unsupervised depth estimation is FA The expression is as follows: L FA =β1L sup +β2L con Among them, β1 and β2 are weight parameters.
