Deep learning-based turbid underwater image restoration method

Abstract

The invention provides a deep learning-based turbid underwater image restoration method, which comprises the following steps: establishing a CycleGAN cyclic consistency network model, respectively adding attention modules in an encoder and a decoder to optimize the model so as to enhance the detail extraction ability of the model, the attention modules comprising simAM and MLCA, and training the CycleGAN cyclic consistency network model by using the data set, adjusting the model configuration, adding a loss function and an optimizer, and performing image quality and visual effect evaluation on the processed image of the CycleGAN cyclic consistency network model. By means of the mode, the problems of blurring, distortion and noise in the underwater image can be better solved, the restored image is clearer and more accurate, the underwater image processing efficiency and quality can be improved, and development of the fields of underwater exploration, underwater archaeology, underwater robots and the like can be promoted.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a turbid underwater image restoration method based on deep learning. Background Art

[0002] The acquisition and processing of information in the ocean mainly rely on acoustic vision or optical vision technology. Information collection based on acoustic vision is mainly achieved by sonar, but its image information collection ability is weak and the generated image is fuzzy; information collection based on optical vision is achieved by optical imaging equipment, with higher data resolution and stronger collection ability. As important data for collecting underwater information, images play a vital role in completing tasks such as marine life identification, underwater environment monitoring, and underwater scene reconstruction.

[0003] Optical line inspection or detection tasks are key to underwater robot operations. However, this task faces huge obstacles. The main reason is that the underwater imaging environment is complex, which causes serious degradation of underwater images acquired by the optical system and serious distortion of target images captured by the camera. Traditional underwater image enhancement methods work well in fixed scenes, but when facing underwater images of different scenes, there will be distortion or under-enhancement problems. When the same device needs to be applied to different underwater scenes, the effect is not ideal.

[0004] At present, the mainstream underwater image restoration methods are mainly divided into image restoration based on priors and image restoration based on deep learning. The image restoration method based on priors can achieve good restoration effects in specific scenarios, but it loses its effect when the degradation characteristics of underwater images change, and cannot adapt to different underwater scenarios. Most image restoration methods based on deep learning also have the problem of not being able to achieve a good balance between noise removal and image color detail restoration. Summary of the invention

[0005] Purpose of the invention: This application provides a turbid underwater image restoration method based on deep learning, which uses a deep learning model to eliminate the degradation effect of water on the image, thereby restoring a clear and realistic underwater scene.

[0006] Technical solution: The present invention provides a turbid underwater image restoration method based on deep learning, comprising the following steps:

[0007] S1, collect a large amount of underwater turbidity image data, and form a data set after preprocessing;

[0008] S2. Establish a CycleGAN cycle consistency network model, wherein the CycleGAN cycle consistency network model includes two generators and two discriminators, and the generator includes an encoder, a residual device and a decoder;

[0009] The input image data is passed through the encoder to output a feature map, and then feature extraction is performed in the residual device. Finally, the decoder outputs a feature map with the same dimension as the input image through deconvolution and depooling. This feature map is the result of the generator's forgery. The discriminator determines whether the data given by the generator is real data or forged data;

[0010] Attention modules are added to the encoder and decoder respectively to optimize the model to enhance the model's ability to extract details. The attention modules include simAM and MLCA.

[0011] S3, using the data set obtained in step S1 to train the CycleGAN cycle consistency network model, and adjust the model configuration, add loss function and optimizer;

[0012] S4. Evaluate the image quality and visual effects of the processed images of the CycleGAN cycle consistency network model.

[0013] Furthermore, step S1 preprocessing includes alignment, cropping, and labeling.

[0014] Further, an attention mechanism SimAM is added to the encoder;

[0015] The decoder includes three convolution modules, and a hybrid attention mechanism SimAM is added after each convolution module of the encoder.

[0016] Furthermore, InstanceNorm normalization is used after each convolution and activation function in the convolution module of the decoder. This normalization method calculates the mean and variance of height and weight in the channel direction in each bitch, and then calculates the result of each feature map.

[0017] Furthermore, the residual device uses multiple Resnet structures, and the input data of a single Resnet structure is output after convolution Conv, InstanceNorm normalization, activation function ReLU, convolution Conv, InstanceNorm normalization, and two convolution kernels are both 3*3.

[0018] Furthermore, an attention mechanism MLCA is added to the decoder. First, the input data is divided into 5*5 blocks, and local average pooling and global average pooling are used for the blocks to output a 5*5*C feature map. Then, two one-dimensional convolutions are used to extract features from the feature map to output a 1*1*C matrix. The sigmoid activation function and adaptive average pooling are used to output the feature map weights. At the same time, MLCA will also resize the 5*5*C feature map to 1*1*(C*5*5), and use one-dimensional convolution to obtain a vector of the same dimension. The vector is resized to 5*5*C, and the obtained feature map is matrix-added with the above-mentioned feature map weights. Finally, the matrix obtained by de-pooling is the weight parameter of the input feature map. The output result of the MLCA attention mechanism can be obtained by using the weight parameter and the input feature map to calculate the Hadamard product.

[0019] Furthermore, the discriminator adopts a discriminator structure in the form of PatchGAN full convolution, inputs the feature map into the discriminator, and outputs the result matrix through the structure of convolution, InstanceNorm normalization and LeakyReLU activation function. Each matrix represents the discrimination result of the generated image receptive field, and finally the discrimination result of the discriminator for the complete image is obtained by averaging the matrix.

[0020] Furthermore, the loss function in step S3 includes adversarial loss and cycle consistency loss. The adversarial loss is used to measure the label value obtained by inputting the image generated by the generator into the corresponding discriminator. The cycle consistency loss is used to measure whether the image in the sample space X is mapped to the sample space Y by the generator, and then generated to the sample space X by another generator. The result is close to the original image.

[0021] Furthermore, the image quality assessment indicators for image quality and visual effect assessment in step S4 include image quality assessment indicators such as PSNR, SSIM, UIQM and UCIQE.

[0022] Beneficial effects: This solution converts one type of image into another type of image by constructing a CycleGAN cycle consistency network model without the need for paired image datasets. Through adversarial training, the generator can learn the mapping relationship between the two domains, thereby realizing image conversion. Therefore, it can be more easily applied to underwater image restoration and reduce the workload of data annotation. Different attention modules are added to the encoder and decoder of the model structure, respectively, so that the model can pay more attention to the clear parts and important features in the image, thereby effectively removing noise and blur and improving the quality and accuracy of the image.

[0023] Through the above method, the blur, distortion and noise problems in underwater images can be better handled, making the restored images clearer and more accurate. This can not only improve the efficiency and quality of underwater image processing, but also promote the development of underwater exploration, underwater archaeology, underwater robots and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 A schematic diagram of a process for restoring a turbid underwater image provided in an embodiment of the present application;

[0026] Figure 2 A schematic diagram of the generator network structure provided in an embodiment of the present application;

[0027] Figure 3 Schematic diagram of the SimAM attention mechanism provided in the embodiment of the present application;

[0028] Figure 4 A diagram of the structure of an encoder provided in an embodiment of the present application;

[0029] Figure 5 A Resnet network structure diagram provided for an embodiment of the present application;

[0030] Figure 6 A decoder network structure diagram provided for an embodiment of the present application;

[0031] Figure 7 Schematic diagram of the MLCA attention mechanism provided for the embodiment of the present application;

[0032] Figure 8 A schematic diagram of the discriminator network structure provided in an embodiment of the present application;

[0033] Fig. 9 A structural diagram of a restoration method provided in an embodiment of the present application;

[0034] Fig.10 A schematic diagram of a mapping process from an x ​​domain to a y domain based on a cross entropy loss function provided in an embodiment of the present application;

[0035] Fig.11 The restored result images of different water depths provided in the embodiment of the present application;

[0036] Fig.12 A schematic diagram of the results of executing the restoration and detection tasks provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be clearly and completely described below in conjunction with the drawings in the present application.

[0038] A turbid underwater image restoration method based on deep learning is mainly aimed at the problems of image distortion and blurring of computer vision imaging systems in underwater environments, so as to improve the clarity and contrast of underwater images and make the images meet the physical imaging characteristics.

[0039] like Figure 1 As shown, the main steps are as follows:

[0040] S1. Collect a large amount of underwater turbidity image data and form a data set after preprocessing.

[0041] The underwater image data collected here can be collected through the Internet or collected on site, and the images are preprocessed. Preprocessing includes operations such as alignment, cropping, and annotation to prepare for the training of the neural network model. The underwater images need to be diverse and representative to reduce the probability of overfitting of the model.

[0042] If underwater image data is obtained by field collection, a homemade camera can be used to take pictures of underwater scenes, and then the images can be transmitted to a PC using 5G transmission to obtain image data. Finally, the image data can be classified according to the sample space to obtain a data set.

[0043] S2. Establish a CycleGAN cycle consistency network model. The CycleGAN cycle consistency network model includes two generators and two discriminators. After the input image data passes through the encoder, a feature map is output. Then, feature extraction is performed in the residual device. Finally, the decoder outputs a feature map with the same dimension as the input image through deconvolution and depooling. This feature map is the result of the generator's forgery. The discriminator determines whether the data given by the generator is real data or forged data.

[0044] Attention modules are added to the encoder and decoder respectively to optimize the model to enhance the model's ability to extract details. The attention modules include simAM and MLCA.

[0045] Specifically, the main function of the generator is to generate data similar to the original data, and the size of its output matrix is ​​consistent with the size of its input image data; the discriminator is used to determine whether the data given by the generator is real data or forged data. Its input is image data and its output is a 30*30*3 matrix. The generator network structure consists of encoder, residual and decoder structures, such as Figure 2As shown in the figure. Among them, after the input image data passes through the encoder, the width and height of the feature map output are reduced, and the number of channels is increased. Then, feature extraction is performed in the residual device, and finally, the decoder outputs a feature map with the same dimension as the input image through deconvolution and depooling. This feature map is the result of the generator's forgery. Such processing will cause the image to lose features due to similar compression operations, so that the result map generated by the generator is significantly different from the original image, and the image detail features cannot be restored. In order to reduce the problem of image feature loss caused by this structure, different types of attention mechanisms are added to the encoder and decoder to extract image spatial features.

[0046] Therefore, a hybrid attention mechanism SimAM is added after each convolution module of the original encoder. Specifically, the original encoder structure of the CycleGAN model uses a padding and three convolution modules (convolution, normalization, and activation function) structure. The network structure is relatively simple and cannot effectively extract features from the image. Therefore, a hybrid attention mechanism SimAM is added after each convolution module. The principle of the attention mechanism SimAM is as follows Figure 3 As shown in the figure. Since existing attention modules usually only focus on one-dimensional channels or two-dimensional domain spaces, and these two attention mechanisms correspond to the feature-based attention and space-based attention in the human brain. Hybrid SimAM fuses the channel and domain space and outputs three-dimensional neuron weights.

[0047] The SimAM attention mechanism believes that when deep learning models process images, they need to estimate the importance of each single neuron in the convolutional layer. The neurons with the richest information often show relatively unique discharge characteristics relative to the surrounding neurons. The easiest way to find such neurons is to use the energy function to measure the linear separability between all neurons and other neurons. If a neuron is linearly separable from all other neurons and the energy function value is the smallest, then the neuron is considered to be the target neuron.

[0048] The minimum energy of each neuron found by simAM can be expressed as follows:

[0049]

[0050] In the formula, represents the energy of the t-th neuron on the *-th channel, is the mean of all neuron inputs, is the variance of all neuron inputs and the mean, and λ is a set constant. Thus, the formula for feature enhancement is:

[0051]

[0052] Using formula (2-2), we can get the output features of the simAM attention module Among them, X represents the input feature, E is the grouping of all channels and spatial dimensions, the sigmoid function is mainly used to constrain E to avoid E being too large, and ⊙ is the scaling operator.

[0053] After combining simAM with the encoder, an improved encoder network structure is formed, such as Figure 4 As shown in the figure. The input image first passes through convolution Conv, activation function ReLU, normalization and simAM encoder to output the feature image of the current input image, and then the output feature image is used as the input to the next level encoding network, and finally the feature image after dimensionality reduction is output. The three convolution kernels are 7*7, 3*3 and 3*3 respectively.

[0054] Special, from Figure 4 As can be seen, InstanceNorm normalization is used after each convolution and activation function in the model. This normalization method calculates the mean and variance of height and weight in the channel direction in each bitch, and then calculates the result of each feature map through formula (2-3). This generated result depends on each individual image instance, so using this normalization structure can accelerate model convergence and maintain the independence between different image instances.

[0055]

[0056] Where x is the bitch data, y is the normalized output, E[x] and Var[x] represent the mean and variance of the bitch data, ε is a very small constant, and γ and β are learnable vectors used to retain the expressive power of the original network.

[0057] As one of the implementation methods, the residual unit in the CycleGAN model of this scheme uses a total of 9 Resnet structures, among which a single Resnet structure is as follows Figure 5As shown. First, the feature image output by the encoding network structure is used as the input of the first Resnet structure in the residual model. Each Resnet structure consists of a 3*3 convolution Conv, an InstanceNorm normalization, an activation function ReLU, a 3*3 convolution Conv, and an InstanceNorm normalization in series. Then, the features of the image output by each Resnet structure are fused with the input of the structure as the input of the next Resnet structure, and finally the features of the image are output. This type of residual device assumes that a certain layer of the network can learn a mapping H(x), where x is the input. Then, after introducing a skip connection, the residual mapping F(x) = H(x)-x can be learned, that is, F(x) can be used to represent the difference between the original mapping and the identity mapping (i.e., x). This design makes it easier for the network to learn the residual because it only needs to add the content learned from the original mapping to the input instead of completely relearning it.

[0058] As one of the implementation methods, the MLCA attention mechanism with parameters is introduced into the decoder structure of the CycleGAN model in this scheme. This is because the decoder module in the original CycleGAN mainly uses deconvolution and ReLU activation functions to perform upsampling-like operations on the feature map. This structure has a relatively good effect in image style conversion, but when completing the image restoration task, the model structure is too simple and cannot effectively extract image features, and the clarity of the generated image is not enough. Therefore, the introduction of the MLCA attention mechanism not only combines the characteristics of the local attention mechanism and the global attention mechanism, but also combines spatial information and channel information at the same time, achieving a good balance between performance and complexity.

[0059] Specifically, the decoder structure that introduces this mechanism is as follows Figure 6 As shown in the figure, the structure contains 2 decoders, 1 convolution layer Conv and activation function Tanh, where each decoder consists of convolution transpose, normalization, activation function ReLU and mixed local channel attention MLCA.

[0060] like Figure 7As shown in the figure, the MLCA attention mechanism first divides the input data into 5*5 blocks, uses local average pooling and global average pooling to output a 5*5*C feature map for the block, then uses two one-dimensional convolutions to extract features and output a 1*1*C matrix for the feature map, and uses the sigmoid activation function and adaptive average pooling to output the feature map weights for the matrix; at the same time, MLCA will also resize the 5*5*C feature map to 1*1*(C*5*5), and use one-dimensional convolution to obtain a vector of the same dimension, resize the vector to 5*5*C, perform matrix addition on the obtained feature map and the above feature map weights, and finally obtain the matrix obtained by one unpooling as the weight parameter of the input feature map. The output result of the MLCA attention mechanism can be obtained by using the weight parameter and the input feature map to calculate the Hadamard product.

[0061] In the decoder structure, the feature map mainly performs upsampling operations. In order to enable the feature map to restore the image details as much as possible, the parameterized attention mechanism MLCA is introduced in the decoder. The depth and breadth of the network are improved by learning parameters, so that it can focus on more detailed features. In the encoder structure, the feature map is downsampled. It is only necessary to make the model pay attention to the important information of the image as quickly as possible, so the parameter-free attention mechanism simAM is used.

[0062] As one of the implementation methods, the discriminator in the CycleGAN model of this scheme adopts the discriminator structure of the PatchGAN full convolution form. The discriminator is used to determine whether the generator result is data in the specified sample space, and its output result is 0 or 1. The discriminator of the traditional generative adversarial network directly maps the result feature map generated by the generator to a number between 0 and 1, and uses the range of the value to judge the category of the image, but this method is greatly affected by the local noise of the image. In order to make the discrimination result unaffected by noise and improve the accuracy of the discrimination result, the PatchGAN network divides the image into many blocks, and then generates an evaluation matrix that is much smaller than the input image under the action of multiple convolutional layers. Any element of the evaluation matrix represents the receptive field of the local image of the input image to obtain more fine-grained local information. This local discriminator method makes the model pay more attention to the details and texture of the image, not just the overall structure. Therefore, the PatchGAN network performs well in generating high-quality, detailed images.

[0063] The discriminator network model is as follows Figure 8As shown in the figure, it consists of two convolutional layers, activation functions ReLU and Tanh, and category incremental learning algorithm CIL. Among them, the two CILs are connected in series, and new features are incorporated under the action of the convolutional layer. Usually, the PatchGAN network inputs a feature map of size 256*256*3 into the discriminator, and outputs a 30*30 result matrix through the structure of convolution, InstanceNorm standardization, and LeakyReLU activation function. Each matrix represents the discrimination result of the receptive field of the generated image. The calculation formula of the receptive field size of each layer is shown in formula (2-4). Finally, the discrimination result of the discriminator for the complete image can be obtained by averaging the matrix.

[0064] input size =(output size -1)×k stride +k size (2-4)

[0065] In the formula, input ste is the receptive field of the current layer, output ste is the receptive field output of the previous layer, k stride is the step length, k ste is the convolution kernel size.

[0066] S3. Use the data set obtained in step S1 to train the CycleGAN cycle consistency network model, adjust the model configuration, add loss functions and optimizers to improve the performance and generalization ability of the model, and set pre-training parameters to train the model.

[0067] The restoration method of this scheme first obtains an underwater image dataset by collecting or searching and classifies it into two sample spaces A and B, where A contains underwater turbid images and B contains restored images. Then, the improved CycleGAN network model is trained using this dataset. During the training process, the discriminator is first trained so that it can correctly output the discrimination results for the image data. Secondly, the generator is trained so that it can forge the data in sample space A (i.e., underwater degraded images) into data in sample space B (i.e., restored images). After the training is completed, the underwater turbid image is input into the generator A2B, and the output result is the corresponding restored image. The restoration process is as follows: Fig. 9 shown.

[0068] As one of the implementation methods, unpaired datasets are used for training and validation. It is only necessary to classify the images in two sample spaces. Therefore, the datasets in this paper use UIEB and UIEBD datasets, and the ratio of training and validation data is 0.9:0.1. The UIEBD dataset contains different underwater environments and diverse underwater image degradation features. Therefore, the model trained using this dataset has good generalization ability.

[0069] Furthermore, the loss function in the CycleGAN model of this scheme is designed, and the overall loss includes adversarial loss (GAN loss) and cycle consistency loss (consitency loss).

[0070] Adversarial loss is used to measure the label value obtained by the corresponding discriminator when the image generated by the generator is input. The mapping process is as follows Fig.10 It is based on the cross entropy loss function, and the expression is as follows:

[0071] l GAN (G,D Y ,X,Y)=E y～Pdata(y) [logD Y (y)]+E x～Pdata(x) [LOG(1-D Y (x))] (3-1)

[0072] In the formula, G, D Y , X, Y are the generator, discriminator, input and output images respectively, where the G generator is responsible for converting the image in the X domain to the image in the Y domain, and E x～Pdata(x) and E y～Pdata(y) are the mean between the input and the real input sample, the output and the real output sample, respectively, Y (x) and D Y (y) represents the discriminator’s discrimination results for input data x and real data y respectively.

[0073] Cycle consistency loss is used to measure whether the image in sample space X is mapped to sample space Y by a generator, and then generated to sample space X by another generator and the resulting image is close to the original image. This loss is based on the L1loss loss function, and the expression is as follows:

[0074] l cyc (G,F)=E X～Pdata(x) [F(G(x))-x]+[|G(F(y))-y] (3-2)

[0075] Where the F generator is responsible for converting images in the Y domain to images in the X domain.

[0076] The total loss function is the sum of two adversarial loss functions and a cycle consistency loss function multiplied by the scaling factor, expressed as follows:

[0077] L(G,F,D x ,D y )=L GAN (G,D Y ,X,Y)+L GAN (F,DX ,Y,X)+λL cyc (G,F) (3-3)

[0078] In the above formula, L(G,F,D x ,D y ) is the total loss, L GAN (G,D Y ,X,Y) represents the adversarial loss in the process of x2y, L GAN (F,D X ,X,Y) represents the adversarial loss in the process of y2x, L cyc (G, F) represents the cycle consistency loss, and λ is the scaling factor of the cycle consistency loss.

[0079] Furthermore, the ADAM optimizer is used in the CycleGAN model of this scheme, the learning rate is 0.001, the training epoch is 350, the input image size is 128*128, and the batch size is 1. The training hardware configuration is i5-11400H CPU, 16GB RAM, and NvidiaRTX 3050TiGPU.

[0080] S4. Evaluate the image quality and visual effects of the processed images of the CycleGAN cycle consistency network model.

[0081] Generally, image quality evaluation indicators such as PSNR, SSIM, UIQM and UCIQE can be used to evaluate the restored image to verify the model performance and effect. In view of the selection of the above data sets, PSNR and SSIM full reference evaluation indicators are used for image evaluation for images with reference images such as UIEB data set. For separate test data sets without reference images, UIQM and UCIQE no reference evaluation indicators are used for image evaluation.

[0082] Based on the above description of the turbid underwater image restoration method, this solution was specifically implemented and verified. First, the 1200TVL5G wireless image transmission camera module was used to complete image acquisition and transmission. The module includes a 1200TVL camera, a transmitter and a receiver. The camera and the transmitter are connected by wire, and the receiver is connected to the PC using a USB. In order to make the 1200YVL image transmission camera waterproof, in actual applications, the camera is sealed in a housing.

[0083] The 1200TVL wireless image transmission camera is connected to the transmitting image transmission for sending data. After it is completely sealed, it is placed underwater. The camera faces the underwater scene where information needs to be collected, and the PC is connected to the receiver via USB. When the camera is powered on, the PC can receive the data sent by the transmitting image transmission. The data is transmitted in the form of a video stream. After receiving the data for a fixed time, the PC saves the video stream in the form of a video file, and then uses the method of extracting video frames to obtain image data. A total of 1,120 underwater image data sets were collected, with a training and verification data ratio of 0.9:0.1.

[0084] In practical applications, the ADAM optimizer is used, the learning rate is 0.001, the training epoch is 350, the input image size is 128*128, and the batch size is 1. The training hardware configuration is i5-11400H CPU, 16GB RAM, and Nvidia RTX 3050Ti GPU.

[0085] The model was tested using a dataset collected in the field. In order to more clearly show the performance of the proposed method under different lighting conditions, the test data was divided into different water depths, namely 30cm, 40cm, and 60cm. The specific restoration result images are shown below: Fig.11 As shown in the figure, it can be seen that the performance of the proposed method is roughly the same at different water depths in the actual lake bottom. The main restoration effect is to make the main texture and edge contour of the target object clearer, and the target becomes more prominent in visual effect, which is easier to be recognized and detected by the equipment.

[0086] The main purpose of this restoration method is to enable the equipment to better complete the detection task. In order to evaluate the impact of the restoration task on the detection task results, the YOLOv5n model is used to perform the goldfish recognition task after the restoration task is completed in the actual underwater environment. The confidence and Map results are improved on the restored image to determine whether the restoration task meets the detection requirements. The results of executing the restoration and detection tasks are shown in Figure 2. Fig.12 As shown, the content in the bounding box with the highest confidence in each image is enlarged and displayed below the image.

[0087] From the above restoration results, it can be seen that this method significantly improves the image quality when performing actual image restoration tasks, and the contrast between the background and foreground in the picture is effectively improved. When performing detection and recognition tasks, the detection results on the restored image are more accurate than those on the original image, and the confidence of the same target on the restored image is significantly improved. This shows that this method has a better ability to interpret underwater scenes.

[0088] In order to more objectively verify the applicability of the restoration method in this paper to underwater degraded images in practical applications, the restored images are evaluated and compared using the non-reference image quality evaluation indicators UIQM and UCIQE.

[0089] The overall restoration result evaluation indicators in practical applications are shown in Table 4-1.

[0090] Table 4-1 Comparison of evaluation indicators of restoration results of this method

[0091]

[0092] As shown in Table 4-1, the proposed method has achieved different degrees of improvement in UIQM and UCIQE, which shows that the proposed method still has good information interpretation ability for underwater images in actual scenarios.

[0093] Since the quality of collected images in different water depth scenes will be affected by different factors such as lighting and floating particles, this article considers the impact of different water depths for image quality evaluation in practical applications, including 30cm, 40cm and 60cm. The evaluation results are shown in Table 4-2.

[0094] Table 4-2 Comparison of evaluation indicators of restoration results at different water depths

[0095]

[0096]

[0097] It can be seen from the above data that in practical applications, the evaluation results of restored images at different water depths are different. The restoration method achieves the best restoration effect in the 30cm water depth scene. The restoration effect in the 60cm water depth scene is relatively poor but still improved. The main reason for this phenomenon is that as the water depth increases, the lighting conditions become worse and the impurities in the water increase, which increases the influence of additional factors on the restoration results. However, the evaluation indicators are still improved, and it can be seen from the visual effect of the image that the restoration effect still exists, and the human eye cannot distinguish a large difference.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A turbid underwater image restoration method based on deep learning, characterized in that: The following steps are involved: S1, collect a large amount of underwater turbidity image data, and form a data set after preprocessing; S2. Establish a CycleGAN cycle consistency network model, wherein the CycleGAN cycle consistency network model includes two generators and two discriminators, and the generator includes an encoder, a residual device and a decoder; The input image data is passed through the encoder to output a feature map, and then feature extraction is performed in the residual device. Finally, the decoder outputs a feature map with the same dimension as the input image through deconvolution and depooling. This feature map is the result of the generator's forgery. The discriminator determines whether the data given by the generator is real data or forged data; Attention modules are added to the encoder and decoder respectively to optimize the model to enhance the model's ability to extract details. The attention modules include simAM and MLCA. S3, using the data set obtained in step S1 to train the CycleGAN cycle consistency network model, and adjust the model configuration, add loss function and optimizer; S4. Evaluate the image quality and visual effects of the processed images of the CycleGAN cycle consistency network model.

2. The turbid underwater image restoration method based on deep learning according to claim 1, characterized in that: Step S1 preprocessing includes alignment, cropping, and labeling.

3. The turbid underwater image restoration method based on deep learning according to claim 1, characterized in that: Add an attention mechanism Sim AM to the encoder; The decoder includes three convolution modules, and a hybrid attention mechanism Sim AM is added after each convolution module of the encoder.

4. The turbid underwater image restoration method based on deep learning according to claim 3 is characterized in that: In the convolution module of the decoder, InstanceNorm normalization is used after each convolution and activation function. This normalization method calculates the mean and variance of height and weight in the channel direction in each bitch, and then calculates the result of each feature map.

5. The turbid underwater image restoration method based on deep learning according to claim 1, characterized in that: The residual device uses multiple Resnet structures. The input data of a single Resnet structure is output after convolution Conv, InstanceNorm normalization, activation function ReLU, convolution Conv, InstanceNorm normalization, and two convolution kernels are both 3*3.

6. The method for restoring turbid underwater images based on deep learning according to claim 1, characterized in that: The attention mechanism MLCA is added to the decoder. First, the input data is divided into 5*5 blocks. Local average pooling and global average pooling are used for the blocks to output a 5*5*C feature map. Then, two one-dimensional convolutions are used to extract features from the feature map to output a 1*1*C matrix. The sigmoid activation function and adaptive average pooling are used to output the feature map weights. At the same time, MLCA will resize the 5*5*C feature map to 1*1*(C*5*5), and use one-dimensional convolution to obtain a vector of the same dimension. The vector is resized to 5*5*C, and the obtained feature map is matrix-added with the above feature map weights. Finally, the matrix obtained by de-pooling is the weight parameter of the input feature map. The output result of the MLCA attention mechanism can be obtained by using the weight parameter and the input feature map to calculate the Hadamard product.

7. The turbid underwater image restoration method based on deep learning according to claim 1 or 3, characterized in that: The discriminator adopts the discriminator structure of PatchGAN full convolution form, inputs the feature map into the discriminator, and outputs the result matrix through the structure of convolution, InstanceNorm normalization and LeakyReLU activation function. Each matrix represents the discrimination result of the generated image receptive field. Finally, the discrimination result of the discriminator for the complete image is obtained by averaging the matrix.

8. The method for restoring turbid underwater images based on deep learning according to claim 1, characterized in that: The loss function in step S3 includes adversarial loss and cycle consistency loss. The adversarial loss is used to measure the label value obtained by inputting the image generated by the generator into the corresponding discriminator. The cycle consistency loss is used to measure whether the image in the sample space X is mapped to the sample space Y by the generator, and then generated to the sample space X by another generator. The result is close to the original image.

9. The method for restoring turbid underwater images based on deep learning according to claim 1, characterized in that: The image quality evaluation indicators used in step S4 to evaluate the image quality and visual effect include PSNR, SSIM, UIQM and UCIQE image quality evaluation indicators.

Images