A network method for AUV underwater image enhancement based on wavelet transform

By combining Haar wavelet decomposition and deep convolutional networks, the problems of color cast and noise in underwater images are solved, thus improving the underwater detection performance of AUVs.

CN115018807BActive Publication Date: 2024-11-22HARBIN ENG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210724011.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-23
Publication Date
2024-11-22
Estimated Expiration
2042-06-23

AI Technical Summary

Technical Problem

Underwater images at different depths suffer from color shifts and noise due to variations in color and lighting, which affects the underwater exploration missions of AUVs.

Method used

We employ Haar wavelet decomposition combined with a deep convolutional network, and enhance underwater images by using wavelet feature fusion, WAF-Haar module, and attention mechanism module to remove noise and color cast.

Benefits of technology

It effectively removes noise and color cast from underwater images, improves image quality, and enhances the underwater detection capabilities of AUVs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115018807B_ABST
    Figure CN115018807B_ABST
Patent Text Reader

Abstract

The present invention provides an AUV underwater image enhancement network method based on wavelet transform, comprising: step 1: the AUV collects data through the underwater visual equipment carried by the AUV to obtain an image F t , and then use Haar wavelet to decompose the image to obtain the frequency component F of the original image after Haar wavelet decomposition x ; Step 2: The two pieces of information F obtained in step 1 t 、F x The image is passed to the designed wavelet feature fusion module to integrate various newly added enhanced information into the basic image; Step 3: Design a WAF-Haar module based on the WAF unit, which contains four WAF units and four residual operations; the image after the Haar wavelet is divided into four times and F t Basic image fusion; Step 4: pass the features obtained in step 3 to the attention mechanism module, and then cut the image into four images; Step 5: pass the information of step 3 to the weight generator, and the generated weight is used to recombine the four images in step 4 to enhance the underwater image. The present invention realizes deblurring, denoising and eliminating color cast of underwater images.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to an AUV underwater image enhancement network method based on wavelet transform (Wavelet transformbased underwater image enhancement network, WIE-Net), belonging to underwater image processing. Background Art

[0002] As human beings' demand for resources continues to increase, the importance of marine resource development and utilization is increasing day by day. In order to speed up the exploration of unknown resources in the ocean, scholars have conducted a lot of research on remotely operated underwater vehicles (ROV) and autonomous underwater vehicles (AUV), and equipped underwater vehicles with imaging systems to improve the perception of the marine environment. However, when AUV or ROV conducts underwater exploration missions, the color and lighting of the waters change with depth, and the images obtained by the visual system contain color deviation and introduce a lot of noise. Summary of the invention

[0003] The purpose of the present invention is to achieve deblurring, de-noising and color cast elimination of underwater images. In the process of algorithm implementation, in order to effectively remove noise, Haar wavelet decomposition is introduced in the program, which is conducive to correcting the spatial domain image, can effectively improve the image enhancement effect, eliminate possible noise, and ensure the underwater detection of AUV and other tasks.

[0004] The object of the present invention is achieved in that the steps are as follows:

[0005] Step 1: The AUV collects data through the underwater visual equipment it carries and obtains image F t , and then use Haar wavelet to decompose the image to obtain the frequency component F of the original image after Haar wavelet decomposition x ;

[0006] Step 2: The two pieces of information F obtained in step 1 t 、F x The information is passed to the designed wavelet feature fusion module to integrate various newly added enhanced information into the basic image;

[0007] Step 3: Design the WAF-Haar module based on the WAF unit, which contains four WAF units and four residual operations; the image after the Haar wavelet is divided into four times and F t Basic image fusion;

[0008] Step 4: Pass the features obtained in step 3 to the attention mechanism module, and then cut the image into four images;

[0009] Step 5: The information from step 3 is passed into the weight generator. The generated weights are used to recombine the four images in step 4 to enhance the underwater image.

[0010] The present invention also includes such structural features:

[0011] 1. The wavelet feature fusion module in step 2 is as follows:

[0012] G x =Maxpool(Relu(φ3φ1(F x ))))

[0013] G t =σ(MLP(F t ))

[0014] G r0 =G x +G t

[0015] G r1 =Maxpool(Relu(φ5(φ1(F t ))))

[0016]

[0017] Where: φ1(·), φ3(·), φ5(·) are convolution operations with kernels of 1×1, 3×3, and 5×5 respectively; G x is the output of the base image after convolution; Relu(·) is the ReLU activation function; G t is the output of the base image after MLP processing; Maxpool(·) is the maximum value pooling; F t is the base image; σ(·) is the Sigmoid activation function; G is the operation of multiplying the corresponding values ​​of pixels; r0 is the base image G x and G t The output of addition; G r1 is the output after the enhanced information convolution; G r2 It is the result of further fusion of the basic image and enhanced information.

[0018] 2. The WAF-Haar module in step 3 is as follows:

[0019] F residual =WAF(F t )+φ3(F t )

[0020] F Cat =Cat(F residual1 +F residual2 )

[0021] Where: WAF(·) is wavelet feature fusion; F residual1 ,F residual2 It is the residual output of two WAF modules; Cat is the channel merging of the image.

[0022] 3. The attention mechanism module in step 4 is as follows:

[0023]

[0024] M s (F) = σ(f 7×7 (Cat(AvgPool(F),MaxPool(F))))

[0025] Where: F is the output of the WAF-Haar module; AvgPool(·) is the average pooling; f 7×7 It is a convolution with a kernel of 7; is the output of spatial attention.

[0026] Compared with the prior art, the invention has the following beneficial effects: the invention proposes a new mode of combining deep convolution with traditional image processing; using Harr wavelet, information with close image space correspondence is extracted from the original image, the key information of the underwater image is retained, and the noise of the image is removed. In the case of an underwater environment with a blue-green background and high-frequency noise, the AUV can obtain a large number of underwater images useful for marine research. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is the network structure diagram of the wavelet feature fusion (WAF) module;

[0028] Figure 2 This is the network structure diagram of the WAF-Haar module;

[0029] Figure 3 It is the network structure diagram of WIE-Net network;

[0030] Figure 4a and Figure 4b It is the edge comparison map before and after image enhancement;

[0031] Figure 5a-5c It is compared with other algorithms on three data sets. Figure 5a is the UFO-120 dataset, Figure 5b is the EUVP dataset, Figure 5cIt is the UIEB dataset;

[0032] Figure 6 It verifies the validity of each module of the WIE-Net network;

[0033] Figure 7 It is the effectiveness of the designed WIE-Net network algorithm including several WAF modules;

[0034] Figure 8 The effectiveness of each part of the designed WAF module;

[0035] Fig. 9 It is a subjective evaluation of the WIE-Net network and other algorithms. DETAILED DESCRIPTION

[0036] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0037] The present invention aims at underwater image blur and color cast phenomena, and the specific implementation of the underwater image processing network is described as follows:

[0038] Step 1: The AUV collects data through the underwater visual equipment it carries and obtains image F t , and then use Haar wavelet to decompose the image to obtain the frequency component F of the original image after Haar wavelet decomposition x ;

[0039] Step 2. Information F from step 1 t is the base image, F x It is the external additional enhancement information. These two types of information are passed into the designed wavelet feature fusion module (WAF), which can integrate a variety of additional enhancement information into the basic image;

[0040] The mathematical formula of the wavelet feature module is characterized by integrating a variety of newly added enhanced information into the basic image.

[0041] Wavelet feature fusion module:

[0042] G x =Maxpool(Relu(φ3(φ1(F x )))) (1)

[0043] G t =σ(MLP(F t )) (2)

[0044] G r0 =G x +G t (3)

[0045] G r1=Maxpool(Relu(φ5(φ1(F t )))) (4)

[0046]

[0047] Where: φ1(·), φ3(·), φ5(·) are convolution operations with kernels of 1×1, 3×3, and 5×5 respectively; G x is the output of the base image after convolution; Relu(·) is the ReLU activation function. MLP is a shared perception network with only one hidden layer; G t is the output of the base image after MLP processing; Maxpool(·) is the maximum value pooling; F t is the base image; σ(·) is the Sigmoid activation function; It is the operation of multiplying the corresponding values ​​of pixels. r0 is the base image G x and G t The output of addition; G r1 is the output after the enhanced information convolution; G r2 It is the result of further fusion of the basic image and enhanced information.

[0048] Step 3: Based on the WAF unit, a WAF-Haar module is designed, which includes four WAF units and four residual operations. The image after the Haar wavelet is divided into four times and F t Basic image fusion;

[0049] The mathematical formula of the WAF-Haar module fusion involved. Its characteristic is that to fuse information groups containing different numbers of sub-graphs, only the hyper-parameters of the modules need to be appropriately changed.

[0050] WAF-Haar module:

[0051] F residual =WAF(F t )+φ3(F t ) (6)

[0052] F Cat =Cat(F residual1 +F residual2 ) (7)

[0053] Where: WAF(·) is wavelet feature fusion; F residual1 ,F residual2 It is the residual output of two WAF modules; Cat is the channel merging of the image.

[0054] Step 4: In order to focus on the impact of water on the local area of ​​the image and the important channel information in the features, the features obtained in step 3 are passed to the attention mechanism module, and then the image is cut into four images;

[0055] The attention mechanism involved pays more attention to local areas and important channel information in the water body.

[0056] Attention mechanism module:

[0057]

[0058] M s (F) = σ(f 7×7 )Cat(AvgPool(F),MaxPool(F)))) (9)

[0059] Where: F is the output of the WAF-Haar module; AvgPool(·) is the average pooling; f 7×7 It is a convolution with a kernel of 7; is the output of spatial attention.

[0060] Step 5: Pass the information from step 3 into the weight generator, and the generated weights are used to recombine the four images in step 4 to enhance the underwater image.

[0061] The image reconstruction module is characterized by helping to remove color casts and artifacts introduced by the network.

[0062] Image reconstruction module:

[0063] I en =R u3 ×C w3 +R u5 ×C w5 +R u7 ×C w7 +R ul ×C wl (10)

[0064] Where: I en To enhance the image; R w3 ,R w5 ,R w7 ,R wl is the output of the weight generator; C w3 ,C w5 ,C w7 ,C wl is the output of the attention mechanism.

Claims

1. A network method for AUV underwater image enhancement based on wavelet transform, It is characterized in that Here are the steps: Step 1: The AUV collects data through the underwater visual equipment it carries and obtains image F t , and then use Haar wavelet to decompose the image to obtain the frequency component F of the original image after Haar wavelet decomposition x ; Step 2: The two pieces of information F obtained in step 1 t 、F x The information is passed to the designed wavelet feature fusion module to integrate various newly added enhanced information into the basic image; Step 3: Design a WAF-Haar module based on the WAF unit, which contains four WAF units and four residual operations; the image after the Haar wavelet is divided into four times and F t Basic image fusion; Step 4: Pass the features obtained in step 3 to the attention mechanism module, and then cut the image into four images; Step 5: The information from step 3 is passed into the weight generator. The generated weights are used to recombine the four images in step 4 to enhance the underwater image.

2. According to claim 1, a wavelet transform-based AUV underwater image enhancement network method, It is characterized in that The wavelet feature fusion module in step 2 is as follows: MR x =Maxpool(Relu(φ 3 (φ 1 (F x )))) G t =σ(MLP(F t )) G r0 =G x +G t MR r1 =Maxpool(Relu(φ 5 (φ 1 (F t )))) Where: φ 1 (·),φ 3 (·),φ 5 (·) is the convolution operation with kernels of 1×1, 3×3, and 5×5 respectively; G x is the output of the base image after convolution; Relu(·) is the ReLU activation function; G t is the output of the base image after MLP processing; Maxpool(·) is the maximum value pooling; F t is the base image; σ(·) is the Sigmoid activation function; G is the operation of multiplying the corresponding values ​​of pixels; r0 is the base image G x and G t The output of addition; G r1 is the output after the enhanced information convolution; G r2 It is the result of further fusion of the basic image and enhanced information.

3. The AUV underwater image enhancement network method based on wavelet transform according to claim 1, It is characterized in that The WAF-Haar module in step 3 is as follows: F residual =WAF(F t )+φ 3 (F t ) F Cat =Cat(F residual1 +F residual2 ) Where: WAF(·) is wavelet feature fusion; F residual1 , F residual2 It is the residual output of two WAF modules; Cat is the channel merging of the image.

4. The AUV underwater image enhancement network method based on wavelet transform according to claim 1, It is characterized in that The attention mechanism module in step 4 is as follows: M s (F)=σ(f 7×7 (Cat(AvgPool(F)),MaxPool(F)))) Where: F is the output of the WAF-Haar module; AvgPool(·) is the average pooling; f 7×7 It is a convolution with a kernel of 7; is the output of spatial attention.