Variable attention low-illumination underwater image enhancement method based on Retinex

Abstract

The invention discloses a variable attention low illumination underwater image enhancement method based on Retinex. The problem of complex degradation caused by insufficient illumination and underwater scattering in a low-illumination underwater image enhancement task is considered. According to the method, an illumination-guided variable attention module is designed, semantic information provided by a brightness feature map is utilized to help to enhance a dark region, and the adaptive enhancement capability of the model is improved; constructing a space-frequency domain feature fusion module, and improving the significance of image texture information through multi-scale feature extraction and feature fusion among different levels; a medium transmission module is introduced, and image artifacts caused by underwater scattering in the enhancement process are further solved. Compared with other frontier methods, the method has the advantages that the image degradation can be effectively recovered while the brightness is improved, and the enhanced image has a better visual perception effect.

Description

Technical Field

[0001] The invention relates to a variable attention low-illumination underwater image enhancement method based on Retinex, and belongs to the technical fields of computer vision, underwater image enhancement, low-illumination enhancement, etc. Background Art

[0002] Underwater images directly reflect underwater scenes, and their acquisition is closely related to imaging equipment. Especially under low-light conditions, some lighting equipment is often needed to provide artificial light sources to assist in acquiring underwater images. However, due to the selective absorption of light by water and the scattering of light by suspended particles, underwater images usually have problems with color shift and blurred details. Under low-light conditions such as at night or in the deep sea, the images taken are generally darker and the recognition of objects is poor. These problems bring difficulties to underwater target recognition, detection and tracking. Therefore, the study of low-light underwater image enhancement can significantly improve the availability and information acquisition capabilities of underwater images, which is of great significance to promoting the development of underwater resource exploration, underwater archaeology, marine science research and other fields.

[0003] Although great progress has been made in underwater image enhancement methods, there are few studies on underwater image enhancement in low-light scenes, which are mainly divided into traditional methods and deep learning-based methods. Traditional methods focus on the process and mechanism of image degradation and use methods such as prior knowledge and physical models to enhance images. Although the visual quality has been improved to a certain extent, these manually designed methods require careful adjustment of parameters when encountering diverse and challenging underwater scenes, which often leads to over- or under-enhancement results and poor generalization ability. With the rapid development of deep learning, some people have tried to combine Retinex theory with deep learning, but these methods usually have cumbersome multi-stage training processes, with multiple CNNs used to decompose the original image and enhance each image component respectively. Moreover, due to ignoring the loss of underwater images hidden in low-light conditions, the enhancement results may lead to changes in texture structure and color distortion.

[0004] Therefore, in order to solve the complexity of the training process and fully consider the complex degradation caused by insufficient illumination and underwater scattering, the present invention proposes a variable attention low-illumination underwater image enhancement method based on Retinex. Summary of the invention

[0005] In view of this, the technical problem to be solved by the present invention is to provide a variable attention low-illuminance underwater image enhancement method based on Retinex, which can effectively restore the degradation caused by underwater scattering while improving the brightness, thereby improving the visual perception effect.

[0006] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0007] A variable attention low-light underwater image enhancement method based on Retinex, including:

[0008] (1) Obtain an initial dataset of underwater images in low-light scenes and perform preprocessing and division on the images in the dataset;

[0009] (2) Input the preprocessed image into the illumination estimator part to obtain the brightness map and brightness features of the low-illumination underwater image, and use the brightness map to obtain the brightened image;

[0010] (3) The brightness features are input into the light-guided variable attention module (LDAT) to obtain the semantic context information of the area with better lighting conditions to help enhance the dark area;

[0011] (4) The brightened image is input into the degradation repairer part, which adopts the classic U-Net structure. After downsampling, it passes through the spatial-frequency domain feature fusion module (SDFF) to enhance the saliency of texture information, and constructs the medium transmission module (MTM) at the jump connection to fuse multi-scale information.

[0012] Furthermore, in step (1), the public low-light underwater image dataset LUIE is re-divided into 2510 pairs for training and 14 pairs for model testing, and real low-light underwater images with enhanced challenges are provided for reference-free quantitative comparison experiments.

[0013] Furthermore, in step (2), the illumination estimator first concatenates the input and the prior map, then performs feature fusion through Conv1×1, uses a deep separable Conv5×5 to model the interaction between regions with different illumination conditions to generate brightness features, then uses Conv1×1 to aggregate features to generate a brightness map, and finally the brightness map is directly multiplied with the input to obtain a brightened image.

[0014] Furthermore, in step (3), the variable attention module is composed of a light-guided multi-head attention mechanism and a feedforward network, which uses the brightness information encoded by the brightness feature and the interaction of areas with different lighting conditions to guide the calculation of self-attention, so that the model focuses on relevant areas and captures more information features. The CB operation is inserted at the end of the feedforward network to provide the dense interaction required by the visual Transformer and enhance the generalization ability of the module.

[0015] Furthermore, in step (4), the degradation repairer adopts a classic U-Net structure, and the brightness feature is fed to each LDAT module. The input is a brightened image. The input feature is first mapped to a high dimension through Conv3×3, and shallow features are extracted through SDFF and a layer of LDAT. Then, downsampling is performed using Conv4×4, and hierarchical features are extracted through SDFF and two layers of LDAT. In the Bottleneck part, deep information is obtained through two layers of LDAT. The upsampling stage is designed as a symmetrical structure, and Deconv2×2 is used to expand the features. Finally, Conv3×3 is used to reduce the dimension and add it to the input to obtain the enhanced image. The MTM module is used at the jump connection to fuse the features while focusing on areas where underwater scattering is more serious. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0017] Figure 1 This is the overall model framework diagram of this method;

[0018] Figure 2 The structure diagram of the illumination-guided variable attention module proposed in this method;

[0019] Figure 3 This is the structure diagram of the spatial-frequency domain feature fusion module proposed in this method;

[0020] Figure 4 This is the structural diagram of the medium transmission module proposed in this method. DETAILED DESCRIPTION

[0021] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0022] The present invention provides a variable attention low-light underwater image enhancement method based on Retinex. The overall model framework proposed by the method is as follows: Figure 1 As shown, the method comprises the following steps:

[0023] (1) Divide the low-light underwater image enhancement dataset;

[0024] (2) Build an illumination estimator;

[0025] (3) Constructing a variable attention module guided by illumination;

[0026] (4) Constructing a spatial-frequency domain feature fusion module;

[0027] (5) Constructing the media transmission module;

[0028] (6) Construct a low-light underwater image enhancement network;

[0029] (7) Training low-light underwater image enhancement network;

[0030] (8) Test the low-light underwater image enhancement network.

[0031] Furthermore, in the step (1), the low-light underwater image dataset LUIE has a total of 2524 pairs of 600×400 low-light underwater images and reference images, and 85 real low-light underwater images with enhanced challenges, covering rich content information and various degrees of light degradation, which can more comprehensively reflect complex low-light underwater scenes, of which 2510 pairs are used for training and 14 pairs are used for model testing.

[0032] Furthermore, in step (2), the input image I is outputted by the illumination estimator to obtain a brightness map The brightness map Multiply the input image by point to get the brightened image I lu The theoretical basis of this invention comes from the Retinex theory. In order to simulate the real underwater exposure scene, the present invention adds disturbance terms to the reflection component R and the incident component L respectively. The process is shown in formula (1)-formula (2).

[0033]

[0034] Where R and L represent disturbance terms.

[0035]

[0036] In the formula, the second term on the right side of the equation represents the underexposure, overexposure and color distortion caused by the brightening process, the third term represents the noise, artifacts and scattering hidden in the underwater low-light environment, and C represents the overall degradation term of the low-light underwater image.

[0037] Furthermore, in step (3), Figure 2 As shown in Figure 1, the module first downsamples the input feature map to generate a uniform grid as the reference point, uses the fully connected layer (fc) to linearly project the input feature x to obtain the query, inputs the query into the offset network to learn multiple sets of offsets, and then adds the offset to the position of the reference point to determine the sampling position of the deformation point, and uses bilinear interpolation to sample features from the important area to obtain Then, the transformed key and value are obtained through linear projection, as shown in formula (3).

[0038]

[0039] Where W q ,W k ,W vrepresents the projection matrix, θ offset represents the offset network, Δp represents the offset, whose size is the same as the reference point, φ represents the bilinear interpolation, and p represents the uniform grid, whose position is based on the size H G ×W G Scaled to [-1, +1], [-1, -1] represents the lower left corner, and [+1, +1] represents the upper right corner.

[0040] like Figure 1 As shown, the brightness feature F lu Input into each LD-MSA of LDAT. In order to realize the multi-head attention guided by illumination, LD-MSA regards a single channel of the feature map as a token, and then calculates the similarity between the query and the deformed key, and weights it with the value of the aggregated brightness feature to obtain Attention. The self-attention of the mth head is shown in formula (4).

[0041]

[0042] In the formula, α m represents a one-dimensional learnable parameter, R represents the relative position encoding, l m Represents the brightness feature. Finally, the self-attention of each head is concatenated and projected to obtain the final Attention, which is then reshaped to obtain the output feature.

[0043] Furthermore, in step (4), if Figure 3 As shown in Figure 1, the SDFF module is divided into two parts: frequency domain processing and spatial domain processing. The frequency domain part is converted to the frequency domain space using Fourier transform, and the weight of each frequency component is determined through the convolution layer to correctly capture the lines and edges of the underwater image. Finally, the inverse Fourier transform is performed to obtain information. The process is shown in formula (5).

[0044]

[0045] In the formula, f FFT represents the fast Fourier transform, f IFFT represents the inverse fast Fourier transform, f1 represents the convolution plus BN operation, σ represents the Relu function, F f is the output of the frequency domain part.

[0046] The spatial domain uses a self-calibration structure to obtain rich contextual information in different scale spaces. By performing convolution in the small latent space after downsampling, features with a larger receptive field are obtained to guide the feature transformation in the original scale space. The process is shown in formula (6).

[0047]

[0048] In the formula, AvgPool represents the average pooling operation, f2, f3, f4 represent the convolution plus BN operation, Up represents the bilinear interpolation, σ represents the Sigmoid function, and F s is the output of the spatial domain. Finally, the feature maps of the frequency domain and the spatial domain are concatenated in the channel dimension, and the output is obtained by feature fusion through point convolution.

[0049] Furthermore, in step (5), Figure 4 As shown in FIG. 7 , the medium transmission module is designed at the jump connection of the codec structure, and the features extracted by each layer of the encoder are weighted by inputting the medium transmission map. Specifically, the medium transmission map first generates spatial attention weights through the convolution layer, and then obtains the pixel-level attention map through the Sigmoid function. The attention map is multiplied by the input feature map to indicate that the area with higher quality degradation should be assigned a larger weight value. The present invention adopts an algorithm based on priors to obtain the medium transmission map, and the process is shown in formula (7).

[0050]

[0051] In the formula, I c (y) represents any color channel of the underwater image, Ω(x) represents the window centered at pixel x, and A c Represents the global background light.

[0052] Furthermore, in step (6), the degradation repairer is first constructed using the modules constructed in steps (3), (4) and (5), and the degradation term C in formula (2) is restored by adopting the U-Net structure and the strategy of multi-module fusion. Overall, the degradation repairer is then fused with the illumination estimator constructed in step (2) to obtain Figure 1 The low-light underwater image enhancement network shown.

[0053] Furthermore, in step (7), the experiment was completed in the Python 3.8 environment, using a Tesla GPU_V100 with a video memory of 32 GB to accelerate the training, and the Pytorch 1.13 deep learning framework was used to train the network model. During training, image blocks of size 128×128 were randomly cropped from the low-light underwater image and its reference image as training samples. The batch size was 8, the initial learning rate was 2e-4, and the cosine annealing strategy was used to steadily decrease to 1e-6. Adam was used as the optimizer, and a total of 474 rounds of training were performed, with a total training time of 11 hours.

[0054] Furthermore, in step (8), the trained weight file is loaded into the low-light underwater image enhancement network, a picture is selected from the test data set for enhancement, and the enhanced underwater image is returned.

[0055] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail through the above embodiments, those skilled in the art should understand that various changes can be made in form and details without departing from the scope defined by the claims of the present invention.

Claims

1. A variable attention low-light underwater image enhancement method based on Retinex, which is characterized by comprising the following steps: (1) Collect and organize the existing publicly available low-light underwater image datasets online and preprocess them: re-divide the LUIE dataset of 2524 pairs of low-light underwater images with a size of 600×400, 2510 pairs are used for model training, and the remaining 14 pairs are used for testing. These images are rich in content and have different degrees of degradation. They can comprehensively show the characteristics of underwater scenes under low-light conditions, so that the model has better generalization ability when processing low-light underwater image enhancement. (2) Constructing the illumination estimator: This part first concatenates the input image with the prior image, implements feature fusion through a 1×1 convolution layer, and then uses a depth-separable 5×5 convolution to model the interaction between different illumination areas to generate brightness features. These features are then aggregated through a 1×1 convolution layer to generate a brightness map, and finally the brightness map is multiplied with the input image to generate an enhanced brightness image. This part uses the Retinex theory to add perturbations to the reflection component and the incident component to simulate a real underwater exposure scene. The purpose is to remove the influence of the incident component and retain the inherent reflection properties of the object itself. (3) Constructing an illumination-guided variable attention module: This module consists of an illumination-guided multi-head attention mechanism and a feedforward network. It uses the brightness information in the brightness feature encoding and the interaction of areas with different lighting conditions to guide the calculation of self-attention, so that the model can focus on relevant areas and capture more features. A CB operation is added at the end of the feedforward network to achieve the dense interaction required by the visual Transformer, thereby enhancing the generalization ability of the module. (4) Constructing a spatial-frequency domain feature fusion module: This module is divided into two parts: frequency domain processing and spatial domain processing. The frequency domain part uses Fourier transform and convolution operations to enhance the saliency of texture information. The spatial domain part first downsamples the features, performs convolution in the downsampled space, and then generates weights through the Sigmoid function to guide the feature extraction process in the original scale space. Finally, the frequency domain information and the spatial domain information are fused and output. (5) Constructing a medium transmission module: This module is based on the underwater dark channel prior theory. It is embedded in the jump connection of the codec structure and aims to improve the overall image quality by accurately adjusting the feature weights of different regions. Specifically, this module dynamically weights the features extracted by each layer of the encoder by inputting the medium transmission map, thereby generating clearer and more information-fidelity images in the decoding stage. (6) Construct a low-illumination underwater image enhancement network: Using the illumination-guided variable attention module, spatial-frequency domain feature fusion module, and medium transmission module constructed in the above steps, a degradation repairer is built with the U-Net structure as the core to restore the degradation items in the brightened image. Finally, the illumination estimator and degradation repairer are used to construct a Retinex-based variable attention low-illumination underwater image enhancement framework. (7) Training the low-illumination underwater image enhancement network: Using pairs of original underwater images and reference underwater images as network input, the Retinex-based variable attention low-illumination underwater image enhancement network consisting of an illumination estimator and a degradation repairer is fully trained to obtain the trained network model weights. (8) Test the low-light underwater image enhancement network: Load the trained weight file into the low-light underwater image enhancement network, use the paired test set images as reference image enhancement tests, and return the enhanced underwater image and corresponding peak signal-to-noise ratio, structural similarity and other indicators.

2. The variable attention low-light underwater image enhancement method based on Retinex according to claim 1, characterized in that: The illumination-guided variable attention module constructed in step 3 processes input features as follows: first, downsample the input feature map to generate a uniform grid as a reference point, use the fully connected layer (fc) to linearly project the input feature to obtain the query, input the query into the offset network to learn multiple sets of offsets, so that the model can flexibly adjust the position relationship in the feature map, then add the offset to the position of the reference point to determine the sampling position of the deformation point, and then use bilinear interpolation to obtain the sampling feature, and then obtain the deformed key and value through linear projection, and then calculate the similarity between the query and the deformed key, and weight it with the value of the aggregated brightness feature to obtain Attention. This module enables the model to focus on important areas of the image more accurately, effectively improving the image enhancement effect in complex low-light underwater environments.

3. The variable attention low-light underwater image enhancement method based on Retinex according to claim 1, characterized in that: The spatial-frequency domain feature fusion module constructed in step 4 consists of two parts: the frequency domain and the spatial domain. In the frequency domain processing part, the Fourier transform is used to convert the image to the frequency domain space, and each frequency component is weighted through the convolution layer to effectively capture the important detail features such as lines and edges in the underwater image. Then, the inverse Fourier transform is performed to restore the processed frequency domain information back to the spatial domain to retain the key information in the image. The spatial domain processing part adopts a self-calibration structure to obtain contextual information at different scales, downsample the image, and then perform convolution operations in a small latent space to obtain features with a larger receptive field, thereby more accurately guiding the feature transformation in the original scale space. Finally, the feature maps obtained in the frequency domain and the spatial domain are spliced ​​in the channel dimension, and the spliced ​​feature maps are fused through point convolution to generate output features.

4. The variable attention low-light underwater image enhancement method based on Retinex according to claim 1, characterized in that: The medium transmission module constructed in step 5 is designed at the jump connection of the codec structure. It uses the input medium transmission map to weight the features extracted by each layer of the encoder. The specific process is as follows: First, the medium transmission map is passed through the convolution layer to generate spatial attention weights, and then the weights are converted into pixel-level attention maps through the Sigmoid function. The attention map is multiplied with the input feature map, indicating that areas with more severe quality degradation in the image should be assigned larger weights to highlight the importance of these areas. In order to ensure that the medium transmission map matches the size of the input feature, the module uses the maximum pooling operation for adjustment. In addition, considering the complexity of the underwater environment and the uncertainty of noise, the module uses a priori-based algorithm to generate the medium transmission map to more accurately reflect the transmission characteristics of underwater light.

Images