Underwater image enhancement method and device based on multiple attention

Abstract

The application relates to the technical field of image enhancement, and particularly provides an underwater image enhancement method and device based on multiple attentions, which comprises the following steps: obtaining an underwater image to be enhanced; performing feature extraction and enhancement processing on the underwater image to obtain an enhanced underwater image; performing layer-by-layer down-sampling on the underwater image to extract multi-scale features; performing layer-by-layer up-sampling on the multi-scale features; performing rectangular window division on a feature map in at least one up-sampling level; performing window attention calculation of a fusion convolution operation on each rectangular window; and fusing the up-sampled features and the down-sampled features of the corresponding level; determining the parameters of the feature extraction and enhancement processing through training; the training comprises constructing a loss function based on a weighted combination of a Charbonnier loss, a gradient loss and a multi-scale structural similarity loss, and adjusting the parameters by taking the loss function as an optimization target; and the application increases the reconstruction of underwater image texture details of the model, thereby improving the quality of the image.

Description

Technical Field

[0001] This application relates to the technical field of image enhancement, and in particular to an underwater image enhancement method and apparatus based on multiple attention. Background Technology

[0002] With the deepening of ocean exploration and underwater activities, the acquisition of underwater imagery has become increasingly important in fields such as marine science, archaeology, marine engineering, and underwater robot navigation. However, due to the complexity of the underwater environment, underwater images are often affected by factors such as light scattering and absorption, resulting in low image quality. These problems mainly manifest as color shift, reduced contrast, blurring, and increased noise, thus significantly reducing the usability and visual appeal of underwater images. Therefore, underwater image enhancement technology has emerged, aiming to improve the visual quality of underwater images through algorithms and technical means, making them more in line with the requirements of human visual perception and computer vision.

[0003] Underwater image enhancement (UIE) has a wide range of applications and is challenging, and research on underwater image enhancement methods is constantly emerging.

[0004] Traditional UIE methods, such as histogram-based and channel-based enhancement, perform the inverse process of image degradation through some priors, the parameters of which need to be estimated. However, the most significant problem with traditional UIE methods is their inability to reverse unknown physical processes, which limits their generalization. They rely on specific parameter estimations and are difficult to adapt to dynamic underwater environments.

[0005] Convolutional Neural Networks (CNNs) are widely used in image enhancement due to their powerful end-to-end representation capabilities. Compared to traditional methods, CNN-based methods do not rely on prior information, and CNNs can learn the probability distributions of degraded and enhanced images through training, thereby achieving the enhancement goal.

[0006] Generative Adversarial Networks (GANs) are also frequently used in underwater image enhancement. A GAN consists of a generative model and a discriminative model. The generative model captures the distribution of sample data, while the discriminative model determines whether the input is real data or a generated sample. GANs excel at enhancing image texture details.

[0007] In recent years, Transformer-based methods have been widely applied in the field of underwater image enhancement. The self-attention mechanism in Transformer can capture a wider range of correlations in the input image and obtain a larger receptive field, resulting in better enhancement effects and performance compared with CNN and GAN-based methods.

[0008] Underwater images are subject to quality degradation and blurring due to light refraction and absorption, making underwater image enhancement an extremely challenging task.

[0009] Traditional methods utilize prior information and statistical information to reverse the degradation process, thereby achieving enhancement. However, this method relies on a large amount of prior information and is limited to transferable intersections.

[0010] Convolutional neural network (CNN)-based methods are adept at learning features from input to output images, and their non-linear expressive power improves to some extent with the increase in the number of neural network layers. However, CNN-based methods are limited by their receptive field, making it difficult to capture relationships across a large area of ​​the image. Furthermore, the fixed convolutional kernels cannot be adjusted to adapt to the changing underwater environment.

[0011] Generative Adversarial Networks (GANs) offer advantages in generating image details and textures. However, they are also limited by the receptive field, require a large amount of data for training, and may fail to converge during training, resulting in unstable image quality.

[0012] To address the shortcomings of existing technologies, this application provides an underwater image enhancement method and apparatus based on multiple attention. Summary of the Invention

[0013] To achieve the above objectives, this application adopts the following technical solution: One aspect of this application provides an underwater image enhancement method based on multiple attention, comprising the following steps: Acquire underwater images to be enhanced; The underwater image is subjected to feature extraction and enhancement processing to obtain an enhanced underwater image. The underwater image is downsampled layer by layer to extract multi-scale features. The multi-scale features are upsampled layer by layer. The feature map is divided into rectangular windows in at least one upsampling level. Window attention calculation is performed on each rectangular window to perform fusion convolution operation. Channel attention calculation is performed in parallel to adaptively allocate channel weights. The window attention calculation results and channel attention calculation results are fused to enhance the features. The upsampled features are fused with the downsampled features of the corresponding level. By determining the parameters of the feature extraction and enhancement process, a loss function is constructed based on a weighted combination of Charbonnier loss, gradient loss, and multi-scale structural similarity loss, and the parameters are adjusted with the loss function as the optimization objective.

[0014] In one optional implementation, the layer-by-layer downsampling is achieved through convolutional layers and pooling operations, wherein the convolutional layers are used to extract local detail features, and the pooling operations are used to reduce the resolution of the feature map to obtain high-level semantic features; The layer-by-layer upsampling involves enlarging the size of the low-resolution feature map and then enhancing the ability to represent details through convolutional layers and nonlinear activation functions.

[0015] In one optional implementation, the rectangular window partitioning involves dividing the input feature map into multiple non-overlapping or partially overlapping rectangular sub-regions along the height and width directions; the in-window self-attention calculation involves independently calculating the key, query, and value, and generating attention weights within each rectangular sub-region. The convolutional fusion unit specifically includes introducing at least one convolutional layer to perform convolution operations on the original feature map or its linear transformation result during the generation of the key, query and / or value, in order to capture local contextual information.

[0016] In one optional implementation, the axial shift operation includes shifting along the height direction and / or shifting along the width direction; the magnitude of the shift is less than or equal to the corresponding side length of the rectangular window; the fusion of the attention calculation results before and after the shift includes adding or concatenating the feature maps obtained from the two attention calculations element by element and then fusing them through a convolutional layer. The axially shifted rectangular window attention unit can be used multiple times in different levels of the decoder, or multiple units can be used in parallel or serially in the same level with different window sizes, different shifting strategies or different fusion methods.

[0017] In one optional implementation, the channel enhancement unit assigns channel weights by performing global average pooling or global max pooling on the input feature map to obtain a channel descriptor, transforming the channel descriptor through at least one fully connected layer or convolutional layer to generate a weight vector with the same number of channels as the input feature map, and multiplying the weight vector with the input feature map channel by channel.

[0018] In one optional implementation, during the training of the multi-attention enhancement network, the loss function L used is a composite loss function that integrates pixel-level reconstruction loss, gradient loss, and multi-scale structural similarity loss, i.e.:

[0019] Where α, β, and γ are preset weight coefficients used to balance the contributions of different loss terms, L_pix is ​​the pixel-level reconstruction loss, L_grad is the gradient loss, and L_ms-ssim is the multi-scale structural similarity loss.

[0020] In one optional implementation, the pixel-level reconstruction loss is a complementary form of mean squared error loss, mean absolute error loss, or structural similarity index loss. The gradient loss is obtained by calculating the difference between the gradient maps in the horizontal and vertical directions of the enhanced image and the reference clear image. The gradient maps can be obtained by the Sobel operator, the Prewitt operator, or the Scharr operator. The multi-scale structural similarity loss calculates the structural similarity index between the enhanced image and the reference clear image at multiple different scales, and uses the mean or weighted sum of these multi-scale SSIM values ​​as a metric, taking their complementary form as the loss.

[0021] Another aspect of this application provides an underwater image enhancement device based on multiple attention, comprising: Image input module acquires the underwater image to be enhanced; A multi-attention enhancement network module, based on the U-NET architecture, includes an encoder submodule, a decoder submodule, and a skip connection submodule; the decoder submodule includes at least one multi-attention submodule, which includes an axially shifted rectangular window attention unit, a convolutional fusion unit, and a channel enhancement unit; The image output module outputs the enhanced underwater image obtained after processing by the multi-attention enhancement network module.

[0022] Another aspect of this application provides an electronic device, comprising: At least one memory stores computer-executable instructions non-transiently; At least one processor, configured to run the computer-executable instructions, The computer-executable instructions are executed by the processor to implement the aforementioned underwater image enhancement method based on multiple attention. In another aspect, this application provides a computer-readable storage medium storing computer-executable instructions that, when executed by at least one processor, implement the aforementioned underwater image enhancement method based on multiple attention.

[0023] The effect of this application: The underwater image to be enhanced is acquired; feature extraction and enhancement processing is performed on the underwater image to obtain an enhanced underwater image. The feature extraction and enhancement processing includes: downsampling the underwater image layer by layer to extract multi-scale features; upsampling the multi-scale features layer by layer; dividing the feature map into rectangular windows in at least one upsampling level; performing window attention calculation with fusion convolution operation on each rectangular window; performing channel attention calculation in parallel to adaptively allocate channel weights; fusing the window attention calculation result and the channel attention calculation result to enhance the features; and fusing the upsampling features with the downsampling features of the corresponding level. The parameters of the feature extraction and enhancement processing are determined through training. The training includes constructing a loss function based on a weighted combination of Charbonnier loss, gradient loss, and multi-scale structural similarity loss, and adjusting the parameters with the loss function as the optimization objective. This application improves the model's ability to reconstruct the texture details of underwater images, thereby improving image quality. Attached Figure Description

[0024] The accompanying drawings are provided to further illustrate the present application and form part of the specification. They are used together with the embodiments of the present application to explain the application and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of an underwater image enhancement method based on multiple attention provided in Embodiment 1 of this application; Figure 2 This is a framework diagram of an underwater image enhancement system based on multiple attention provided in Embodiment 3 of this application; Figure 3 This is a block diagram of the electronic device provided in Embodiment 4 of this application; Figure 4 This is a block diagram of a computer-readable storage medium provided in Embodiment 4 of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0026] In the following description, the terms "first," "second," etc., are used for descriptive convenience only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0027] In this application, unless otherwise expressly specified and limited, the term "connection" should be interpreted broadly. For example, "connection" can be a fixed mechanical connection, a detachable mechanical connection, or an integral part; or, "connection" can be a direct connection or an indirect connection through an intermediate medium. Furthermore, unless otherwise expressly specified and limited, the term "coupling" should be interpreted broadly. For example, "coupling" can be a direct electrical connection, such as physical contact and electrical conduction between two components; it can also be understood as an electrical connection between different components in a circuit structure through physical lines capable of transmitting electrical signals, such as copper foil or wires on a printed circuit board (PCB), to transmit electrical signals; or, "coupling" can be an indirect electrical connection between two components through an intermediate medium; or, "coupling" can be an electrical connection between two components in a non-contact manner, such as an electrical connection between two components using capacitive coupling to transmit electrical signals.

[0028] In the embodiments of this application, directional terms such as "up," "down," "left," and "right" may be defined relative to the orientation of the components shown in the accompanying drawings. It should be understood that these directional terms can be relative concepts, used for relative description and clarification, and can change accordingly depending on the orientation of the components in the accompanying drawings.

[0029] Example 1: like Figure 1 As shown, this application provides an underwater image enhancement method based on multiple attention, comprising the following steps: S1: Obtain the underwater image to be enhanced; S2: Perform feature extraction and enhancement processing on the underwater image to obtain an enhanced underwater image. Perform downsampling layer by layer on the underwater image to extract multi-scale features. Perform upsampling layer by layer on the multi-scale features. Perform rectangular window division on the feature map in at least one upsampling level. Perform window attention calculation for each rectangular window to perform fusion convolution operation. Perform channel attention calculation in parallel to adaptively allocate channel weights. Fuse the window attention calculation result and the channel attention calculation result to enhance the features. Fuse the upsampled features with the downsampled features of the corresponding level. S3: By determining the parameters of the feature extraction and enhancement processing, a loss function is constructed based on a weighted combination of Charbonnier loss, gradient loss and multi-scale structural similarity loss, and the parameters are adjusted with the loss function as the optimization objective.

[0030] In the above embodiments, a multi-attention network is designed based on U-NET. Utilizing U-NET's hierarchical upsampling and downsampling operations, information from feature maps at different scales is effectively extracted, thereby enhancing the model's ability to process images of various complex underwater environments. To address the computational burden of Transformer, the feature map is divided into a series of rectangular windows, and attention is calculated within each window to effectively extract features in both the horizontal and vertical directions. Further, the feature map is divided into shift-axis rectangular windows to enhance the connection between the windows and expand the receptive field. Convolution operations are integrated into the window attention calculation to extract richer local information and improve the reconstruction effect of structural regions in underwater images. A channel enhancement module is added, and different weight coefficients are assigned to each channel to improve channel modeling capabilities. A novel loss function is designed, adding gradient loss and multi-scale structural similarity loss to the pixel-based reconstruction loss to improve the model's robustness in recovering texture details and enhance the visual quality of the image.

[0031] Example 2: like Figure 1 As shown, based on Embodiment 1, the steps provided in this application embodiment include the encoder module comprising multiple convolutional blocks and downsampling layers, each convolutional block containing at least one convolutional layer and a nonlinear activation function; the downsampling layer is implemented using max pooling or strided convolution; The decoder module employs transposed convolution or bilinear interpolation during upsampling, and after each upsampling, it concatenates or adds and fuses the feature maps from the corresponding level of the encoder, which have been processed by the skip connection module.

[0032] The layer-by-layer downsampling is achieved through convolutional layers and pooling operations. The convolutional layers are used to extract local detail features, and the pooling operations are used to reduce the resolution of the feature map to obtain high-level semantic features. The layer-by-layer upsampling involves enlarging the size of the low-resolution feature map and then enhancing the ability to represent details through convolutional layers and nonlinear activation functions.

[0033] In one optional implementation, the rectangular window partitioning involves dividing the input feature map into multiple non-overlapping or partially overlapping rectangular sub-regions along the height and width directions; the in-window self-attention calculation involves independently calculating the key, query, and value, and generating attention weights within each rectangular sub-region. The convolutional fusion unit specifically includes introducing at least one convolutional layer to perform convolution operations on the original feature map or its linear transformation result during the generation of the key, query and / or value, in order to capture local contextual information.

[0034] In one optional implementation, the axial shift operation includes shifting along the height direction and / or shifting along the width direction; the magnitude of the shift is less than or equal to the corresponding side length of the rectangular window; the fusion of the attention calculation results before and after the shift includes adding or concatenating the feature maps obtained from the two attention calculations element by element and then fusing them through a convolutional layer. The axially shifted rectangular window attention unit can be used multiple times in different levels of the decoder, or multiple units can be used in parallel or serially in the same level with different window sizes, different shifting strategies or different fusion methods.

[0035] In one optional implementation, the channel enhancement unit assigns channel weights by performing global average pooling or global max pooling on the input feature map to obtain a channel descriptor, transforming the channel descriptor through at least one fully connected layer or convolutional layer to generate a weight vector with the same number of channels as the input feature map, and multiplying the weight vector with the input feature map channel by channel.

[0036] In one optional implementation, during the training of the multi-attention enhancement network, the loss function L used is a composite loss function that integrates pixel-level reconstruction loss, gradient loss, and multi-scale structural similarity loss, i.e.:

[0037] Where α, β, and γ are preset weight coefficients used to balance the contributions of different loss terms, L_pix is ​​the pixel-level reconstruction loss, L_grad is the gradient loss, and L_ms-ssim is the multi-scale structural similarity loss.

[0038] In one optional implementation, the pixel-level reconstruction loss is a complementary form of mean squared error loss, mean absolute error loss, or structural similarity index loss. The gradient loss is obtained by calculating the difference between the gradient maps in the horizontal and vertical directions of the enhanced image and the reference clear image. The gradient maps can be obtained by the Sobel operator, the Prewitt operator, or the Scharr operator. The multi-scale structural similarity loss calculates the structural similarity index between the enhanced image and the reference clear image at multiple different scales, and uses the mean or weighted sum of these multi-scale SSIM values ​​as a metric, taking their complementary form as the loss.

[0039] In the above embodiments, a multi-attention fusion U-Net network (MAAU-UIE) was designed for underwater image enhancement. The network structure is as follows: Figure 1As shown, the network mainly consists of an encoder, a bottleneck layer, and a decoder. A novel MultipleAttentionBlock (MAB) module was designed as the foundational module of the network.

[0040] Adjust the dimensions of the original underwater image to , recorded as First, adopt a Convolutional layers map the number of channels to And through the PatchEmbed layer, to The image is divided into a series of patches of varying sizes to obtain shallow-level encoded features. .

[0041] The encoder consists of three feature maps (MABs). Each MAB is followed by a PatchMerging down-sampling (DS) module, which halves the width and height of the feature map while doubling the number of channels. As input, the output feature maps of the three DS in the encoder are denoted as follows: , and The bottleneck layer further enhances the features through one MAB, outputting the features. .

[0042] The decoder's structure is symmetrical to the encoder, also containing three feature maps (MABs). An up-sampling (UP) module is added after each MAB, which doubles the width and height of the feature map using bicubic interpolation, while halving the number of channels. The output feature maps of the three UP modules in the decoder are denoted as follows: , and Simultaneously, the feature maps output by the two MABs at symmetrical positions of the encoder and decoder are concatenated, and a fully connected (Linear) layer is used to halve the number of channels to enhance the information transmission of the network.

[0043] To preserve more image details and improve the visual quality of the reconstructed image, an Enhanced Upsampling Block (EUB) is used after the decoder. The feature map's width and height are doubled through parallel sub-pixel convolution (PixelShuffle) and bilinear interpolation, followed by channel concatenation. Then... Convolutional layers extract spatial features and restore the number of channels to [previous value]. Output features The network endpoint uses Convolutional layers will The channel dimension is mapped to 3, resulting in an enhanced underwater image, denoted as . .

[0044] The Multiple Attention Module (MAB) includes The MAB employs a rectangular window attention layer (RWAL) and a shifted rectangular window attention layer (SRWAL). These two window attention layers alternate, learning local features and extracting global contextual information. To simplify the labeling, the input features of the MAB are denoted as... The inputs are fed in parallel to the Convolution-AxialRectangleWindowAttention (Conv-ARWA) and the ChannelEnhancedModule (CEM) of the fused convolution. Conv-ARWA focuses on calculating the correlation within the rectangular window, extracting information in the horizontal and vertical directions, and using convolutional layers to enhance the learning ability of local features. CEM establishes the correlation between feature channels, and its output is weighted and summed with the calculation result of Conv-ARWA to control the weights of spatial features and channel features when they are fused. The overall process of RWAL is shown in formulas (1) to (4): (1) (2) (3) (4) in, Representation layer normalization. This indicates a feed-forward network. This represents the weights of the channel features. SRWAL replaces Conv-ARWA in RWAL with Convolution-ShiftAxialRectangleWindowAttention (Conv-SARWA) that is fused with the convolution, and the rest of the steps are the same as RWAL.

[0045] The standard Transformer computes self-attention across pixels of the entire feature map, which suffers from high computational cost and insufficient ability to extract local features. To reduce computation while maintaining the model's feature extraction capabilities, a method from the literature is adopted, dividing the feature map into a series of non-overlapping rectangular windows. The width and height of each rectangular window are denoted as... , .according to and Based on their size relationship, rectangular windows are divided into two categories. If... Then it is defined as a horizontal window. If If so, it is defined as a vertical window.

[0046] Rectangle Window Attention (RWA) uses feature maps Divided equally along the channel dimension and Then and Divide into a series of horizontal and vertical windows respectively. and View as Each dimension is The token, then each window can be seen as being made of It consists of several tokens. Figure 2 In the feature map, the green boxes represent tokens, and the blue rectangles represent the divided windows used for subsequent attention calculations. Compared to square windows, rectangular windows capture more information in the horizontal and vertical directions for each pixel, which enhances the model's ability to extract features. Considering that RWA focuses on information interaction within the window and lacks correlation between windows, Shifted Rectangular Window Attention (SRWA) is added to further expand the receptive field. Figure 2 As shown, a circular shift is used to move the windows divided in RWA downwards. 1 pixel, move to the right Each pixel is used to obtain a newly divided window. Finally, the calculation results of the horizontal SRWA and the vertical SRWA are concatenated to obtain the final window attention output.

[0047] Furthermore, the length of one side of the rectangular window is extended to the side length of the input feature map. or The other edge takes the smaller value, denoted as . .like Figure 3As shown, the orange pixel can interact with all pixels on its horizontal and vertical axes, i.e., the orange area. This type of attention calculation is denoted as AxialRectangleWindowAttention (ARWA). Similarly, ShiftAxialRectangleWindowAttention (SARWA) is added after ARWA, and the two alternate.

[0048] In terms of computational complexity, the standard Transformer has a size of The similarity between pixels is calculated on the entire feature map. The computational cost is quadratic with the size of the feature map, as shown in formula (5). The rectangular window attention is focused on a series of pixels of size 1000. Self-attention is calculated within a rectangular window, where the number of rectangular windows is [number missing]. The computational complexity is shown in formula (6).

[0049] (5) (6) Similarly, the computational cost of the axial rectangular window attention ARWA is shown in Equation (7): (7) Although ARWA is more computationally expensive than RWA, it significantly reduces computational cost compared to the standard Transformer. Furthermore, ARWA's size varies with the feature map, making it more flexible. ARWA also has a larger attention area, capturing all information in both the horizontal and vertical directions. Therefore, this paper alternates between ARWA and SARWA in MAB.

[0050] Transformers focus on extracting global dependencies, while CNNs possess good locality and translation invariance, effectively capturing the two-dimensional local structure of images. Therefore, this paper integrates convolution operations into the computation of rectangular window attention, proposing Conv-ARWA and Conv-SARWA to enhance local information within the window, enabling the model to better reconstruct image details.

[0051] Conventional window attention computation generates query, key, and value matrices from the input window feature map using a fully connected layer. This paper uses two convolutional layers on the partitioned window to extract local features in the channel and spatial dimensions, respectively, to obtain the three matrices corresponding to each window. Figure 4 The window attention computation process for fused convolutions is demonstrated.

[0052] After dividing the feature map into a series of rectangular windows as described in Section 2.3, the previous... The horizontal rectangular window divided on the feature map of each channel is denoted as... .

[0053] For windows Firstly, adopt Convolution interacts with information along the channel dimension of the window, while simultaneously increasing the number of channels by a factor of three. Then, it employs... Convolution enhances the extraction of local spatial features, thus obtaining feature extraction capabilities. .Will Dimensional reorganization Then, by dividing the data equally along the channel dimension, we obtain the Query matrix, Key matrix, and Value matrix of the window, denoted as […]. , , . , , Divided equally along the channel dimension Each head has a channel dimension of [number]. , Each unit performs parallel computation of window attention.

[0054] 2.4 Channel Enhancement Module The channel dimension of an image contains rich color information, and enhancing the correlation between feature map channels is beneficial for learning the color and brightness variations of the underwater environment. This paper adds a ChannelEnhancedModule (CEM) in parallel with the rectangular window attention mechanism.

[0055] Input features Through 2 Convolutional layers and GELU activation function for feature extraction , Furthermore, channel attention is used to establish connections between the channels of the feature map and adaptively allocate channel weights. Specifically, the feature map is... See as The size is The feature map, denoted as Global average pooling is used to collect global information for each channel. Convolutional layers reduce the number of channels to ,in The channel compression ratio is set to 4. Then, the GELU activation function is used to enhance the non-linearity of the features, followed by... Convolutional layers restore the number of channels to .

[0056] This paper adopts a multi-task loss function, the loss term of which includes Charbonnier loss, gradient loss and multi-scale structural similarity (MS-SSIM) loss, and the weighted combination of the three is shown in formula (8): (8) in, and The balancing coefficients for the loss term take values ​​of 2 and 1.

[0057] Charbonnier loss aims to reduce the size of the enhanced image output by the model. Compared with reference image The difference in pixels is shown in equation (9): (9) in, and Enhanced images of the reconstruction Compared with reference image The distribution of . Set as This prevents the gradient from being zero, thus improving robustness to some extent. Considering that the Charbonnier loss lacks attention to high-frequency information in the image, gradient loss is used to enhance the image through constraints. Compared with reference image Differences in spatial gradients are used to enhance the model's ability to reconstruct high-frequency details.

[0058] To further improve To improve the perceived quality of the human eye, multi-scale structural similarity loss is employed to reduce the impact at multiple scales. and Differences in brightness, contrast, and structure.

[0059] The original image is subjected to multiple Gaussian filters and a 2x downsampling to obtain images at multiple scales. The original image is labeled as scale 1. The image generated in the next iteration is labeled with scale. .

[0060] This approach was validated on the underwater image benchmark datasets UIEB and UCCS. The UIEB dataset contains 890 pairs of underwater images with references, of which 800 pairs were used as the training set (Train-800) and 90 pairs as the test set (Test-90). UIEB also includes 60 unlabeled raw underwater images, designated as the challenge set (C60). The UCCS dataset contains 300 unlabeled raw underwater images, with 100 images each for three color-shifted underwater environments: blue, green, and blue-green. On these datasets, this approach was compared with several state-of-the-art methods. PSNR and SSIM metrics were used to measure the pixel similarity between two images; higher values ​​indicate higher similarity. UIQM and UCIQE were used to evaluate the saturation, chroma, and brightness of a single image, assessing the image generation quality in the absence of reference images; higher values ​​indicate higher generated image quality. The specific results are shown in Table 1. Ucolor, FiveA+, U-shape, PuGan, Semi-UIR, XCAUNET, and Pixmamba are all cutting-edge methods in the field of underwater image enhancement in recent years. The table shows that our method achieved leading results on the PSNR metric of the Test-90 dataset, the UIQM metric of the C60 dataset, and the UCIQE metric of the UCCS dataset. On several other datasets and related metrics, it is comparable to most other methods.

[0061] Table 1 Performance Comparison on Test-90

[0062] Example 3: like Figure 2 As shown, based on Embodiment 1, this application provides an underwater image enhancement device based on multiple attention, comprising: Image input module acquires the underwater image to be enhanced; A multi-attention enhancement network module, based on the U-NET architecture, includes an encoder submodule, a decoder submodule, and a skip connection submodule; the decoder submodule includes at least one multi-attention submodule, which includes an axially shifted rectangular window attention unit, a convolutional fusion unit, and a channel enhancement unit; The image output module outputs the enhanced underwater image obtained after processing by the multi-attention enhancement network module. Example 4: Figure 3 A block diagram of an exemplary electronic device suitable for implementing embodiments of this application is shown.

[0063] The electronic device may include a central processing unit / microprocessor / main control chip, etc. 4; and a storage medium 5, coupled to the central processing unit / microprocessor / main control chip, etc. 4, and storing computer-executable instructions therein for performing the steps of the various methods of the embodiments of this application when executed by the processor.

[0064] The central processing unit / microprocessor / main control chip, etc., can include, but are not limited to, one or more processors or microprocessors.

[0065] Storage medium 5 may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, computer storage media (e.g., hard disk, floppy disk, solid-state drive, removable disk, CD-ROM, DVD-ROM, Blu-ray disc, etc.).

[0066] In addition, the electronic device may also include (but is not limited to) a data bus 6, an input / output bus / external bus / device bus 7, a display 8, and input / output devices 9 (e.g., keyboard, mouse, speaker, etc.).

[0067] The central processing unit / microprocessor / main control chip, etc. 4 can communicate with external devices (8, 9, etc.) via I / O bus 7 through wired or wireless network (not shown).

[0068] The storage medium 5 may also store at least one computer-executable instruction for performing the steps of various functions and / or methods in the embodiments described herein when the central processing unit / microprocessor / main control chip, etc., 4 is running.

[0069] In one embodiment, the at least one computer-executable instruction may also be compiled into or comprise a software product, wherein one or more computer-executable instructions are executed by a processor to perform the steps of the various functions and / or methods in the embodiments described herein.

[0070] Figure 4 A schematic diagram of a computer-readable storage medium according to an embodiment of this application is shown.

[0071] like Figure 4As shown, the non-transitory computer-readable storage medium 11 stores instructions, such as computer-readable instructions 10. When the computer-readable instructions 10 are executed by a processor, the various methods described above can be performed. The non-transitory computer-readable storage medium includes, but is not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-transitory non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. For example, the non-transitory computer-readable storage medium 11 can be connected to a computing device such as a computer, and then, when the computing device executes the computer-readable instructions 10 stored on the computer-readable storage medium 11, the various methods described above can be performed.

[0072] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0073] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0074] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0075] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for executing all or part of the steps of the methods of the various embodiments of this application through a computer device (which may be a personal computer, server, or network device, etc.). The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0076] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An underwater image enhancement method based on multiple attention, characterized in that, Includes the following steps: Acquire underwater images to be enhanced; The underwater image is subjected to feature extraction and enhancement processing to obtain an enhanced underwater image. The underwater image is downsampled layer by layer to extract multi-scale features. The multi-scale features are upsampled layer by layer. The feature map is divided into rectangular windows in at least one upsampling level. Window attention calculation is performed on each rectangular window to perform fusion convolution operation. Channel attention calculation is performed in parallel to adaptively allocate channel weights. The window attention calculation results and channel attention calculation results are fused to enhance the features. The upsampled features are fused with the downsampled features of the corresponding level. By determining the parameters of the feature extraction and enhancement process, a loss function is constructed based on a weighted combination of Charbonnier loss, gradient loss, and multi-scale structural similarity loss, and the parameters are adjusted with the loss function as the optimization objective.

2. The underwater image enhancement method based on multiple attention as described in claim 1, characterized in that, The layer-by-layer downsampling is achieved through convolutional layers and pooling operations. The convolutional layers are used to extract local detail features, and the pooling operations are used to reduce the resolution of the feature map to obtain high-level semantic features. The layer-by-layer upsampling involves enlarging the size of the low-resolution feature map and then enhancing the ability to represent details through convolutional layers and nonlinear activation functions.

3. The underwater image enhancement method based on multiple attention as described in claim 1, characterized in that, The rectangular window partitioning involves dividing the input feature map into multiple non-overlapping or partially overlapping rectangular sub-regions along the height and width directions; the in-window self-attention calculation involves independently calculating the key, query, and value, and generating attention weights within each rectangular sub-region. The fusion convolution specifically includes introducing at least one convolutional layer to perform convolution operations on the original feature map or its linear transformation result during the generation of the key, query and / or value, in order to capture local contextual information.

4. The underwater image enhancement method based on multiple attention as described in claim 1, characterized in that, The channel attention calculation includes an axial shift operation, which includes shifting along the height direction and / or shifting along the width direction; the shifting magnitude is less than or equal to the corresponding side length of the rectangular window; The attention calculation results before and after the fusion shift include adding or concatenating the feature maps obtained from the two attention calculations element by element and then fusing them through a convolutional layer; The axial shift operation includes a rectangular window attention unit, which is used multiple times in different levels of the decoder, or multiple units are used in parallel or serially in the same level with different window sizes, different shift strategies, or different fusion methods.

5. The underwater image enhancement method based on multiple attention as described in claim 1, characterized in that, The channel attention calculation includes a channel enhancement unit, which assigns channel weights by performing global average pooling or global max pooling on the input feature map to obtain channel descriptors, transforms the channel descriptors through at least one fully connected layer or convolutional layer to generate a weight vector with the same number of channels as the input feature map, and multiplies the weight vector with the input feature map channel by channel.

6. The underwater image enhancement method based on multiple attention as described in claim 1, characterized in that, During the training of the multi-attention enhancement network, the loss function L used is a composite loss function that integrates pixel-level reconstruction loss, gradient loss, and multi-scale structural similarity loss, namely: Where α, β, and γ are preset weight coefficients used to balance the contributions of different loss terms, L_pix is ​​the pixel-level reconstruction loss, L_grad is the gradient loss, and L_ms-ssim is the multi-scale structural similarity loss.

7. The underwater image enhancement method based on multiple attention as described in claim 6, characterized in that, The pixel-level reconstruction loss is a complementary form of mean squared error loss, mean absolute error loss, or structural similarity index loss. The gradient loss is obtained by calculating the difference between the gradient maps in the horizontal and vertical directions of the enhanced image and the reference clear image. The gradient maps can be obtained by the Sobel operator, the Prewitt operator, or the Scharr operator. The multi-scale structural similarity loss calculates the structural similarity index between the enhanced image and the reference clear image at multiple different scales, and uses the mean or weighted sum of these multi-scale SSIM values ​​as a metric, taking their complementary form as the loss.

8. A multi-attention-based underwater image enhancement apparatus for the multi-attention-based underwater image enhancement method as described in any one of claims 1 to 7, characterized in that, include: Image input module acquires the underwater image to be enhanced; A multi-attention enhancement network module, which is based on the U-NET architecture and includes an encoder submodule, a decoder submodule, and a skip connection submodule; The decoder submodule includes at least one multi-attention submodule, which includes an axially shifted rectangular window attention unit, a convolutional fusion unit, and a channel enhancement unit. The image output module outputs the enhanced underwater image obtained after processing by the multi-attention enhancement network module.

9. An electronic device, comprising: At least one memory stores computer-executable instructions non-transiently; At least one processor, configured to run the computer-executable instructions, The computer-executable instructions are executed by the processor to implement the underwater image enhancement method based on multiple attention according to any one of claims 1-7.

10. A computer-readable storage medium, wherein, The computer-readable storage medium stores computer-executable instructions that, when executed by at least one processor, implement the underwater image enhancement method based on multiple attention as described in any one of claims 1-7.

Images