Underwater image enhancement method and device based on multi-color space and frequency domain guidance

Abstract

The invention discloses an underwater image enhancement method and device based on multi-color space and frequency domain guidance, and the method comprises the steps: converting an inputted underwater image into three color spaces, namely RGB, HSV and LAB, based on a color extraction module in a multi-color space recovery sub-network, so as to carry out the color recovery of the image through the full utilization of the advantages of each color space; a color fusion module in the multi-color space recovery sub-network is used to adaptively correct color distortion of the underwater image by using a color attention mechanism; extracting, enhancing and fusing high and low frequency information of the image through a frequency domain enhancement sub-network; further enhancing the details and contrast of the image through a detail enhancement sub-network; and fusing the output feature maps of the three sub-networks, and finally outputting an enhanced underwater image. According to the method, the problems of color distortion, detail blurring and contrast reduction in the underwater image are effectively solved, and the color and texture details of the image can be truly restored.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an underwater image enhancement method and device based on multi-color space and frequency domain guidance. Background Art

[0002] Underwater imaging has important applications in areas such as marine resource exploration, underwater robot navigation, and marine science research. However, due to the propagation characteristics of light in water, underwater images often suffer from color distortion, blurred details, and reduced contrast. Underwater image enhancement plays a crucial role in improving the visual quality and fidelity of images, contributing to a more accurate understanding of the underwater environment. Underwater image degradation can be attributed to two primary factors. First, the wavelength of light varies with propagation distance, unlike the uniform attenuation of terrestrial light. As propagation distance increases, longer wavelengths, such as red and orange, attenuate more dramatically, resulting in a blue-green shift associated with shorter wavelengths. This effect becomes more pronounced as the distance between the subject and the camera increases. Second, suspended particles in the water, including organic particles and planktonic microorganisms, scatter light, deflecting its propagation direction and reducing image contrast and clarity. Numerous methods have emerged to improve the quality of underwater images, broadly categorized into two types: traditional methods and deep learning-based methods. Traditional underwater image enhancement methods include those based on priors and those based on physical models. Prior-based methods leverage rich priors to explore the spatial relationships between pixel values ​​in raw underwater images and enhance them by adjusting contrast, brightness, and saturation, such as the gray-world assumption, RGB maximization, and white balancing. However, these methods often ignore the physical imaging process, limiting their enhancement effectiveness. Physical model-based underwater image enhancement methods focus on accurately estimating the underwater image formation model or medium transmission parameters. These methods aim to obtain a clean image by inverting the physical underwater imaging model. Despite their potential, the performance of traditional underwater image enhancement methods is often limited by the complexity and diversity of real-world underwater conditions. Recently, deep learning has driven significant progress in underwater image enhancement, with major methods based on convolutional neural networks (CNNs), generative adversarial networks (GANs), and transformers. In this work, the existing state-of-the-art focuses on CNN architectures due to their powerful visual feature learning capabilities and flexible training procedures. However, CNN-based underwater image enhancement methods encounter challenges in real underwater scenes with complex distortions and diverse image content, hindering accurate and adaptive restoration. As described by the underwater imaging model, global color distortion caused by global illumination and local complex distortion within scene brightness pose a major obstacle to underwater image enhancement and significantly reduce image clarity.

[0003] In view of the above problems, it can be seen that how to deal with serious degradation problems such as blurred details and color cast in the process of underwater image enhancement is an urgent problem to be solved. Summary of the Invention

[0004] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology and provide an underwater image enhancement method and device based on multi-color space and frequency domain guidance. By combining multi-color space processing and frequency domain enhancement technology, the problems of color distortion, blurred details and reduced contrast in underwater images are effectively solved, thereby improving the visual quality and practicality of underwater images.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] In a first aspect, the present invention provides an underwater image enhancement method based on multi-color space and frequency domain guidance, comprising the following steps:

[0007] Based on the color extraction module in the multi-color space restoration sub-network MCR, the input underwater image is converted into three color spaces: RGB, HSV, and LAB, to extract the features of each color space;

[0008] The color fusion module in the multi-color space restoration sub-network (MCR) is used to further extract and fuse the color information of the three color spaces. The color distortion of the underwater image is adaptively corrected using the color attention mechanism, and finally a fused color feature map is obtained.

[0009] The underwater original image is passed through the frequency domain enhancement sub-network FE, the input features are decomposed into low-frequency features and high-frequency features, the expression ability of high- and low-frequency features is enhanced, and a feature map after frequency domain enhancement is obtained; the frequency domain enhancement sub-network FE decomposes the input features into low-frequency features and high-frequency features through a dynamically learned filter, and improves the performance of the underwater image enhancement task by enhancing the expression ability of low-frequency features and high-frequency features;

[0010] The underwater original image is passed through the detail enhancement sub-network DE to further enhance the key area information in the image and increase the context relevance, thereby obtaining a feature map after detail enhancement. The detail enhancement sub-network DE downsamples the input image by applying NAF-Block layer by layer to extract high-level semantic features, and restores the image detail information through upsampling operations, thereby achieving refined optimization of image features at both global and local levels.

[0011] The fused color feature map, the frequency domain enhanced feature map, and the detail enhanced feature map are fused to output a final underwater image enhancement result.

[0012] As a preferred technical solution, the processing process of the color extraction module is specifically as follows:

[0013] The convolutional layer Conv is used to extract features related to the color space, thereby obtaining a preliminary color feature representation;

[0014] The obtained color features are passed to the color attention block (CAB) to extract and enhance key color information. In CAB, two convolutional layers are first used to further extract features. Nonlinearity is then introduced through the ReLU activation function to enhance the model's expressiveness. Attention weights are then generated through a convolutional layer and a Sigmoid activation function. The weighted features are then fused with the original features through element-wise multiplication to output feature maps in RGB, HSV, and LAB color spaces.

[0015] Add the feature maps of RGB, HSV, and LAB color spaces element by element to generate a fused feature map F sum .

[0016] As a preferred technical solution, the processing process of the color fusion module is specifically as follows:

[0017] First, pass the convolution layer to represent the fusion feature map F after element-by-element addition sum Further feature extraction is performed to obtain a deep fusion feature map

[0018] Then, the channel attention mechanism CA is used to enable the network to adaptively adjust the importance of different channel features. In CA, the deep fusion feature map is firstly integrated into the global average pooling operation. Compress in the spatial dimension to obtain the global features of each channel, and then generate channel attention weights through two convolutional layers and ReLU activation function;

[0019] The channel attention weight is multiplied element-wise with the deep fusion feature map, and then the feature extraction and fusion are performed through the convolution layer to output the color-corrected fused color feature map.

[0020] As a preferred technical solution, the formula for generating channel attention weights is as follows:

[0021]

[0022] Among them, GAP represents global average pooling, f ch It is a convolution operation using filters. σ and δ represent the Sigmoid activation function and the ReLU activation function respectively. Global average pooling GAP can integrate the global spatial information of the channel into a channel description. The specific process is as follows:

[0023]

[0024] in, Indicates the cth channel X at position (i, j) c value.

[0025] As a preferred technical solution, the processing process of the frequency domain enhancement sub-network is as follows:

[0026] The shallow features are extracted through the convolution layer to obtain the feature map F s ;

[0027] For the feature map F s Perform global average pooling and convolution operations to generate a low-pass filter F l ; Subtract the low-pass filter from the identity matrix to obtain the high-pass filter F h ;

[0028] Use the generated low-pass filter F l For the input feature F s Perform convolution operation to obtain low-frequency features F lo ; Use the generated high-pass filter F h For the input feature F s Perform convolution operation to obtain high-frequency features F hi ;

[0029] The low-frequency feature F lo and high-frequency features F hi The input is respectively fed into the multi-branch convolutional unit and processed by convolution kernels of different sizes. Each convolution is followed by batch normalization and ReLU activation function to form a set of convolution-batch normalization-activation CBR units.

[0030] Add the output feature maps of all branches element by element to generate fused features;

[0031] Perform global average pooling (GAP) on the fused features to generate a global descriptor, and generate channel descriptors for each scale through two fully connected layers. The channel descriptors are reshaped and normalized to generate weights for each scale.

[0032] Apply the weight of each scale to the corresponding convolution output and generate scale-enhanced features by weighted summation;

[0033] Perform global average pooling on the scale enhancement features along the width and height directions respectively to obtain the width direction feature map F X and height direction feature map F Y ;

[0034] The width feature map F X and height direction feature map F Y Stacked in the channel dimension, the first shape feature layer F is obtained X+Y, further processed by convolution, BatchNorm and activation function to obtain the second shape feature layer F′ X+Y ;

[0035] After the Split operation, the feature layer F′ X+Y Divide into two parts and adjust them into shapes and

[0036] Use convolution to adjust the number of channels and generate attention weights in width and height directions respectively through Sigmoid function. Extended to width direction features, Expanded to the height directional features, both directional feature maps are multiplied with the original feature map to obtain the final frequency domain enhancement information F enhance ;

[0037] After obtaining the enhanced high-frequency features and low-frequency features, they are fused through the Concat operation to obtain the high- and low-frequency fusion feature map F lo+hi Finally, add the original input features element by element to obtain the output feature map F of the frequency domain enhancement sub-network fe .

[0038] As a preferred technical solution, the processing process of the detail enhancement sub-network DE is specifically as follows:

[0039] In NAF-Block, the underwater original image is processed by layer normalization LayerNorm to reduce internal covariate shift;

[0040] The number of channels is reduced through 1×1 convolutional layers, which reduces computational complexity.

[0041] Extract local features through 3×3 depth-wise separable convolution, which decomposes the standard convolution into depth-wise convolution and point-wise convolution, reducing the number of parameters and computational cost;

[0042] Chunk operation is performed on the convolution output, evenly splitting it into two feature blocks along the channel dimension, and generating nonlinear features through element-by-element product;

[0043] The spatial information of the feature map is aggregated through the global average pooling layer to generate a channel attention map to capture global information. The number of channels is adjusted through 1×1 convolution and multiplied element-by-element with the original feature map to achieve channel attention weighting.

[0044] The weighted feature map is passed through a 1×1 convolutional layer to adjust the number of channels, and the layer normalization LayerNorm is used to stabilize the training;

[0045] The trained results are chunked and weighted with channel attention again, and added to the input feature map through skip connections to output the enhanced detail feature map.

[0046] As a preferred technical solution, the fused color feature map, the frequency domain enhanced feature map and the detail enhanced feature map are added and fused element by element to finally obtain an output result.

[0047] In a second aspect, the present invention provides an underwater image enhancement system based on multi-color space and frequency domain guidance, which is applied to the underwater image enhancement method based on multi-color space and frequency domain guidance, including a multi-color space restoration subnetwork, a frequency domain enhancement subnetwork module, and a detail enhancement subnetwork module. The multi-color space restoration subnetwork includes a color extraction module and a color fusion module.

[0048] The color extraction module is used to convert the input underwater image into three color spaces: RGB, HSV and LAB, so as to extract the features of each color space;

[0049] The color fusion module is used to further extract and fuse the color information of the three color spaces, and use the color attention mechanism to adaptively correct the color distortion of the underwater image, and finally obtain a fused color feature map;

[0050] The frequency domain enhancement sub-network module is used to decompose the input features into low-frequency features and high-frequency features, enhance the expressiveness of high- and low-frequency features, and obtain a feature map after frequency domain enhancement; the frequency domain enhancement sub-network FE decomposes the input features into low-frequency features and high-frequency features through a dynamically learned filter, and improves the performance of underwater image enhancement tasks by enhancing the expressiveness of low-frequency features and high-frequency features;

[0051] The detail enhancement sub-network module is used to further enhance the key area information in the image and increase the context relevance to obtain a feature map after detail enhancement; the frequency domain enhancement sub-network FE downsamples the input image by applying NAF-Block layer by layer to extract high-level semantic features, and restores the image detail information through upsampling operations, thereby achieving refined optimization of image features at both global and local levels;

[0052] The fused color feature map, the frequency domain enhanced feature map, and the detail enhanced feature map are fused to output a final underwater image enhancement result.

[0053] In a third aspect, the present invention provides an electronic device, comprising:

[0054] at least one processor; and,

[0055] a memory communicatively connected to the at least one processor; wherein,

[0056] The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to perform the underwater image enhancement method based on multi-color space and frequency domain guidance.

[0057] In a fourth aspect, the present invention provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the underwater image enhancement method based on multi-color space and frequency domain guidance.

[0058] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0059] (1) This paper provides a multi-color space restoration subnetwork that adaptively corrects various types of color distortion in underwater images through multi-color space analysis and a color attention mechanism. This design combines the advantages of RGB, LAB, and HSV color spaces to more comprehensively restore the image's color information, improve the adaptability of color restoration, and make the output image closer to human visual perception.

[0060] (2) The frequency domain enhancement subnetwork provided by the present invention effectively extracts and enhances high- and low-frequency information through a designed network, enhances the details and texture information of the image, and thus improves the visual quality and clarity of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0062] Figure 1 This is a flow chart of an underwater image enhancement method based on multi-color space and frequency domain guidance according to an embodiment of the present invention;

[0063] Figure 2 2. It is a schematic diagram of an underwater image enhancement network based on multi-color space and frequency domain guidance according to an embodiment of the present invention;

[0064] Figure 3 Schematic diagram of the color extraction module structure in the multi-color space restoration subnetwork in an embodiment of the present invention;

[0065] Figure 4 is a schematic diagram of a frequency domain enhancement module in a frequency domain enhancement subnetwork according to an embodiment of the present invention;

[0066] Figure 5 This is a comparison chart of the enhancement effects of the present invention and several advanced underwater image enhancement algorithms on a reference dataset;

[0067] Figure 6 This is a comparison chart of the enhancement effects of the present invention and several advanced underwater image enhancement algorithms on a no-reference dataset;

[0068] Figure 7 Schematic diagram of the structure of an underwater image enhancement system based on multi-color space and frequency domain guidance according to an embodiment of the present invention;

[0069] Figure 8 2 is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0070] In order to enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0071] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0072] See also Figure 1 This embodiment proposes an underwater image enhancement method based on multi-color space and frequency domain guidance, including the following steps:

[0073] S1. Extract color information from the underwater original image based on the color extraction module in the multi-color space restoration sub-network.

[0074] The color extraction module in the multi-color space restoration subnetwork first extracts features related to the color space through the convolution layer (Conv), thereby obtaining a preliminary color feature representation. These color features are then passed to the Color Attention Block (CAB) to further extract and enhance key color information. In CAB, features are first further extracted through two convolutional layers, and then nonlinearity is introduced through the ReLU activation function to enhance the model's expressiveness. Next, attention weights are generated through a convolutional layer and a Sigmoid activation function. These weights are used to weight the features to highlight important color information. Finally, the weighted features are fused with the original features through element-wise multiplication to generate the final output features. Through this design, the color extraction module can effectively capture key color information in different color spaces, providing rich feature representations for the subsequent color fusion module.

[0075] After the color extraction module, feature maps from RGB, HSV, and LAB color spaces can be obtained. These feature maps contain key color information in their respective color spaces. For example, the RGB color space is more sensitive to changes in illumination, while the HSV color space has advantages in representing the hue and saturation of colors, and the LAB color space performs better in representing the brightness and chromaticity of colors. The present invention performs element-wise summation on the feature maps of these color spaces to fuse the feature information of different color spaces. This process can be expressed as:

[0076]

[0077] Among them, F sum Represents the fused feature map after element-by-element addition, F RGB 、F HSV and F Lab Respectively represent the feature maps output by the color extraction module for RGB, HSV and Lab color space information. It is element-wise addition.

[0078] Feature map F after element-by-element addition sum Contains complementary information from different color spaces and can more comprehensively represent the color features of the image. sum It is input into the color fusion module for feature enhancement and fusion.

[0079] S2, the color fusion module in the multi-color space restoration subnetwork (MCR) further extracts and fuses the color information of the above color spaces, and uses the color attention mechanism to adaptively correct the color distortion of the underwater image, and finally obtains the fused color feature map.

[0080] The color fusion module in the multi-color space restoration sub-network first performs a series of convolutional layers (Conv) on F sum Further feature extraction is performed to obtain Then, the channel attention mechanism (CA) is used to enable the network to adaptively adjust the importance of different channel features. In the CA structure, first, the feature is pooled by the global average pooling (GAP) operation. Compress in the spatial dimension to obtain the global features of each channel. Then, through two convolutional layers and ReLU activation function, generate channel attention weights. The specific formula is as follows:

[0081]

[0082] Among them, GAP represents global average pooling, and f ch is a convolution operation using filters. σ and δ represent the Sigmoid activation function and the ReLU activation function, respectively. Global average pooling (GAP) can integrate the global spatial information of the channel into a channel description. The specific process is as follows:

[0083]

[0084] in, Indicates the cth channel X at position (i, j) c value.

[0085] Channel attention feature map F c Is to input feature map and the channel attention vector M replicated in the spatial dimension c The specific process is as follows:

[0086]

[0087] Where ⊙ represents element-wise multiplication.

[0088] The weighted feature map F c Feature extraction and fusion are performed through the convolution layer to obtain the fused feature map

[0089] Through the above steps, the color fusion module can effectively fuse the feature information of different color spaces and adaptively adjust the weight of each channel through the channel attention mechanism, thereby enhancing the key color features and suppressing unimportant features. Finally, the fusion feature map F fusion The enhanced image is then output by fusing the output feature maps of the frequency domain enhancement sub-network and the detail enhancement sub-network.

[0090] By combining analysis in multiple color spaces, the MCR subnetwork can capture and process color features in images from different perspectives, avoiding the potential limitations of a single color space. This allows for a more comprehensive restoration of the image's original color information and effectively improves color deviation in underwater images. The color attention mechanism enables the MCR subnetwork to automatically adjust its focus on different color regions based on the specific image context, enhancing the model's adaptability to underwater images with varying types and degrees of color distortion. Whether in deep waters with a predominantly blue hue or shallow waters with a predominantly green hue, the MCR subnetwork flexibly performs color correction without human intervention or pre-setting specific parameters.

[0091] S3. At the same time, the underwater original image passes through the frequency domain enhancement sub-network (FE), which decomposes the input features into low-frequency and high-frequency parts, enhances the expression ability of high- and low-frequency features, and obtains the feature map after frequency domain enhancement.

[0092] The frequency domain enhancement sub-network (FE) decomposes the input features into low-frequency and high-frequency parts through dynamically learned filters, and improves the performance of underwater image enhancement tasks by enhancing the expressive power of these features. The sub-network first performs shallow feature extraction through a convolutional layer, aiming to capture the basic features of the image and obtain a feature map. In order to decompose the input features into different frequency domain parts, a low-pass filter is first generated by global average pooling (GAP) and 1×1 convolution operations. Among them, the Sigmoid activation function is mainly used to suppress unimportant elements in the filter, thereby enhancing the effect of the low-pass filter. Then, the high-pass filter is obtained by subtracting the low-pass filter from the unit matrix. Then use the generated low-pass filter F l For the input feature F s Perform convolution operation to obtain low-frequency features Use the generated high-pass filter F h For the input feature F s Perform convolution operation to obtain high-frequency features Next, the sub-network adopts a frequency domain enhancement module to further enhance the high and low frequency features.

[0093] The frequency domain enhancement module first processes the input features through convolution kernels of different sizes. Each convolution is followed by batch normalization (Batch Normalization) and ReLU activation function to form a set of convolution-batch normalization-activation (CBR) units. The feature maps of all branches are then added element by element to form a richer feature representation. The summed result is globally average pooled to compress the features into a global descriptor. The pooled features are processed through two fully connected layers (first dimensionality reduction and then dimensionality increase) to generate channel descriptors for each scale. The channel descriptor is reshaped to the size of the original input and normalized by softmax to generate weights for each scale. Next, the weight of each scale is applied to the corresponding convolution output, and these features are combined by weighted summation. In this way, the network can focus on the most effective feature scale at the moment, thereby optimizing the processing results. Subsequently, in the COOA module, the features are globally average pooled in the width and height directions respectively to obtain a width-wise feature map. and height directional feature map The feature mapping to high and wide dimensions is completed. Then, these two feature layers are stacked in the channel dimension to form a shape of The feature layer is further processed by convolution, BatchNorm and activation function to obtain a shape of The feature layer F′ is then split by a Split operation. X+Y Divide into two parts and adjust them into shapes and Finally, 1x1 convolution is used to adjust the number of channels and the attention weights in the width and height directions are generated by the Sigmoid function. Extended to width direction features, Expanded to the height directional features, both directional feature maps are multiplied with the original feature map to obtain the final frequency domain enhancement information

[0094] After obtaining the enhanced high-frequency features and low-frequency features, they are fused through the Concat operation to obtain the high- and low-frequency fusion feature map. Finally, add the original input features element by element to obtain the output feature map of the frequency domain enhancement sub-network

[0095] Frequency domain processing, after multi-color space processing, can further optimize image quality from the perspective of spatial frequency characteristics. Frequency domain processing effectively removes noise from images, preventing the loss of image detail caused by denoising. Furthermore, by adjusting frequency domain information, it can highlight details such as edges and textures, enhancing image detail and contrast for a clearer and more vivid image. Combining multi-color space processing with frequency domain processing can more comprehensively restore the original image information, making the processed image more realistic in color, detail, and structure, and consistent with human visual perception.

[0096] S4. At the same time, the detail enhancement sub-network (DE) further enhances the key area information in the image and increases the context relevance to obtain a feature map after detail enhancement.

[0097] The detail enhancement subnetwork (DE) is an ultra-simple baseline solution that removes nonlinear activation units, reducing computational complexity while delivering excellent performance. It utilizes a UNet architecture with skip connections, which better preserves image detail and texture information. Furthermore, when processing images, the network not only focuses on local region features but also captures the correlation between any two points in the image, effectively enhancing image detail. By combining multi-color space processing, frequency domain processing, and the detail enhancement subnetwork, comprehensive optimization of image color, structure, and detail can be achieved, resulting in an output image that is closer to the original scene in every respect and consistent with human visual perception.

[0098] Specifically, the subnetwork downsamples the input image by applying NAF-Block layer by layer to extract high-level semantic features; then, it restores the image's detail information with the help of upsampling operations, thereby achieving refined optimization of image features at both global and local levels.

[0099] In NAF-Block, feature maps first undergo LayerNorm processing, which normalizes the feature maps of each channel, effectively reducing internal covariate shift and ensuring network stability during training. Next, the feature maps pass through a 1x1 convolutional layer to reduce the number of channels, lower computational complexity, and improve efficiency. Subsequently, the feature maps pass through a 3x3 depthwise separable convolutional layer. This convolution decomposes the standard convolution into depthwise and pointwise convolutions, significantly reducing the number of parameters and computational cost while maintaining the expressive power of convolution. This makes NAF-Block more efficient during training and effectively extracts local image features. The features are evenly divided into two feature blocks through the Chunk operation, and a simple element-wise product operation replaces traditional nonlinear activation functions (such as Sigmoid, ReLU, and GELU). This operation simplifies the model structure and reduces computational complexity while maintaining the nonlinear transformation capability of the features. Next, a global average pooling (GAP) layer aggregates the spatial information of the feature maps to generate a channel attention map that captures global information. The channel attention map passes through a 1x1 convolutional layer to map back to its original number of channels, ensuring the same dimensionality as the original feature map, facilitating subsequent feature fusion. The channel attention map is element-wise multiplied with the original feature map to generate the final feature map. The channel attention map weights the original feature map to enhance the expressiveness of the features. The feature map then passes through another 1x1 convolutional layer to further adjust the number of channels, followed by LayerNorm processing to further stabilize the training process. Next, the feature map passes through another 1x1 convolutional layer to adjust the number of channels, followed by another Chunk and element-wise product operation, a 1x1 convolutional layer, and finally, a skip connection to fuse the original and processed feature maps. This design preserves image details and enhances the network's ability to capture key image regions. This series of design steps ensures that NAF-Block effectively extracts and enhances image features while maintaining efficient computation, improving network performance and stability, and achieving excellent underwater image enhancement results.

[0100] S5. Add and fuse the fused color feature map, the frequency domain enhanced feature map, and the detail enhanced feature map element by element, and output a final underwater image enhancement result.

[0101] In one embodiment, training is performed on RTX3090 with batchsize=8, using the Adam optimizer and a cosine annealing strategy. The present invention crops each image into a size of 256×256 and uses the obtained image as the input network for training.

[0102] This embodiment can better cope with various degradation factors of underwater scenes, such as uneven lighting, color degradation, low brightness, severe detail loss, color cast and other problems, and obtain underwater enhanced images with rich detail textures and correct color cast processing. Figure 5 In the figure, although the coral colonies in UWnet, USUIR, PUIE-Net and PUGAN appear dark gray, and BCTA-Net appears dark red, the proposed method can generate more natural colors, making the enhancement effect closer to the real label. Similarly, in correcting the blue distortion ( Figure 5 , the sixth column image), USUIR has a clear blue color cast, while UWnet, PUIE-Net, PUGAN and BCTA-Net do not have a blue color cast, but these methods restore the image as a whole to a gray-black tone, and do not accurately restore the colors of the fish body and fins. The method of the present invention achieves better color restoration in both the overall tone and key areas. This shows that the method of the present invention can restore more natural and vivid colors compared to the existing technology. Figure 5 In the fourth column of the figure, in the complex fish face scene rich in texture details and color changes, the original image is not only blurred in detail due to the scattering of underwater suspended particles, but also affected by the overall hue of the light blue background light underwater, resulting in serious degradation of the complex color changes of the fish face, presenting a single hue distribution of blue, white and black. Although the numerous fish face parts of different colors and shades in this scene pose a huge challenge to the image restoration task, the method of the present invention performs outstandingly in these aspects, while other methods appear to be unable to cope. For example, USUIR is still disturbed by the underwater background light, resulting in a light blue overall hue; although UWnet, PUIR-Net, PUGAN, and BCTA-Net have gotten rid of the influence of background light, there are problems in detail recovery and local color changes. For example, the color detail changes of the fish eyes, forehead, brow bones and other parts are not restored, and the overall color appears in black and white. In contrast, the method of the present invention is closest to the real image. It not only adjusts the background light hue, but also restores the detailed changes and local color changes of the fish face, thereby improving the clarity. Similarly, Figure 6 As shown in the third row, UWnet suffers from a gray-green color cast, while USUIR, PUIE-Net, PUGAN, and BCTA-Net all suffer from blurred details. However, the proposed method overcomes the dark green underwater color cast while effectively restoring the detailed texture features of the sculpture, even clearly reproducing the rust marks on the body and the lines on the face, greatly improving the image quality.

[0103] Table 1

[0104]

[0105] Note: __ and The labeled values ​​represent the best and second-best results, respectively. GDCP, HLRP, MLLE, and UNTV are traditional methods, while the remaining methods are based on deep learning.

[0106] In order to more specifically reflect the technical effects of this embodiment, as shown in the objective evaluation results of the method in Table 1, peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are used as evaluation indicators, where red and blue represent the first and second ranked indicators respectively. The higher the indicator value, the closer the generated image is to the standard clear image. In the table, this embodiment is trained on the UIEB training set and tested on the UIEB test set, and trained on the EUVP training set and tested on the EUVP test set. Due to the limitations of the expressive power of manually designed features, the performance of traditional methods is generally inferior to that of deep learning-based technologies. Among the deep learning methods, the method proposed in the present invention achieved the best PSNR and SSIM scores on both the UIEB and EUVP datasets. On the UIEB and EUVP datasets, the PSNR was 2.0% and 3.5% higher, and the SSIM was 3.3% and 1.1% higher, respectively, than the second-ranked method. These significant performance improvements fully demonstrate the powerful advantages of the method of the present invention in effectively addressing the challenges of underwater image degradation.

[0107] As shown in the objective evaluation results of the methods in Tables 2 and 3, the single-image metrics UIQM and ILNIQE were used as evaluation metrics, where red and blue represent the first and second ranked metrics, respectively. The higher the UIQM or the lower the ILNIQE, the better the image quality. In the table, this embodiment was trained on the UIEB training set and tested on the UIEB test set, EUVP test set, MABLS dataset, U45 dataset, and SUIM dataset. The MCFG-Net proposed in this embodiment achieved the best UIQM score on the EUVP, MABLS, U45, and SUIM datasets, and the second best score on the UIEB dataset, demonstrating its superiority in enhancing key image attributes such as color, sharpness, and contrast. In addition, MCFG-Net ranked first among learning-based methods based on the ILNIQE metric (used to measure the perceptual quality of natural images) on the UIEB, EUVP, U45, and SUIM datasets, and ranked second on the MABLS dataset. These comprehensive results verify the superiority of our method in terms of UIQM and ILNIQE indicators, demonstrate its consistently outstanding performance on various datasets, and demonstrate the effectiveness of MCFG-Net in comprehensively improving the quality of underwater images.

[0108] Table 2

[0109]

[0110] Note: __ and The labeled values ​​represent the best and second best results among the deep learning-based methods. GDCP, HLRP, MLLE, and UNTV are traditional methods, while the rest are deep learning-based methods.

[0111] Table 3

[0112]

[0113] Note: __ and The labeled values ​​represent the best and second best results among deep learning based methods, respectively.

[0114] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0115] Based on the same concept as the underwater image enhancement method based on multi-color space and frequency domain guidance in the above-mentioned embodiment, the present invention also provides an underwater image enhancement system based on multi-color space and frequency domain guidance, which can be used to implement the above-mentioned underwater image enhancement method based on multi-color space and frequency domain guidance. For ease of explanation, the structural diagram of the embodiment of the underwater image enhancement system based on multi-color space and frequency domain guidance only shows the parts related to the embodiment of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation of the device, and it may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0116] See also Figure 7 In another embodiment of the present application, an underwater image enhancement system 100 based on multi-color space and frequency domain guidance is provided, which includes a color extraction module 101 in a multi-color space restoration subnetwork (MCR), a color fusion module 102 in a multi-color space restoration subnetwork (MCR), a frequency domain enhancement subnetwork module 103, and a detail enhancement subnetwork module 104;

[0117] The color extraction module 101 in the multi-color space restoration sub-network (MCR) is used to convert the input underwater image into three color spaces: RGB, HSV, and LAB, so as to fully utilize the advantages of each color space to perform color restoration of the image;

[0118] The color fusion module 102 in the multi-color space restoration sub-network (MCR) is used to further extract and fuse the color information of each color space, and use the color attention mechanism to adaptively correct the color distortion of the underwater image, and finally obtain a fused color feature map;

[0119] The frequency domain enhancement sub-network module 103 is used to decompose the input features into low-frequency and high-frequency parts, enhance the expressiveness of high- and low-frequency features, and obtain a feature map after frequency domain enhancement; the frequency domain enhancement sub-network FE decomposes the input features into low-frequency features and high-frequency features through a dynamically learned filter, and improves the performance of the underwater image enhancement task by enhancing the expressiveness of the low-frequency features and high-frequency features;

[0120] The detail enhancement sub-network module 104 is used to further enhance the key area information in the image and increase the context relevance to obtain a feature map after detail enhancement; the detail enhancement sub-network DE downsamples the input image by applying NAF-Block layer by layer to extract high-level semantic features, and restores the image detail information with the help of upsampling operations, thereby achieving refined optimization of image features at both global and local levels.

[0121] The fused color feature map, the frequency domain enhanced feature map, and the detail enhanced feature map are fused to output a final underwater image enhancement result.

[0122] It should be noted that the underwater image enhancement system based on multi-color space and frequency domain guidance of the present invention corresponds one-to-one to the underwater image enhancement method based on multi-color space and frequency domain guidance of the present invention. The technical features and beneficial effects described in the above-mentioned embodiment of the underwater image enhancement method based on multi-color space and frequency domain guidance are all applicable to the embodiment of underwater image enhancement based on multi-color space and frequency domain guidance. For specific contents, please refer to the description in the embodiment of the method of the present invention, which will not be repeated here. This is hereby declared.

[0123] In addition, in the implementation of the underwater image enhancement system based on multi-color space and frequency domain guidance in the above embodiment, the logical division of each program module is only an example. In actual application, the above functions can be assigned to different program modules as needed, for example, for the configuration requirements of the corresponding hardware or the convenience of software implementation. That is, the internal structure of the underwater image enhancement system based on multi-color space and frequency domain guidance is divided into different program modules to complete all or part of the functions described above.

[0124] See also Figure 8 In one embodiment, an electronic device for implementing an underwater image enhancement method based on multi-color space and frequency domain guidance is provided. The electronic device 200 may include a first processor 201, a first memory 202 and a bus, and may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as an underwater image enhancement program 203 based on multi-color space and frequency domain guidance.

[0125] The first memory 202 includes at least one type of readable storage medium, including flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the first memory 202 may be an internal storage unit of the electronic device 200, such as a mobile hard disk of the electronic device 200. In other embodiments, the first memory 202 may also be an external storage device of the electronic device 200, such as a plug-in mobile hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 200. Furthermore, the first memory 202 may include both an internal storage unit of the electronic device 200 and an external storage device. The first memory 202 can be used not only to store application software installed in the electronic device 200 and various data, such as the code of the underwater image enhancement program 203 based on color perception fusion attention and background light exclusion contrast learning, but also to temporarily store data that has been output or is about to be output.

[0126] In some embodiments, the first processor 201 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including a combination of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 201 is the control core (Control Unit) of the electronic device, connecting the various components of the entire electronic device using various interfaces and lines, and executing or executing programs or modules stored in the first memory 202, as well as calling data stored in the first memory 202, to perform various functions of the electronic device 200 and process data.

[0127] Figure 8 Only the electronic device with components is shown, and it can be understood by those skilled in the art that Figure 8 The structure shown does not constitute a limitation on the electronic device 200 , and the electronic device 200 may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0128] The underwater image enhancement program 203 based on color perception fusion attention and background light drive-off contrast learning stored in the first memory 202 of the electronic device 200 is a combination of multiple instructions. When running in the first processor 201, it can achieve the following:

[0129] Based on the color extraction module in the multi-color space restoration sub-network MCR, the input underwater image is converted into three color spaces: RGB, HSV, and LAB, to extract the features of each color space;

[0130] The color fusion module in the multi-color space restoration sub-network (MCR) is used to further extract and fuse the color information of the three color spaces. The color distortion of the underwater image is adaptively corrected using the color attention mechanism, and finally a fused color feature map is obtained.

[0131] The underwater original image is passed through the frequency domain enhancement sub-network FE, the input features are decomposed into low-frequency features and high-frequency features, the expression ability of high- and low-frequency features is enhanced, and a feature map after frequency domain enhancement is obtained; the frequency domain enhancement sub-network FE decomposes the input features into low-frequency features and high-frequency features through a dynamically learned filter, and improves the performance of the underwater image enhancement task by enhancing the expression ability of low-frequency features and high-frequency features;

[0132] The underwater original image is passed through the detail enhancement sub-network DE to further enhance the key area information in the image and increase the context relevance, thereby obtaining a feature map after detail enhancement; the frequency domain enhancement sub-network FE downsamples the input image by applying NAF-Block layer by layer to extract high-level semantic features, and restores the image detail information through upsampling operations, thereby achieving refined optimization of image features at both global and local levels;

[0133] The fused color feature map, the frequency domain enhanced feature map, and the detail enhanced feature map are fused to output a final underwater image enhancement result.

[0134] Furthermore, if the modules / units integrated in the electronic device 200 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0135] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0136] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0137] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. An underwater image enhancement method based on multi-color space and frequency domain guidance, characterized in that: The steps include: The color extraction module in the multi-color space restoration sub-network (MCR) converts the input underwater image into three color spaces: RGB, HSV, and LAB, to extract features from each color space. The color fusion module in the multi-color space restoration sub-network (MCR) is used to further extract and fuse the color information of the three color spaces. The color distortion of the underwater image is adaptively corrected using the color attention mechanism, and finally a fused color feature map is obtained. The underwater original image is passed through the frequency domain enhancement sub-network FE, the input features are decomposed into low-frequency features and high-frequency features, the expression ability of high- and low-frequency features is enhanced, and a feature map after frequency domain enhancement is obtained; the frequency domain enhancement sub-network FE decomposes the input features into low-frequency features and high-frequency features through a dynamically learned filter, and improves the performance of the underwater image enhancement task by enhancing the expression ability of low-frequency features and high-frequency features; The underwater original image is passed through the detail enhancement sub-network DE to further enhance the key area information in the image and increase the context relevance, thereby obtaining a feature map after detail enhancement. The detail enhancement sub-network DE downsamples the input image by applying NAF-Block layer by layer to extract high-level semantic features, and restores the image detail information through upsampling operations, thereby achieving refined optimization of image features at both global and local levels. The fused color feature map, the frequency domain enhanced feature map, and the detail enhanced feature map are fused to output a final underwater image enhancement result.

2. The underwater image enhancement method based on multi-color space and frequency domain guidance according to claim 1, characterized in that: The processing process of the color extraction module is specifically as follows: The convolutional layer Conv is used to extract features related to the color space, thereby obtaining a preliminary color feature representation; The obtained color features are passed to the color attention block (CAB) to extract and enhance key color information. In CAB, two convolutional layers are first used to further extract features. Nonlinearity is then introduced through the ReLU activation function to enhance the model's expressiveness. Attention weights are then generated through a convolutional layer and a Sigmoid activation function. The weighted features are then fused with the original features through element-wise multiplication to output feature maps in RGB, HSV, and LAB color spaces. Add the feature maps of RGB, HSV, and LAB color spaces element by element to generate a fused feature map F sum .

3. The underwater image enhancement method based on multi-color space and frequency domain guidance according to claim 1, characterized in that: The processing process of the color fusion module is specifically as follows: First, pass the convolution layer to represent the fusion feature map F after element-by-element addition sum Further feature extraction is performed to obtain a deep fusion feature map Then, the channel attention mechanism CA is used to enable the network to adaptively adjust the importance of different channel features. In CA, the deep fusion feature map is firstly integrated into the global average pooling operation. Compress in the spatial dimension to obtain the global features of each channel, and then generate channel attention weights through two convolutional layers and ReLU activation function; The channel attention weight is multiplied element-wise with the deep fusion feature map, and then the feature extraction and fusion are performed through the convolution layer to output the color-corrected fused color feature map.

4. The underwater image enhancement method based on multi-color space and frequency domain guidance according to claim 3, characterized in that: The formula for generating channel attention weights is as follows: Among them, GAP represents global average pooling, f ch It is a convolution operation using filters. σ and δ represent the Sigmoid activation function and the ReLU activation function respectively. Global average pooling GAP can integrate the global spatial information of the channel into a channel description. The specific process is as follows: in, Indicates the cth channel X at position (i, j) c The value of .

5. The underwater image enhancement method based on multi-color space and frequency domain guidance according to claim 1, characterized in that: The processing process of the frequency domain enhancement sub-network is as follows: The shallow features are extracted through the convolution layer to obtain the feature map F s ; For the feature map F s Perform global average pooling and convolution operations to generate a low-pass filter F l ; Subtract the low-pass filter from the identity matrix to obtain the high-pass filter F h ; Use the generated low-pass filter F l For the input feature F s Perform convolution operation to obtain low-frequency features F lo ; Use the generated high-pass filter F h For the input feature F s Perform convolution operation to obtain high-frequency features F hi ; The low-frequency feature F lo and high-frequency features F hi The input is respectively fed into the multi-branch convolutional unit and processed by convolution kernels of different sizes. Each convolution is followed by batch normalization and ReLU activation function to form a set of convolution-batch normalization-activation CBR units. Add the output feature maps of all branches element by element to generate fused features; Perform global average pooling (GAP) on the fused features to generate a global descriptor, and generate channel descriptors for each scale through two fully connected layers. The channel descriptors are reshaped and normalized to generate weights for each scale. Apply the weight of each scale to the corresponding convolution output and generate scale-enhanced features by weighted summation; Perform global average pooling on the scale enhancement features along the width and height directions respectively to obtain the width direction feature map F X and height directional feature map F Y ; The width feature map F X and height directional feature map F Y Stacked in the channel dimension, the first shape feature layer F is obtained X+Y , further processed by convolution, BatchNorm and activation function to obtain the second shape feature layer F′ X+Y ; After the Split operation, the feature layer F′ X+Y Divide into two parts and adjust them into shapes and Use convolution to adjust the number of channels and generate attention weights in width and height directions respectively through Sigmoid function. Extended to width direction features, Expanded to the height directional features, both directional feature maps are multiplied with the original feature map to obtain the final frequency domain enhancement information F enhance ; After obtaining the enhanced high-frequency features and low-frequency features, they are fused through the Concat operation to obtain the high- and low-frequency fusion feature map F lo+hi Finally, add the original input features element by element to obtain the output feature map F of the frequency domain enhancement sub-network fe .

6. The underwater image enhancement method based on multi-color space and frequency domain guidance according to claim 1, characterized in that: The processing process of the detail enhancement sub-network DE is specifically as follows: In NAF-Block, the underwater original image is processed by layer normalization LayerNorm to reduce internal covariate shift; The number of channels is reduced through 1×1 convolutional layers, which reduces computational complexity. Extract local features through 3×3 depth-wise separable convolution, which decomposes the standard convolution into depth-wise convolution and point-wise convolution, reducing the number of parameters and computational cost; Chunk operation is performed on the convolution output, evenly splitting it into two feature blocks along the channel dimension, and generating nonlinear features through element-by-element product; The spatial information of the feature map is aggregated through the global average pooling layer to generate a channel attention map to capture global information. The number of channels is adjusted through 1×1 convolution and multiplied element-by-element with the original feature map to achieve channel attention weighting. The weighted feature map is passed through a 1×1 convolutional layer to adjust the number of channels, and the layer normalization LayerNorm is used to stabilize the training; The trained results are chunked and weighted with channel attention again, and added to the input feature map through skip connections to output the enhanced detail feature map.

7. The underwater image enhancement method based on multi-color space and frequency domain guidance according to claim 6, characterized in that: The fused color feature map, the frequency domain enhanced feature map, and the detail enhanced feature map are added and fused element by element to finally obtain an output result.

8. An underwater image enhancement system based on multi-color space and frequency domain guidance, characterized by: The underwater image enhancement method based on multi-color space and frequency domain guidance applied to any one of claims 1-7 comprises a multi-color space restoration subnetwork, a frequency domain enhancement subnetwork module, and a detail enhancement subnetwork module, wherein the multi-color space restoration subnetwork comprises a color extraction module and a color fusion module; The color extraction module is used to convert the input underwater image into three color spaces: RGB, HSV and LAB, so as to extract the features of each color space; The color fusion module is used to further extract and fuse the color information of the three color spaces, and use the color attention mechanism to adaptively correct the color distortion of the underwater image, and finally obtain a fused color feature map; The frequency domain enhancement sub-network module is used to decompose the input features into low-frequency features and high-frequency features, enhance the expressiveness of high- and low-frequency features, and obtain a feature map after frequency domain enhancement; the frequency domain enhancement sub-network FE decomposes the input features into low-frequency features and high-frequency features through a dynamically learned filter, and improves the performance of underwater image enhancement tasks by enhancing the expressiveness of low-frequency features and high-frequency features; The detail enhancement sub-network module is used to further enhance the key area information in the image and increase the context relevance to obtain a feature map after detail enhancement; the frequency domain enhancement sub-network FE downsamples the input image by applying NAF-Block layer by layer to extract high-level semantic features, and restores the image detail information through upsampling operations, thereby achieving refined optimization of image features at both global and local levels; The fused color feature map, the frequency domain enhanced feature map, and the detail enhanced feature map are fused to output a final underwater image enhancement result.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the underwater image enhancement method based on multi-color space and frequency domain guidance as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the underwater image enhancement method based on multi-color space and frequency domain guidance according to any one of claims 1 to 7 is implemented.

Images