An underwater image enhancement method and system based on global pyramid linear modulation

Abstract

This invention discloses an underwater image enhancement method and system based on global pyramid linear modulation. The method includes: acquiring the original underwater image to be enhanced; extracting multi-scale features from the original underwater image using a global pyramid module to obtain a first feature map; modulating the feature map using a global linear modulation module to obtain a modulated second feature map; enhancing the second feature map using a MixDehazeNet backbone network to obtain a third feature map; enhancing the third feature map using an adaptive dynamic feature aggregation channel spatial attention module to obtain a fourth feature map; training a model based on the fourth feature map; and enhancing the original underwater image to be enhanced using the trained image enhancement model to obtain the enhanced underwater image. This invention can significantly improve the effect of underwater image enhancement while maintaining high computational efficiency, and can be widely applied in the field of image processing technology.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an underwater image enhancement method and system based on global pyramid linear modulation. Background Technology

[0002] Underwater image enhancement is a crucial technology essential for improving the visual quality and legibility of underwater images. This technology has wide applications in marine resource development, marine ecological research, underwater infrastructure monitoring, and naval operations. However, the light absorption and scattering characteristics of the underwater environment pose significant challenges to underwater image enhancement. These phenomena lead to problems such as color distortion, reduced contrast, blurred outlines, and increased noise in the images.

[0003] To overcome the challenges of underwater imaging, the academic community has proposed various image enhancement techniques. However, existing techniques suffer from the following drawbacks:

[0004] 1. Limited feature representation capability:

[0005] Existing underwater image enhancement methods, such as FiLM-based methods, mainly employ two-dimensional feature modulation, which cannot fully capture the complex features of the image. This limits the model's ability to process subtle details and complex structures in underwater images.

[0006] 2. Insufficient utilization of multi-scale information:

[0007] While some methods, such as FPN, have considered multi-scale features, they typically employ simple feature fusion strategies that fail to fully utilize information from different scales. This results in inconsistent performance when dealing with underwater targets of varying sizes and complexities.

[0008] 3. The attention mechanism is inefficient:

[0009] Existing attention mechanisms, such as channel attention and spatial attention, can improve model performance, but they are computationally complex and not suitable for real-time processing. In addition, they are often applied independently, lacking effective integration strategies.

[0010] 4. Lack of adaptability:

[0011] Most existing methods use fixed network structures and parameters, making it difficult to adapt to changes in different underwater environments and imaging conditions. This limits the effectiveness of the models in complex and variable real-world underwater scenarios.

[0012] 5. Insufficient generalization ability to real-world data:

[0013] Many methods perform well on synthetic datasets but poorly on real-world underwater images. This reflects the insufficient generalization ability of existing methods when dealing with complex and variable real-world underwater environments. Summary of the Invention

[0014] The main objective of this invention is to propose an efficient underwater image enhancement method and system based on global pyramid linear modulation, which can significantly improve the effect of underwater image enhancement.

[0015] To achieve the above objectives, one aspect of this invention proposes an underwater image enhancement method based on global pyramid linear modulation, comprising the following steps:

[0016] Obtain the original underwater image to be enhanced;

[0017] The first feature map is obtained by extracting multi-scale features from the original underwater image using the global pyramid module;

[0018] The feature map is modulated by a global linear modulation module to obtain a modulated second feature map.

[0019] The second feature map is enhanced using the MixDehazeNet backbone network to obtain the third feature map; wherein, the MixDehazeNet backbone network integrates a global pyramid linear modulation block;

[0020] The third feature map is enhanced by the adaptive dynamic feature aggregation channel spatial attention module to obtain the fourth feature map;

[0021] The model is trained based on the fourth feature map, and the trained image enhancement model is used to enhance the original underwater image to be enhanced, resulting in an enhanced underwater image.

[0022] In some embodiments, the step of extracting multi-scale features from the original underwater image using a global pyramid module to obtain a first feature map includes the following steps:

[0023] Feature extraction is performed on the original underwater image using depth-separable convolutional blocks; wherein, the depth-separable convolutional blocks consist of depthwise convolution and pointwise convolution; the expression for this process is:

[0024] Y=PointwiseConv(DepthwiseConv(X))

[0025] Where X is the input feature map and Y is the output feature map;

[0026] Based on the feature pyramid network structure, interpolation and element-wise addition are used to perform multi-scale feature fusion on the extracted features. The feature pyramid network structure fuses features at different levels through top-down paths and lateral connections. The expression for this process is:

[0027] P l =Conv(C l +Upsample(P l+1 ))

[0028] Among them, P l It is the feature map of the l-th layer, C l It is the corresponding convolutional feature map;

[0029] The fused features are augmented using a triple attention module to obtain the first feature map. This triple attention module comprises three parallel attention branches, focusing on the width and height of the channel dimension and spatial dimension, respectively. The expression for this process is as follows:

[0030]

[0031] in, SpatialAtt represents element-wise multiplication. W SpatialAtt H ChannelAtt and ChannelAtt represent spatial attention in the width direction, spatial attention in the height direction, and channel attention, respectively.

[0032] In some embodiments, the modulation processing of the feature map by a global linear modulation module to obtain a modulated second feature map includes the following steps:

[0033] Global feature extraction is performed on the first feature map using a feature extraction network;

[0034] Modulation parameters are generated using a modulation parameter generation network;

[0035] Based on the extracted global features and generated modulation parameters, feature modulation is performed through a feature modulation unit; the expression for this process is:

[0036] Y = γ⊙X + β

[0037] Where X is the input feature map, Y is the modulated feature map, γ and β are the learned modulation parameters, and ⊙ represents element-wise multiplication.

[0038] In some embodiments, the step of enhancing the second feature map according to the MixDehazeNet backbone network to obtain the third feature map includes the following steps:

[0039] The global pyramid linear modulation block is integrated into the MixDehazeNet backbone network;

[0040] Image features are extracted using the encoder of the MixDehazeNet backbone network;

[0041] The enhanced image is reconstructed using the decoder of the MixDehazeNet backbone network;

[0042] Skip connection blocks in the MixDehazeNet backbone are used to preserve detailed information during the image enhancement process.

[0043] In some embodiments, the adaptive dynamic feature aggregation channel spatial attention module includes a global average pooling and global max pooling module, a multilayer perceptron, and channel attention and spatial attention units;

[0044] The expression for the processing procedure of the adaptive dynamic feature aggregation channel spatial attention module is as follows:

[0045]

[0046] Where X is the feature map before enhancement, Y is the feature map after enhancement; W c and W s The weights are learned through a multilayer perceptron.

[0047] In some embodiments, the global average pooling and global max pooling modules are used to extract global statistical information from the input third feature map; specifically, they include global average pooling and global max pooling operations:

[0048] The global average pooling operation is used to calculate the average value for each channel, capturing the overall activation level of the feature map. The expression for this process is:

[0049]

[0050] Where GAP represents global average pooling; X is the input feature map, and H and W are the height and width of the feature map, respectively; global max pooling is used to extract the maximum value of each channel, highlighting salient features, and the expression for this process is:

[0051] GMP(X) = max(X)

[0052] GMP stands for Global Max Pooling Operation;

[0053] The multilayer perceptron is used to learn adaptive weights for channel attention and spatial attention; specifically, it includes:

[0054] 1) Receive feature vectors from the global pooling module, perform nonlinear transformations, and learn higher-level feature representations;

[0055] 2) Generate parameters for dynamically adjusting channel attention and spatial attention weights; the expression for this process is:

[0056] W c W s =MLP(concat(GAP(X),GMP(X)))

[0057] Among them, W c and W s These are the weights for channel attention and spatial attention, respectively.

[0058] 3) By using an adaptive weight learning mechanism, attention allocation is dynamically adjusted according to the characteristics of the input features, thereby improving the flexibility and adaptability of the model;

[0059] The channel attention and spatial attention units are used to perform fine-grained feature enhancement on the feature map.

[0060] In some embodiments, the method further includes a training step for the image enhancement model, which includes the following steps:

[0061] The underwater image data to be trained is preprocessed to obtain training samples;

[0062] Perform model initialization operations to obtain the initial model;

[0063] The training samples are input into the initial model and forward propagation, loss calculation, back propagation and parameter update are performed sequentially to train the image enhancement model.

[0064] The image enhancement model is evaluated, and the image enhancement model that meets the evaluation requirements is saved.

[0065] Another aspect of this invention provides an underwater image enhancement system based on global pyramid linear modulation, comprising:

[0066] The first module is used to acquire the original underwater image to be enhanced;

[0067] The second module is used to extract multi-scale features from the original underwater image through the global pyramid module to obtain a first feature map.

[0068] The third module is used to modulate the feature map through a global linear modulation module to obtain a modulated second feature map.

[0069] The fourth module is used to enhance the second feature map according to the MixDehazeNet backbone network to obtain the third feature map; wherein, the MixDehazeNet backbone network integrates a global pyramid linear modulation block;

[0070] The fifth module is used to enhance the third feature map through the adaptive dynamic feature aggregation channel spatial attention module to obtain the fourth feature map;

[0071] The sixth module is used to train the model based on the fourth feature map, and to enhance the original underwater image to be enhanced based on the trained image enhancement model, so as to obtain the enhanced underwater image.

[0072] Another aspect of the present invention provides an electronic device, including a processor and a memory;

[0073] The memory is used to store programs;

[0074] The processor executes the program to implement the method described above.

[0075] Another aspect of this invention provides a computer-readable storage medium storing a program that is executed by a processor to implement the methods described above.

[0076] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0077] The embodiments of this invention include at least the following beneficial effects: This invention provides an underwater image enhancement method and system based on global pyramid linear modulation. The invention acquires the original underwater image to be enhanced; extracts multi-scale features from the original underwater image using a global pyramid module to obtain a first feature map; modulates the feature map using a global linear modulation module to obtain a modulated second feature map; enhances the second feature map using a MixDehazeNet backbone network to obtain a third feature map; wherein the MixDehazeNet backbone network integrates a global pyramid linear modulation block; enhances the third feature map using an adaptive dynamic feature aggregation channel spatial attention module to obtain a fourth feature map; trains a model based on the fourth feature map, and enhances the original underwater image to be enhanced using the trained image enhancement model to obtain the enhanced underwater image. This invention can significantly improve the effect of underwater image enhancement while maintaining high computational efficiency. Attached Figure Description

[0078] Figure 1 This is a schematic diagram of an implementation environment provided by an embodiment of the present invention;

[0079] Figure 2 This is a flowchart of the overall steps provided in the embodiments of the present invention;

[0080] Figure 3 This is a schematic diagram of the overall framework provided in an embodiment of the present invention;

[0081] Figure 4 This is a schematic diagram of the structure of the Global Pyramid Module (GPM) provided in an embodiment of the present invention;

[0082] Figure 5 This is a schematic diagram of the adaptive dynamic feature aggregation channel spatial attention module provided in an embodiment of the present invention;

[0083] Figure 6 This is a flowchart of the model training process provided in an embodiment of the present invention;

[0084] Figure 7 This is a comparison chart of the enhancement results on the EUVP-515 and UFO-120 datasets provided in this embodiment of the invention;

[0085] Figure 8 This is a comparison chart of the enhanced results on the UIEB90 dataset provided in this embodiment of the invention. Detailed Implementation

[0086] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.

[0087] It is understood that the terms “first,” “second,” etc., used in this invention may be used herein to describe various concepts, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another. For example, first information may also be referred to as second information without departing from the scope of embodiments of the invention, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to determination” as used herein may be interpreted as “when…” or “when…” or “in response to determination.”

[0088] The terms “at least one,” “multiple,” “each,” “any,” etc., used in this invention, “at least one” includes one, two, or more than two; “multiple” includes two or more than two; “each” refers to each of the corresponding multiple; and “any” refers to any one of the multiple.

[0089] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0090] Before providing a detailed description of the embodiments of the present invention, some related technologies involved in the embodiments of the present invention will be described first, as follows:

[0091] 1. Global Pyramid Linear Modulation (GPLM): A novel neural network modulation method for underwater image enhancement models. It employs a Feature Pyramid Network (FPN) structure, combining self-attention and feature fusion mechanisms to generate 3D feature maps, and then modulates these feature maps through affine transformations, thereby affecting the computation of the neural network.

[0092] 2. Feature-wise Linear Modulation (FiLM): A technique for regulating neural network computation, achieved through feature affine transformation based on external information.

[0093] 3. Feature Pyramid Network (FPN): A network architecture that utilizes the inherent multi-scale, pyramidal hierarchical structure of convolutional neural networks to construct feature pyramids with strong semantic information.

[0094] 4. Global Pyramid Module (GPM): A key component of GPLM, used to extract 3D feature maps from the input image.

[0095] 5. Global Linear Modulation (GLM): Another key component in GPLM, used to apply affine transformations to feature maps.

[0096] 6. Adaptive Dynamic Feature Aggregation Channel-Spatial Attention Module (ADFA-CSAM): An innovative feature fusion technique for adaptively learning aggregation weights for high-dimensional and low-dimensional features.

[0097] 7. Peak Signal-to-Noise Ratio (PSNR): An objective metric used to evaluate image quality, measured in decibels (dB).

[0098] 8. Underwater Image Quality Measure (UIQM): A non-reference metric specifically designed to evaluate the quality of underwater images, taking into account color, sharpness, and contrast.

[0099] 9. Underwater Color Image Quality Evaluation (UCIQE): Another non-reference metric for evaluating the quality of underwater images.

[0100] 10. Depthwise Separable Convolution: A computationally efficient convolution operation that breaks down standard convolution into two steps: depthwise convolution and pointwise convolution.

[0101] 11. Triplet Attention Module (TA): A lightweight self-attention mechanism used to enhance feature representations.

[0102] 12. MixDehazeNet: A deep learning model for image dehazing. This invention integrates GPLM into the model to improve performance.

[0103] The underwater image enhancement method and system based on global pyramid linear modulation provided in this invention relates to the field of image processing technology. The underwater image enhancement method based on global pyramid linear modulation provided in this invention can be applied to a terminal, a server, or software running on a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the underwater image enhancement method based on global pyramid linear modulation, but is not limited to the above forms.

[0104] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0105] like Figure 1 The diagram shown is a schematic representation of an implementation environment provided by an embodiment of the present invention. (Refer to...) Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected via a network, either wirelessly or via a wired connection, to complete data transmission and exchange.

[0106] Server 101 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0107] Additionally, server 101 can also be a node server in a blockchain network. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms.

[0108] Terminal 102 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc. It can also be a vehicle-mounted terminal of the various device types described above, but is not limited to these. Terminal 102 and server 101 can be directly or indirectly connected via wired or wireless communication, and this embodiment of the invention does not impose any limitations.

[0109] Exemplary based on Figure 1 The implementation environment shown in this embodiment of the invention provides an underwater image enhancement method based on global pyramid linear modulation. The following description uses the application of this underwater image enhancement method based on global pyramid linear modulation in server 101 as an example. It can be understood that this method can also be applied to terminal 102.

[0110] Reference Figure 2 , Figure 2 The flowchart illustrates an underwater image enhancement method based on global pyramid linear modulation applied to a server, as provided in this embodiment of the invention. The execution entity of this method can be any of the aforementioned computer devices (including a server or a terminal). (Refer to...) Figure 2 The method may include the following steps:

[0111] Obtain the original underwater image to be enhanced;

[0112] The first feature map is obtained by extracting multi-scale features from the original underwater image using the global pyramid module;

[0113] The feature map is modulated by a global linear modulation module to obtain a modulated second feature map.

[0114] The second feature map is enhanced using the MixDehazeNet backbone network to obtain the third feature map; wherein, the MixDehazeNet backbone network integrates a global pyramid linear modulation block;

[0115] The third feature map is enhanced by the adaptive dynamic feature aggregation channel spatial attention module to obtain the fourth feature map;

[0116] The model is trained based on the fourth feature map, and the trained image enhancement model is used to enhance the original underwater image to be enhanced, resulting in an enhanced underwater image.

[0117] In some embodiments, the step of extracting multi-scale features from the original underwater image using a global pyramid module to obtain a first feature map includes the following steps:

[0118] Feature extraction is performed on the original underwater image using depth-separable convolutional blocks; wherein, the depth-separable convolutional blocks consist of depthwise convolution and pointwise convolution; the expression for this process is:

[0119] Y=PointwiseConv(DepthwiseConv(X))

[0120] Where X is the input feature map and Y is the output feature map;

[0121] Based on the feature pyramid network structure, interpolation and element-wise addition are used to perform multi-scale feature fusion on the extracted features. The feature pyramid network structure fuses features at different levels through top-down paths and lateral connections. The expression for this process is:

[0122] P l =Conv(C l +Upsample(P l+1 ))

[0123] Among them, P l It is the feature map of the l-th layer, C l It is the corresponding convolutional feature map;

[0124] The fused features are augmented using a triple attention module to obtain the first feature map. This triple attention module comprises three parallel attention branches, focusing on the width and height of the channel dimension and spatial dimension, respectively. The expression for this process is as follows:

[0125]

[0126] in, SpatialAtt represents element-wise multiplication. W SpatialAtt H ChannelAtt and ChannelAtt represent spatial attention in the width direction, spatial attention in the height direction, and channel attention, respectively.

[0127] In some embodiments, the modulation processing of the feature map by a global linear modulation module to obtain a modulated second feature map includes the following steps:

[0128] Global feature extraction is performed on the first feature map using a feature extraction network;

[0129] Modulation parameters are generated using a modulation parameter generation network;

[0130] Based on the extracted global features and generated modulation parameters, feature modulation is performed through a feature modulation unit; the expression for this process is:

[0131] Y = γ⊙X + β

[0132] Where X is the input feature map, Y is the modulated feature map, γ and β are the learned modulation parameters, and ⊙ represents element-wise multiplication.

[0133] In some embodiments, the step of enhancing the second feature map according to the MixDehazeNet backbone network to obtain the third feature map includes the following steps:

[0134] The global pyramid linear modulation block is integrated into the MixDehazeNet backbone network;

[0135] Image features are extracted using the encoder of the MixDehazeNet backbone network;

[0136] The enhanced image is reconstructed using the decoder of the MixDehazeNet backbone network;

[0137] Skip connection blocks in the MixDehazeNet backbone are used to preserve detailed information during the image enhancement process.

[0138] In some embodiments, the adaptive dynamic feature aggregation channel spatial attention module includes a global average pooling and global max pooling module, a multilayer perceptron, and channel attention and spatial attention units;

[0139] The expression for the processing procedure of the adaptive dynamic feature aggregation channel spatial attention module is as follows:

[0140]

[0141] Where X is the feature map before enhancement, Y is the feature map after enhancement; W c and W s The weights are learned through a multilayer perceptron.

[0142] In some embodiments, the global average pooling and global max pooling modules are used to extract global statistical information from the input third feature map; specifically, they include global average pooling and global max pooling operations:

[0143] The global average pooling operation is used to calculate the average value for each channel, capturing the overall activation level of the feature map. The expression for this process is:

[0144]

[0145] Where GAP represents global average pooling; X is the input feature map, and H and W are the height and width of the feature map, respectively; global max pooling is used to extract the maximum value of each channel, highlighting salient features, and the expression for this process is:

[0146] GMP(X) = max(X)

[0147] GMP stands for Global Max Pooling Operation;

[0148] The multilayer perceptron is used to learn adaptive weights for channel attention and spatial attention; specifically, it includes:

[0149] 1) Receive feature vectors from the global pooling module, perform nonlinear transformations, and learn higher-level feature representations;

[0150] 2) Generate parameters for dynamically adjusting channel attention and spatial attention weights; the expression for this process is:

[0151] W c W s =MLP(concat(GAP(X),GMP(X)))

[0152] Among them, W c and W s These are the weights for channel attention and spatial attention, respectively.

[0153] 3) By using an adaptive weight learning mechanism, attention allocation is dynamically adjusted according to the characteristics of the input features, thereby improving the flexibility and adaptability of the model;

[0154] The channel attention and spatial attention units are used to perform fine-grained feature enhancement on the feature map.

[0155] In some embodiments, the method further includes a training step for the image enhancement model, which includes the following steps:

[0156] The underwater image data to be trained is preprocessed to obtain training samples;

[0157] Perform model initialization operations to obtain the initial model;

[0158] The training samples are input into the initial model and forward propagation, loss calculation, back propagation and parameter update are performed sequentially to train the image enhancement model.

[0159] The image enhancement model is evaluated, and the image enhancement model that meets the evaluation requirements is saved.

[0160] The implementation process of the method of the present invention in a specific application scenario will be described in detail below with reference to the accompanying drawings:

[0161] This invention proposes an underwater image enhancement method and system based on Global Pyramid Linear Modulation (GPLM). The core idea of ​​this method is to extend Feature Linear Modulation (FiLM) from two dimensions to four dimensions and combine it with a Feature Pyramid Network (FPN) structure to achieve finer-grained and multi-scale feature modulation.

[0162] like Figure 3 As shown, the overall framework of this invention includes the following main components:

[0163] ① Input module;

[0164] ② Global Pyramid Module (GPM);

[0165] ③ Global Linear Modulation (GLM);

[0166] ④MixDehazeNet backbone network;

[0167] ⑤ Output module.

[0168] The following is a detailed description of each major component:

[0169] 2. Global Pyramid Module (GPM)

[0170] The global pyramid module is one of the core components of this invention, used to extract multi-scale features from the input image. For example... Figure 4 As shown, the structure of GPM includes:

[0171] ①Depthly separable convolutional (DSC) blocks: used for efficient feature extraction;

[0172] ② Feature Pyramid Network (FPN) structure: used for multi-scale feature fusion;

[0173] ③ Triplet Attention (TA) module: used to enhance feature representation.

[0174] 2.1 Depthwise Separable Convolutional (DSC) Blocks

[0175] Depthwise separable convolutional blocks consist of depthwise convolution and pointwise convolution, which can maintain good performance while reducing computational complexity. The mathematical expression for a DSC block is as follows:

[0176] Y=PointwiseConv(DepthwiseConv(X))

[0177] Where X is the input feature map and Y is the output feature map.

[0178] 2.2 Feature Pyramid Network (FPN) Structure

[0179] The FPN structure fuses features from different levels through a top-down path and lateral connections. In this invention, embodiments use interpolation operations and element-wise addition to achieve feature fusion:

[0180] P l =Conv(C l +Upsample(P l+1 ))

[0181] Among them, P l It is the feature map of the l-th layer, C l It is the corresponding convolutional feature map.

[0182] 2.3 Triple Attention (TA) Module

[0183] The Triple Attention module contains three parallel attention branches, focusing on the width and height of the channel dimension, spatial dimension, and so on, respectively. The output of the TA module can be represented as:

[0184]

[0185] in, SpatialAtt represents element-wise multiplication. W SpatialAtt H ChannelAtt and ChannelAtt represent spatial attention in the width direction, spatial attention in the height direction, and channel attention, respectively.

[0186] 3. Global Linear Modulation Module (GLM)

[0187] The global linear modulation module is responsible for modulating the feature map generated by GPM, such as... Figure 3 As shown, GLM includes:

[0188] ① Feature extraction network: used to extract global features from the input image;

[0189] ② Modulation parameter generation network: used to generate modulation parameters γ and β;

[0190] ③ Feature modulation unit: used to perform the actual feature modulation operation.

[0191] The mathematical expression for GLM can be represented as:

[0192] Y = γ⊙X + β

[0193] Where X is the input feature map, Y is the modulated feature map, γ and β are the learned modulation parameters, and ⊙ represents element-wise multiplication.

[0194] 4. MixDehazeNet backbone network

[0195] This invention integrates GPLM into MixDehazeNet to improve the performance of underwater image enhancement. The main components of MixDehazeNet include:

[0196] ① Encoder: Used to extract image features;

[0197] ② Decoder: Used to reconstruct the enhanced image;

[0198] ③ Skip connections: used to preserve detailed information.

[0199] After each MixStructureBlock, embodiments of the present invention insert a GPLM module to modulate the features. Figure 3 The MixDehazeNet architecture integrating GPLM is demonstrated.

[0200] 5. Adaptive Dynamic Feature Aggregation Channel Spatial Attention Module (ADFA-CSAM)

[0201] To further improve the feature fusion effect, this invention proposes an ADFA-CSAM module. For example... Figure 5 As shown, this module includes:

[0202] ① Global Average Pooling (GAP) and Global Max Pooling (GMP);

[0203] ② Multilayer Perceptron (MLP);

[0204] ③ Channel attention and spatial attention units.

[0205] The mathematical expression for ADFA-CSAM can be represented as:

[0206]

[0207] Among them, W c and W sThe weights are obtained through MLP learning.

[0208] Specifically, the global average pooling and global max pooling modules are used to extract global statistical information from the input third feature map. Specifically:

[0209] 1) Global Average Pooling (GAP) is used to calculate the average value for each channel, capturing the overall activation level of the feature map. Its operation can be represented as:

[0210]

[0211] Where X is the input feature map, and H and W are the height and width of the feature map, respectively.

[0212] 2) Global Max Pooling (GMP) is used to extract the maximum value of each channel, highlighting salient features. Its operation can be represented as:

[0213] GMP(X) = max(X)

[0214] By using GAP and GMP simultaneously, this module can comprehensively capture global information of the feature map, providing important statistical basis for subsequent attention mechanisms.

[0215] The multilayer perceptron is used to learn adaptive weights for channel attention and spatial attention. Specific functions include:

[0216] 1) Receive feature vectors from the global pooling module, perform nonlinear transformations, and learn higher-level feature representations.

[0217] 2) Generate parameters for dynamically adjusting channel attention and spatial attention weights. This operation can be represented as:

[0218] W c W s =MLP(concat(GAP(X),GMP(W)))

[0219] Among them, W c and W s These are the weights for channel attention and spatial attention, respectively.

[0220] 3) Through this adaptive weight learning mechanism, ADFA-CSAM can dynamically adjust the attention allocation according to the characteristics of the input features, thereby improving the flexibility and adaptability of the model.

[0221] The channel attention and spatial attention units are used for fine-grained feature enhancement of the feature map. Specifically:

[0222] 1) The channel attention unit focuses on "what" are useful features, highlighting important features by recalibrating the dependencies between channels. Its operation can be represented as:

[0223] ChannelAtt(X)=σ(MLP(GAP(X)))

[0224] Where σ represents the sigmoid activation function.

[0225] 2) Spatial attention units focus on features that are useful "where," enhancing key areas by learning the importance of spatial location. Its operation can be represented as:

[0226] SpatialAtt(X)=σ(Conv(concat(AvgPool(X),MaxPool(X))))

[0227] 3) The final attention enhancement features are obtained in the following way:

[0228]

[0229] in, This indicates element-wise multiplication.

[0230] For the third feature map generated earlier, ADFA-CSAM can extract and fuse multi-scale feature information to enhance the model's ability to process targets of different sizes; through an adaptive attention mechanism, it highlights important features and key regions and suppresses irrelevant information; and dynamically adjusts the weights of high-dimensional and low-dimensional features to achieve more effective feature fusion.

[0231] For subsequent model training steps, the ADFA-CSAM module can provide richer and more discriminative feature representations, which helps the model learn the essential features of underwater images better; it improves the model's generalization ability through an adaptive mechanism, enabling it to better adapt to different underwater environments and imaging conditions; and it reduces computational complexity while maintaining high performance, thereby improving the model's training and inference efficiency.

[0232] 6. Training Process

[0233] The training process of this invention is as follows: Figure 6 The flowchart shown below. The main steps include:

[0234] ① Data preprocessing;

[0235] ② Model initialization;

[0236] ③ Forward propagation;

[0237] ④ Loss calculation;

[0238] ⑤ Backpropagation and parameter update;

[0239] ⑥ Model evaluation and storage.

[0240] During training, this embodiment of the invention uses the L1 loss function to optimize the model:

[0241] l = |I out -I GT |1

[0242] Among them, I out It is the model output, I GT It's a real label.

[0243] 7. Experimental Results

[0244] To verify the effectiveness of the method of the present invention, experiments were conducted on multiple datasets in the embodiments of the present invention. Figure 7 and Figure 8 This section showcases some of the enhanced results on the EUVP, UFO, and UIEB datasets.

[0245] Experimental results show that the method of the present invention has achieved significant improvements in metrics such as peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and underwater image quality metric (UIQM).

[0246] The underwater image enhancement method proposed in this invention, based on global pyramid linear modulation, effectively improves the quality of underwater images by introducing innovative modules such as GPLM, GPM, and ADFA-CSAM. This method not only achieves excellent performance on multiple benchmark datasets but also demonstrates good generalization ability and robustness in real-world underwater scenarios.

[0247] In summary, compared with the prior art, the present invention has the following characteristics:

[0248] 1. Improve feature representation capabilities:

[0249] This invention aims to enhance the model's ability to represent complex underwater image features by introducing the Global Pyramid Linear Modulation (GPLM) method to extend feature modulation from two dimensions to four dimensions.

[0250] 2. Make full use of multi-scale information:

[0251] The purpose of this invention is to design an efficient Global Pyramid Module (GPM) that combines a Feature Pyramid Network (FPN) structure with a self-attention mechanism to better integrate and utilize multi-scale feature information.

[0252] 3. Improve the efficiency of attention mechanisms:

[0253] The present invention aims to introduce a lightweight and efficient triple attention module (TA) to reduce computational complexity while maintaining performance, thereby achieving more efficient feature enhancement.

[0254] 4. Enhance the model's adaptability:

[0255] By designing an Adaptive Dynamic Feature Aggregation Channel Spatial Attention Module (ADFA-CSAM), the purpose of this invention is to enable the model to dynamically adjust the processing strategy according to the features of the input image, thereby improving its adaptability to different underwater environments.

[0256] 5. Improve the ability to process real-world data:

[0257] This invention aims to improve the generalization ability and robustness of the model in real underwater environments by validating and optimizing it on multiple benchmark datasets and real-world datasets (such as DanGan1190).

[0258] By implementing these technologies, this invention aims to significantly improve the effect of underwater image enhancement while maintaining high computational efficiency, providing new ideas and methods for the development of underwater image processing technology.

[0259] Compared to existing technologies, the Global Pyramid Linear Modulation (GPLM) method proposed in this invention represents a qualitative leap in feature representation and modulation capabilities. Compared to existing Feature Linear Modulation (FiLM)-based methods, GPLM extends feature modulation from two dimensions to four dimensions and incorporates a Feature Pyramid Network (FPN) structure, achieving finer-grained and multi-scale feature modulation. This innovative design enables the model to better capture and process complex features and details in underwater images, resulting in superior enhancement effects across various underwater environments and imaging conditions. Particularly when processing real-world underwater images, this invention demonstrates stronger generalization ability and robustness, which is lacking in many existing methods.

[0260] Secondly, this invention achieves a better balance between computational efficiency and performance. By introducing depthwise separable convolution and a lightweight triple attention module, as well as an innovative adaptive dynamic feature aggregation channel spatial attention module (ADFA-CSAM), this invention significantly reduces computational complexity while maintaining high performance. This allows the method to achieve excellent enhancement results not only on high-end devices but also to run efficiently on resource-constrained devices, greatly expanding its application scope. Furthermore, through ingenious integration with MixDehazeNet, this invention achieves layer-by-layer feature modulation and enhancement, further improving the model's ability to process underwater image details and structures. Its performance on multiple benchmark datasets outperforms state-of-the-art methods such as PUGAN and DGNet. In particular, this invention achieves significant improvements of over 3dB and 1 in key metrics such as PSNR and UIQM, fully demonstrating its leading position in the field of underwater image enhancement.

[0261] Another aspect of this invention provides an underwater image enhancement system based on global pyramid linear modulation, comprising:

[0262] The first module is used to acquire the original underwater image to be enhanced;

[0263] The second module is used to extract multi-scale features from the original underwater image through the global pyramid module to obtain a first feature map.

[0264] The third module is used to modulate the feature map through a global linear modulation module to obtain a modulated second feature map.

[0265] The fourth module is used to enhance the second feature map according to the MixDehazeNet backbone network to obtain the third feature map; wherein, the MixDehazeNet backbone network integrates a global pyramid linear modulation block;

[0266] The fifth module is used to enhance the third feature map through the adaptive dynamic feature aggregation channel spatial attention module to obtain the fourth feature map;

[0267] The sixth module is used to train the model based on the fourth feature map, and to enhance the original underwater image to be enhanced based on the trained image enhancement model, so as to obtain the enhanced underwater image.

[0268] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0269] This invention also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned underwater image enhancement method based on global pyramid linear modulation. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0270] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0271] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described underwater image enhancement method based on global pyramid linear modulation.

[0272] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0273] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0274] The embodiments described in this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0275] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present invention, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0276] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0277] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0278] The terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0279] It should be understood that in this invention, "at least one (item)" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0280] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0281] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0282] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0283] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0284] The preferred embodiments of the present invention have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of the present invention should be within the scope of the claims of the present invention.

Claims

1. An underwater image enhancement method based on global pyramid linear modulation, characterized in that, The method comprises the following steps: obtaining an original underwater image to be enhanced; extracting multi-scale features from the original underwater image through a global pyramid module to obtain a first feature map; modulating the feature map through a global linear modulation module to obtain a second feature map after modulation; enhancing the second feature map according to a MixDehazeNet backbone network to obtain a third feature map; wherein the global pyramid linear modulation block is integrated into the MixDehazeNet backbone network; enhancing the third feature map through an adaptive dynamic feature aggregation channel and spatial attention module to obtain a fourth feature map; training a model according to the fourth feature map, and enhancing the original underwater image to be enhanced according to an image enhancement model obtained through training to obtain an enhanced underwater image; the global pyramid module extracts multi-scale features from the original underwater image to obtain a first feature map, which comprises the following steps: extracting features from the original underwater image through a depth separable convolution block; wherein the depth separable convolution block is composed of a depth convolution and a point-by-point convolution; performing multi-scale feature fusion on the extracted features through an interpolation operation and an element-level addition according to a feature pyramid network structure; wherein the feature pyramid network structure fuses features of different levels through a top-down path and a horizontal connection; performing feature enhancement representation on the fused features through a triple attention module to obtain a first feature map; wherein the triple attention module contains three parallel attention branches, which respectively focus on the channel dimension, the width and the height of the spatial dimension; the expression of the process is: wherein, denotes element-wise multiplication, , and denote spatial attention in the width direction, spatial attention in the height direction, and channel attention, respectively. the global linear modulation module modulates the feature map to obtain a second feature map after modulation, which comprises the following steps: extracting global features from the first feature map through a feature extraction network; generating modulation parameters through a modulation parameter generation network; performing feature modulation operation through a feature modulation unit according to the extracted global features and the generated modulation parameters; the MixDehazeNet backbone network enhances the second feature map to obtain a third feature map, which comprises the following steps: integrating the global pyramid linear modulation block into the MixDehazeNet backbone network; extracting image features through the encoder of the MixDehazeNet backbone network; reconstructing the enhanced image through the decoder of the MixDehazeNet backbone network; retaining the detail information in the image enhancement process through the skip connection block in the MixDehazeNet backbone network; the adaptive dynamic feature aggregation channel and spatial attention module comprises a global average pooling and global maximum pooling module, a multi-layer perception, and a channel attention and spatial attention unit; the expression of the processing process of the adaptive dynamic feature aggregation channel and spatial attention module is: wherein, and are weights obtained by learning with a multi-layer perceptron.

2. The underwater image enhancement method based on global pyramid linear modulation according to claim 1, wherein The global average pooling and global maximum pooling module is configured to extract global statistical information from the input third feature map, and specifically includes a global average pooling operation and a global maximum pooling operation. The global average pooling operation is configured to calculate the average value of each channel to capture the overall activation level of the feature map. The global maximum pooling operation is configured to extract the maximum value of each channel to highlight the significant features. The multi-layer perceptron is configured to learn adaptive weights of channel attention and spatial attention, and specifically includes: 1) receiving the feature vector from the global pooling module, performing nonlinear transformation, and learning higher-level feature representation; 2) generating parameters for dynamically adjusting the weights of channel attention and spatial attention; 3) dynamically adjusting the attention allocation according to the characteristics of the input features through an adaptive weight learning mechanism, thereby improving the flexibility and adaptability of the model; The channel attention and spatial attention unit is configured to perform fine-grained feature enhancement on the feature map.

3. The underwater image enhancement method based on global pyramid linear modulation according to claim 1, characterized in that, The method further includes a training step of the image enhancement model, which includes the following steps: Preprocessing the underwater image data to be trained to obtain training samples; Performing model initialization to obtain an initial model; Inputting the training samples into the initial model to sequentially perform forward propagation processing, loss calculation, back propagation, and parameter updating, thereby training the image enhancement model; Model evaluation is performed on the image enhancement model, and the image enhancement model that meets the evaluation requirements is saved.

4. A system for implementing the method of underwater image enhancement based on global pyramid linear modulation according to any one of claims 1-3, characterized in that, It includes: A first module for obtaining an original underwater image to be enhanced; A second module for performing multi-scale feature extraction on the original underwater image through a global pyramid module to obtain a first feature map; A third module for performing modulation processing on the feature map through a global linear modulation module to obtain a second feature map after modulation; A fourth module for performing enhancement processing on the second feature map according to a MixDehazeNet backbone network to obtain a third feature map; wherein the MixDehazeNet backbone network integrates a global pyramid linear modulation block; A fifth module for performing enhancement processing on the third feature map through an adaptive dynamic feature aggregation channel and spatial attention module to obtain a fourth feature map; A sixth module for model training according to the fourth feature map, and enhancing the original underwater image to be enhanced according to the trained image enhancement model to obtain an enhanced underwater image.

5. An electronic device, comprising: It includes a processor and a memory; The memory is configured to store a program; The processor executes the program to implement the method of any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The storage medium stores a program, and the program is executed by the processor to implement the method of any one of claims 1-3.

Images