Underwater image enhancement method and system based on global pyramid linear modulation

Through the combination of global pyramid linear modulation and adaptive dynamic feature aggregation channel spatial attention module, the problem of insufficient utilization of feature representation and multi-scale information in underwater image enhancement is solved, efficient underwater image enhancement is achieved, and the model is improved Adaptability and generalization ability.

CN119067869AActive Publication Date: 2024-12-03SUN YAT SEN UNIV

Patent Information

Application Number
CN202411083523.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-08
Publication Date
2024-12-03
Estimated Expiration
2044-08-08

Smart Images

  • Figure CN119067869A_ABST
    Figure CN119067869A_ABST
Patent Text Reader

Abstract

The invention discloses an underwater image enhancement method and system based on global pyramid linear modulation. The method comprises the following steps: acquiring an original underwater image to be enhanced; performing multi-scale feature extraction from the original underwater image through a global pyramid module to obtain a first feature map; modulating the feature map through a global linear modulation module to obtain a modulated second feature map; performing enhancement processing on the second feature map according to a MixDehazeNet backbone network to obtain a third feature map; performing enhancement processing on the third feature map through an adaptive dynamic feature aggregation channel space attention module to obtain a fourth feature map; and performing model training according to the fourth feature map, and enhancing the to-be-enhanced original underwater image according to an image enhancement model obtained by training to obtain an enhanced underwater image. The underwater image enhancement effect can be remarkably improved, meanwhile, high calculation efficiency is kept, and the method can be widely applied to the technical field of image processing.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an underwater image enhancement method and system based on global pyramid linear modulation. Background Art

[0002] Underwater image enhancement is a key technology that is essential for improving the visual quality and recognizability of underwater images. This technology has a wide range of applications in the fields of marine resource development, marine ecological research, underwater infrastructure monitoring, and naval military operations. However, the absorption and scattering characteristics of light in the underwater environment pose great challenges to underwater image enhancement. These phenomena lead to problems such as color distortion, reduced contrast, blurred outlines, and increased noise in the image.

[0003] In order to overcome the challenges faced by underwater images, the academic community has proposed a variety of image enhancement technologies. The relevant existing technologies have the following disadvantages:

[0004] 1. Limited feature representation capabilities:

[0005] Existing underwater image enhancement methods, such as FiLM-based methods, mainly use two-dimensional feature modulation, which cannot fully capture the complex features of the image. This limits the model's ability to handle subtle details and complex structures in underwater images.

[0006] 2. Insufficient use of multi-scale information:

[0007] Although some methods such as FPN have considered multi-scale features, they usually adopt a simple feature fusion strategy and fail to fully utilize information at different scales. This leads to inconsistent results when dealing with underwater targets of different sizes and complexities.

[0008] 3. The attention mechanism is inefficient:

[0009] Existing attention mechanisms such as channel attention and spatial attention can improve model performance, but they have high computational complexity and are not conducive to real-time processing. In addition, they are often applied independently and lack effective integration strategies.

[0010] 4. Lack of adaptability:

[0011] Most existing methods use fixed network structures and parameters, which are difficult to adapt to changes in different underwater environments and imaging conditions. This limits the application effect of the model in complex and changeable actual underwater scenes.

[0012] 5. Insufficient generalization ability of real-world data:

[0013] Many methods perform well on synthetic datasets but poorly on real-world underwater images, which reflects the lack of generalization ability of existing methods when dealing with complex and changeable real underwater environments. Summary of the invention

[0014] The main purpose of the embodiments of the present invention is to propose an efficient underwater image enhancement method and system based on global pyramid linear modulation, which can significantly improve the effect of underwater image enhancement.

[0015] To achieve the above object, an embodiment of the present invention provides an underwater image enhancement method based on global pyramid linear modulation, comprising the following steps:

[0016] Acquire an original underwater image to be enhanced;

[0017] Extracting multi-scale features from the original underwater image by a global pyramid module to obtain a first feature map;

[0018] Modulating the feature map through a global linear modulation module to obtain a modulated second feature map;

[0019] Performing enhancement processing on the second feature map according to a MixDehazeNet backbone network to obtain a third feature map; wherein the MixDehazeNet backbone network is integrated with a global pyramid linear modulation block;

[0020] Performing enhancement processing on the third feature map through an adaptive dynamic feature aggregation channel spatial attention module to obtain a fourth feature map;

[0021] Model training is performed according to the fourth feature map, and the original underwater image to be enhanced is enhanced according to the image enhancement model obtained through training to obtain an enhanced underwater image.

[0022] In some embodiments, extracting multi-scale features from the original underwater image using a global pyramid module to obtain a first feature map comprises the following steps:

[0023] The feature extraction of the original underwater image is performed through a depth-separable convolution block; wherein the depth-separable convolution block is composed of depth-wise convolution and point-wise convolution; the expression of this process is:

[0024] Y=PointwiseConv(DepthwiseConv(X))

[0025] Among them, X is the input feature map, and Y is the output feature map;

[0026] According to the feature pyramid network structure, interpolation operations and element-level addition are used to perform multi-scale feature fusion on the extracted features; the feature pyramid network structure fuses features at different levels through top-down paths and lateral connections; the expression of this process is:

[0027] P l =Conv(C l +Upsample(P l+1 ))

[0028] Among them, P l is the feature map of layer l, C l is the corresponding convolutional feature map;

[0029] The fused features are enhanced by the triple attention module to obtain the first feature map. The triple attention module contains three parallel attention branches, which focus on the width and height of the channel dimension and the spatial dimension respectively. The expression of this process is:

[0030]

[0031] in, Represents element-by-element multiplication, SpatialAtt W 、SpatialAtt H and ChannelAtt represent the spatial attention in width direction, spatial attention in height direction and channel attention respectively.

[0032] In some embodiments, the modulating the characteristic graph by a global linear modulation module to obtain a modulated second characteristic graph comprises the following steps:

[0033] Performing global feature extraction on the first feature map through a feature extraction network;

[0034] Generate modulation parameters through a modulation parameter generation network;

[0035] According to the extracted global features and the generated modulation parameters, the feature modulation operation is performed through the feature modulation unit; the expression of this process is:

[0036] Y=γ⊙X+β

[0037] Among them, X is the input feature map, Y is the modulated feature map, γ and β are the learned modulation parameters, and ⊙ represents element-by-element multiplication.

[0038] In some embodiments, the step of enhancing the second feature map according to the MixDehazeNet backbone network to obtain a third feature map comprises the following steps:

[0039] Integrate the global pyramid linear modulation block into the MixDehazeNet backbone network;

[0040] Extract image features through the encoder of the MixDehazeNet backbone network;

[0041] The enhanced image is reconstructed through the decoder of the MixDehazeNet backbone network;

[0042] The detail information in the image enhancement process is preserved through the skip connection blocks in the MixDehazeNet backbone network.

[0043] In some embodiments, the adaptive dynamic feature aggregation channel spatial attention module includes global average pooling and global maximum pooling modules, a multi-layer perceptron, and channel attention and spatial attention units;

[0044] The expression of the processing process of the adaptive dynamic feature aggregation channel space attention module is:

[0045]

[0046] Among them, X is the feature map before enhancement processing, Y is the feature map after enhancement processing; W c and W s are the weights learned by a multi-layer perceptron.

[0047] In some embodiments, the global average pooling and global maximum pooling modules are used to extract global statistical information from the input third feature map; specifically including a global average pooling operation and a global maximum pooling operation:

[0048] Among them, the global average pooling operation is used to calculate the average value of each channel and capture the overall activation level of the feature map. The expression of this process is:

[0049]

[0050] Among them, GAP represents the global average pooling operation; X is the input feature map, H and W are the height and width of the feature map respectively; the global maximum pooling operation is used to extract the maximum value of each channel and highlight the significant features. The expression of this process is:

[0051] GMP(X)=max(X)

[0052] GMP stands for global maximum pooling operation;

[0053] The multi-layer perceptron is used to learn the adaptive weights of channel attention and spatial attention; specifically includes:

[0054] 1) Receive feature vectors from the global pooling module, perform nonlinear transformation, and learn higher-level feature representations;

[0055] 2) Generate parameters for dynamically adjusting channel attention and spatial attention weights; the expression of this process is:

[0056] W c ,W s =MLP(concat(GAP(X),GMP(X)))

[0057] Among them, W c and W s are the weights of channel attention and spatial attention respectively;

[0058] 3) Through the adaptive weight learning mechanism, attention allocation is dynamically adjusted according to the characteristics of the input features to improve the flexibility and adaptability of the model;

[0059] The channel attention and spatial attention units are used to perform fine-grained feature enhancement on the feature map.

[0060] In some embodiments, the method further comprises a step of training an image enhancement model, the step comprising the following steps:

[0061] Preprocess the underwater image data to be trained to obtain training samples;

[0062] Perform model initialization operation to obtain the initial model;

[0063] Inputting the training samples into the initial model to sequentially perform forward propagation processing, loss calculation, back propagation and parameter update, and training to obtain the image enhancement model;

[0064] Perform model evaluation on the image enhancement model, and save the image enhancement model that meets the evaluation requirements.

[0065] Another aspect of the embodiment of the present invention further provides an underwater image enhancement system based on global pyramid linear modulation, comprising:

[0066] The first module is used to obtain the original underwater image to be enhanced;

[0067] A second module is used to extract multi-scale features from the original underwater image through a global pyramid module to obtain a first feature map;

[0068] A third module is used to modulate the feature map through a global linear modulation module to obtain a modulated second feature map;

[0069] A fourth module is used to enhance the second feature map according to a MixDehazeNet backbone network to obtain a third feature map; wherein the MixDehazeNet backbone network is integrated with a global pyramid linear modulation block;

[0070] A fifth module is used to enhance the third feature map through an adaptive dynamic feature aggregation channel space attention module to obtain a fourth feature map;

[0071] The sixth module is used to perform model training according to the fourth feature map, and enhance the original underwater image to be enhanced according to the trained image enhancement model to obtain an enhanced underwater image.

[0072] Another aspect of an embodiment of the present invention further provides an electronic device, including a processor and a memory;

[0073] The memory is used to store programs;

[0074] The processor executes the program to implement the method described above.

[0075] Another aspect of the embodiments of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the method described above.

[0076] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.

[0077] The embodiments of the present invention include at least the following beneficial effects: the present invention provides an underwater image enhancement method and system based on global pyramid linear modulation, the present invention obtains an original underwater image to be enhanced; performs multi-scale feature extraction from the original underwater image through a global pyramid module to obtain a first feature map; performs modulation processing on the feature map through a global linear modulation module to obtain a modulated second feature map; performs enhancement processing on the second feature map according to a MixDehazeNet backbone network to obtain a third feature map; wherein a global pyramid linear modulation block is integrated in the MixDehazeNet backbone network; performs enhancement processing on the third feature map through an adaptive dynamic feature aggregation channel spatial attention module to obtain a fourth feature map; performs model training according to the fourth feature map, and enhances the original underwater image to be enhanced according to the image enhancement model obtained by the training to obtain an enhanced underwater image. The present invention can significantly improve the effect of underwater image enhancement while maintaining a high computational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present invention;

[0079] Figure 2 is a flow chart of overall steps provided by an embodiment of the present invention;

[0080] Figure 3 It is a schematic diagram of the overall framework provided by an embodiment of the present invention;

[0081] Figure 4 is a schematic diagram of the structure of a global pyramid module (GPM) provided in an embodiment of the present invention;

[0082] Figure 5 is a schematic diagram of an adaptive dynamic feature aggregation channel spatial attention module provided by an embodiment of the present invention;

[0083] Figure 6 is a model training flow chart provided by an embodiment of the present invention;

[0084] Figure 7 It is a comparison diagram of enhancement results on EUVP-515 and UFO-120 datasets provided by an embodiment of the present invention;

[0085] Figure 8 It is a comparison chart of enhancement results on the UIEB90 dataset provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0086] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention. They are only examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the attached claims.

[0087] It is understood that the terms "first", "second", etc. used in the present invention may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0088] The terms "at least one", "multiple", "each", "any", etc. used in the present invention, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0089] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0090] Before describing the embodiments of the present invention in detail, some related technologies involved in the embodiments of the present invention are first described as follows:

[0091] 1. Global Pyramid Linear Modulation (GPLM): A novel neural network adjustment method for underwater image enhancement models. It uses the Feature Pyramid Network (FPN) structure, combined with self-attention and feature fusion mechanisms to generate three-dimensional feature maps, and then modulates these feature maps through affine transformations to affect the calculation of the neural network.

[0092] 2. Feature-wise Linear Modulation (FiLM): A technique for regulating neural network calculations by affine transformation of features based on external information.

[0093] 3. Feature Pyramid Network (FPN): A network architecture that uses the multi-scale, pyramidal hierarchical structure inherent in convolutional neural networks to construct a feature pyramid with strong semantic information.

[0094] 4. Global Pyramid Module (GPM): A key component in GPLM used to extract 3D feature maps from the input image.

[0095] 5. Global Linear Modulation (GLM): Another key component in GPLM, used to apply affine transformation to feature maps.

[0096] 6. Adaptive Dynamic Feature Aggregation Channel-Spatial Attention Module (ADFA-CSAM): An innovative feature fusion technique for adaptively learning the aggregation weights of high-dimensional and low-dimensional features.

[0097] 7.Peak Signal-to-Noise Ratio (PSNR): An objective indicator used to evaluate image quality, measured in decibels (dB).

[0098] 8. Underwater Image Quality Measure (UIQM): A reference-free metric specifically designed to evaluate underwater image quality, taking into account image color, clarity, and contrast.

[0099] 9. Underwater Color Image Quality Evaluation (UCIQE): Another reference-free metric for evaluating underwater image quality.

[0100] 10. Depthwise Separable Convolution: A computationally efficient convolution operation that decomposes the standard convolution into two steps: depthwise convolution and pointwise convolution.

[0101] 11. Triplet Attention Module (TA): A lightweight self-attention mechanism for enhanced feature representation.

[0102] 12.MixDehazeNet: A deep learning model for image dehazing. We integrate GPLM into the model to improve performance.

[0103] The underwater image enhancement method and system based on global pyramid linear modulation provided by the embodiment of the present invention relate to the field of image processing technology. The underwater image enhancement method based on global pyramid linear modulation provided by the embodiment of the present invention can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements an underwater image enhancement method based on global pyramid linear modulation, etc., but is not limited to the above forms.

[0104] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0105] like Figure 1 FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present invention. Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected to a network wirelessly or wired to complete data transmission and exchange.

[0106] Server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0107] In addition, the server 101 can also be a node server in the blockchain network. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.

[0108] The terminal 102 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc. The terminal 102 may also be a vehicle-mounted terminal of various device types described above, but is not limited thereto. The terminal 102 and the server 101 may be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment of the present invention.

[0109] Based on the example Figure 1 In the implementation environment shown, an embodiment of the present invention provides an underwater image enhancement method based on global pyramid linear modulation. The underwater image enhancement method based on global pyramid linear modulation is applied to the server 101 as an example for explanation below. It can be understood that the method can also be applied to the terminal 102.

[0110] Reference Figure 2 , Figure 2 The flowchart of the underwater image enhancement method based on global pyramid linear modulation applied to the server provided by the embodiment of the present invention, the execution subject of the method can be any of the aforementioned computer devices (including servers or terminals). Figure 2 , the method may include the following steps:

[0111] Acquire an original underwater image to be enhanced;

[0112] Extracting multi-scale features from the original underwater image by a global pyramid module to obtain a first feature map;

[0113] Modulating the feature map through a global linear modulation module to obtain a modulated second feature map;

[0114] Performing enhancement processing on the second feature map according to a MixDehazeNet backbone network to obtain a third feature map; wherein the MixDehazeNet backbone network is integrated with a global pyramid linear modulation block;

[0115] Performing enhancement processing on the third feature map through an adaptive dynamic feature aggregation channel spatial attention module to obtain a fourth feature map;

[0116] Model training is performed according to the fourth feature map, and the original underwater image to be enhanced is enhanced according to the image enhancement model obtained through training to obtain an enhanced underwater image.

[0117] In some embodiments, extracting multi-scale features from the original underwater image using a global pyramid module to obtain a first feature map comprises the following steps:

[0118] The feature extraction of the original underwater image is performed through a depth-separable convolution block; wherein the depth-separable convolution block is composed of depth-wise convolution and point-wise convolution; the expression of this process is:

[0119] Y=PointwiseConv(DepthwiseConv(X))

[0120] Among them, X is the input feature map, and Y is the output feature map;

[0121] According to the feature pyramid network structure, interpolation operations and element-level addition are used to perform multi-scale feature fusion on the extracted features; the feature pyramid network structure fuses features at different levels through top-down paths and lateral connections; the expression of this process is:

[0122] P l =Conv(C l +Upsample(P l+1 ))

[0123] Among them, P l is the feature map of layer l, C l is the corresponding convolutional feature map;

[0124] The fused features are enhanced by the triple attention module to obtain the first feature map. The triple attention module contains three parallel attention branches, which focus on the width and height of the channel dimension and the spatial dimension respectively. The expression of this process is:

[0125]

[0126] in, Represents element-by-element multiplication, SpatialAtt W 、SpatialAtt H and ChannelAtt represent the spatial attention in width direction, spatial attention in height direction and channel attention respectively.

[0127] In some embodiments, the modulating the characteristic graph by a global linear modulation module to obtain a modulated second characteristic graph comprises the following steps:

[0128] Performing global feature extraction on the first feature map through a feature extraction network;

[0129] Generate modulation parameters through a modulation parameter generation network;

[0130] According to the extracted global features and the generated modulation parameters, the feature modulation operation is performed through the feature modulation unit; the expression of this process is:

[0131] Y=γ⊙X+β

[0132] Among them, X is the input feature map, Y is the modulated feature map, γ and β are the learned modulation parameters, and ⊙ represents element-by-element multiplication.

[0133] In some embodiments, the step of enhancing the second feature map according to the MixDehazeNet backbone network to obtain a third feature map comprises the following steps:

[0134] Integrate the global pyramid linear modulation block into the MixDehazeNet backbone network;

[0135] Image features are extracted through the encoder of the MixDehazeNet backbone network;

[0136] The enhanced image is reconstructed through the decoder of the MixDehazeNet backbone network;

[0137] The detail information in the image enhancement process is preserved through the skip connection blocks in the MixDehazeNet backbone network.

[0138] In some embodiments, the adaptive dynamic feature aggregation channel spatial attention module includes global average pooling and global maximum pooling modules, a multi-layer perceptron, and channel attention and spatial attention units;

[0139] The expression of the processing process of the adaptive dynamic feature aggregation channel space attention module is:

[0140]

[0141] Among them, X is the feature map before enhancement processing, Y is the feature map after enhancement processing; W c and W s are the weights learned by a multi-layer perceptron.

[0142] In some embodiments, the global average pooling and global maximum pooling modules are used to extract global statistical information from the input third feature map; specifically including a global average pooling operation and a global maximum pooling operation:

[0143] Among them, the global average pooling operation is used to calculate the average value of each channel and capture the overall activation level of the feature map. The expression of this process is:

[0144]

[0145] Among them, GAP represents the global average pooling operation; X is the input feature map, H and W are the height and width of the feature map respectively; the global maximum pooling operation is used to extract the maximum value of each channel and highlight the significant features. The expression of this process is:

[0146] GMP(X)=max(X)

[0147] GMP stands for global maximum pooling operation;

[0148] The multi-layer perceptron is used to learn the adaptive weights of channel attention and spatial attention; specifically includes:

[0149] 1) Receive feature vectors from the global pooling module, perform nonlinear transformation, and learn higher-level feature representations;

[0150] 2) Generate parameters for dynamically adjusting channel attention and spatial attention weights; the expression of this process is:

[0151] W c ,W s =MLP(concat(GAP(X),GMP(X)))

[0152] Among them, W c and W s are the weights of channel attention and spatial attention respectively;

[0153] 3) Through the adaptive weight learning mechanism, attention allocation is dynamically adjusted according to the characteristics of the input features to improve the flexibility and adaptability of the model;

[0154] The channel attention and spatial attention units are used to perform fine-grained feature enhancement on the feature map.

[0155] In some embodiments, the method further comprises a step of training an image enhancement model, the step comprising the following steps:

[0156] Preprocess the underwater image data to be trained to obtain training samples;

[0157] Perform model initialization operation to obtain the initial model;

[0158] Inputting the training samples into the initial model to sequentially perform forward propagation processing, loss calculation, back propagation and parameter update, and training to obtain the image enhancement model;

[0159] Perform model evaluation on the image enhancement model, and save the image enhancement model that meets the evaluation requirements.

[0160] The implementation process of the method of the present invention in a specific application scenario is described in detail below in conjunction with the accompanying drawings of the specification:

[0161] This paper proposes an underwater image enhancement method and system based on global pyramid linear modulation (GPLM). The core idea of ​​this method is to expand feature linear modulation (FiLM) from two dimensions to four dimensions, and combine it with the feature pyramid network (FPN) structure to achieve finer-grained and multi-scale feature modulation.

[0162] like Figure 3 As shown, the overall framework of the present invention includes the following main components:

[0163] ① Input module;

[0164] ②Global Pyramid Module (GPM);

[0165] ③Global Linear Modulation Module (GLM);

[0166] ④MixDehazeNet backbone network;

[0167] ⑤Output module.

[0168] Each major component is described in detail below:

[0169] 2. Global Pyramid Module (GPM)

[0170] The global pyramid module is one of the core components of the present invention, which is used to extract multi-scale features from the input image. Figure 4 As shown, the structure of GPM includes:

[0171] ① Depthwise Separable Convolution (DSC) block: for efficient feature extraction;

[0172] ②Feature Pyramid Network (FPN) structure: used for multi-scale feature fusion;

[0173] ③Triplet Attention (TA) module: used to enhance feature representation.

[0174] 2.1 Depthwise Separable Convolution (DSC) Block

[0175] The depth-wise separable convolution block consists of depth-wise convolution and point-wise convolution, which can reduce the computational complexity while maintaining good performance. The mathematical expression of the DSC block is as follows:

[0176] Y=PointwiseConv(DepthwiseConv(X))

[0177] Among them, X is the input feature map and Y is the output feature map.

[0178] 2.2 Feature Pyramid Network (FPN) Structure

[0179] The FPN structure fuses features at different levels through top-down paths and lateral connections. In the present invention, the embodiment of the present invention uses interpolation operations and element-level addition to achieve feature fusion:

[0180] P l =Conv(C l +Upsample(P l+1 ))

[0181] Among them, P l is the feature map of layer l, C l is the corresponding convolutional feature map.

[0182] 2.3 Triple Attention (TA) Module

[0183] The triple attention module contains three parallel attention branches, focusing on the channel dimension, the width and the height of the spatial dimension respectively. The output of the TA module can be expressed as:

[0184]

[0185] in, Represents element-by-element multiplication, SpatialAtt W 、SpatialAtt H and ChannelAtt represent the spatial attention in width direction, spatial attention in height direction and channel attention respectively.

[0186] 3. Global Linear Modulation Module (GLM)

[0187] The global linear modulation module is responsible for modulating the feature map generated by GPM, such as Figure 3 As shown, the GLM includes:

[0188] ① Feature extraction network: used to extract global features from the input image;

[0189] ② Modulation parameter generation network: used to generate modulation parameters γ and β;

[0190] ③ Feature modulation unit: used to perform actual feature modulation operations.

[0191] The mathematical expression of GLM can be expressed as:

[0192] Y=γ⊙X+β

[0193] Among them, X is the input feature map, Y is the modulated feature map, γ and β are the learned modulation parameters, and ⊙ represents element-by-element multiplication.

[0194] 4.MixDehazeNet backbone network

[0195] The present invention integrates GPLM into MixDehazeNet to improve the performance of underwater image enhancement. The main components of MixDehazeNet include:

[0196] ①Encoder: used to extract image features;

[0197] ②Decoder: used to reconstruct the enhanced image;

[0198] ③ Skip connection: used to retain detail information.

[0199] After each MixStructureBlock, the embodiment of the present invention inserts a GPLM module to modulate the characteristics. Figure 3 The MixDehazeNet structure with integrated GPLM is shown.

[0200] 5. Adaptive Dynamic Feature Aggregation Channel Spatial Attention Module (ADFA-CSAM)

[0201] In order to further improve the effect of feature fusion, the embodiment of the present invention proposes an ADFA-CSAM module. Figure 5 As shown, the module includes:

[0202] ① Global average pooling (GAP) and global maximum pooling (GMP);

[0203] ②Multilayer Perceptron (MLP);

[0204] ③Channel attention and spatial attention units.

[0205] The mathematical expression of ADFA-CSAM can be expressed as:

[0206]

[0207] Among them, W c and W sare the weights learned through MLP.

[0208] Specifically, the global average pooling and global maximum pooling modules are used to extract global statistical information from the input third feature map. Specifically:

[0209] 1) Global Average Pooling (GAP) is used to calculate the average value of each channel and capture the overall activation level of the feature map. Its operation can be expressed as:

[0210]

[0211] Where X is the input feature map, H and W are the height and width of the feature map respectively.

[0212] 2) Global Max Pooling (GMP) is used to extract the maximum value of each channel and highlight the significant features. Its operation can be expressed as:

[0213] GMP(X)=max(X)

[0214] By using GAP and GMP simultaneously, this module can comprehensively capture the global information of the feature map and provide important statistical basis for the subsequent attention mechanism.

[0215] The multi-layer perceptron is used to learn the adaptive weights of channel attention and spatial attention. Specific functions include:

[0216] 1) Receive feature vectors from the global pooling module, perform nonlinear transformation, and learn higher-level feature representations.

[0217] 2) Generate parameters for dynamically adjusting channel attention and spatial attention weights. Its operation can be expressed as:

[0218] W c ,W s =MLP(concat(GAP(X),GMP(W)))

[0219] Among them, W c and W s are the weights of channel attention and spatial attention, respectively.

[0220] 3) Through this adaptive weight learning mechanism, ADFA-CSAM can dynamically adjust attention allocation according to the characteristics of the input features, thereby improving the flexibility and adaptability of the model.

[0221] The channel attention and spatial attention units are used to perform fine-grained feature enhancement on the feature map. Specifically:

[0222] 1) The channel attention unit focuses on "what" is a useful feature and highlights important features by recalibrating the dependencies between channels. Its operation can be expressed as:

[0223] ChannelAtt(X)=σ(MLP(GAP(X)))

[0224] Among them, σ represents the sigmoid activation function.

[0225] 2) The spatial attention unit focuses on "where" useful features and enhances key areas by learning the importance of spatial locations. Its operation can be expressed as:

[0226] SpatialAtt(X)=σ(Conv(concat(AvgPool(X),MaxPool(X))))

[0227] 3) The final attention-enhanced features are obtained as follows:

[0228]

[0229] in, Represents element-wise multiplication.

[0230] For the third feature map generated earlier, ADFA-CSAM can extract and fuse multi-scale feature information to enhance the model's ability to process targets of different sizes; through the adaptive attention mechanism, it can highlight important features and key areas and suppress irrelevant information; and dynamically adjust the weights of high-dimensional and low-dimensional features to achieve more effective feature fusion.

[0231] For the subsequent steps of training the model, the ADFA-CSAM module can provide richer and more discriminative feature representations, which helps the model better learn the essential characteristics of underwater images; improve the generalization ability of the model through adaptive mechanisms, so that it can better adapt to different underwater environments and imaging conditions; while maintaining high performance, reduce computational complexity and improve the training and reasoning efficiency of the model.

[0232] 6. Training Process

[0233] The training process of the present invention is as follows Figure 6 The main steps include:

[0234] ①Data preprocessing;

[0235] ②Model initialization;

[0236] ③Forward propagation;

[0237] ④Loss calculation;

[0238] ⑤Back propagation and parameter update;

[0239] ⑥Model evaluation and preservation.

[0240] During the training process, the embodiment of the present invention uses the L1 loss function to optimize the model:

[0241] l=|I out -I GT |1

[0242] Among them, I out is the model output, I GT is the true label.

[0243] 7. Experimental Results

[0244] In order to verify the effectiveness of the method of the present invention, the embodiments of the present invention conducted experiments on multiple data sets. Figure 7 and Figure 8 Some enhancement results on EUVP, UFO and UIEB datasets are shown.

[0245] Experimental results show that the proposed method has achieved significant improvements in indicators such as peak signal-to-noise ratio (PSNR), structural similarity (SSIM) and underwater image quality metric (UIQM).

[0246] The underwater image enhancement method based on global pyramid linear modulation proposed in this paper effectively improves the quality of underwater images by introducing innovative modules such as GPLM, GPM and ADFA-CSAM. This method not only achieves excellent performance on multiple benchmark datasets, but also shows good generalization ability and robustness in actual underwater scenes.

[0247] In summary, compared with the prior art, the present invention has the following characteristics:

[0248] 1. Improve feature representation capabilities:

[0249] The present invention aims to expand feature modulation from two dimensions to four dimensions by introducing a global pyramid linear modulation (GPLM) method, so as to enhance the model's ability to represent complex underwater image features.

[0250] 2. Make full use of multi-scale information:

[0251] The purpose of this invention is to design an efficient global pyramid module (GPM) to better fuse and utilize multi-scale feature information by combining the feature pyramid network (FPN) structure and the self-attention mechanism.

[0252] 3. Improve the efficiency of the attention mechanism:

[0253] This paper aims to introduce a lightweight and efficient triple attention module (TA) to reduce the computational complexity while maintaining performance and achieve more efficient feature enhancement.

[0254] 4. Enhance the adaptability of the model:

[0255] By designing an adaptive dynamic feature aggregation channel spatial attention module (ADFA-CSAM), the present invention aims to enable the model to dynamically adjust the processing strategy according to the characteristics of the input image and improve the adaptability to different underwater environments.

[0256] 5. Improve the processing ability of real-world data:

[0257] The present invention aims to improve the generalization ability and robustness of the model in actual underwater environments by validating and optimizing it on multiple benchmark datasets and real-world datasets (such as DanGan1190).

[0258] By implementing these technologies, the present invention is expected to significantly improve the effect of underwater image enhancement while maintaining high computational efficiency, and provide new ideas and methods for the development of underwater image processing technology.

[0259] Compared with the prior art, first of all, the global pyramid linear modulation (GPLM) method proposed in the present invention has made a qualitative leap in feature representation and modulation capabilities. Compared with the existing method based on feature linear modulation (FiLM), GPLM expands feature modulation from two dimensions to four dimensions, and combines the feature pyramid network (FPN) structure to achieve finer-grained and multi-scale feature modulation. This innovative design enables the model to better capture and process complex features and details in underwater images, thereby achieving excellent enhancement effects in various underwater environments and imaging conditions. Especially when processing real-world underwater images, the present invention exhibits stronger generalization and robustness, which is lacking in many existing methods.

[0260] Secondly, the present invention achieves a better balance between computational efficiency and performance. By introducing deep separable convolutions and lightweight triple attention modules, as well as an innovative adaptive dynamic feature aggregation channel spatial attention module (ADFA-CSAM), the present invention significantly reduces computational complexity while maintaining high performance. This enables the method to not only achieve excellent enhancement effects on high-end devices, but also run efficiently on resource-constrained devices, greatly expanding its scope of application. In addition, through the clever integration with MixDehazeNet, the present invention realizes layer-by-layer feature modulation and enhancement, further improving the model's ability to handle underwater image details and structures, and outperforms existing state-of-the-art methods such as PUGAN and DGNet on multiple benchmark datasets. In particular, in terms of key indicators such as PSNR and UIQM, the present invention has achieved significant improvements of 3dB and more than 1, respectively, fully demonstrating its leading position in the field of underwater image enhancement.

[0261] Another aspect of the embodiment of the present invention further provides an underwater image enhancement system based on global pyramid linear modulation, comprising:

[0262] The first module is used to obtain the original underwater image to be enhanced;

[0263] A second module is used to extract multi-scale features from the original underwater image through a global pyramid module to obtain a first feature map;

[0264] A third module is used to modulate the feature map through a global linear modulation module to obtain a modulated second feature map;

[0265] A fourth module is used to enhance the second feature map according to a MixDehazeNet backbone network to obtain a third feature map; wherein the MixDehazeNet backbone network is integrated with a global pyramid linear modulation block;

[0266] A fifth module is used to enhance the third feature map through an adaptive dynamic feature aggregation channel space attention module to obtain a fourth feature map;

[0267] The sixth module is used to perform model training according to the fourth feature map, and enhance the original underwater image to be enhanced according to the trained image enhancement model to obtain an enhanced underwater image.

[0268] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0269] The embodiment of the present invention further provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the underwater image enhancement method based on global pyramid linear modulation when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0270] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0271] An embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the underwater image enhancement method based on global pyramid linear modulation is implemented.

[0272] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0273] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0274] The embodiments described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art can appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0275] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present invention and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0276] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0277] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0278] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0279] It should be understood that in the present invention, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0280] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0281] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0282] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0283] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including multiple instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store programs.

[0284] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the embodiments of the present invention is not limited thereby. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the embodiments of the present invention shall be within the scope of the rights of the embodiments of the present invention.

Claims

1. An underwater image enhancement method based on global pyramid linear modulation, characterized in that: The following steps are involved: Acquire an original underwater image to be enhanced; Extracting multi-scale features from the original underwater image by a global pyramid module to obtain a first feature map; Modulating the feature map through a global linear modulation module to obtain a modulated second feature map; Performing enhancement processing on the second feature map according to a MixDehazeNet backbone network to obtain a third feature map; wherein the MixDehazeNet backbone network is integrated with a global pyramid linear modulation block; Performing enhancement processing on the third feature map through an adaptive dynamic feature aggregation channel spatial attention module to obtain a fourth feature map; Model training is performed according to the fourth feature map, and the original underwater image to be enhanced is enhanced according to the image enhancement model obtained through training to obtain an enhanced underwater image.

2. The underwater image enhancement method based on global pyramid linear modulation according to claim 1, characterized in that: The method of extracting multi-scale features from the original underwater image by using a global pyramid module to obtain a first feature map comprises the following steps: The feature extraction of the original underwater image is performed through a depth-separable convolution block; wherein the depth-separable convolution block is composed of depth-wise convolution and point-wise convolution; the expression of this process is: Y=PointwiseConv(DepthwiseConv(X)) Among them, X is the input feature map, and Y is the output feature map; According to the feature pyramid network structure, interpolation operations and element-level addition are used to perform multi-scale feature fusion on the extracted features; the feature pyramid network structure fuses features at different levels through top-down paths and lateral connections; the expression of this process is: P l =Conv(C l +Upsample(P l+1 )0 Among them, P l is the feature map of layer l, C l is the corresponding convolutional feature map; The fused features are enhanced by the triple attention module to obtain the first feature map. The triple attention module contains three parallel attention branches, which focus on the width and height of the channel dimension and the spatial dimension respectively. The expression of this process is: in, Represents element-by-element multiplication, SpatialAtt W 、SpatialAtt H and ChannelAtt represent the spatial attention in width direction, spatial attention in height direction and channel attention respectively.

3. The underwater image enhancement method based on global pyramid linear modulation according to claim 1, characterized in that: The method of modulating the feature map by a global linear modulation module to obtain a modulated second feature map includes the following steps: Performing global feature extraction on the first feature map through a feature extraction network; Generate modulation parameters through a modulation parameter generation network; According to the extracted global features and the generated modulation parameters, the feature modulation operation is performed through the feature modulation unit; the expression of this process is: Y=γ⊙X+β Among them, X is the input feature map, Y is the modulated feature map, γ and β are the learned modulation parameters, and ⊙ represents element-by-element multiplication.

4. The underwater image enhancement method based on global pyramid linear modulation according to claim 1, characterized in that: The step of performing enhancement processing on the second feature map according to the MixDehazeNet backbone network to obtain a third feature map comprises the following steps: Integrate the global pyramid linear modulation block into the MixDehazeNet backbone network; Image features are extracted through the encoder of the MixDehazeNet backbone network; The enhanced image is reconstructed through the decoder of the MixDehazeNet backbone network; The detail information in the image enhancement process is preserved through the skip connection blocks in the MixDehazeNet backbone network.

5. The underwater image enhancement method based on global pyramid linear modulation according to claim 1, characterized in that: The adaptive dynamic feature aggregation channel spatial attention module includes global average pooling and global maximum pooling modules, a multi-layer perceptron, and channel attention and spatial attention units; The expression of the processing process of the adaptive dynamic feature aggregation channel space attention module is: Among them, X is the feature map before enhancement processing, Y is the feature map after enhancement processing; W c and W s are the weights learned by a multi-layer perceptron.

6. The underwater image enhancement method based on global pyramid linear modulation according to claim 5, characterized in that: The global average pooling and global maximum pooling modules are used to extract global statistical information from the input third feature map; specifically include global average pooling operations and global maximum pooling operations: Among them, the global average pooling operation is used to calculate the average value of each channel and capture the overall activation level of the feature map. The expression of this process is: Where GAP stands for global average pooling operation; X is the input feature map, H and W are the height and width of the feature map respectively; The global maximum pooling operation is used to extract the maximum value of each channel and highlight the significant features. The expression of this process is: GMP(X)=max(X) GMP stands for global maximum pooling operation; The multi-layer perceptron is used to learn the adaptive weights of channel attention and spatial attention; specifically includes: 1) Receive feature vectors from the global pooling module, perform nonlinear transformation, and learn higher-level feature representations; 2) Generate parameters for dynamically adjusting channel attention and spatial attention weights; the expression of this process is: W c ,W s =MLP(concat(GAP(X),GMP(X))) Among them, W c and W s are the weights of channel attention and spatial attention respectively; 3) Through the adaptive weight learning mechanism, attention allocation is dynamically adjusted according to the characteristics of the input features to improve the flexibility and adaptability of the model; The channel attention and spatial attention units are used to perform fine-grained feature enhancement on the feature map.

7. The underwater image enhancement method based on global pyramid linear modulation according to claim 1, characterized in that: The method also includes a step of training the image enhancement model, which includes the following steps: Preprocess the underwater image data to be trained to obtain training samples; Perform model initialization operation to obtain the initial model; Inputting the training samples into the initial model to sequentially perform forward propagation processing, loss calculation, back propagation and parameter update, and training to obtain the image enhancement model; Perform model evaluation on the image enhancement model, and save the image enhancement model that meets the evaluation requirements.

8. An underwater image enhancement system based on global pyramid linear modulation, characterized in that: include: The first module is used to obtain the original underwater image to be enhanced; A second module is used to extract multi-scale features from the original underwater image through a global pyramid module to obtain a first feature map; A third module is used to modulate the feature map through a global linear modulation module to obtain a modulated second feature map; A fourth module is used to enhance the second feature map according to a MixDehazeNet backbone network to obtain a third feature map; wherein the MixDehazeNet backbone network is integrated with a global pyramid linear modulation block; A fifth module is used to enhance the third feature map through an adaptive dynamic feature aggregation channel space attention module to obtain a fourth feature map; The sixth module is used to perform model training according to the fourth feature map, and enhance the original underwater image to be enhanced according to the trained image enhancement model to obtain an enhanced underwater image.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Fast underwater image enhancement method based on two-stage attention mechanism

    CN116416157A

  • Underwater image enhancement method based on multistage subnet cascade network

    CN117830770A

Cited By

  • Visual model closed-loop enhancement method based on physical simulation and parameter perception

    CN120894503A

  • A visual model closed-loop enhancement method based on physical simulation and parameter perception

    CN120894503B