Underwater image enhancement method based on regional differentiation fusion network
By using a regional differential fusion network, the degradation differences in different regions of underwater images are explicitly modeled. By utilizing the cross-fusion Swin Transformer module and the joint loss function, the problem of uneven visual effects between foreground and background in underwater image enhancement is solved, achieving high-quality image enhancement results.
Patent Information
- Application Number
- CN202610108957.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-27
- Publication Date
- 2026-05-15
AI Technical Summary
Existing technologies struggle to simultaneously enhance both foreground and background images under conditions of uneven underwater degradation, leading to problems such as local over-enhancement or insufficient detail in the background.
We employ a region-differentiated fusion network approach, which explicitly models the degradation differences in different regions of underwater images through a dual-branch learning framework and a cross-fusion learning strategy. We utilize the cross-fusion Swin Transformer module for feature fusion and optimize the results through a joint loss function.
It effectively corrects color deviation and detail blur in underwater images, improves overall visual quality, and generates enhanced images with vibrant colors and clear details.
Smart Images
Figure CN122048692A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image information processing technology, specifically an underwater image enhancement method based on a regional differential fusion network. Background Technology
[0002] Currently, the technologies related to this invention include three aspects: first, enhancement algorithms based on visual priors; second, recovery algorithms based on physical models; and third, enhancement algorithms based on deep learning.
[0003] Visual prior-based enhancement algorithms typically rely on global image properties or statistical experience to segment image regions and adjust pixel distribution to improve overall visual quality. Ancuti et al., in their paper "Color channel compensation (3C): A fundamental pre-processing step for image enhancement," observed wavelength-dependent attenuation in specific scenes and compensated for color loss by reducing the contrasting colors of severely attenuated channels to their local averages. An et al., in their paper "HFM: A hybrid fusion method for underwater image enhancement," introduced the gray-world principle and a nonlinear color mapping function to correct color and white balance, and then used a perceptual fusion strategy to fuse images that restore visibility and enhance contrast. Zhou et al., in their paper "Underwater camera: Improving visual perception via adaptive dark pixel prior and color correction," addressed specular issues through channel intensity-based depth map estimation and used adaptive dark pixels to remove backscattering.
[0004] Physically-based image restoration algorithms construct underwater imaging models to model absorption and scattering during light propagation, and then use the estimation of physical parameters to inversely restore degraded images. Liang et al., in their paper "GUDCP: Generalization of underwater dark channel prior for underwater imagerestoration," integrated multiple prior knowledge sets to estimate backscattering components and incorporated the spectral characteristics of different water body types into the dark channel model. Varghese et al., in their paper "Self-supervised monocular underwaterdepth recovery, image restoration, and a real-sea video dataset," combined fog cues and geometric constraints to estimate scene depth and used multiple sub-networks to collaboratively predict and restore images and depth maps.
[0005] In recent years, deep learning-based underwater image enhancement algorithms have been extensively studied. These methods construct end-to-end network models, utilizing large amounts of data to learn the mapping relationship between degradation features and enhancement in underwater images. Li et al., in their paper "Underwater image enhancement via medium transmission-guided multi-colorspace embedding," combined multiple color spaces to enrich network features and guided the network to focus on degraded regions through attention mechanisms and transmission graphs. Liu et al., in their paper "A multi-level wavelet-based underwater image enhancement network with color compensation prior," used the green channel to compensate for severely attenuated channels, and then adjusted the image's naturalness through contrastive learning in the decoder. To reduce the requirements for training data, some studies further introduce unsupervised learning or cross-domain transfer strategies, achieving underwater image enhancement through domain adaptation or adversarial learning. For example, Jiang et al., in their paper "Two-step domain adaptation for underwater image enhancement," used paired datasets to transform underwater images to a terrestrial style, and then further enhanced the images using shared dehazing weights. In their paper "Twin adversarial contrastive learning for underwater image enhancement and beyond", Liu et al. proposed a twin adversarial contrastive learning network with a task-aware feedback module to generate enhanced images that are beneficial for detection.
[0006] According to underwater imaging models, color attenuation and detail degradation in underwater images typically exhibit significant spatial non-uniformity. In actual imaging, areas closer to the camera usually possess clearer structural details, while more distant areas are more susceptible to color shifts, decreased contrast, and hazy blurring. In this context, employing a uniform global enhancement strategy often fails to simultaneously address the degradation characteristics of different regions, potentially leading to localized over-enhancement or insufficient detail in distant areas in certain scenarios. A potential solution is to develop region-specific enhancement algorithms to eliminate non-uniform degradation, thereby providing a more comprehensive image quality improvement. However, the lack of explicit modeling of near and far regions means that deep feature extraction cannot effectively characterize the degradation differences between regions, easily resulting in over-enhancement of near areas or blurred detail in distant areas when enhancing complex underwater images. Summary of the Invention
[0007] The purpose of this invention is to address the problem of high-quality image enhancement under non-uniform underwater degradation conditions. It provides an underwater image enhancement method based on a region-differentiated fusion network. This method introduces explicit enhancement priors and constructs a dual-branch learning framework to model the degradation differences in different regions of the underwater image. It also employs a cross-fusion learning strategy to achieve synergistic enhancement between regional features, thereby effectively correcting non-uniform color deviations and detail blurring, and improving the overall visual quality of the underwater image.
[0008] To achieve the above objectives, the technical solution of the present invention is: an underwater image enhancement method based on a regional differential fusion network, comprising:
[0009] Input the original underwater image and perform contrast-limited adaptive histogram equalization to generate the corresponding adaptive enhanced image;
[0010] The original underwater image and the adaptively enhanced image are input into the original encoder, respectively. and adaptive encoder The shallow feature extraction module generates initial raw features and adaptive enhanced features;
[0011] The original features and adaptive enhanced features are input into the cross-fusion Swin Transformer module to perform regional differential feature fusion;
[0012] The fused regional difference features are input into the reconstruction decoder. The underwater image is gradually decoded and reconstructed through interpolation upsampling and convolution activation operations, and the enhanced underwater image is output.
[0013] During the training phase, a joint loss function is used to supervise and optimize the augmentation results. The joint loss function includes Charbonnier loss and VGG loss.
[0014] Furthermore, both the original encoder and the adaptive enhancement encoder include a multi-layer structure, wherein:
[0015] The first coding layer is a shallow feature extraction module consisting of convolutional layers, LReLU activation functions, and convolutional layers.
[0016] Subsequent encoding layers are all downsampling modules consisting of a max pooling layer, a convolutional layer, and an LReLU activation function connected in sequence.
[0017] Furthermore, the cross-fusion Swin Transformer module specifically executes the following process:
[0018] Raw features of the input and adaptive enhancement features Layer-by-layer normalization is performed, and the normalized features are then input into a window-based multi-head cross-attention layer. Then, residual connections are performed to generate the first region's differential features. Difference characteristics of the second region :
[0019]
[0020] The first and second region difference features are processed by layer normalization, residual connection, and multilayer perceptron (MLP) to obtain the enhanced first region difference features. Difference characteristics of the second region :
[0021]
[0022] The enhanced first and second region difference features are subjected to layer normalization and then input into a sliding window-based multi-head cross-attention layer. Global aggregation is performed, and the final fused features are generated through residual connections, layer normalization, multilayer perceptron (MLP) processing, and residual connections. :
[0023] .
[0024] Furthermore, the operations based on the windowed multi-head cross-attention mechanism include:
[0025] The input features are divided into multiple non-overlapping fixed windows;
[0026] Within each window, the cross-correlation between the query matrix and key matrix between the original features and the adaptively enhanced features is calculated, and a learnable relative position embedding vector is introduced to label the feature mapping position.
[0027] The correlation matrix is normalized using the Softmax function, then multiplied by the value matrix of another feature, and the cross-attention distribution within the two windows is obtained through linear mapping.
[0028] Furthermore, the operations based on the sliding window-based multi-head cross-attention mechanism include:
[0029] The input features are divided into multiple sliding windows, which are offset by half the window size in the height and width directions relative to the fixed window, so as to realize information interaction between adjacent windows;
[0030] Within each sliding window, the correlation between the query matrix and key matrix between the original region contrast features and the adaptively enhanced region contrast features is calculated, and a learnable relative position embedding vector is introduced to label the feature mapping position.
[0031] The correlation matrix is normalized using the Softmax function, then multiplied by the value matrix, and the attention distribution within the window is obtained through linear mapping.
[0032] Furthermore, the reconstruction decoder contains a multi-layer upsampling structure, wherein:
[0033] Each upsampling layer consists of interpolation operations, convolutional layers, and LReLU activation functions connected in sequence;
[0034] The final layer is an image reconstruction layer consisting of convolutional layers, LReLU activation functions, and convolutional layers, which is used to output the final enhanced image.
[0035] Furthermore, the joint loss function The specific expression is:
[0036]
[0037] in, and For stabilizing parameters and balancing weights; This represents the convolutional feature representation in the VGG-199 network. Indicates an enhanced image; This refers to a reference image.
[0038] Furthermore, the cross-fusion Swin Transformer module is called once after each coding level of the original encoder and the adaptive augmentation encoder to achieve multi-scale regional differential feature fusion.
[0039] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to perform the steps of the method described above.
[0040] The present invention also provides a computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor, wherein when the processor executes the computer program instructions, it can implement the steps of the method described above.
[0041] Compared with the prior art, the present invention has the following beneficial effects: Attached Figure Description
[0042] Figure 1 This is a schematic diagram of the overall process of a regionally differentiated integration network.
[0043] Figure 2 This is a schematic diagram of the cross-fusion Swin Transformer.
[0044] Figure 3 This is a schematic diagram of a multi-head attention structure based on windowing and sliding windowing. Detailed Implementation
[0045] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.
[0046] This invention provides an underwater image enhancement method based on a regional differential fusion network, comprising:
[0047] Input the original underwater image and perform contrast-limited adaptive histogram equalization to generate the corresponding adaptive enhanced image;
[0048] The original underwater image and the adaptively enhanced image are input into the original encoder, respectively. and adaptive encoder The shallow feature extraction module generates initial raw features and adaptive enhanced features;
[0049] The original features and adaptive enhanced features are input into the cross-fusion Swin Transformer module to perform regional differential feature fusion;
[0050] The fused regional difference features are input into the reconstruction decoder. The underwater image is gradually decoded and reconstructed through interpolation upsampling and convolution activation operations, and the enhanced underwater image is output.
[0051] During the training phase, a joint loss function is used to supervise and optimize the augmentation results. The joint loss function includes Charbonnier loss and VGG loss.
[0052] The following is a detailed implementation process of the present invention.
[0053] This invention discloses an underwater image enhancement method based on a regional differential fusion network. Specifically, it is an underwater degraded image enhancement method that uses a regional cross-fusion mechanism to aggregate dual-branch differential features, and the steps are as follows:
[0054] (1) Image differential modeling and reconstruction task
[0055] This image differentiation modeling and reconstruction task introduces a contrast-limited adaptive histogram equalization method to enhance the regional differences in the input underwater image. The enhanced image and the original underwater image are then input into the network to perceive the imaging differences between different regions and model the scene degradation distribution. To this end, a dual-branch network structure is constructed, consisting of a raw encoder, an adaptive enhancement encoder, and a reconstruction decoder. The first layer of both the raw encoder and the adaptive enhancement encoder is a shallow feature extraction module composed of convolutional layers, LReLU functions, and convolutional layers. The remaining encoding layers are downsampling layers composed of max pooling layers, convolutional layers, and LReLU functions. The decoding layers of the reconstruction decoder are upsampling layers composed of interpolation operations, convolutional layers, and LReLU functions. The last layer is an image reconstruction layer composed of convolutional layers, LReLU functions, and convolutional layers. Instead of a direct skip connection for feature fusion between the raw encoder and the adaptive enhancement encoder, a cross-fusion Swing Transformer is introduced to capture the spatial differences in the distribution of features between the two branches, thereby achieving dynamic interaction and adaptive fusion of regional differentiation features. In the reconstruction and decoding stage, the propagation attenuation characteristics are perceived by aggregating multi-round cross-fusion features step by step, and the non-uniform scene degradation is corrected by a progressively refined learning method. This allows for targeted correction of color deviation and detail blurring in different areas of the underwater image, thereby obtaining an underwater image with comprehensive enhancement.
[0056] (2) Cross-disciplinary learning tasks
[0057] The cross-fusion learning task is implemented by a SwinTransformer that cross-fuses the original encoder and the adaptive augmentation encoder, with the input being the original features corresponding to the i-th layer encoder. and adaptive enhancement features To address the two types of input features mentioned above, the Swin Transformer employs a two-stage process to establish an information interaction and learning mechanism between regional difference features. This mechanism includes layer normalization, a multi-head cross-attention mechanism based on windowing and sliding windows, and a multilayer perceptron. In the first stage, the two-branch features are first processed by layer normalization, and then jointly input into a window-based multi-head cross-attention module to compare and learn the distribution differences of the two features in different regions. Next, the obtained regional difference features are fused with the original input features through two residual connections, and layer normalization and a multilayer perceptron are further introduced to identify feature distribution differences. The above process is represented as follows:
[0058]
[0059]
[0060] in, Presentation layer normalization operation, This indicates a windowed multi-head attention mechanism. This represents a multilayer perceptron.
[0061] In the second stage, the two features are again aligned through layer normalization and then input into a sliding window-based multi-head cross-attention module to aggregate regional difference information, achieving interactive fusion of the bi-branch modeling features. Subsequently, regional difference features are captured through layer normalization and a multilayer perceptron, and the difference distribution information is progressively labeled through two residual connections. The process is represented as follows:
[0062]
[0063] Each round of the network's encoding process incorporates a cross-fusion Swin Transformer to capture multi-stage regional disparity information. The regional disparity features generated in each round are residually concatenated with the corresponding original encoded features and then input into the corresponding reconstruction decoder. They are then fused through interpolation, convolution, and LReLU functions to provide more comprehensive reconstructed features for modeling.
[0064] (3) Cross-attention task based on windowing and sliding windowing
[0065] The cross-attention task based on windowing and sliding windowing consists of two parts of the cross-learning process. Each step first divides the input features into two regions using both fixed and sliding windowing. In fixed windowing, the features are uniformly divided into multiple non-overlapping windows, with the window size determined by a preset number of windows, confining attention computation within each window. In sliding windowing, based on the previous window division, half-windows are moved along the height and width directions, enabling information interaction between adjacent windows to expand the network's receptive range. Subsequently, the windowed feature sequence is input into a linear mapping layer to generate a query matrix, key matrix, and value matrix. The computation process is as follows:
[0066]
[0067] in, and These represent the division between windowed and sliding windowing, respectively. Represents a linear mapping; , as well as These represent the query matrix, key matrix, and value matrix corresponding to the original input and the adaptive augmentation input, respectively. , as well as These represent the query matrix, key matrix, and value matrix corresponding to the original differential features and the adaptive differential features, respectively.
[0068] Based on this, multi-head cross-attention adopts a dual-branch structure, calculating the query-key relevance between the original features and the adaptive enhancement features respectively, and introducing a learnable relative position embedding vector to mark the feature mapping position. This process is accomplished by direct addition. Simultaneously, the fused relevance matrix is normalized to [0, 1] using the Softmax function. Then, the normalized weights are multiplied by the value matrix of another feature, and the attention distribution result is obtained through linear mapping. In the two corresponding window partitioning methods, windowed multi-head cross-attention learns the regional contrast distribution information of the two features twice, thereby obtaining the regional contrast relationship between the original input and the adaptive enhancement input, and between the adaptive enhancement input and the original input. In a further sliding window-based multi-head cross-attention, the two regional contrast features are again interactively aggregated into a global attention distribution related to uneven degradation, thereby expanding the network's ability to model regional differential degradation. The above process can be described as follows:
[0069]
[0070]
[0071] in, and This represents cross-attention based on windowing and sliding windowing; This represents the softmax function. This indicates the dimension of the input matrix. This represents a learnable relative position embedding. A two-step deep interaction mechanism is established based on cross-attention using windowing and sliding windowing, enabling the network to have a clearer perception of the regional enhancement differences represented by the two features. This further promotes the effective fusion between encoding and decoding features, thereby generating underwater enhanced images with good visibility.
[0072] In summary, the supervision for training the augmented network comes from the joint optimization of Charbonnier loss and VGG loss:
[0073]
[0074] in, and For stabilizing parameters and balancing weights; This represents the convolutional feature representation in the VGG-199 network. Indicates an enhanced image; This refers to a reference image.
[0075] The specific implementation of the method of the present invention is as follows:
[0076] Figure 1 This is the overall network flowchart. The first step involves performing contrast-limited adaptive histogram equalization on the input raw underwater image to generate a corresponding adaptively enhanced image, used to explicitly introduce region enhancement prior information. The second step involves inputting the raw underwater image and the adaptively enhanced image into the original encoder, respectively. and adaptive encoder Both encoders first generate initial raw features and adaptive enhanced features through a shallow feature extraction module, and then input these two types of features into... Figure 2 The cross-fusion Swin Transformer first performs region comparison modeling as shown in formulas (1) and (2) sequentially. The cross-fusion Swin Transformer first unifies the pixel distribution of the two features through layer normalization, and then inputs both features together into... Figure 3 The network uses window-based multi-head cross attention and maps the two features to a query matrix, a key matrix, and a value matrix according to formula (4). The query matrix is used to calculate the correlation between regions, the key matrix is used to store regional feature information, and the value matrix is used to store the feature content to be aggregated. Then, the bidirectional cross attention between the original feature and the adaptive enhancement feature is calculated according to formula (5) to infer the regional distribution comparison between the two features for the other feature. Next, two residual connections are made in the two branches of the cross-fusion Swin Transformer to model the two regional comparison information and generate the corresponding regional difference features. Then, the two features are further interacted and aggregated according to the regional difference degradation modeling shown in formula (3). This process uses multi-head cross attention based on sliding window partitioning to model the global attention related to uneven degradation, i.e., formula (4) and formula (6). The network encodes the features step by step through multiple rounds of pooling downsampling and convolutional activation operations. Each round of encoding introduces the cross-fusion Swin Transformer to output the regional difference fusion features of the corresponding level. The above features are input to the reconstruction decoder after residual connection. The reconstruction decoder progressively decodes and reconstructs the regional differential fusion features through interpolation upsampling and convolutional activation operations to perceive the attenuation distribution of the underwater scene and specifically correct color deviations and detail degradation in different regions, ultimately outputting an enhanced underwater image. During the training phase, the network employs the joint loss function shown in formula (7). The enhancement results are then supervised and optimized. During the testing phase, simply inputting the underwater image to be enhanced into the regional differential fusion network outputs a vibrant, detailed enhanced image.
[0077] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to perform the steps of the method described above.
[0078] The present invention also provides a computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor, wherein when the processor executes the computer program instructions, it can implement the steps of the method described above.
[0079] The above are preferred embodiments of the present invention. Any changes made to the technical solution of the present invention that do not exceed the scope of the technical solution of the present invention shall fall within the protection scope of the present invention.
Claims
1. An underwater image enhancement method based on a regional differential fusion network, characterized in that, include: Input the original underwater image and perform contrast-limited adaptive histogram equalization to generate the corresponding adaptive enhanced image; The original underwater image and the adaptively enhanced image are input into the original encoder, respectively. and adaptive encoder The shallow feature extraction module generates initial raw features and adaptive enhanced features; The original features and adaptive enhanced features are input into the cross-fusion Swin Transformer module to perform regional differential feature fusion; The fused regional difference features are input into the reconstruction decoder. The underwater image is gradually decoded and reconstructed through interpolation upsampling and convolution activation operations, and the enhanced underwater image is output. During the training phase, a joint loss function is used to supervise and optimize the augmentation results. The joint loss function includes Charbonnier loss and VGG loss.
2. The underwater image enhancement method based on a regional differential fusion network according to claim 1, characterized in that, Both the original encoder and the adaptive enhancement encoder include a multi-layer structure, in which: The first coding layer is a shallow feature extraction module consisting of convolutional layers, LReLU activation functions, and convolutional layers. Subsequent encoding layers are all downsampling modules consisting of a max pooling layer, a convolutional layer, and an LReLU activation function connected in sequence.
3. The underwater image enhancement method based on a regional differential fusion network according to claim 1, characterized in that, The cross-integration of the Swin Transformer module specifically executes the following process: Raw features of the input and adaptive enhancement features Layer-by-layer normalization is performed, and the normalized features are then input into a window-based multi-head cross-attention layer. Then, residual connections are performed to generate the first region's differential features. Difference characteristics of the second region : The first and second region difference features are processed by layer normalization, residual connection, and multilayer perceptron (MLP) to obtain the enhanced first region difference features. Difference characteristics of the second region : The enhanced first and second region difference features are subjected to layer normalization and then input into a sliding window-based multi-head cross-attention layer. Global aggregation is performed, and the final fused features are generated through residual connections, layer normalization, multilayer perceptron (MLP) processing, and residual connections. : 。 4. The underwater image enhancement method based on a regional differential fusion network according to claim 3, characterized in that, Operations based on the windowed multi-head cross-attention mechanism include: The input features are divided into multiple non-overlapping fixed windows; Within each window, cross-correlation is calculated between the query matrix and key matrix of the original features and the adaptive augmented features, and a learnable relative position embedding vector is introduced to label the feature mapping position. The correlation matrix is normalized using the Softmax function, then multiplied by the value matrix of another feature, and the cross-attention distribution within the two windows is obtained through linear mapping.
5. The underwater image enhancement method based on a regional differential fusion network according to claim 3, characterized in that, Operations based on the sliding window-based multi-head cross-attention mechanism include: The input features are divided into multiple sliding windows, which are offset by half the window size in the height and width directions relative to the fixed window, so as to realize information interaction between adjacent windows; Within each sliding window, the correlation between the query matrix and key matrix between the original region contrast features and the adaptively enhanced region contrast features is calculated, and a learnable relative position embedding vector is introduced to label the feature mapping position. The correlation matrix is normalized using the Softmax function, then multiplied by the value matrix, and the attention distribution within the window is obtained through linear mapping.
6. The underwater image enhancement method based on a regional differential fusion network according to claim 1, characterized in that, The reconstruction decoder contains a multi-layer upsampling structure, in which: Each upsampling layer consists of interpolation operations, convolutional layers, and LReLU activation functions connected in sequence; The final layer is an image reconstruction layer consisting of convolutional layers, LReLU activation functions, and convolutional layers, which is used to output the final enhanced image.
7. The underwater image enhancement method based on a regional differential fusion network according to claim 1, characterized in that, Joint loss function The specific expression is: in, and For stabilizing parameters and balancing weights; This represents the convolutional feature representation in the VGG-199 network. Indicates an enhanced image; This refers to a reference image.
8. The underwater image enhancement method based on a regional differential fusion network according to claim 1, characterized in that, The cross-fusion Swin Transformer module is called once after each coding level of the original encoder and the adaptive augmentation encoder to achieve the fusion of regionally differentiated features at multiple scales.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1-8.
10. A computer-readable storage medium having stored thereon computer program instructions executable by a processor, wherein when the processor executes the computer program instructions, it is able to implement the steps of the method as described in any one of claims 1-8.