Underwater image enhancement methods, devices, equipment and media

Abstract

This application relates to underwater image enhancement methods, devices, equipment, and media. The method, through innovative design, implements a progressive image inpainting structure with three stages: a Transformer encoder, U-net, and FSnet, and the TCFF and CSFF mechanisms between these stages. This method achieves performance breakthroughs in three dimensions: global semantic information modeling, local semantic information extraction, and spatial detail preservation. It exhibits superior performance in these three dimensions. Based on a Transformer and CNN fusion architecture, this method also possesses high inference efficiency, making it valuable for underwater robot operations that meet the requirements of high-precision image enhancement and real-time processing. Balanced optimization in multi-dimensional semantic representation and detail preservation improves the model's generalization ability, making it applicable to various image inpainting tasks such as rain removal, defogging, and deblurring.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology and relates to an underwater image enhancement method, apparatus, device, and medium. Background Technology

[0002] With the iterative development of underwater robot technology, its applications in underwater surveillance, seabed mapping, underwater archaeology, underwater rescue, and deep-sea sampling are becoming increasingly widespread. The ability to achieve clear perception of the underwater environment directly affects the operational efficiency and application boundaries of marine robots. However, during underwater imaging, light is affected by forward and backward scattering from particles in the water, significantly reducing image quality and causing blurring and degradation during transmission. Furthermore, the different attenuation coefficients of different wavelengths of light result in a noticeable blue-green bias in the image.

[0003] These objective factors not only weaken the environmental perception capabilities of underwater robots, but also restrict their ability to perform downstream tasks based on underwater vision. Therefore, it is essential to develop underwater image enhancement (UIE) methods to improve degraded underwater image signals and make them more suitable for practical applications.

[0004] Traditional UIE methods are typically based on prior physical or statistical knowledge. However, these methods heavily rely on empirical priors, and their design process is quite challenging. Furthermore, because of their excessive reliance on empirical priors, these methods generally lack universality.

[0005] In recent years, advanced UIE methods have shifted towards deep learning techniques, encompassing architectures such as CNN networks and Transformer networks. These methods capture the statistical properties of natural images from large-scale data, thereby learning prior knowledge that is difficult for humans to perceive, ultimately achieving high-quality UIE results.

[0006] Image augmentation networks based on CNNs have spawned various classic architectures, including encoder-decoder structures, densely connected structures, residual connected structures, multi-scale feature fusion structures, and attention mechanisms. While these architectures offer significant advantages in computational efficiency, they suffer from insufficient global semantic information capture capabilities due to the constraint of convolutional kernel size on the receptive field, making it difficult to directly model long-range dependencies in images. In the UIE task, the global color shift caused by light attenuation requires the model to capture the global relationships between pixels, an area where CNNs have an inherent disadvantage.

[0007] In contrast, image augmentation networks based on Transformers, with their global modeling capabilities through self-attention mechanisms, overcome the limitations of local receptive fields in CNNs, demonstrating significant advantages in capturing long-distance dependencies and fusing cross-modal features. However, compared to CNNs, the global modeling nature of Transformers weakens their ability to capture local semantic details, severely restricting the performance of high-precision downstream tasks such as underwater robot target detection and fine manipulation. Furthermore, the design of its self-attention mechanism leads to an exponential increase in the number of model parameters, significantly increasing computational overhead and making it difficult to meet the stringent real-time requirements of applications such as underwater robots.

[0008] Although many cutting-edge works have emerged in the field of image enhancement, and multi-scale feature fusion strategies using complex CNN architectures or Transformers have demonstrated strong image enhancement capabilities by achieving synergy between local and global information, such methods still face the problem of excessive computational complexity. This makes it difficult to overcome the computational bottleneck of real-time inference in UIE tasks, especially to meet the needs of scenarios such as mobile devices and video streaming processing that require millisecond-level response. Summary of the Invention

[0009] To address the problems existing in the above-mentioned traditional methods, this invention proposes an underwater image enhancement method, apparatus, device, and medium. This method significantly improves the multi-scale representation capability of the model while ensuring high inference efficiency.

[0010] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0011] On the one hand, an underwater image enhancement method is provided, the method comprising:

[0012] Acquire underwater images to be processed.

[0013] In stage 1, the underwater image is preprocessed, linearly mapped and positionally encoded, and then the global semantic features of the image are extracted using a Transformer encoder. Based on the global semantic features of the image, the image is reconstructed using an image reconstruction head and then inpainted to obtain the first image.

[0014] In stage 2, after fusing the first image with the underwater image, a feature extraction network based on the Unet architecture is used to extract local semantic features of the image. The TCFF mechanism is introduced between stage 1 and stage 2 to integrate the intermediate global semantic features extracted by the Transformer encoder into stage 2. The local semantic features of the image are processed by convolution and activation functions, and image inpainting is performed to generate the second image. The TCFF mechanism is used to deeply fuse the intermediate global semantic features extracted by the Transformer encoder in stage 1 with the local semantic information extracted by the corresponding encoding stage of the feature extraction network in stage 2.

[0015] In stage 3, the second image and the underwater image are fused and the FSnet model is used to extract spatial detail features. The CSFF mechanism is introduced between stage 2 and stage 3 to achieve feature fusion and output the third image, which is the final enhanced underwater image. The CSFF mechanism is used to fuse the features extracted in the same encoding and decoding stage of the feature extraction network into the intermediate spatial detail features extracted by the FSnet model in stage 3 using convolution operations.

[0016] On the other hand, an underwater image enhancement device is also provided, the device comprising:

[0017] The underwater image acquisition module is used to acquire underwater images to be processed.

[0018] The first enhancement unit is used in stage 1 to preprocess, linearly map, and position-encode the underwater image, and then use a Transformer encoder to extract global semantic features of the image; based on the global semantic features of the image, an image reconstruction head is used to reconstruct the image and perform image inpainting to obtain the first image.

[0019] The second enhancement unit is used in stage 2 to extract local semantic features of the image by fusing the first image with the underwater image and then using a feature extraction network based on the Unet architecture. The TCFF mechanism is introduced between stage 1 and stage 2 to integrate the intermediate global semantic features extracted by the Transformer encoder into stage 2. The local semantic features of the image are processed by convolution and activation functions, and image inpainting is performed to generate the second image. The TCFF mechanism is used to deeply fuse the intermediate global semantic features extracted by the Transformer encoder in stage 1 with the local semantic information extracted by the corresponding encoding stage of the feature extraction network in stage 2.

[0020] The third enhancement unit is used in stage 3 to fuse the second image and the underwater image and then use the FSnet model to extract spatial detail features. The CSFF mechanism is introduced between stage 2 and stage 3 to achieve feature fusion and output the third image, which is the final enhanced underwater image. The CSFF mechanism is used to use convolution operations to fuse the features extracted in the same encoding and decoding stage of the feature extraction network into the intermediate spatial detail features extracted by the FSnet model in stage 3.

[0021] In another aspect, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the underwater image enhancement method described above.

[0022] In another aspect, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the underwater image enhancement method described above.

[0023] One of the above technical solutions has the following advantages and beneficial effects:

[0024] The aforementioned underwater image enhancement method, apparatus, equipment, and medium, through innovative design, realize a progressive image inpainting structure with three stages: Transformer encoder, U-net, and FSnet, and the TCFF and CSFF mechanisms between these three stages. This method achieves performance breakthroughs in three dimensions: global semantic information modeling, local semantic information extraction, and spatial detail preservation. It exhibits superior performance in these three dimensions; furthermore, based on a Transformer and CNN fusion architecture, it possesses high inference efficiency, making it valuable for underwater robot operations that meet the requirements of high-precision image enhancement and real-time processing. The balanced optimization of multi-dimensional semantic representation and detail preservation improves the model's generalization ability, making it applicable to various image inpainting tasks such as rain removal, defogging, and deblurring. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart illustrating an underwater image enhancement method in one embodiment;

[0027] Figure 2This is a schematic diagram of the network structure of an underwater image enhancement method in one embodiment;

[0028] Figure 3 This is a schematic diagram showing the module design details in one embodiment, wherein... Figure 3 (a) Detailed schematic diagram of the Conv module design. Figure 3 (b) Detailed schematic diagram of the Transformer layer design. Figure 3 (c) Schematic diagram showing details of the image reconstruction head design;

[0029] Figure 4 This is a schematic diagram of the migration experiment results in one embodiment;

[0030] Figure 5 This is a schematic diagram of the generalization experiment results in one embodiment. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0033] It should be noted that, in this document, the reference to "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The presentation of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will understand that the embodiments described herein can be combined with other embodiments. The term "and / or" as used herein refers to any combination of one or more of the associated listed items, and all possible combinations, including such combinations.

[0034] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0035] In one embodiment, such as Figure 1 As shown, an underwater image enhancement method is provided, which may include the following processing steps 100 to 106:

[0036] Step 100: Obtain the underwater image to be processed.

[0037] Specifically, the underwater images to be processed can be images taken by underwater robots during operations.

[0038] Step 102: In stage 1, after preprocessing, linear mapping and position encoding of the underwater image, the Transformer encoder is used to extract the global semantic features of the image; based on the global semantic features of the image, the image reconstruction head is used to reconstruct the image and perform image repair to obtain the first image.

[0039] Specifically, Stage 1 employs a Transformer architecture, facilitating the network's modeling of global semantic information. This stage first segments the input image into several image patches and maps them to a one-dimensional sequence that meets the requirements of Transformer processing. Next, the sequence is positionally encoded to incorporate spatial location information. Then, a standard Transformer encoder is used for feature learning to capture cross-regional semantic dependencies. Finally, an Image Reconstruction Head (IRH) is used to reconstruct the image from features. This design leverages the long-range modeling advantage of the Transformer to enhance the network's ability to represent the global semantics of the image.

[0040] Step 104: In stage 2, after fusing the first image with the underwater image, a feature extraction network based on the Unet architecture is used to extract local semantic features of the image. The TCFF mechanism is introduced between stage 1 and stage 2 to integrate the intermediate global semantic features extracted by the Transformer encoder into stage 2. The local semantic features of the image are processed by convolution and activation functions, and image inpainting is performed to generate the second image. The TCFF mechanism is used to deeply fuse the intermediate global semantic features extracted by the Transformer encoder in stage 1 with the local semantic information extracted by the corresponding encoding stage of the feature extraction network in stage 2.

[0041] Specifically, Stage 2 employs the Unet architecture to enhance the network's ability to capture local details. This stage's computation is divided into an encoder and a decoder. The input image is first processed by the encoder through multiple convolutional layers to extract features. Max pooling downsampling is then used to progressively reduce the feature map size and increase the number of channels, capturing multi-scale semantic information. The decoder restores the spatial dimensions through deconvolution and simultaneously concatenates the features from the corresponding encoder layers with CAB and skip connections, fusing shallow spatial details with deep semantic information. Finally, convolution and activation functions are used to generate the image. This design leverages Unet's strengths in deeply extracting local semantic information from images, enhancing the network's ability to represent local image semantics.

[0042] To achieve deep fusion of global semantic information, local semantic information, and spatial detail information, a novel feature fusion mechanism was designed between stage 1 and stage 2. Specifically, the intermediate sequence generated by the Transformer encoder is reconstructed by the Image Reconstruction Head (IRH) and then processed by convolution before being integrated into stage 2, forming the TCFF mechanism.

[0043] The Transformer-CNN fusion architecture aims to combine the local feature extraction capabilities of CNNs with the long-range dependency modeling advantages of Transformers to solve more complex tasks. In the field of image inpainting, Uformer introduces a Transformer module into the U-Net architecture. The lower layers use CNNs to extract multi-scale features, while the higher layers use Transformers to model global semantic dependencies, improving the ability to restore complex textures and structures in image inpainting tasks. Inspired by this, this method adopts a hybrid architecture where Transformers and CNNs work together, and introduces the TCFF mechanism to achieve deep fusion of global and local semantic information, ultimately achieving high-quality restoration of degraded images and accurate restoration of visual details.

[0044] Step 106: In stage 3, the second image and the underwater image are fused and the FSnet model is used to extract spatial detail features. The CSFF mechanism is introduced between stage 2 and stage 3 to achieve feature fusion and output the third image, which is the final enhanced underwater image. The CSFF mechanism is used to fuse the features extracted by the same encoding and decoding stage of the feature extraction network into the intermediate spatial detail features extracted by the FSnet model in stage 3 using convolution operation.

[0045] Specifically, this method employs a multi-stage network architecture. This architecture breaks down complex tasks into lightweight, sequential stages, progressively enhancing the model's information processing capabilities and task accuracy. In the field of image inpainting, MPRNet uses a multi-stage progressive inpainting strategy, moving from coarse global structural inpainting to fine-tuning local textures, achieving high-quality reconstruction of degraded images. This design is effective because it breaks down challenging image inpainting tasks into smaller, easier sub-tasks. Therefore, this method references this type of network architecture design, aiming to achieve high-quality inpainting and detail restoration of degraded images through a deep fusion mechanism of multi-scale information.

[0046] Considering that downsampling operations in the encoder structure of Stage 2 would lead to the loss of spatial details, Stage 3 was designed. This stage effectively preserves the spatial details of the image by using a feedforward subnetwork that does not contain any downsampling.

[0047] The advantages of the CSFF (Cross-stage Feature Fusion) mechanism are: it alleviates the information loss caused by the downsampling operation of the encoder structure in stage 2, making the network less fragile; the multi-scale features of the second stage help enrich the features of the third stage; and it eases the information flow, making the network optimization process more stable.

[0048] The underwater image enhancement model includes: the Transformer encoder, a feature extraction network based on the Unet architecture, the FSnet model, the TCFF mechanism, and the CSFF mechanism; the structure of the underwater image enhancement model is as follows: Figure 2 As shown.

[0049] The aforementioned underwater image enhancement method, through innovative design, implements a progressive image inpainting structure with three stages: Transformer encoder, U-net, and FSnet, as well as TCFF and CSFF mechanisms between these stages. This method achieves performance breakthroughs in three dimensions: global semantic information modeling, local semantic information extraction, and spatial detail preservation. It exhibits superior performance in these three dimensions; furthermore, based on a Transformer and CNN fusion architecture, it possesses high inference efficiency, making it valuable for underwater robot operations that meet the requirements of both high-precision image enhancement and real-time processing. Balanced optimization in multi-dimensional semantic representation and detail preservation improves the model's generalization ability, making it applicable to various image inpainting tasks such as rain removal, defogging, and deblurring.

[0050] In one embodiment, the Transformer encoder includes: a plurality of stacked Transformer layers (the Transformer layers are designed as follows...). Figure 3 As shown in (b). It achieves efficient modeling of global semantic information by leveraging the collaborative architecture of self-attention mechanism and fully connected feedforward network; the Transformer layer includes several sequentially connected Transformer modules and a layer normalization layer; the Transformer module includes a multi-head self-attention mechanism and a fully connected feedforward network; step 102 includes: segmenting the underwater image into several image blocks and mapping them into a one-dimensional sequence that meets the requirements of Transformer processing; after position encoding the one-dimensional sequence, global semantic information is modeled through several stacked Transformer layers to obtain global semantic features; based on the global semantic features, an image reconstruction head is used to reconstruct the image from features; based on the reconstructed image and the underwater image, the SAM module is used to perform image inpainting to obtain the first image.

[0051] In one embodiment, the TCFF mechanism includes an image reconstruction head and a convolutional layer; step 104 includes: fusing the first image with the underwater image and then using a Unet-based feature extraction network to extract local semantic features of the image; and then processing the intermediate sequences generated by each Transformer encoder in stage 1 through the image reconstruction head and convolutional layer, and then integrating them into the local semantic features output by the corresponding layer of the encoder in stage 2 of the Unet-based feature extraction network to obtain local semantic features of the image; processing the local semantic features of the image using convolution and activation functions; and using the SAM module to perform image inpainting based on the obtained processing results and the underwater image to generate a second image.

[0052] In one embodiment, the Unet-based feature extraction network includes: an encoder and a decoder; in the Unet-based feature extraction network: stitching a first image with an underwater image to obtain a feature image; and applying a Double... After processing by Conv, the output features are input into the first stage of the encoder to obtain the output features of the first stage. The output features of the first stage of the encoder are fused with the intermediate sequence generated by the first layer of the Transformer encoder and then max-pooled downsampled. The downsampled result is input into the second stage of the encoder to obtain the output features of the second stage. The output features of the second stage of the encoder are fused with the intermediate sequence generated by the second layer of the Transformer encoder and then max-pooled downsampled. The downsampled result is input into the third stage of the encoder to obtain the output features of the third stage. The output features of the third stage of the encoder are fused with the intermediate sequence generated by the third layer of the Transformer encoder and then processed by the CAB module. The result is then input into the first stage of the decoder through skip connections to obtain the output features of the first stage of the decoder. The output features of the first and second stages of the encoder are processed by the CAB module and then input into the third and second stages of the decoder through skip connections to obtain the output features of the third and second stages of the decoder, respectively. The decoder restores the spatial dimensions through deconvolution operations, fusing shallow spatial details with deep semantic information.

[0053] In one embodiment, the FSnet model includes three feedforward sub-networks; each feedforward sub-network includes one Conv module and two CAB modules; the Conv module includes multiple convolutional modules; each convolutional module includes a convolutional layer, a ReLU activation function, and a batch normalization layer; the CSFF mechanism includes multiple convolutional layers; step 106 includes: fusing the features obtained by fusing the second image and the underwater image, the output features of the encoder of the Unet architecture feature extraction network after processing by the first convolutional layer of the CSFF mechanism, and the output features of the decoder of the Unet architecture feature extraction network after processing by the second convolutional layer of the CSFF mechanism to obtain a first fused feature; inputting the first fused feature into the first feedforward sub-network, processing the first fused feature through the Conv module and the first CAB module respectively, and then fusing them together; processing the fused result through the second CAB module to obtain a first spatial detail feature; and combining the result of the first spatial detail feature processed by the CAB module with the second-stage output features of the encoder of the Unet architecture feature extraction network. The results of the third convolutional layer processing by the CSFF mechanism and the output features of the second stage of the decoder of the Unet architecture feature extraction network processed by the fourth convolutional layer of the CSFF mechanism are added and fused to obtain the second fused feature. The second fused feature is then input into the second feedforward sub-network to obtain the second spatial detail feature. The results of the second spatial detail feature processed by the CAB module, the output features of the third stage of the encoder of the Unet architecture feature extraction network processed by the fifth convolutional layer of the CSFF mechanism, and the output features of the third stage of the decoder of the Unet architecture feature extraction network processed by the sixth convolutional layer of the CSFF mechanism are added and fused to obtain the third fused feature. The third fused feature is then input into the third feedforward sub-network to obtain the third spatial detail feature. The third spatial detail feature is then processed by the CAB module and added and fused with the features obtained from the fusion of the second image and the underwater image to obtain the final fused feature. The final fused feature is then processed by the seventh CAB module and then convolved to obtain the output third image, which is the final enhanced underwater image.

[0054] Specifically, the Conv module is designed as follows: Figure 3 As shown in (a), this module is constructed by cascading multiple convolutional blocks and using a residual connection mechanism. It can efficiently capture and preserve the spatial details of the image while adhering to a lightweight design concept, introducing only a very small number of parameters.

[0055] In one embodiment, the image reconstruction head is used to achieve structured reconstruction of a sequence into a feature map by cascading a Linear layer and a Rearrange layer.

[0056] Specifically, the design of the Image Reconstruction Head (IRH) is as follows: Figure 3 As shown in (c), it achieves structured reconstruction of the sequence into feature maps through the cascading of the Linear layer and the Rearrange layer, which constitutes the key basic component of the TCFF mechanism.

[0057] In one embodiment, the underwater image enhancement model includes: the Transformer encoder, a Unet-based feature extraction network, an FSnet model, a TCFF mechanism, and a CSFF mechanism. In underwater images, significant structural and textural information is highly correlated with human subjective perception. Therefore, during model training, it is desirable to guide the model to focus on high-frequency texture structure information in the image through loss function design. The loss function used in the underwater image enhancement model training process is:

[0058] ;

[0059] ;

[0060] ;

[0061] in, Represents the loss function. X Represents the predicted image. Y Represents real images. Let be an experimental constant. Represents marginal loss. Represents L1 loss, represents the Laplace operator, N represents the total number of data points, and ||.||1 represents the L1 norm.

[0062] In some implementations, experimental examples are also provided. In this embodiment, the following five datasets were selected to evaluate the performance, transferability, and generalization ability of this model.

[0063] The EUVP dataset is a large-scale dataset designed specifically for underwater visual perception enhancement tasks. It contains multiple paired and unpaired images, comprehensively covering both low-quality and high-quality perception samples. This dataset was collected in the field by the research team during explorations of various marine environments, covering different seawater visibility conditions. The paired samples were generated by degrading real-world images using an underwater image distortion model based on the CycleGAN architecture, providing training data that combines realism and diversity for underwater image enhancement tasks.

[0064] The UFO-120 dataset contains 1620 pairs of underwater images, all collected during marine exploration in different aquatic environments. Researchers then used style transfer techniques to generate corresponding distorted images, making the dataset suitable for evaluating the performance of underwater image enhancement and super-resolution models.

[0065] The UIEB dataset contains 890 real-world underwater images, all captured under varying lighting conditions, exhibiting diverse color distributions and contrast levels. Researchers have provided corresponding reference images, which show no color deviation and demonstrate significantly improved visibility and brightness compared to the source images.

[0066] Rain12600 and Rain1400. Rain12600 and Rain1400 contain 12,600 pairs of training images and 1,400 pairs of test images, respectively. Both are based on scenes from datasets such as BSD and are synthesized using rain trail simulation technology according to 14 rain condition parameters, covering diverse scenes such as natural and urban areas, providing key resources for image deraining algorithm research.

[0067] The partitioning of the training and test sets for all datasets during training is shown in Table 1, and all images were reshaped to 256 pixels × 256 pixels.

[0068] Table 1. Division of training and test sets during training.

[0069]

[0070] (1) Evaluation indicators

[0071] To quantitatively analyze the quality of the images generated by this method, this embodiment uses Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) as evaluation metrics. PSNR is calculated based on pixel mean square error; a higher value indicates less image distortion. SSIM measures image similarity from three aspects: brightness, contrast, and structure, which is more consistent with human perception of image structure. Its value ranges from 0 to 1, with values ​​closer to 1 indicating higher image quality.

[0072] (2) Experimental equipment and hyperparameters

[0073] The hardware and software environment and hyperparameter configuration for model training are as follows: the operating system is Ubuntu 22.04, the hardware is equipped with an Intel(R) Xeon(R) Gold 6330 CPU and an Nvidia GTX 3090 GPU, and the software environment uses Python 3.12 and the PyTorch 2.3.0 framework; the training hyperparameters are set as follows: learning rate 0.0001, batch size 8, and number of training epochs 100.

[0074] (3) UIE Experiment

[0075] To evaluate our proposed model, qualitative and quantitative comparisons were performed on underwater images with various state-of-the-art image enhancement methods. All learnable models were trained for 100 epochs under identical conditions.

[0076] Table 2 systematically evaluates the enhancement effect and inference efficiency of the proposed model compared to other models from four dimensions: model parameter count, average inference time per image, PSNR, and SSIM. Experimental data show that the proposed model achieves state-of-the-art (SOTA) performance on both the EUVP and UFO-120 datasets. Notably, the performance of the Transformer-based model is even weaker than that of the CNN-based model. In-depth analysis reveals that this phenomenon stems from an inherent deficiency in the Transformer model's ability to recover local details, leading to a lower structure-related index.

[0077] Although CNN-based models outperform Transformer models slightly on structural evaluation metrics, from Figure 3 It is evident that in color restoration tasks, CNNs outperform pure Transformer models. This difference is primarily attributed to the Transformer model's stronger long-range global semantic modeling capabilities, giving it a significant advantage in capturing and restoring image color information. Qualitative analysis also shows that our proposed model achieves state-of-the-art (SOTA) color restoration performance.

[0078] In summary, the hybrid architecture design employed in this method effectively compensates for the shortcomings of the Transformer model in local semantic information extraction and the limitations of CNN in global semantic modeling. This design organically integrates the advantages of both architectures, achieving synergistic optimization of global semantic information modeling capabilities and local semantic information recovery capabilities.

[0079] (4) Transfer experiment

[0080] To evaluate the cross-dataset transfer performance of the proposed model, this embodiment directly applies the model trained on the EUVP dataset to the UIEB dataset for inference testing. The transfer experiment results are as follows: Figure 4As shown in Table 2, the experimental results demonstrate that, under training conditions on small to medium-sized datasets, the proposed model not only achieves the highest evaluation metric scores on the original training datasets EUVP and UFO-120, but also significantly outperforms most comparative models in cross-dataset transfer tests. Its transfer capability is comparable to the lightweight model Shallow-UWnet (PSNR decreases by only 0.08 dB, and SSIM decreases by only 0.015). The transfer experiments fully validate the dual advantages of the proposed model: excellent basic modeling capabilities and outstanding cross-domain transfer adaptability, providing solid experimental support for its application in real-world transfer scenarios.

[0081] Table 2. Results of the migration experiment

[0082]

[0083] The model of this method achieves state-of-the-art (SOTA) quantization metrics while maintaining high inference performance, with an average inference time of only 0.032 seconds per image.

[0084] (5) Generalization experiment

[0085] This method's model is designed for the UIE task, primarily due to its outstanding performance in global semantic modeling, local feature extraction, and computational efficiency, giving it a more comprehensive advantage in UIE. In fact, thanks to the model's efficient modeling of global semantic information and accurate extraction of local detail features, this method's model possesses generalizability for various image restoration tasks (such as deraining and dehazing). For these reasons, no special optimization strategies for the UIE task were adopted when designing the loss function and quantization metrics. This design choice both verifies the model architecture's versatility and ensures consistent performance across different image restoration scenarios.

[0086] To verify the generalization ability of the proposed model in different types of image inpainting tasks, experiments were conducted on the classic rain removal datasets Rain12600 and Rain1400. Considering that CBMSF, GUDCP, and UDnet were not originally designed for rain removal tasks, these three comparison models were removed from this part of the experiment. The Uformer model was introduced as a new comparison model, and all models were trained for 100 epochs. The generalization experiment results are shown in Table 3. Figure 5 As shown, this method still performs excellently on the rain removal task, even significantly outperforming models specifically designed for image restoration (MPRnet, Uformer).

[0087] Table 3 Generalization Experiment Results

[0088]

[0089] (6) Ablation test

[0090] To verify the effectiveness of each module in the proposed model, this embodiment designed a series of ablation experiments. During the experiments, an ablation strategy was employed to remove modules one by one, sequentially removing TCFF, CSFF, SAM, FSnet, and Unet to generate five comparative models: E, D, C, B, and A. All models were trained and tested on the UFO-120 dataset for 100 training epochs.

[0091] The results of the ablation experiments are shown in Table 4. It can be seen that as each module is removed sequentially, the score index of each model decreases accordingly. This proves that each module contributes to the image enhancement effect, and the model design of this method does not have redundant structures. It is noteworthy that Model A shows a larger score reduction compared to Model B (PSNR decreases by 3.18 dB, SSIM decreases by 0.082), which further demonstrates that using a fusion architecture of Transformer and CNN to design a neural network is beneficial for image enhancement tasks.

[0092] Table 4 Ablation Experiment Results

[0093]

[0094] It should be understood that, although the above process Figure 1 The steps in the diagram are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order in which these steps are executed; they can be performed in other orders. Furthermore, the above process... Figure 1 At least some of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0095] In one embodiment, an underwater image enhancement device is also provided, the device comprising:

[0096] The underwater image acquisition module is used to acquire underwater images to be processed.

[0097] The first enhancement unit is used in stage 1 to preprocess, linearly map, and position-encode the underwater image, and then use a Transformer encoder to extract global semantic features of the image; based on the global semantic features of the image, an image reconstruction head is used to reconstruct the image and perform image inpainting to obtain the first image.

[0098] The second enhancement unit is used in stage 2 to extract local semantic features of the image by fusing the first image with the underwater image and then using a feature extraction network based on the Unet architecture. The TCFF mechanism is introduced between stage 1 and stage 2 to integrate the intermediate global semantic features extracted by the Transformer encoder into stage 2. The local semantic features of the image are processed by convolution and activation functions, and image inpainting is performed to generate the second image. The TCFF mechanism is used to deeply fuse the intermediate global semantic features extracted by the Transformer encoder in stage 1 with the local semantic information extracted by the corresponding encoding stage of the feature extraction network in stage 2.

[0099] The third enhancement unit is used in stage 3 to fuse the second image and the underwater image and then use the FSnet model to extract spatial detail features. The CSFF mechanism is introduced between stage 2 and stage 3 to achieve feature fusion and output the third image, which is the final enhanced underwater image. The CSFF mechanism is used to use convolution operations to fuse the features extracted in the same encoding and decoding stage of the feature extraction network into the intermediate spatial detail features extracted by the FSnet model in stage 3.

[0100] In one embodiment, the Transformer encoder includes: a plurality of stacked Transformer layers; the Transformer layer in the first enhancement unit includes a plurality of sequentially connected Transformer modules and a layer normalization layer; the Transformer module includes a multi-head self-attention mechanism and a fully connected feedforward network; the first enhancement unit is further configured to segment the underwater image into a plurality of image patches and map them into a one-dimensional sequence that meets the requirements of Transformer processing; after position encoding of the one-dimensional sequence, global semantic information is modeled through the plurality of stacked Transformer layers to obtain global semantic features; based on the global semantic features, an image reconstruction head is used to reconstruct the image from features; based on the reconstructed image and the underwater image, an SAM module is used to perform image inpainting to obtain a first image.

[0101] In one embodiment, the TCFF mechanism includes an image reconstruction head and a convolutional layer; the second enhancement unit is further configured to fuse the first image with the underwater image and then use a Unet-based feature extraction network to extract local semantic features of the image, and to integrate the intermediate sequences generated by each Transformer encoder in stage 1 into the local semantic features output by the corresponding layer of the encoder in stage 2 after processing by the image reconstruction head and convolutional layer, thereby obtaining local semantic features of the image; to process the local semantic features of the image using convolution and activation functions; and to perform image inpainting using the SAM module based on the obtained processing results and the underwater image, thereby generating a second image.

[0102] In one embodiment, the Unet-based feature extraction network in the second enhancement unit includes an encoder and a decoder; in the Unet-based feature extraction network: the first image is stitched together with the underwater image to obtain a feature image; the feature image is then processed using a Double... After processing by Conv, the output features are input into the first stage of the encoder to obtain the output features of the first stage. The output features of the first stage of the encoder are fused with the intermediate sequence generated by the first layer of the Transformer encoder and then max-pooled downsampled. The downsampled result is input into the second stage of the encoder to obtain the output features of the second stage. The output features of the second stage of the encoder are fused with the intermediate sequence generated by the second layer of the Transformer encoder and then max-pooled downsampled. The downsampled result is input into the third stage of the encoder to obtain the output features of the third stage. The output features of the third stage of the encoder are fused with the intermediate sequence generated by the third layer of the Transformer encoder and then processed by the CAB module. The result is then input into the first stage of the decoder through skip connections to obtain the output features of the first stage of the decoder. The output features of the first and second stages of the encoder are processed by the CAB module and then input into the third and second stages of the decoder through skip connections to obtain the output features of the third and second stages of the decoder, respectively. The decoder restores the spatial dimensions through deconvolution operations, fusing shallow spatial details with deep semantic information.

[0103] In one embodiment, the FSnet model includes three feedforward sub-networks; each feedforward sub-network includes one Conv module and two CAB modules; the Conv module includes multiple convolutional modules; each convolutional module includes a convolutional layer, a ReLU activation function, and a batch normalization layer; the CSFF mechanism includes multiple convolutional layers; a third enhancement unit is further used to add and fuse the features of the fused second image and underwater image, the output features of the encoder of the Unet architecture feature extraction network after processing by the first convolutional layer of the CSFF mechanism, and the output features of the decoder of the Unet architecture feature extraction network after processing by the second convolutional layer of the CSFF mechanism, to obtain a first fused feature; the first fused feature is input into the first feedforward sub-network, and the first fused feature is processed by the Conv module and the first CAB module respectively and then added and fused; the fused result is processed by the second CAB module to obtain the first spatial detail feature; the result of the first spatial detail feature processed by the CAB module and the feature of the encoder of the Unet architecture feature extraction network after processing by the second convolutional layer are added and fused to obtain the first spatial detail feature; the first spatial detail feature processed by the CAB module and the first spatial detail feature of the encoder of the Unet architecture feature extraction network are added and fused to obtain the first spatial detail feature. The results of the features processed by the third convolutional layer of the CSFF mechanism and the output features of the second stage of the decoder of the Unet architecture feature extraction network processed by the fourth convolutional layer of the CSFF mechanism are added and fused to obtain the second fused feature. The second fused feature is then input into the second feedforward sub-network to obtain the second spatial detail feature. The results of the second spatial detail feature processed by the CAB module, the output features of the third stage of the encoder of the Unet architecture feature extraction network processed by the fifth convolutional layer of the CSFF mechanism, and the output features of the third stage of the decoder of the Unet architecture feature extraction network processed by the sixth convolutional layer of the CSFF mechanism are added and fused to obtain the third fused feature. The third fused feature is then input into the third feedforward sub-network to obtain the third spatial detail feature. The third spatial detail feature is then processed by the CAB module and added and fused with the features obtained by fusing the second image and the underwater image to obtain the final fused feature. The final fused feature is then processed by the seventh CAB module and then convolved to obtain the output third image, which is the final enhanced underwater image.

[0104] In one embodiment, the image reconstruction head in the first enhancement unit is used to achieve structured reconstruction of the sequence into a feature map by cascading a Linear layer and a Rearrange layer.

[0105] In one embodiment, the underwater image enhancement model includes: the Transformer encoder, a Unet-based feature extraction network, an FSnet model, a TCFF mechanism, and a CSFF mechanism; the loss function during the training process of the underwater image enhancement model is:

[0106] ;

[0107] ;

[0108] ;

[0109] in, Represents the loss function. X Represents the predicted image. Y Represents real images. Let be an experimental constant. Represents marginal loss. Represents L1 loss, represents the Laplace operator, N represents the total number of data points, and ||.||1 represents the L1 norm.

[0110] It is understood that for detailed explanations of the underwater image enhancement device, please refer to the corresponding explanations of the various embodiments of the underwater image enhancement method above, and will not be repeated here. Each module in the above-described underwater image enhancement device can be implemented entirely or partially through software, hardware, or a combination thereof. Each module can be embedded in hardware or independently of a device with data processing capabilities, or stored in software in the memory of the aforementioned device, so that the processor can call and execute the operations corresponding to each module. The aforementioned device can be, but is not limited to, various types of data processing computer devices already existing in the art.

[0111] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in any of the above method embodiments.

[0112] It is understood that, in addition to the memory and processor mentioned above, the computer equipment described above also includes other hardware and software components not listed in this specification. The specific components can be determined according to the model of the image processing computer in different application scenarios, and will not be listed and described in detail in this specification.

[0113] In one embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps in any of the above method embodiments.

[0114] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), memory bus DRAM (RDRAM), and interface DRAM (DRDRAM), etc.

[0115] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0116] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and all such modifications and improvements fall within the scope of protection of this application.

Claims

1. An underwater image enhancement method, characterized in that, The method includes: Acquire underwater images to be processed; In stage 1, the underwater image is preprocessed, linearly mapped, and positionally encoded. Then, a Transformer encoder is used to extract global semantic features of the image. Based on the global semantic features of the image, an image reconstruction head is used to reconstruct the image and perform image inpainting to obtain the first image. In stage 2, after fusing the first image with the underwater image, a feature extraction network based on the Unet architecture is used to extract local semantic features of the image. A TCFF mechanism is introduced between stage 1 and stage 2 to integrate the intermediate global semantic features extracted by the Transformer encoder into stage 2. The local semantic features of the image are processed using convolution and activation functions, and image inpainting is performed to generate the second image. The TCFF mechanism is used to deeply fuse the intermediate global semantic features extracted by the Transformer encoder in stage 1 with the local semantic information extracted by the corresponding encoding stage of the feature extraction network in stage 2. The TCFF mechanism includes an image reconstruction head and convolutional layers. In stage 3, the second image and the underwater image are fused and then spatial detail features are extracted using the FSnet model. A CSFF mechanism is introduced between stage 2 and stage 3 to achieve feature fusion, outputting a third image, i.e., the final enhanced underwater image. The CSFF mechanism is used to fuse features extracted in the same encoding and decoding stage of the feature extraction network into the intermediate spatial detail features extracted by the FSnet model in stage 3 using convolutional operations. The FSnet model includes three feedforward sub-networks. The feedforward sub-networks include one Conv module and two CAB modules. The Conv module includes multiple convolutional modules. Each convolutional module includes a convolutional layer, a ReLU activation function, and a batch normalization layer. The CSFF mechanism includes multiple convolutional layers.

2. The underwater image enhancement method according to claim 1, characterized in that, The Transformer encoder includes: several stacked Transformer layers; each Transformer layer includes several sequentially connected Transformer modules and a layer normalization layer; the Transformer module includes a multi-head self-attention mechanism and a fully connected feedforward network; After preprocessing, linear mapping, and position encoding of the underwater image, a Transformer encoder is used to extract global semantic features. Based on the global semantic features, an image reconstruction head is used to reconstruct the image and perform image inpainting to obtain a first image, including: The underwater image is segmented into several image blocks and mapped into a one-dimensional sequence that meets the requirements of Transformer processing. After position encoding of the one-dimensional sequence, global semantic information is modeled through several stacked Transformer layers to obtain global semantic features; Based on the global semantic features, an image reconstruction head is used to reconstruct the image from the features. Based on the reconstructed image and the underwater image, the SAM module is used to perform image restoration to obtain the first image.

3. The underwater image enhancement method according to claim 1, characterized in that, After fusing the first image with the underwater image, a feature extraction network based on the Unet architecture is used to extract local semantic features of the image. A TCFF mechanism is introduced between stage 1 and stage 2 to incorporate the intermediate global semantic features extracted by the Transformer encoder into stage 2. The local semantic features of the image are processed using convolution and activation functions, and image inpainting is performed to generate a second image, including: After fusing the first image with the underwater image, a feature extraction network based on the Unet architecture is used to extract local semantic features of the image. The intermediate sequence generated by each Transformer encoder in stage 1 is processed by the image reconstruction head and convolutional layer and then integrated into the local semantic features output by the corresponding layer of the encoder of the feature extraction network based on the Unet architecture in stage 2 to obtain the local semantic features of the image. Local semantic features of the image are processed using convolution and activation functions; Based on the processing results and the underwater image, the SAM module is used to perform image restoration to generate a second image.

4. The underwater image enhancement method according to claim 3, characterized in that, The feature extraction network based on the Unet architecture includes: an encoder and a decoder; In feature extraction networks based on the Unet architecture: The first image is stitched together with the underwater image to obtain the feature image; The feature image is processed using Double Conv and then input into the first stage of the encoder to obtain the output features of the first stage of the encoder. The output features of the first stage of the encoder are fused with the intermediate sequence generated by the first layer of the Transformer encoder and then max pooled downsampled. The downsampled result is then input into the second stage of the encoder to obtain the output features of the second stage of the encoder. The output features of the second stage of the encoder are fused with the intermediate sequence generated by the second layer of the Transformer encoder and then max-pooled downsampled. The downsampled result is then input into the third stage of the encoder to obtain the output features of the third stage of the encoder. The output features of the third stage of the encoder are fused with the intermediate sequence generated by the third layer of the Transformer encoder and then processed by the CAB module. The result is then input into the first stage of the decoder through a skip connection to obtain the output features of the first stage of the decoder. The output features of the first and second stages of the encoder are processed by the CAB module and then fed into the third and second stages of the decoder via skip connections to obtain the output features of the third and second stages of the decoder. The decoder restores the spatial dimensions through deconvolution operations and fuses shallow spatial details with deep semantic information.

5. The underwater image enhancement method according to claim 1, characterized in that, In stage 3, the second image and the underwater image are fused, and the FSnet model is used to extract spatial detail features. A CSFF mechanism is introduced between stage 2 and stage 3 to achieve feature fusion, outputting a third image, i.e., the final enhanced underwater image, including: The first fused feature is obtained by adding and fusing the features obtained by fusing the second image and the underwater image, the output features of the encoder of the Unet architecture feature extraction network after processing the first convolutional layer of the CSFF mechanism, and the output features of the decoder of the Unet architecture feature extraction network after processing the second convolutional layer of the CSFF mechanism. The first fused feature is input into the first feedforward sub-network. The first fused feature is processed by the Conv module and the first CAB module respectively and then added together. The fused result is processed by the second CAB module to obtain the first spatial detail feature. The first spatial detail feature is processed by the CAB module, the second stage output feature of the encoder of the Unet architecture feature extraction network is processed by the third convolutional layer of the CSFF mechanism, and the second stage output feature of the decoder of the Unet architecture feature extraction network is processed by the fourth convolutional layer of the CSFF mechanism. The two features are then added together and fused to obtain the second fused feature. The second fused feature is input into the second feedforward sub-network to obtain the second spatial detail feature; The results of the second spatial detail features processed by the CAB module, the output features of the third stage of the encoder of the Unet architecture feature extraction network processed by the fifth convolutional layer of the CSFF mechanism, and the output features of the third stage of the decoder of the Unet architecture feature extraction network processed by the sixth convolutional layer of the CSFF mechanism are added and fused to obtain the third fused feature. The third fusion feature is input into the third feedforward sub-network to obtain the third spatial detail feature; The third spatial detail features are processed by the CAB module and then fused with the features obtained from the fusion of the second image and the underwater image to obtain the final fused features; The final fused features are processed by the seventh CAB module and then convolved to obtain the third output image, which is the final enhanced underwater image.

6. The underwater image enhancement method according to claim 1, characterized in that, The image reconstruction head is used to achieve structured reconstruction of sequences into feature maps by cascading linear and rearrange layers.

7. The underwater image enhancement method according to claim 1, characterized in that, The underwater image enhancement model includes: the Transformer encoder, a feature extraction network based on the Unet architecture, the FSnet model, the TCFF mechanism, and the CSFF mechanism; the loss function during the training process of the underwater image enhancement model is: in, Represents the loss function. X Represents the predicted image. Y Represents real images. Let be an experimental constant. Represents marginal loss. Represents L1 loss, represents the Laplace operator, N represents the total number of data points, and ||.||1 represents the L1 norm.

8. An underwater image enhancement device, characterized in that, The device includes: The underwater image acquisition module is used to acquire underwater images to be processed. The first enhancement unit is used in stage 1 to preprocess, linearly map, and positionally encode the underwater image, and then use a Transformer encoder to extract global semantic features of the image; based on the global semantic features of the image, an image reconstruction head is used to reconstruct the image and perform image inpainting to obtain the first image; The second enhancement unit is used in stage 2 to fuse the first image with the underwater image and then use a feature extraction network based on the Unet architecture to extract local semantic features of the image. A TCFF mechanism is introduced between stage 1 and stage 2 to integrate the intermediate global semantic features extracted by the Transformer encoder into stage 2. The local semantic features of the image are processed using convolution and activation functions, and image inpainting is performed to generate the second image. The TCFF mechanism is used to deeply fuse the intermediate global semantic features extracted by the Transformer encoder in stage 1 with the local semantic information extracted by the corresponding encoding stage of the feature extraction network in stage 2. The TCFF mechanism includes an image reconstruction head and convolutional layers. The third enhancement unit, in stage 3, fuses the second image and the underwater image and then uses the FSnet model to extract spatial detail features. A CSFF mechanism is introduced between stages 2 and 3 to achieve feature fusion, outputting a third image, i.e., the final enhanced underwater image. The CSFF mechanism uses convolutional operations to fuse features extracted from the same encoding / decoding stage of the feature extraction network into the intermediate spatial detail features extracted by the FSnet model in stage 3. The FSnet model includes three feedforward sub-networks; each feedforward sub-network includes one Conv module and two CAB modules; each Conv module includes multiple convolutional modules; each convolutional module includes a convolutional layer, a ReLU activation function, and a batch normalization layer; the CSFF mechanism includes multiple convolutional layers.

9. A computer device, comprising a memory and a processor, characterized in that, The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the underwater image enhancement method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the underwater image enhancement method as described in any one of claims 1 to 7.

Images