Parallel transformer CNN mixing proportion underwater image enhancement method based on PTCHS module

Through the PTCHS module combined with CNN and Transformer, the local and global features of underwater images are extracted and enhanced, which solves the problem of low underwater image quality, achieves the balance of high-quality image generation and computing efficiency, and significantly improves the underwater image color and contrast.

CN119067871APending Publication Date: 2024-12-03GUANGDONG OCEAN UNIVERSITY

Patent Information

Application Number
CN202411270223.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-11
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The quality of underwater image is affected by the absorption and scattering of light in the underwater environment, resulting in color shift and contrast reduction. It is difficult for the prior art to effectively carry out image enhancement and super-resolution processing in complex environments, affecting the visual effect and subsequent image Accuracy of the analysis.

Method used

Using a parallel transformer CNN mixed proportional underwater image enhancement method based on PTCHS module, combined with convolutional neural network (CNN) and Transformer, image features are extracted and reconstructed through multi-level processing, and local and global features are extracted using convolutional layer and Transformer, respectively. And enhance feature transfer through residual connection and skip connection mechanisms to generate high-quality underwater images.

Benefits of technology

It effectively improves the color accuracy and contrast of underwater images, maintains calculation efficiency and feature extraction capabilities, significantly improves image quality, and is better than the existing technology in multiple evaluation indicators.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119067871A_ABST
    Figure CN119067871A_ABST
Patent Text Reader

Abstract

The invention discloses a parallel transformer CNN mixing proportion underwater image enhancement method based on a PTCHS module, and the method achieves the independent parallel processing of a local feature and a global feature through the effective coding and decoding of underwater image features through the PTCHS module, the unfolding of a channel adjustment dimension through 1 * 1 convolution, and the uniform segmentation of an input tensor into a CNN part and a Transform part. And finally, fusing the features and establishing skip connection to obtain an enhanced underwater image. The invention relates to the technical field of image processing. According to the parallel transformer CNN mixing proportion underwater image enhancement method based on the PTCHS module, the balance between the calculation efficiency and the feature extraction capability is kept while the image quality is improved, the color accuracy and the contrast ratio of the underwater image are effectively improved, through parallel processing of the PTCHS module, the calculation efficiency is improved, and the accuracy and the contrast ratio of the underwater image are improved. And the detail information of the image is reserved through residual connection and skip connection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the technical field of image processing, and in particular to a parallel transformer CNN mixed ratio underwater image enhancement method based on a PTCHS module. Background Art

[0002] With the rapid development of autonomous underwater vehicles and remotely operated submersibles, the demand for underwater exploration has also grown rapidly, including underwater remote sensing analysis, marine environmental monitoring, and detection of underwater organisms. Recently, underwater imaging has become the most effective means to discover the endless treasures hidden in the depths of the ocean. However, the quality of underwater images is often severely affected by the absorption and scattering of light propagation by particles in the underwater environment. In addition, different wavelengths of light can also affect the imaging quality. Even in high-end cameras, color shift, color artifacts, and blurred details are very common.

[0003] To make matters worse, the performance limitations of underwater image acquisition equipment usually result in lower resolution. Although many super-resolution (SISR) models are able to generate high UCIQE resolution images from low resolution, a small amount of noise and optical distortion may lead to further amplification of information loss in complex underwater environments. Therefore, considering the unique optical properties of the underwater domain, it is more appropriate for a single network to perform image enhancement (UIE) and super-resolution (SR, i.e., SESR) simultaneously. More importantly, with the popularity of Transformer, CNN seems to be overshadowed by Transformer. Many networks replace the convolutional layers used to extract features with Transformer blocks. However, revisiting the self-attention design in Transformer shows that reintroducing convolution into the position of Transformer is a way to combine the advantages of both due to the lack of the correct inductive bias to guide the inductive bias of image degradation under inhomogeneous medium distribution.

[0004] As an important carrier for understanding and perceiving the underwater environment, it is crucial to obtain clear underwater images. However, underwater images are often affected by light absorption and scattering caused by particles in the water, resulting in a significant decrease in image color and contrast, which limits underwater scene recognition and understanding, such as underwater biological detection, underwater remote sensing analysis, and underwater archaeology. In recent years, in order to improve the quality of underwater images, underwater images have important application value in the fields of ocean exploration and underwater operations. However, due to the scattering and absorption of light in the underwater environment, underwater images usually have problems such as color deviation and reduced contrast, which seriously affects the visual effect of the image and the accuracy of subsequent processing and analysis. Summary of the invention

[0005] 1. Problems to be solved In view of the shortcomings of the prior art, the present invention provides a parallel transformer CNN hybrid ratio underwater image enhancement method based on the PTCHS module, which combines the advantages of convolutional neural network (CNN) and Transformer, and extracts and reconstructs image features through multi-level processing.

[0006] (II) Technical solution

[0007] To achieve the above objectives, the present invention is implemented by the following technical scheme: a parallel transformer CNN hybrid ratio underwater image enhancement method based on a PTCHS module, specifically comprising the following steps: S1, preprocess the input underwater image, map the image data to the feature space through the input projection layer, and use the convolution operation to extract preliminary features; S2, feature representation is processed by PTCHS module; S3. In the PTCHS module, local features and global features are fused through convolutional layers to combine local details and global context; S4, using residual connection and skip connection mechanism to enhance feature transfer and retain the original image information; S5. Map the fused features back to the original image resolution through the output projection layer to generate an enhanced image.

[0008] Preferably, the processing by the PTCHS module in step S2 specifically includes the following steps: A1. Use convolutional layers to adjust the channel dimension and split the input tensor into CNN and Transformer processing paths. A2,CNN path focuses on extracting local features such as edges and textures; A3. The Transformer path focuses on capturing global features, such as the overall structure of the image.

[0009] Preferably, the specific processing process of the PTCHS module is as follows: The PTCHS module is the core of the present invention: B1. Input separation: adjust the channel dimension of the input feature through 1×1 convolution to achieve uniform feature segmentation; B2, CNN branch: uses deep convolutional neural network to extract local features of the image, including edges, textures, etc.; B3, Transformer branch: captures long-distance dependencies of images through self-attention mechanism and extracts global features; B4, Feature fusion: The output features of CNN and Transformer are fused through 1×1 convolution to enhance the feature expression capability; B5. Residual connection: Add residual connection to the PTCHS module to improve the training stability and feature transfer efficiency of the network.

[0010] Preferably, in order to train the proposed network model, the following loss function is used: C1, Structural Similarity Loss (SSIM Loss): measures the structural similarity between the enhanced image and the high-quality reference image; C2, Perceptual Loss: Based on the pre-trained VGG network, calculate the feature differences between the enhanced image and the reference image at different levels; C3, total loss: Combine C1 and C2 losses, and obtain the final loss function through weighted summation for end-to-end training of the network.

[0011] Preferably, after the enhanced image is generated in step S5, an experimental verification is performed as follows: The experimental part describes in detail the experimental settings, data sets, evaluation indicators and result analysis of the method of the present invention: D1. Dataset: The UIEB dataset is used for training and testing, including 800 pairs of training images and 90 pairs of test images; D2. Evaluation indicators: PSNR, SSIM, UIQM, UCIQE and NIQE are used for comprehensive evaluation; D3. Result analysis: It outperforms the existing technology in multiple evaluation indicators and shows excellent performance.

[0012] Preferably, the experimental verification adopts the PyTorch framework, is performed on an NVIDIA RTX3090 GPU, and is trained for 500 epochs.

[0013] The PTCHS module is the core component of the present invention, which combines the advantages of CNN and Transformer through parallel processing, extracts local and global features respectively, and fuses these features through convolutional layers. The design of the PTCHS module allows the model to consider both local details and global context at the same time to achieve more accurate feature representation.

[0014] (III) Beneficial effects The present invention provides a parallel transformer CNN mixed ratio underwater image enhancement method based on a PTCHS module. Compared with the prior art, it has the following beneficial effects: the parallel transformer CNN mixed ratio underwater image enhancement method based on the PTCHS module effectively encodes and decodes underwater image features by utilizing the PTCHS module, adjusts the dimension by 1×1 convolution expansion, and evenly divides the input tensor into two parts, CNN and Transformer, to achieve independent parallel processing of local features and global features. Finally, by fusing these features and establishing skip connections, an enhanced underwater image is obtained. While improving image quality, the method of the present invention maintains a balance between computational efficiency and feature extraction capabilities, effectively improves the color accuracy and contrast of underwater images, improves computational efficiency through parallel processing of PTCHS modules, and retains image detail information through residual connections and skip connections. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic diagram of the structure of the present invention. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] See also Figure 1 The embodiment of the present invention provides six technical solutions: a parallel transformer CNN hybrid ratio underwater image enhancement method based on a PTCHS module, specifically including the following embodiments: Embodiment 1: A parallel transformer CNN hybrid ratio underwater image enhancement method based on a PTCHS module, characterized in that it specifically comprises the following steps: S1, preprocess the input underwater image, map the image data to the feature space through the input projection layer, and use the convolution operation to extract preliminary features; S2, feature representation is processed by PTCHS module; S3. In the PTCHS module, local features and global features are fused through convolutional layers to combine local details and global context; S4, using residual connection and skip connection mechanism to enhance feature transfer and retain the original image information; S5. Map the fused features back to the original image resolution through the output projection layer to generate an enhanced image.

[0018] The processing by the PTCHS module in step S2 of the embodiment of the present invention specifically includes the following steps: A1. Use convolutional layers to adjust the channel dimension and split the input tensor into CNN and Transformer processing paths. A2,CNN path focuses on extracting local features such as edges and textures; A3. The Transformer path focuses on capturing global features, such as the overall structure of the image.

[0019] In the embodiment of the present invention, the specific processing process of the PTCHS module is as follows: The PTCHS module is the core of the present invention: B1. Input separation: adjust the channel dimension of the input feature through 1×1 convolution to achieve uniform feature segmentation; B2, CNN branch: uses deep convolutional neural network to extract local features of the image, including edges, textures, etc.; B3, Transformer branch: captures long-distance dependencies of images through self-attention mechanism and extracts global features; B4, Feature fusion: The output features of CNN and Transformer are fused through 1×1 convolution to enhance the feature expression capability; B5. Residual connection: Add residual connection to the PTCHS module to improve the training stability and feature transfer efficiency of the network.

[0020] Embodiment 2: Embodiment 2 of the present invention is different from the technical solution of Embodiment 1 in that: in order to train the proposed network model, the following loss function is adopted: C1, Structural Similarity Loss (SSIM Loss): measures the structural similarity between the enhanced image and the high-quality reference image; C2, Perceptual Loss: Based on the pre-trained VGG network, calculate the feature differences between the enhanced image and the reference image at different levels; C3, total loss: Combine C1 and C2 losses, and obtain the final loss function through weighted summation for end-to-end training of the network.

[0021] Embodiment 3: The difference between the embodiment of the present invention and embodiment 2 is that: after the enhanced image is generated in step S5, an experimental verification is performed as follows: The experimental part describes in detail the experimental setting, data set, evaluation index and result analysis of the method of the present invention: D1. Dataset: The UIEB dataset is used for training and testing, including 800 pairs of training images and 90 pairs of test images; D2. Evaluation indicators: PSNR, SSIM, UIQM, UCIQE and NIQE are used for comprehensive evaluation; D3. Result analysis: It outperforms the existing technology in multiple evaluation indicators and shows excellent performance.

[0022] In this embodiment of the present invention, the experimental verification adopts the PyTorch framework, is carried out on the NVIDIA RTX3090 GPU, and is trained for 500 epochs.

[0023] Detailed description of the embodiments The following embodiments are described in detail to show the specific application of the present invention: Example 4: In this example, the collaborative working mechanism of the CNN branch and the Transformer branch in the PTCHS module is further explored. The CNN branch uses its multi-layer convolutional network structure to focus on extracting local features of the image, such as edge and texture information. These local features are then combined with the global features extracted by the Transformer branch to achieve a comprehensive feature representation of the underwater image.

[0024] Example 5: In this example, the working principles of the multi-head inter-region self-attention module (MISA) and the multi-head channel self-attention module (MCSA) are described in detail. The MISA module captures global information between regions through average pooling and reshaping operations, while the MCSA module focuses on detail enhancement within a single region and models cross-channel dependencies through a gated separable convolutional feedforward network (GDFN).

[0025] Example 6: This example focuses on the design and function of the full-scale feature fusion module. This module achieves effective integration of features of different scales through feature fusion and channel adjustment. Feature fusion uses pixel rearrangement and 1×1 convolution to align and fuse feature maps, while channel adjustment is based on the channel attention mechanism, generating attention scores through global average pooling and maximum pooling to optimize feature representation.

[0026] Additional Experimental Results In addition to the aforementioned experimental verification, the present invention also conducted the following additional experiments: Additional Experiment 1: The image quality improvement effect of the method of the present invention was tested under different underwater environmental conditions. The experimental results show that the method of the present invention can stably improve the image quality even when the lighting conditions vary greatly and the colors of underwater objects vary.

[0027] Additional Experiment 2: The algorithm's generalization ability was tested by applying the proposed method to underwater image datasets from different sources. The test results confirmed that the algorithm has good generalization and can work effectively in a variety of underwater environments.

[0028] In summary, the present invention effectively encodes and decodes underwater image features by utilizing the PTCHS module, adjusts the dimension by 1×1 convolution expansion, and evenly divides the input tensor into two parts, CNN and Transformer, to achieve independent parallel processing of local features and global features. Finally, by fusing these features and establishing skip connections, an enhanced underwater image is obtained. While improving image quality, the method of the present invention maintains a balance between computational efficiency and feature extraction capabilities, effectively improves the color accuracy and contrast of underwater images, improves computational efficiency through parallel processing of the PTCHS module, and retains image detail information through residual connections and skip connections. The underwater image enhancement method based on the PTCHS module proposed in the present invention effectively improves the quality of underwater images by combining the advantages of CNN and Transformer, and has broad application prospects.

[0029] Meanwhile, the contents not described in detail in this specification belong to the prior art known to those skilled in the art.

[0030] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0031] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A parallel transformer CNN hybrid ratio underwater image enhancement method based on PTCHS module, characterized by: The specific steps include: S1, preprocess the input underwater image, map the image data to the feature space through the input projection layer, and use the convolution operation to extract preliminary features; S2, feature representation is processed by PTCHS module; S3. In the PTCHS module, local features and global features are fused through convolutional layers to combine local details and global context; S4, using residual connection and skip connection mechanism to enhance feature transfer and retain the original image information; S5. Map the fused features back to the original image resolution through the output projection layer to generate an enhanced image.

2. The parallel transformer CNN hybrid ratio underwater image enhancement method based on the PTCHS module according to claim 1 is characterized in that: The processing by the PTCHS module in step S2 specifically includes the following steps: A1. Use convolutional layers to adjust the channel dimension and split the input tensor into CNN and Transformer processing paths. A2,CNN path focuses on extracting local features; A3. Transformer path focuses on capturing global features.

3. The parallel transformer CNN hybrid ratio underwater image enhancement method based on the PTCHS module according to claim 2 is characterized in that: The specific processing process of the PTCHS module is as follows: B1. Input separation: adjust the channel dimension of the input feature through 1×1 convolution to achieve uniform feature segmentation; B2, CNN branch: uses deep convolutional neural network to extract local features of the image, including edges and textures; B3, Transformer branch: captures long-distance dependencies of images through self-attention mechanism and extracts global features; B4, Feature fusion: The output features of CNN and Transformer are fused through 1×1 convolution to enhance the feature expression capability; B5. Residual connection: Add residual connection to the PTCHS module to improve the training stability and feature transfer efficiency of the network.

4. The parallel transformer CNN hybrid ratio underwater image enhancement method based on the PTCHS module according to claim 1 is characterized in that: To train the proposed network model, the following loss function is adopted: C1, structural similarity loss: measures the structural similarity between the enhanced image and the high-quality reference image; C2, perceptual loss: based on the pre-trained VGG network, calculate the feature differences between the enhanced image and the reference image at different levels; C3, total loss: Combine C1 and C2 losses, and obtain the final loss function through weighted summation for end-to-end training of the network.

5. The parallel transformer CNN hybrid ratio underwater image enhancement method based on the PTCHS module according to claim 1 is characterized in that: After the enhanced image is generated in step S5, an experimental verification is performed as follows: D1. Dataset: The UIEB dataset is used for training and testing, including 800 pairs of training images and 90 pairs of test images; D2. Evaluation indicators: PSNR, SSIM, UIQM, UCIQE and NIQE indicators are used for comprehensive evaluation; D3. Result analysis: It outperforms the existing technology in multiple evaluation indicators and shows excellent performance.

6. The parallel transformer CNN hybrid ratio underwater image enhancement method based on the PTCHS module according to claim 5 is characterized by: The experimental verification was conducted using the PyTorch framework on an NVIDIA RTX3090 GPU and trained for 500 epochs.

Citation Information

Patent Citations

  • Semi-supervised underwater image semantic segmentation method based on generative adversarial learning

    CN116051840A

  • Underwater image enhancement method based on Transform and generative adversarial network

    CN116309107A

  • Underwater image enhancement method based on CNN and Transform mixed structure

    CN117670687A

  • Underwater image enhancement method based on cascade adaptive network

    CN117689592A

  • Remote sensing image segmentation method based on dual-branch multi-scale feature fusion

    CN118314353A

Cited By

  • Transform network-based underwater image enhancement method, system and device, and medium

    CN119477778A

  • Battery system fault identification method and system based on time sequence contrast learning encoder

    CN120892801A