Parallel transformer CNN mixing proportion underwater image enhancement method based on PTCHS module
Through the PTCHS module combined with CNN and Transformer, the local and global features of underwater images are extracted and enhanced, which solves the problem of low underwater image quality, achieves the balance of high-quality image generation and computing efficiency, and significantly improves the underwater image color and contrast.
Patent Information
- Application Number
- CN202411270223.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-11
- Publication Date
- 2024-12-03
AI Technical Summary
The quality of underwater image is affected by the absorption and scattering of light in the underwater environment, resulting in color shift and contrast reduction. It is difficult for the prior art to effectively carry out image enhancement and super-resolution processing in complex environments, affecting the visual effect and subsequent image Accuracy of the analysis.
Using a parallel transformer CNN mixed proportional underwater image enhancement method based on PTCHS module, combined with convolutional neural network (CNN) and Transformer, image features are extracted and reconstructed through multi-level processing, and local and global features are extracted using convolutional layer and Transformer, respectively. And enhance feature transfer through residual connection and skip connection mechanisms to generate high-quality underwater images.
It effectively improves the color accuracy and contrast of underwater images, maintains calculation efficiency and feature extraction capabilities, significantly improves image quality, and is better than the existing technology in multiple evaluation indicators.
Smart Images

Figure CN119067871A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of image processing, and in particular to a parallel transformer CNN mixed ratio underwater image enhancement method based on a PTCHS module. Background Art
[0002] With the rapid development of autonomous underwater vehicles and remotely operated submersibles, the demand for underwater exploration has also grown rapidly, including underwater remote sensing analysis, marine environmental monitoring, and detection of underwater organisms. Recently, underwater imaging has become the most effective means to discover the endless treasures hidden in the depths of the ocean. However, the quality of underwater images is often severely affected by the absorption and scattering of light propagation by particles in the underwater environment. In addition, different wavelengths of light can also affect the imaging quality. Even in high-end cameras, color shift, color artifacts, and blurred details are very common.
[0003] To make matters worse, the performance limitations of underwater image acquisition equipment usually result in lower resolution. Although many super-resolution (SISR) models are able to generate high UCIQE resolution images from low resolution, a small amount of noise and optical distortion may lead to further amplification of information loss in complex underwater environments. Therefore, considering the unique optical properties of the underwater domain, it is more appropriate for a single network to perform image enhancement (UIE) and super-resolution (SR, i.e., SESR) simultaneously. More importantly, with the popularity of Transformer, CNN seems to be overshadowed by Transformer. Many networks replace the convolutional layers used to extract features with Transformer blocks. However, revisiting the self-attention design in Transformer shows that reintroducing convolution into the position of Transformer is a way to combine the advantages of both due to the lack of the correct inductive bias to guide the inductive bias of image degradation under inhomogeneous medium distribution.
[0004] As an important carrier for understanding and perceiving the underwater environment, it is crucial to obtain clear underwater images. However, underwater images are often affected by light absorption and scattering caused by particles in the water, resulting in a significant decrease in image color and contrast, which limits underwater scene recognition and understanding, such as underwater biological detection, underwater remote sensing analysis, and underwater archaeology. In recent years, in order to improve the quality of underwater images, underwater images have important application value in the fields of ocean exploration and underwater operations. However, due to the scattering and absorption of light in the underwater environment, underwater images usually have problems such as color deviation and reduced contrast, which seriously affects the visual effect of the image and the accuracy of subsequent processing and analysis. Summary of the invention
[0005] 1. Problems to be solved In view of the shortcomings of the prior art, the present invention provides a parallel transformer CNN hybrid ratio underwater image enhancement method based on the PTCHS module, which combines the advantages of convolutional neural network (CNN) and Transformer, and extracts and reconstructs image features through multi-level processing.
[0006] (II) Technical solution
[0007] To achieve the above objectives, the present invention is implemented by the following technical scheme: a parallel transformer CNN hybrid ratio underwater image enhancement method based on a PTCHS module, specifically comprising the following steps: S1, preprocess the input underwater image, map the image data to the feature space through the input projection layer, and use the convolution operation to extract preliminary features; S2, feature representation is processed by PTCHS module; S3. In the PTCHS module, local features and global features are fused through convolutional layers to combine local details and global context; S4, using residual connection and skip connection mechanism to enhance feature transfer and retain the original image information; S5. Map the fused features back to the original image resolution through the output projection layer to generate an enhanced image.
[0008] Preferably, the processing by the PTCHS module in step S2 specifically includes the following steps: A1. Use convolutional layers to adjust the channel dimension and split the input tensor into CNN and Transformer processing paths. A2,CNN path focuses on extracting local features such as edges and textures; A3. The Transformer path focuses on capturing global features, such as the overall structure of the image.
[0009] Preferably, the specific processing process of the PTCHS module is as follows: The PTCHS module is the core of the present invention: B1. Input separation: adjust the channel dimension of the input feature through 1×1 convolution to achieve uniform feature segmentation; B2, CNN branch: uses deep convolutional neural network to extract local features of the image, including edges, textures, etc.; B3, Transformer branch: captures long-distance dependencies of images through self-attention mechanism and extracts global features; B4, Feature fusion: The output features of CNN and Transformer are fused through 1×1 convolution to enhance the feature expression capability; B5. Residual connection: Add residual connection to the PTCHS module to improve the training stability and feature transfer efficiency of the network.
[0010] Preferably, in order to train the proposed network model, the following loss function is used: C1, Structural Similarity Loss (SSIM Loss): measures the structural similarity between the enhanced image and the high-quality reference image; C2, Perceptual Loss: Based on the pre-trained VGG network, calculate the feature differences between the enhanced image and the reference image at different levels; C3, total loss: Combine C1 and C2 losses, and obtain the final loss function through weighted summation for end-to-end training of the network.
[0011] Preferably, after the enhanced image is generated in step S5, an experimental verification is performed as follows: The experimental part describes in detail the experimental settings, data sets, evaluation indicators and result analysis of the method of the present invention: D1. Dataset: The UIEB dataset is used for training and testing, including 800 pairs of training images and 90 pairs of test images; D2. Evaluation indicators: PSNR, SSIM, UIQM, UCIQE and NIQE are used for comprehensive evaluation; D3. Result analysis: It outperforms the existing technology in multiple evaluation indicators and shows excellent performance.
[0012] Preferably, the experimental verification adopts the PyTorch framework, is performed on an NVIDIA RTX3090 GPU, and is trained for 500 epochs.
[0013] The PTCHS module is the core component of the present invention, which combines the advantages of CNN and Transformer through parallel processing, extracts local and global features respectively, and fuses these features through convolutional layers. The design of the PTCHS module allows the model to consider both local details and global context at the same time to achieve more accurate feature representation.
[0014] (III) Beneficial effects The present invention provides a parallel transformer CNN mixed ratio underwater image enhancement method based on a PTCHS module. Compared with the prior art, it has the following beneficial effects: the parallel transformer CNN mixed ratio underwater image enhancement method based on the PTCHS module effectively encodes and decodes underwater image features by utilizing the PTCHS module, adjusts the dimension by 1×1 convolution expansion, and evenly divides the input tensor into two parts, CNN and Transformer, to achieve independent parallel processing of local features and global features. Finally, by fusing these features and establishing skip connections, an enhanced underwater image is obtained. While improving image quality, the method of the present invention maintains a balance between computational efficiency and feature extraction capabilities, effectively improves the color accuracy and contrast of underwater images, improves computational efficiency through parallel processing of PTCHS modules, and retains image detail information through residual connections and skip connections. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 It is a schematic diagram of the structure of the present invention. DETAILED DESCRIPTION
[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0017] See also Figure 1 The embodiment of the present invention provides six technical solutions: a parallel transformer CNN hybrid ratio underwater image enhancement method based on a PTCHS module, specifically including the following embodiments: Embodiment 1: A parallel transformer CNN hybrid ratio underwater image enhancement method based on a PTCHS module, characterized in that it specifically comprises the following steps: S1, preprocess the input underwater image, map the image data to the feature space through the input projection layer, and use the convolution operation to extract preliminary features; S2, feature representation is processed by PTCHS module; S3. In the PTCHS module, local features and global features are fused through convolutional layers to combine local details and global context; S4, using residual connection and skip connection mechanism to enhance feature transfer and retain the original image information; S5. Map the fused features back to the original image resolution through the output projection layer to generate an enhanced image.
[0018] The processing by the PTCHS module in step S2 of the embodiment of the present invention specifically includes the following steps: A1. Use convolutional layers to adjust the channel dimension and split the input tensor into CNN and Transformer processing paths. A2,CNN path focuses on extracting local features such as edges and textures; A3. The Transformer path focuses on capturing global features, such as the overall structure of the image.
[0019] In the embodiment of the present invention, the specific processing process of the PTCHS module is as follows: The PTCHS module is the core of the present invention: B1. Input separation: adjust the channel dimension of the input feature through 1×1 convolution to achieve uniform feature segmentation; B2, CNN branch: uses deep convolutional neural network to extract local features of the image, including edges, textures, etc.; B3, Transformer branch: captures long-distance dependencies of images through self-attention mechanism and extracts global features; B4, Feature fusion: The output features of CNN and Transformer are fused through 1×1 convolution to enhance the feature expression capability; B5. Residual connection: Add residual connection to the PTCHS module to improve the training stability and feature transfer efficiency of the network.
[0020] Embodiment 2: Embodiment 2 of the present invention is different from the technical solution of Embodiment 1 in that: in order to train the proposed network model, the following loss function is adopted: C1, Structural Similarity Loss (SSIM Loss): measures the structural similarity between the enhanced image and the high-quality reference image; C2, Perceptual Loss: Based on the pre-trained VGG network, calculate the feature differences between the enhanced image and the reference image at different levels; C3, total loss: Combine C1 and C2 losses, and obtain the final loss function through weighted summation for end-to-end training of the network.
[0021] Embodiment 3: The difference between the embodiment of the present invention and embodiment 2 is that: after the enhanced image is generated in step S5, an experimental verification is performed as follows: The experimental part describes in detail the experimental setting, data set, evaluation index and result analysis of the method of the present invention: D1. Dataset: The UIEB dataset is used for training and testing, including 800 pairs of training images and 90 pairs of test images; D2. Evaluation indicators: PSNR, SSIM, UIQM, UCIQE and NIQE are used for comprehensive evaluation; D3. Result analysis: It outperforms the existing technology in multiple evaluation indicators and shows excellent performance.
[0022] In this embodiment of the present invention, the experimental verification adopts the PyTorch framework, is carried out on the NVIDIA RTX3090 GPU, and is trained for 500 epochs.
[0023] Detailed description of the embodiments The following embodiments are described in detail to show the specific application of the present invention: Example 4: In this example, the collaborative working mechanism of the CNN branch and the Transformer branch in the PTCHS module is further explored. The CNN branch uses its multi-layer convolutional network structure to focus on extracting local features of the image, such as edge and texture information. These local features are then combined with the global features extracted by the Transformer branch to achieve a comprehensive feature representation of the underwater image.
[0024] Example 5: In this example, the working principles of the multi-head inter-region self-attention module (MISA) and the multi-head channel self-attention module (MCSA) are described in detail. The MISA module captures global information between regions through average pooling and reshaping operations, while the MCSA module focuses on detail enhancement within a single region and models cross-channel dependencies through a gated separable convolutional feedforward network (GDFN).
[0025] Example 6: This example focuses on the design and function of the full-scale feature fusion module. This module achieves effective integration of features of different scales through feature fusion and channel adjustment. Feature fusion uses pixel rearrangement and 1×1 convolution to align and fuse feature maps, while channel adjustment is based on the channel attention mechanism, generating attention scores through global average pooling and maximum pooling to optimize feature representation.
[0026] Additional Experimental Results In addition to the aforementioned experimental verification, the present invention also conducted the following additional experiments: Additional Experiment 1: The image quality improvement effect of the method of the present invention was tested under different underwater environmental conditions. The experimental results show that the method of the present invention can stably improve the image quality even when the lighting conditions vary greatly and the colors of underwater objects vary.
[0027] Additional Experiment 2: The algorithm's generalization ability was tested by applying the proposed method to underwater image datasets from different sources. The test results confirmed that the algorithm has good generalization and can work effectively in a variety of underwater environments.
[0028] In summary, the present invention effectively encodes and decodes underwater image features by utilizing the PTCHS module, adjusts the dimension by 1×1 convolution expansion, and evenly divides the input tensor into two parts, CNN and Transformer, to achieve independent parallel processing of local features and global features. Finally, by fusing these features and establishing skip connections, an enhanced underwater image is obtained. While improving image quality, the method of the present invention maintains a balance between computational efficiency and feature extraction capabilities, effectively improves the color accuracy and contrast of underwater images, improves computational efficiency through parallel processing of the PTCHS module, and retains image detail information through residual connections and skip connections. The underwater image enhancement method based on the PTCHS module proposed in the present invention effectively improves the quality of underwater images by combining the advantages of CNN and Transformer, and has broad application prospects.
[0029] Meanwhile, the contents not described in detail in this specification belong to the prior art known to those skilled in the art.
[0030] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.
[0031] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A parallel transformer CNN hybrid ratio underwater image enhancement method based on PTCHS module, characterized by: The specific steps include: S1, preprocess the input underwater image, map the image data to the feature space through the input projection layer, and use the convolution operation to extract preliminary features; S2, feature representation is processed by PTCHS module; S3. In the PTCHS module, local features and global features are fused through convolutional layers to combine local details and global context; S4, using residual connection and skip connection mechanism to enhance feature transfer and retain the original image information; S5. Map the fused features back to the original image resolution through the output projection layer to generate an enhanced image.
2. The parallel transformer CNN hybrid ratio underwater image enhancement method based on the PTCHS module according to claim 1 is characterized in that: The processing by the PTCHS module in step S2 specifically includes the following steps: A1. Use convolutional layers to adjust the channel dimension and split the input tensor into CNN and Transformer processing paths. A2,CNN path focuses on extracting local features; A3. Transformer path focuses on capturing global features.
3. The parallel transformer CNN hybrid ratio underwater image enhancement method based on the PTCHS module according to claim 2 is characterized in that: The specific processing process of the PTCHS module is as follows: B1. Input separation: adjust the channel dimension of the input feature through 1×1 convolution to achieve uniform feature segmentation; B2, CNN branch: uses deep convolutional neural network to extract local features of the image, including edges and textures; B3, Transformer branch: captures long-distance dependencies of images through self-attention mechanism and extracts global features; B4, Feature fusion: The output features of CNN and Transformer are fused through 1×1 convolution to enhance the feature expression capability; B5. Residual connection: Add residual connection to the PTCHS module to improve the training stability and feature transfer efficiency of the network.
4. The parallel transformer CNN hybrid ratio underwater image enhancement method based on the PTCHS module according to claim 1 is characterized in that: To train the proposed network model, the following loss function is adopted: C1, structural similarity loss: measures the structural similarity between the enhanced image and the high-quality reference image; C2, perceptual loss: based on the pre-trained VGG network, calculate the feature differences between the enhanced image and the reference image at different levels; C3, total loss: Combine C1 and C2 losses, and obtain the final loss function through weighted summation for end-to-end training of the network.
5. The parallel transformer CNN hybrid ratio underwater image enhancement method based on the PTCHS module according to claim 1 is characterized in that: After the enhanced image is generated in step S5, an experimental verification is performed as follows: D1. Dataset: The UIEB dataset is used for training and testing, including 800 pairs of training images and 90 pairs of test images; D2. Evaluation indicators: PSNR, SSIM, UIQM, UCIQE and NIQE indicators are used for comprehensive evaluation; D3. Result analysis: It outperforms the existing technology in multiple evaluation indicators and shows excellent performance.
6. The parallel transformer CNN hybrid ratio underwater image enhancement method based on the PTCHS module according to claim 5 is characterized by: The experimental verification was conducted using the PyTorch framework on an NVIDIA RTX3090 GPU and trained for 500 epochs.
Citation Information
Patent Citations
Semi-supervised underwater image semantic segmentation method based on generative adversarial learning
CN116051840A
Underwater image enhancement method based on Transform and generative adversarial network
CN116309107A
Underwater image enhancement method based on CNN and Transform mixed structure
CN117670687A
Underwater image enhancement method based on cascade adaptive network
CN117689592A
Remote sensing image segmentation method based on dual-branch multi-scale feature fusion
CN118314353A
Cited By
Transform network-based underwater image enhancement method, system and device, and medium
CN119477778A
Battery system fault identification method and system based on time sequence contrast learning encoder
CN120892801A
