Underwater image collaborative enhancement method

Abstract

The application provides an underwater image cooperative enhancement method, acquires an underwater original image pair, and then extracts a dark channel feature map corresponding to the underwater original image based on a dark channel prior. Then, the dark channel feature map is used as a guide, and a self-attention Transformer module is fused to extract an original feature map pair of the underwater original image pair. Then, a channel and space Transformer module based on second-order statistics is used to extract a global feature map pair from the original feature map pair. Further, a feature Transformer interaction module is used to obtain feature map pair correlation to obtain an enhanced feature map pair. Finally, the enhanced feature map pair is restored to obtain an underwater enhanced image pair. The application utilizes similar degradation features between multiple images collected in the same underwater scene, adopts a cooperative processing and joint learning strategy, thereby complementing the underwater images, achieving cooperative enhancement, and effectively improving the quality of the enhanced underwater images.

Description

Technical Field

[0001] This invention relates to the field of marine information processing technology, and specifically to a method for collaborative enhancement of underwater images. Background Technology

[0002] With the development of underwater robots and imaging technology, people are increasingly interested in exploring the ocean. However, the quality of underwater imaging is affected by various factors such as absorption, scattering, and attenuation, leading to problems such as blurring, color distortion, and low contrast, which greatly limits the development of ocean exploration and marine technology. Therefore, underwater image enhancement (UIE) methods have become a key area of ​​research and application.

[0003] Traditional underwater image enhancement methods rely on statistical regularities, using underwater imaging models to estimate parameters to improve the quality of underwater images. However, these parameters vary greatly in different types of underwater environments, limiting their applicability in complex and diverse underwater environments and making them unreliable. Summary of the Invention

[0004] Based on the above situation, the main objective of this invention is to provide an underwater image collaborative enhancement method to overcome the shortcomings of low reliability in existing underwater image enhancement methods.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] An underwater image collaborative enhancement method includes:

[0007] Acquire underwater raw image pairs, the underwater raw image pairs including a first underwater raw image and a second underwater raw image, the underwater raw image pairs being acquired in the same underwater scene;

[0008] Based on the dark channel prior, the first dark channel feature map of the first underwater original image and the second dark channel feature map of the second underwater original image are extracted respectively. The first dark channel feature map and the second dark channel feature map contain the dark channel information of their respective underwater original images.

[0009] Based on the first dark channel feature map and the second dark channel feature map, the self-attention Transformer module is fused to extract the first original feature map of the first underwater original image and the second original feature map of the second underwater original image.

[0010] The first global feature map is extracted from the first original feature map by the CS-Transformer module based on second-order statistics, and the second global feature map is extracted from the second original feature map.

[0011] A second enhanced feature map is obtained by selecting several first structural features that meet the preset conditions of correlation from the first global feature map and fusing them into the second global feature map; and a first enhanced feature map is obtained by selecting several second structural features that meet the preset conditions of correlation from the second global feature map and fusing them into the first global feature map.

[0012] The first enhanced feature map and / or the second enhanced feature map are restored to obtain the first underwater enhanced image and the second underwater enhanced image.

[0013] Furthermore, in the step of extracting the first dark channel feature map of the first underwater original image and the second dark channel feature map of the second underwater original image based on the dark channel prior, the step of extracting the first dark channel feature map of the first underwater original image based on the dark channel prior is as follows:

[0014] Obtain the dark channel prior features corresponding to the first underwater original image;

[0015] The underwater scene depth prior information is calculated based on the aforementioned dark channel prior features;

[0016] The underwater scene depth prior information is introduced into the first underwater original image through an attention mechanism to obtain the first initial feature map corresponding to the first underwater original image.

[0017] Based on the attention mechanism and the first initial feature map, the dark channel prior is incorporated into the mapping of the first underwater original image to obtain the first dark channel feature map.

[0018] Furthermore, the first initial feature map includes a first initial query term, a first initial key value, and a first initial value term. The step of integrating the dark channel prior features into the mapping of the first underwater original image based on the attention mechanism and the first initial feature map to obtain the first dark channel feature map includes:

[0019] The first initial key value and the first initial value item are mapped to obtain the first prior key value and the first prior value item;

[0020] Using the first initial query term and the first prior key value, the correlation between the first underwater original image and the dark channel prior features is constructed, and the dark channel information is fused into the first initial query term by multiplying the first initial query term and the first prior term value to obtain the first dark channel feature map.

[0021] Furthermore, the step of extracting the first original feature map of the first underwater original image by fusing the self-attention Transformer module based on the first dark channel feature map includes:

[0022] The self-attention Transformer module fuses the first dark channel feature map, the first initial feature map, and the first underwater original image to obtain the first original feature map. The self-attention Transformer module employs a channel-based self-attention mechanism.

[0023] Furthermore, the self-attention Transformer module includes a feature fusion multi-head self-attention module and a feedforward neural network, which combines deep convolution to encode information about the positions of adjacent pixels in order to extract local information of the image.

[0024] Furthermore, the step of extracting the first global feature map from the first original feature map using the CS-Transformer module based on second-order statistics includes:

[0025] The first original feature map of the linear mapping is calculated based on the channel attention mechanism and the spatial attention mechanism respectively to obtain the first channel feature map and the first spatial feature map.

[0026] The first channel feature map and the first spatial feature map are adaptively added together to obtain the first global feature map.

[0027] Furthermore, the CS-Transformer includes a spatial Transformer and a channel Transformer.

[0028] The first channel feature map is obtained by processing the channel Transformer module, which is used to calculate channel attention;

[0029] The first spatial feature map is obtained by the spatial Transformer module, which is used to calculate spatial attention;

[0030] The channel Transformer module and the spatial Transformer module run in parallel. When the spatial Transformer module calculates spatial attention, the first original feature map is downsampled to reduce spatial computation costs, and then upsampled to return the same feature scale as the output features of the channel Transformer module.

[0031] Furthermore, the spatial Transformer module includes a first multi-head attention module and a second multi-head attention module;

[0032] The first multi-head attention module is window-based and has second-order statistical properties, used to calculate the similarity of pixels within the window;

[0033] The second multi-head attention module is based on a moving window and has second-order statistical properties, used to calculate the similarity of pixels within the moving window.

[0034] Furthermore, the step of selecting several first structural features from the first global feature map that meet preset relevance conditions and fusing them into the second global feature map to obtain a second enhanced feature map includes:

[0035] Calculate the correlation coefficient between the first global feature map and the second global feature map;

[0036] Based on the correlation coefficient and feature-transfer attention, the first structural feature is obtained by filtering from the first global feature map;

[0037] The first structural feature is associated with the corresponding second global feature in the second global feature map, and then convolved to obtain the first fused feature;

[0038] The first fused feature is aggregated with the second global feature in the second global feature map and then residual-connected to obtain the second enhanced feature map.

[0039] Furthermore, the step of filtering the first structural features from the first global feature map based on the correlation coefficient and feature-transfer attention includes:

[0040] The correlation coefficient is used to obtain a transfer attention map, which includes position-related elements, and the position-related elements represent the most position-related elements in the first global feature map and the second global feature map.

[0041] Using the location-related elements as indexes, an index selection operation is applied to the expanded patch of the first initial value item of the first global feature map to filter and obtain the first structural feature.

[0042] [Beneficial Effects] This application provides a collaborative underwater image enhancement method. First, it acquires pairs of original underwater images. Then, based on dark channel priors, it extracts dark channel feature maps corresponding to the original underwater images. Next, guided by the dark channel feature maps and fused with a self-attention Transformer module, it extracts original feature map pairs from the original underwater image pairs. Then, from the original feature map pairs, it extracts global feature map pairs using a channel and spatial Transformer module based on second-order statistics. Further, it obtains the correlation between feature map pairs through a feature Transformer interaction module, resulting in enhanced feature map pairs. Finally, it restores the enhanced feature map pairs to obtain enhanced underwater image pairs. This application utilizes similar degradation features between multiple images acquired from the same underwater scene, employing collaborative processing and joint learning strategies to complement underwater images, achieving joint enhancement and effectively improving the quality of the enhanced underwater images.

[0043] Other beneficial effects of the present invention will be explained in detail through the introduction of specific technical features and technical solutions in specific embodiments. Those skilled in the art should be able to understand the beneficial technical effects brought about by these technical features and technical solutions through the introduction of these technical features and technical solutions. Attached Figure Description

[0044] The preferred embodiments of the present invention will now be described with reference to the accompanying drawings. In the drawings:

[0045] Figure 1 This is a schematic diagram of the overall framework of an underwater image collaborative enhancement method according to a preferred embodiment of the present invention;

[0046] Figure 2 This is a schematic diagram illustrating the steps of an underwater image collaborative enhancement method according to a preferred embodiment of the present invention. Detailed Implementation

[0047] The present invention is described below based on embodiments, but the present invention is not limited to these embodiments. In the following detailed description of the present invention, some specific details are described in detail, but well-known methods, processes, procedures, and elements are not described in detail in order to avoid obscuring the essence of the present invention.

[0048] Furthermore, those skilled in the art should understand that the accompanying drawings provided herein are for illustrative purposes only and are not necessarily drawn to scale.

[0049] Unless the context explicitly requires it, the words "comprising," "including," and similar terms throughout the specification and claims should be interpreted as encompassing rather than being exclusive or exhaustive; that is, meaning "including but not limited to."

[0050] In the description of this invention, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0051] like Figure 1 As shown, the topmost diagram illustrates the overall framework of UICE-PTI: (a) CS-TransformerBlock; (b) FTIM; (c) CTM; and (d) STM. The underwater image collaborative enhancement method (UICE-PT) provided in this embodiment integrates physical prior information and utilizes an encoder-decoder structure with feature interactions. The proposed UICE-PTI is an end-to-end trainable model for underwater image collaborative enhancement tasks. Figure 1 The overall framework of the proposed UICE-PTI is presented, including an encoder-decoder structure with long-range skip connections. Specifically, the proposed UICE-PTI framework consists of three basic units: a Transformer module (DCPT) for fusing dark channel priors, a CS-Transformer block with second-order statistics, where the CS-Transformer block includes channel and spatial Transformer modules, and a Feature Transformer Interaction (FTIM) module to capture the correlation between two branches in the network bottleneck layer. The network backbone is constructed in encoder-decoder order, with the CS-Transformer block serving as the basic module (except for the bottleneck section); the FTIM module is designed in the bottleneck section of the network backbone, conveying the feature correlation between two branches and providing complementary information to the other branch.

[0052] like Figure 2 As shown, this embodiment provides an underwater image collaborative enhancement method, including:

[0053] S1: Acquire underwater raw image pairs, the underwater raw image pairs including a first underwater raw image and a second underwater raw image, the underwater raw image pairs being acquired in the same underwater scene.

[0054] In this embodiment, the system acquires a pair of original underwater images that need to be enhanced. This pair consists of at least two original underwater images. This embodiment uses a pair of original underwater images including a first and a second original underwater image as an example. The original underwater images in the pair were all acquired in the same underwater scene using a camera or other device. That is, the first and second original underwater images were captured in the same underwater scene. Original underwater images captured in the same underwater scene typically exhibit similar degradation characteristics, which can provide richer complementary information to each other.

[0055] S2: Based on the dark channel prior, extract the first dark channel feature map of the first underwater original image and the second dark channel feature map of the second underwater original image respectively. The first dark channel feature map and the second dark channel feature map contain the dark channel information of their respective underwater original images.

[0056] Furthermore, in the step of extracting the first dark channel feature map of the first underwater original image and the second dark channel feature map of the second underwater original image based on the dark channel prior, the step of extracting the first dark channel feature map of the first underwater original image based on the dark channel prior is as follows:

[0057] S201: Obtain the dark channel prior features corresponding to the first underwater original image;

[0058] S202: Calculate the underwater scene depth prior information based on the prior features of the dark channel;

[0059] S203: The prior information of the underwater scene depth is introduced into the first underwater original image through an attention mechanism to obtain the first initial feature map corresponding to the first underwater original image;

[0060] S204: Based on the attention mechanism and the first initial feature map, the dark channel prior is incorporated into the mapping of the first underwater original image to obtain the first dark channel feature map.

[0061] Furthermore, the first initial feature map includes a first initial query term, a first initial key value, and a first initial value term. The step of integrating the dark channel prior features into the mapping of the first underwater original image based on the attention mechanism and the first initial feature map to obtain the first dark channel feature map includes:

[0062] S20401: Map the first initial key value and the first initial value item to obtain the first prior key value and the first prior value item;

[0063] S20402: Using the first initial query term and the first prior key value, construct the correlation between the first underwater original image and the dark channel prior features, and fuse the dark channel information into the first initial query term by multiplying the first initial query term and the first prior term value to obtain the first dark channel feature map.

[0064] S3: Based on the first dark channel feature map and the second dark channel feature map, the self-attention Transformer module is fused to extract the first original feature map of the first underwater original image and the second original feature map of the second underwater original image.

[0065] Furthermore, the step of extracting the first original feature map of the first underwater original image by fusing the self-attention Transformer module based on the first dark channel feature map includes:

[0066] S301: The first dark channel feature map, the first initial feature map, and the first underwater original image are fused through the self-attention Transformer module to obtain the first original feature map. The self-attention Transformer module adopts a channel-based self-attention mechanism.

[0067] Furthermore, the self-attention Transformer module includes a multi-head self-attention module for feature fusion and a feedforward neural network. The feedforward neural network combines deep convolutions to encode information about the locations of neighboring pixels in order to extract local information from the image.

[0068] In this embodiment, the first underwater raw image and the second underwater raw image are respectively entered into their respective DCPTs. The calculation and processing logic of the DCPT for the first underwater raw image and the second underwater raw image is the same. Here, only the first underwater raw image is used as an example for specific explanation. The processing process of the second underwater raw image will not be described in detail.

[0069] To improve the performance of the proposed UCIE-PTI and recover high-quality images by leveraging prior physical knowledge, the physical prior information of the original underwater image is incorporated into the proposed UICE-PTI. The underwater imaging model is shown below:

[0070] Where λ∈{R, G, B}, I λ(x) J represents the original underwater image. λ(x) The image represents the recovered underwater image, where A represents global atmospheric light, and T represents... λ(x)=exp(-βd(x)) represents the transmittance, reflecting the percentage of scene radiation reflected in the medium and reaching the camera. For underwater imaging models, in local regions of underwater images, at least one color channel contains pixels with very low intensity; these pixels are called dark channels. To incorporate this prior knowledge, the dark channel prior feature DCP is introduced into the proposed UCIE-PTI. The DCP of underwater images can be defined as follows:

[0071] DCP(I) = min y∈Ω(x) (min c∈{r,g,b} I c (y)), where Ω(x) is a local block centered at x. DCP(I) is the dark channel prior, assuming the atmospheric background light value is 1. Combining the above two formulas, the transmittance can be calculated. in It is the transmittance of a local block centered at x, which is related to the depth of the underwater scene, i.e. This provides prior information on the depth of the underwater scene.

[0072] DCPT consists of a feature fusion multi-head self-attention module (FF-MSA) and a feedforward neural network (FFN). FF-MSA incorporates prior underwater scene depth information into the original input images (i.e., the first and second original underwater images) through an attention mechanism, and then applies FFN for feature transformation after the attention mechanism. For FF-MSA, considering the first original underwater image I... A ∈R H×W×3 The information distribution among the three channels (RGB) is usually non-uniform. FF-MSA first uses a self-attention mechanism in the first underwater raw image I A This establishes long-range dependencies between channels, thereby capturing degradation differences between different channels. The self-attention mechanism produces results. (i.e., the first initial feature map) can be represented as: Among them, Q A ,K A V A These are the first underwater original images I A The results after point-wise convolution and depth-wise convolution are, in order, the first initial query term, the first initial key value, and the first initial value term.

[0073] Meanwhile, in order to incorporate the prior information of the dark channel into the network, the calculated first initial key value and first initial value term are mapped to obtain their respective first prior key value and first prior value term. And Q A From the first raw underwater image I A The mapping. By utilizing Q A , To create the first raw underwater image I A With dark channel prior features DCP(I) A )∈R H×W×1 The relevance, and then the first initial query term Q. A Multiply by the first prior value term To integrate the dark channel information into the first initial query item Q A In the process, the first dark channel feature map is obtained. Finally, the first initial feature map is obtained by applying an attention mechanism to the first original underwater image. Compared with the first dark channel feature map containing dark channel information The fusion process yields a first original feature map containing dark channel information. Formulated as:

[0074]

[0075] It is DCP(I) A The result after point-wise convolution and depth-wise convolution; This represents a linear mapping. The Attention mechanism here uses a channel-based self-attention mechanism, which can be generalized as: Attention(Q,K,V)=Softmax(K·Q' / α)·V, where α represents the scaling factor.

[0076] Compared to traditional FFN, the FFN proposed in this embodiment combines depthwise convolution to encode information about the positions of neighboring pixels, thereby better extracting local information of the image. Therefore, the output of DCPT is the first original feature map. for:

[0077] S4: The first global feature map is extracted from the first original feature map by the CS-Transformer module based on second-order statistics, and the second global feature map is extracted from the second original feature map.

[0078] Furthermore, the step of extracting the first global feature map from the first original feature map using the CS-Transformer module based on second-order statistics includes:

[0079] S401: The first original feature map of the linear mapping is calculated based on the channel attention mechanism and the spatial attention mechanism respectively to obtain the first channel feature map and the first spatial feature map;

[0080] S402: Adaptively add the first channel feature map and the first spatial feature map to obtain the first global feature map.

[0081] Furthermore, CS-Transformer includes Spatial Transformer and Channel Transformer.

[0082] The first channel feature map is obtained by processing the channel Transformer module, which is used to calculate the channel attention.

[0083] The first spatial feature map is obtained by processing the spatial Transformer module, which is used to calculate spatial attention.

[0084] The channel Transformer module and the spatial Transformer module run in parallel. When the spatial Transformer module computes spatial attention, the first original feature map is downsampled to reduce spatial computation costs, and then upsampled to return the same feature scale as the output features of the channel Transformer module.

[0085] Furthermore, the spatial Transformer module includes a first multi-head attention module and a second multi-head attention module;

[0086] The first multi-head attention module is window-based and has second-order statistical properties, used to calculate the similarity of pixels within the window;

[0087] The second multi-head attention module is based on a moving window and has second-order statistical properties, used to calculate the similarity of pixels within the moving window.

[0088] In this embodiment, after the DCPT module, the first original feature map After linear mapping, the input is fed into the CS-Transformer block, which combines channel and spatial Transformer modules. Specifically, the CS-Transformer block mainly consists of two parallel Transformer modules: a channel Transformer module (CTM) for calculating channel attention and a spatial Transformer module (STM) for calculating spatial attention. Then, the outputs of the channel and spatial Transformer modules (i.e., the first channel feature map and the second spatial feature map) are adaptively added to obtain the final output feature map of the CS-Transformer block, i.e., the first global feature map (corresponding to the first original underwater image). It is worth noting that when calculating spatial attention, the original feature map is downsampled to reduce the computational cost of the STM in space, and then upsampled to return to the normal feature scale for feature fusion with the CTM output, thereby obtaining feature maps with different dimensional feature information. In the self-attention mechanism, query(Q), key(K), and value(V) first establish local associations of image information through pointwise and depthwise convolutions.

[0089] Furthermore, the CTM module consists of Conv-Attention and FFN components. Considering that the Transformer only consists of linear layers and a self-attention module, lacking positional encoding, the patch output only depends on the corresponding input and cannot capture changes in local features, which is unsuitable for visual tasks. Therefore, absolute positional encoding is designed through deep convolutional modules and residual connections to generate the input tensor. Then, pointwise convolutions are used to aggregate pixel-level cross-channel contextual information, and depthwise convolutions are designed to encode the spatial context within the channels, generating projected query terms (Q), key terms (K), and value terms (V) to establish connections between local contextual information. In the original Transformer, the attention computation space complexity is O(WH). 2 The time complexity is O((WH)). 2 C). Aside from normalization, dot product attention involves two consecutive matrix multiplications. The first S = QK T Calculate pairwise similarity between pixels and per-pixel attention maps. Then aggregate the values ​​V into the per-pixel attention map to generate the output SV. Since matrix multiplication is correlated, from (QK) T )V to Q(K T V) has no impact on the effect, but the space complexity becomes O((WH)C+C). 2 The time complexity becomes O((WH)C). 2Therefore, Conv-attention calculates attention in the following form: Convatt(X) = Q(ρ(K) T V), where ρ is the normalization function. It's worth noting that this attention is not a direct approximation of scaled dot product attention, but it can still be viewed as a generalized attention mechanism that models feature interactions using query, key, and value vectors. Simply embedding absolute positions into the input can limit the modeling of relationships between patches. Therefore, to address this issue, integrating relative position encoding... For a window size of M, obtain the relative attention map. Depthwise convolution can be viewed as a special case of relative position encoding. RAM is computed using a two-dimensional depthwise convolution with kernel weights P: Where ο represents the Hadamard product. Therefore, the first channel feature map Conv-Attention can be written as: Convatt(X)=Q(ρ(K) T V))+RAM.

[0090] Furthermore, considering the spatial differences in underwater degraded images, the Transformer module STM was used spatially. The STM module mainly consists of two parts. First, a first multi-head attention module (W-SMSA) based on windows and with second-order statistical properties was designed to calculate the similarity of pixels within a window. Then, a second multi-head attention module (SW-SMSA) based on moving windows and with second-order statistical properties was designed. By designing moving windows to establish connections between windows, interactions across local windows are achieved to obtain non-local receptive fields, thereby establishing long-distance dependencies in space. Therefore, the definition of STM is as follows:

[0091]

[0092]

[0093]

[0094]

[0095] Where x m Describes the output characteristics of W-SMSA, x m+1 The output features of SW-SMSA are represented, and the two are combined to obtain the first spatial feature map.

[0096] It is important to note that when calculating the correlation of spatial attention, the proposed UICE-PTI uses the second-order statistic Pearson correlation to calculate the correlation between Q and K, instead of the dot product. To achieve better feature relevance learning; among which These represent the height, width, and number of channels of the feature map before reshaping, respectively. Any patches in Q and K... The formula for the Pearson correlation coefficient is as follows: Where r i,j Let represent the correlation between the i-th patch in Q and the j-th patch in K. Then the correlation coefficient matrix of Q and K is: Therefore, the multi-head self-attention mechanism (SMSA) with second-order statistical properties can be formulated as: SMSA(x) = φ p (P(Q,K,V)), where φ p represents linear projection, while P represents attention with Pearson correlation coefficient, which is formulated as: P(Q,K,V)=Softmax(corr(Q,K))·V.

[0097] S5: Select several first structural features that meet the preset conditions from the first global feature map and fuse them into the second global feature map to obtain a second enhanced feature map; select several second structural features that meet the preset conditions from the second global feature map and fuse them into the first global feature map to obtain a first enhanced feature map.

[0098] Furthermore, the step of selecting several first structural features from the first global feature map that meet preset relevance conditions and fusing them into the second global feature map to obtain a second enhanced feature map includes:

[0099] S501: Calculate the correlation coefficient between the first global feature map and the second global feature map;

[0100] S502: Based on the correlation coefficient and feature transfer attention, the first structural feature is obtained by filtering from the first global feature map;

[0101] S503: Associate the first structural feature with the corresponding second global feature in the second global feature map, and obtain the first fused feature after convolution;

[0102] S504: Aggregate the first fused feature with the second global feature in the second global feature map and perform residual connection to obtain the second enhanced feature map.

[0103] Furthermore, the step of filtering the first structural features from the first global feature map based on the correlation coefficient and feature-transfer attention includes:

[0104] S50201: Use the correlation coefficient to obtain a transfer attention map, the transfer attention map including position-related elements, the position-related elements representing the most position-related elements in the first global feature map and the second global feature map;

[0105] S50202: Using the location-related elements as indexes, apply an index selection operation to the expanded patch of the first initial value item of the first global feature map to filter and obtain the first structural feature.

[0106] In this embodiment, considering that underwater images of similar scenes often exhibit similar degradation, they can provide valuable complementary information to each other for mutually beneficial enhancement. To leverage this fact, an FTIM module is proposed to convey the correlation between two branches (such as the branch corresponding to the first original underwater image and the branch corresponding to the second original underwater image, as exemplified in this embodiment) and provide effective information for better enhancement results. It is important to note that the spatial resolution and number of channels are different for each encoder / decoder layer. In the encoder stage, the spatial resolution gradually decreases to half the input, and the number of channels doubles. Patch merging is used during downsampling to reduce information loss. The decoder stage follows the opposite trend, with the spatial resolution gradually increasing to double and the number of channels decreasing to half the input.

[0107] Taking branch B (first global feature map) corresponding to the first original underwater image and branch A (second global feature map) corresponding to the second original underwater image as examples, correlation embedding aims to embed correlation information from branch B into branch A by estimating the similarity between Q and K. The correlation coefficient Re between branch A and branch B is... i,j It can be defined as follows: in, These are the patches for Q and K, respectively.

[0108] The goal of feature transfer attention is to transfer structural features of an image from branch B(A) to branch A(B) to supplement the information in branch A(B) and achieve collaborative enhancement. To transfer feature information from the most relevant location in branch B(A), the correlation coefficient Re is used. i,j To obtain the attention shift graph T, the i-th element in T It can be represented as: t i =argmax j (Re i,j ), where t i Let represent the position in branch B(A) that is most relevant to the i-th position in branch A(B). To obtain a clear first structural feature C from branch B(A), t is used. iApply the index selection operation to the expanded patches of index V: c i =v ti , where c i This represents the value at the i-th position in C, which is equal to the t-th position in V. i The value at each position.

[0109] To fuse the features of the two branches in the model, features Q and C are first concatenated, and then passed through a convolutional layer to obtain the first fused feature Z = Conv(Concat(C, Q)). Furthermore, a soft attention module is used to aggregate the first fused feature Z and the second global feature Q, enhancing the transmitted structural information. The soft attention map S represents the confidence level of the transmitted structural features at each position in C, and its calculation is as follows: s i =max j (r i,j ), where s i This is the i-th position of the soft attention map S. To fully utilize the effective information of branch A(B), a residual connection is used to combine the synthetic feature Z with the second global feature Q of branch A(B). Finally, the second enhanced feature map F output by FTIM is... out It can be represented as: in, This indicates element-wise summation. This indicates element-wise multiplication.

[0110] S6: Restore the first enhanced feature map and / or the second enhanced feature map to obtain the first underwater enhanced image and the second underwater enhanced image.

[0111] In this embodiment, the system restores the first enhanced feature map and the second enhanced feature map to obtain the first underwater enhanced image and the second underwater enhanced image; furthermore, the system can enhance only the first underwater original image, only the second underwater original image, or both the first and second underwater original images according to the user's needs.

[0112] The underwater image collaborative enhancement method in this embodiment utilizes the similar degradation features among multiple images acquired from the same underwater scene, and employs collaborative processing and joint learning strategies to complement the underwater images, thereby achieving joint enhancement and effectively improving the quality of the enhanced underwater images.

[0113] Furthermore, networks trained using the l1 loss function outperform those trained using the l2 loss function in both PSNR and SSIM image evaluation metrics. Therefore, this embodiment chooses the l1 loss function to calculate the feature difference between the output image and the reference image. The l1 loss function is expressed as follows: Where ||□||1 represents the l1 norm, This represents the final enhanced image pair. Represents a pair of real images.

[0114] Multi-scale structural similarity (MS-SSIM) is an image quality paradigm based on structural similarity (SSIM). Compared to single-scale SSIM, it offers greater flexibility in fusing variations in image resolution and visual conditions. Therefore, this embodiment defines the corresponding loss function as follows:

[0115]

[0116] To compute the perceptual loss, a pre-trained VGG-16 network, trained on the ImageNet dataset, is used. Then, the distance between the final augmented image pair and the ground truth image pair is calculated, as follows: Where φ is the relevant feature extraction function.

[0117] Finally, the total loss function of the proposed UICE-PTI can be expressed as: Loss = λ1L1 + λ M L M +λ Per L Per Where λ is the weighting coefficient, when λ out =1, and λ M and λ Per When the value is set to 0.5, the proposed UICE-PTI achieves the best test results on the test dataset.

[0118] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media provided in this disclosure and in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-speed SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0119] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, first object, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, first object, or method. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, first object, or method that includes that element.

[0120] The above description is only a preferred embodiment of this disclosure and does not limit the patent scope of this disclosure. Any equivalent structural or procedural changes made based on the content of this disclosure and its drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this disclosure.

Claims

1. A method of underwater image synergistic enhancement, characterized in that, The method comprises the following steps: obtaining an underwater original image pair, the underwater original image pair comprising a first underwater original image and a second underwater original image, the underwater original image pair being captured at the same underwater scene; extracting a first dark channel feature map of the first underwater original image and a second dark channel feature map of the second underwater original image based on dark channel prior, the first dark channel feature map and the second dark channel feature map containing dark channel information of the respective corresponding underwater original image; based on the first dark channel feature map and the second dark channel feature map, fusing a self-attention Transformer module to extract a first original feature map of the first underwater original image and a second original feature map of the second underwater original image; extracting a first global feature map from the first original feature map and a second global feature map from the second original feature map through a CS-Transformer module based on second-order statistics; screening a plurality of first structural features with a correlation meeting a preset condition from the first global feature map and fusing them to the second global feature map to obtain a second enhanced feature map, and screening a plurality of second structural features with a correlation meeting a preset condition from the second global feature map and fusing them to the first global feature map to obtain a first enhanced feature map; restoring the first enhanced feature map and / or the second enhanced feature map to obtain a first underwater enhanced image and a second underwater enhanced image.

2. The underwater image synergistic enhancement method according to claim 1, characterized in that, In the step of extracting the first dark channel feature map of the first underwater original image and the second dark channel feature map of the second underwater original image based on dark channel prior, the step of extracting the first dark channel feature map of the first underwater original image based on dark channel prior comprises the following steps: obtaining a dark channel prior feature corresponding to the first underwater original image; calculating underwater scene depth prior information based on the dark channel prior feature; introducing the underwater scene depth prior information into the first underwater original image through an attention mechanism to obtain a first initial feature map corresponding to the first underwater original image; fusing the dark channel prior into the mapping of the first underwater original image based on the attention mechanism and the first initial feature map to obtain the first dark channel feature map.

3. The underwater image collaborative enhancement method according to claim 2, characterized in that, The first initial feature map comprises a first initial query item, a first initial key value and a first initial value item, and the step of fusing the dark channel prior feature into the mapping of the first underwater original image based on the attention mechanism and the first initial feature map to obtain the first dark channel feature map comprises the following steps: mapping the first initial key value and the first initial value item to obtain a first prior key value and a first prior value item; using the first initial query item and the first prior key value to construct the correlation between the first underwater original image and the dark channel prior feature, and fusing the dark channel information into the first initial query item through the product of the first initial query item and the first prior item value to obtain the first dark channel feature map.

4. The method of collaborative enhancement of underwater images according to claim 2, characterized in that, The step of fusing the first underwater original image based on the first dark channel feature map includes: The first dark channel feature map, the first initial feature map and the first underwater original image are fused through the self-attention Transformer module to obtain the first original feature map, wherein the self-attention Transformer module adopts a channel-based self-attention mechanism.

5. The underwater image collaborative enhancement method according to claim 4, characterized in that, The self-attention Transformer module includes a multi-head self-attention module for feature fusion and a feedforward neural network that combines deep convolution to encode information of adjacent pixel positions to extract local information of the image.

6. The underwater image collaborative enhancement method of claim 1, wherein, The step of extracting the first global feature map from the first original feature map through the CS-Transformer module based on second-order statistics includes: The first original feature map is calculated based on a channel attention mechanism and a spatial attention mechanism to obtain a first channel feature map and a first spatial feature map; The first channel feature map and the first spatial feature map are adaptively added to obtain the first global feature map.

7. The underwater image collaborative enhancement method according to claim 6, characterized in that: The CS-Transformer includes a spatial Transformer and a channel Transformer The first channel feature map is obtained by processing the channel Transformer module, and the channel Transformer module is used to calculate channel attention; The first spatial feature map is obtained by processing the spatial Transformer module, and the spatial Transformer module is used to calculate spatial attention; The channel Transformer module and the spatial Transformer module are parallel, and when calculating spatial attention, the first original feature map is down-sampled to reduce spatial calculation consumption and then up-sampled to return to the same feature scale as the output feature of the channel Transformer module.

8. The underwater image collaborative enhancement method of claim 7, wherein, The spatial Transformer module includes a first multi-head attention module and a second multi-head attention module; The first multi-head attention module is based on a window and has a second-order statistical property, and is used to calculate the similarity of pixels within the window; The second multi-head attention module is based on a moving window and has a second-order statistical property, and is used to calculate the similarity of pixels within the moving window.

9. The underwater image collaborative enhancement method of claim 1, wherein, The step of screening a plurality of first structural features with a correlation meeting a preset condition from the first global feature map and fusing them into the second global feature map to obtain a second enhanced feature map includes: The correlation coefficient between the first global feature map and the second global feature map is calculated; The first structural features are screened from the first global feature map based on the correlation coefficient and feature transfer attention; The first structure feature is associated with a corresponding second global feature in the second global feature map, and a first fusion feature is obtained after convolution; The first fusion feature is aggregated with the second global feature in the second global feature map and connected with a residual, and a second enhanced feature map is obtained.

10. The underwater image collaborative enhancement method of claim 9, wherein, Based on the correlation coefficient and the feature transfer attention, the first structure feature is filtered from the first global feature map, including: Using the correlation coefficient to obtain a transfer attention map, the transfer attention map includes position-related elements, and the position-related elements represent the most relevant elements in positions in the first global feature map and the second global feature map; Using the position-related elements as an index, an index selection operation is applied to an unfolded patch of a first initial value item of the first global feature map to filter the first structure feature.

Images