A non-paired underwater image enhancement method based on structure perception multi-modal fusion

Abstract

This invention discloses an unpaired underwater image enhancement method based on structure-aware multimodal fusion, belonging to the field of underwater image processing technology. It includes constructing a generator network based on a U-shaped Transformer; training the generator network using an unpaired dataset, employing a combination of generative adversarial loss and multi-scale patch NCE contrastive loss as constraints; concatenating the underwater RGB image to be processed with a spatial depth map and a structural prior map to construct a multimodal input tensor, which is then fed into the trained generator network, outputting the enhanced underwater image after forward propagation. This invention enhances the network's ability to perceive geometric contours by introducing depth maps and structural prior maps. Combined with global modeling of the Transformer and unpaired contrastive learning, it achieves high-fidelity restoration of high-turbidity underwater images without the need for paired data, and can be widely applied in fields such as marine exploration and underwater monitoring.

Description

Technical Field

[0001] This invention belongs to the field of underwater image enhancement technology, and specifically relates to an unpaired underwater image enhancement method based on structure-aware multimodal fusion. Background Technology

[0002] Underwater imaging has significant applications in marine resource exploration, environmental monitoring, and underwater archaeology. However, due to the selective absorption of light by water and the strong scattering effect caused by suspended particles, underwater images generally suffer from color distortion, low contrast, and blurred details. This scattering is particularly severe in highly turbid waters, leading to non-uniform fogging and significantly weakening the edge and geometric information of objects, severely hindering subsequent visual analysis and intelligent recognition tasks.

[0003] To improve underwater image quality, existing technologies mainly fall into two categories: physical model-based restoration methods and deep learning-based enhancement methods. Physical model-based methods (such as dark channel priors and underwater light propagation models) recover clear images by reversing the underwater imaging process. These methods rely on specific physical assumptions and parameter estimations; however, the real underwater environment is complex and variable. Under conditions of high turbidity or dynamic scattering, accurate estimation of parameters such as transmittance and background light is extremely difficult, easily leading to problems such as incomplete dehazing, color distortion, or edge artifacts. In recent years, deep learning-based image enhancement methods, especially convolutional neural networks (CNNs), have made significant progress in this field. CNNs can effectively fit the mapping relationship between underwater degradation and clear images through end-to-end learning. However, the inherent limitation of the local receptive field in CNNs makes it difficult to model large-scale global scattering distributions, resulting in poor performance when dealing with non-uniform fog. Furthermore, to alleviate the difficulty of obtaining paired training data, some studies have introduced unpaired image translation frameworks (such as Cycle-GAN). However, existing unpaired methods often lack explicit constraints on image structural information, and in extremely turbid scenes, the geometric contours of objects are easily distorted or lost.

[0004] Recently, the Transformer architecture has been introduced into the field of underwater image enhancement due to its powerful ability to model long-range dependencies. However, existing Transformer-based methods still mainly rely on a single RGB spectral input, ignoring the implicit physical geometric cues (such as depth information and local edge structures) in turbid water. In highly turbid media, the edge structures of objects are often severely obscured by scattered light, making it difficult to recover accurate geometric contours using only RGB features.

[0005] In summary, there is an urgent need for an underwater image enhancement method that can integrate multimodal geometric cues, possess global scattering perception capabilities, and maintain structural consistency under unpaired data conditions, in order to overcome the problems of incomplete defogging and loss of detail in existing technologies under strong scattering and non-uniform fogging scenarios. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides an unpaired underwater image enhancement method based on structure-aware multimodal fusion. This method aims to fully utilize scene geometric depth and edge structure cues under unpaired training data conditions, overcome non-uniform fogging interference in high-turbidity, strong scattering environments, and achieve high-fidelity restoration of underwater image color and high-frequency geometric details.

[0007] To achieve the above objectives, the technical solution of the present invention is as follows:

[0008] An unpaired underwater image enhancement method based on structure-aware multimodal fusion includes the following steps:

[0009] S1: Construct a generator network based on a U-shaped Transformer, which includes an encoder, a bottleneck layer, and a decoder. The encoder receives multimodal input tensors and performs hierarchical encoding through a window-based multi-head self-attention mechanism and a local enhancement feedforward network to extract multi-scale deep features. The decoder upsamples the deep features layer by layer and uses skip connections to stitch together high-frequency features of the same scale from the encoder to reconstruct the enhanced underwater image.

[0010] S2: The generator network is trained using a pre-built unpaired dataset. During the training process, generative adversarial loss and multi-scale Patch NCE contrastive loss of multi-level feature node sampling are used for joint constraint optimization until the model converges, and the trained generator network is obtained.

[0011] S3: Acquire the underwater RGB image to be processed, and retrieve the corresponding spatial depth map and structure prior map; combine the RGB image, spatial depth map, and structure prior map... Figure 3 The modal data are spliced ​​and fused along the channel dimension to construct a multimodal input tensor;

[0012] S4: Feed the multimodal input tensor into the trained generator network, and output the enhanced underwater image after forward propagation.

[0013] In the above scheme, after the encoder of the generator network completes feature extraction at each encoding level, it uses strided convolution with a stride of 2 to perform spatial downsampling and channel dimension expansion, so that the spatial resolution of the feature map is smoothly halved and the number of channels is doubled, in order to construct multi-scale hierarchical features.

[0014] In the above scheme, the window-based multi-head self-attention mechanism introduces a learnable relative position bias B when calculating attention, and its attention calculation expression is as follows:

[0015] ;

[0016] Where Q, K, and V are the query matrix, key matrix, and value matrix, respectively, and Softmax is the normalization exponential function; is the channel dimension of a single attention head; T is the matrix transpose.

[0017] In the above scheme, the computational logic of the local enhancement feedforward network is as follows: after channel expansion, the features are reshaped into two-dimensional tensors, and local edge details are captured using 3×3 depthwise separable convolutions in the spatial domain. Then, the channel weights are adaptively recalibrated using an efficient channel attention mechanism before outputting. Its transformation expression is:

[0018] ;

[0019] ;

[0020] ;

[0021] in, These are the features of the intermediate hidden layer after the first step of linear mapping; Z represents a non-linear activation function; Z is a one-dimensional multimodal feature sequence input after processing by a window-based multi-head self-attention mechanism. This is the first-layer linear fully connected weight matrix used for channel expansion; This represents the intermediate features after local structure extraction; For 3×3 depthwise separable convolution operations used to capture local structural details; Indicates 2D dimensional reshaping; The enhanced features are the output of the local enhanced feedforward network level; ECA represents the efficient channel attention function, which is used to adaptively recalibrate the fused features; This means flattening the two-dimensional features back into a one-dimensional sequence; This is the weight matrix of the second-layer linear fully connected layer used for channel dimensionality reduction.

[0022] In the above scheme, the generator network is further provided with an input projection module before the encoder, which is used to perform cross-modal feature aggregation and high-dimensional mapping on the multimodal input tensor; the input projection module adopts a 3×3 two-dimensional convolutional layer with no bias term and a LeakyReLU activation function with a negative slope set to 0.2.

[0023] In the above scheme, the decoder upsampling layer by layer is achieved by bilinear interpolation, and the upsampled features are spliced ​​and fused with the same-scale features from the encoder to restore the image spatial resolution.

[0024] In the above scheme, the generative adversarial loss is constructed using a least-squares generative adversarial network, and its generator loss function is... With discriminator loss function They are defined as follows:

[0025] ;

[0026] ;

[0027] in, y represents the expected value of the calculation; y~Y represents the reference sample y sampled from the real image domain Y; D is the confidence score output by the discriminator. Represents the source image x sampled from the turbid underwater image domain X; Enhanced image output by the generator.

[0028] In the above scheme, the multi-scale Patch NCE contrastive loss calculation process includes:

[0029] (1) Define a multi-scale feature set participating in the joint contrast space in the generator network. , ,in, The initial features are the output of the input projection module. These are shallow features. This is a mid-level feature. For deep features, Features output by the bottleneck layer;

[0030] (2) For each layer of feature map in the feature set, a fixed number of discrete spatial coordinate points are randomly selected in the spatial dimension as sampling tile anchor points; the feature vector extracted from the enhanced image features output by the generator at the anchor points is defined as the query sample z, and the feature vector extracted from the source domain turbidity input features at the same spatial coordinate points is defined as the positive sample z. The feature vectors extracted from the turbid input features of the source domain at other random coordinate points on the same feature map are defined as negative samples. ; Calculate the multi-scale Patch NCE contrastive loss based on the query sample, positive sample, and negative sample.

[0031] In a further technical solution, the multi-scale Patch NCE contrastive loss is calculated as follows:

[0032] ;

[0033] in, For multi-scale Patch NCE contrast loss; This represents the expectation of the sampled data from the source domain; M is the total number of layers extracting features from the joint contrast space; This represents the feature layer index in the joint comparison space; Indicates the first The total number of spatial locations in the layer feature map; s is the current spatial location index; Temperature hyperparameters for controlling distribution smoothness; This represents the total number of negative samples.

[0034] In a further technical solution, the total loss function of the training process Defined as:

[0035] ;

[0036] in, For generator loss, and They are respectively and The weights; The calculation process for the identity contrast loss calculated for the target domain image Y is as follows: Completely identical, except that the input source for the forward propagation is replaced with the target domain image Y.

[0037] Through the above technical solution, the unpaired underwater image enhancement method based on structure-aware multimodal fusion provided by the present invention has the following beneficial effects:

[0038] 1. This invention proposes a structure-aware multimodal fusion strategy. By explicitly introducing spatial depth maps and structural prior maps, scene geometric depth information and high-frequency edge cues are used as additional modalities and stitched together with RGB images in the channel dimension. This provides the network with clear physical geometric constraints, effectively making up for the problem of feature information loss in strong scattering environments of a single RGB modality, and significantly enhancing the network's ability to perceive and recover the geometric contours of underwater objects.

[0039] 2. This invention designs a generator architecture based on a U-shaped Transformer, and uses a window-based multi-head self-attention mechanism to model the global scattering dependency, enabling the model to perceive the spatial distribution of non-uniform fogging. Combined with the depthwise separable convolution in the local enhancement feedforward network, it accurately captures high-frequency details obscured by scattered light, achieving spatially adaptive dehazing and sharpness restoration, overcoming the problem of incomplete dehazing caused by the local receptive field limitation of traditional CNNs.

[0040] 3. Under the non-paired training framework, this invention constructs a joint contrast space covering multi-scale feature nodes. Through the joint constraints of multi-scale Patch NCE contrast loss and generative adversarial loss, it effectively ensures the spatial structure consistency between the source domain and the enhanced image without the need for paired ground truth data. This prevents geometric distortion and color shift in high-turbidity scenes and greatly improves the training efficiency and restoration fidelity in practical engineering applications.

[0041] 4. By introducing an identity contrast loss for clear images in the target domain, this invention constrains the network from excessive enhancement when faced with originally clear images, further improving the model's generalization ability and the stability of the enhancement results, and providing high-quality image data support for downstream tasks such as underwater visual perception, target detection and recognition. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0043] Figure 1 This is a schematic diagram of a generator network architecture based on a U-shaped Transformer disclosed in an embodiment of the present invention;

[0044] Figure 2 This is a schematic diagram illustrating the multi-scale tensor data flow and dimensional evolution disclosed in the embodiments of the present invention;

[0045] Figure 3 This is a schematic diagram of a locally enhanced feedforward network disclosed in an embodiment of the present invention;

[0046] Figure 4 This is a schematic diagram of the input projection module disclosed in the embodiments of the present invention;

[0047] Figure 5 This is a schematic diagram of the PatchNCE joint sampling node disclosed in the embodiments of the present invention. Detailed Implementation

[0048] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0049] This invention provides an unpaired underwater image enhancement method based on structure-aware multimodal fusion. This method enhances high-turbidity underwater images by constructing and training a generator network based on a U-shaped Transformer and utilizing multimodal inputs (RGB image, spatial depth map, and structure prior map) and a joint optimization strategy.

[0050] Example 1: Generator Network Construction and Image Enhancement Method

[0051] This embodiment describes in detail how to construct and use a trained generator network to enhance the underwater image to be processed. (Refer to...) Figure 1 The network architecture shown is Figure 2 The method, which illustrates the multi-scale tensor data flow, includes the following steps:

[0052] Step S1: Construct a generator network based on a U-shaped Transformer.

[0053] The generator network has a U-shaped symmetrical structure, including an encoder, a bottleneck layer, and a decoder.

[0054] (1) Input Projection Module: An input projection module is provided at the entry point of the generator network. For example... Figure 4 As shown, this module consists of a 3×3 two-dimensional convolutional layer with no bias term and a LeakyReLU activation function with a negative slope (alpha) of 0.2, cascaded together. Its function is to perform preliminary cross-modal feature aggregation and high-dimensional mapping on the concatenated multimodal input tensor to prevent the generation of neuronal "dead zones" due to negative truncation of weak dark response features, and finally output the initial feature map. Its physical mapping process is represented as follows:

[0055] ;

[0056] in, This represents an unbiased two-dimensional convolution operation. =0.2 is a negative slope parameter. The initial features are the output of the input projection module.

[0057] (2) Encoder: The encoder consists of four levels (l=1,2,3,4). Each level includes a LeWinTransformer module group and a strided convolutional downsampling layer.

[0058] LeWin Transformer module group: Each module group contains two core sub-layers, namely the Window Multi-Head Self-Attention (W-MSA) module and the Locally-Enhanced Feed-Forward (LeFF) network.

[0059] like Figure 3 As shown, firstly, the input features of the previous level. (in , which are the multi-scale features output from the previous level, especially when =1 After layer normalization (LN), a window-based multi-head self-attention mechanism (W-MSA) is fed in to capture the global dependencies of the non-uniform underwater scattering field. W-MSA reduces computational complexity by dividing the feature map into non-overlapping windows and independently computing self-attention within each window. When computing attention, a learnable relative position bias B is introduced, enabling the model to perceive the relative positional relationships between pixels. The calculation formula is as follows:

[0060] ;

[0061] Where Q, K, and V are the query matrix, key matrix, and value matrix, respectively, and Softmax is the normalization exponential function; is the channel dimension of a single attention head; T is the matrix transpose.

[0062] W-MSA output and original input Perform residual connections to obtain intermediate features. The expression for this process is:

[0063] ;

[0064] Subsequently, intermediate features After a second layer normalization, the data is fed into a Locally Enhanced Feedforward Network (LeFF) to accurately extract and restore the high-frequency geometric details obscured by the water body. The structure of the LeFF is as follows: Figure 3 As shown, the computational process is as follows: First, the channel dimension is expanded (e.g., expanded by 4 times) through a first fully connected layer. After GELU activation, the features are reshaped into a two-dimensional tensor, and local spatial details are accurately extracted using a 3×3 depthwise separable convolution (DWConv). Then, the channel features are adaptively recalibrated using an efficient channel attention mechanism (ECA). Finally, the channel dimension is restored through a second fully connected layer. The matrix transformation expression is:

[0065] ;

[0066] ;

[0067] ;

[0068] in, These are the features of the intermediate hidden layer after the first step of linear mapping; Z represents a non-linear activation function; Z is a one-dimensional multimodal feature sequence input after processing by a window-based multi-head self-attention mechanism. This is the first-layer linear fully connected weight matrix used for channel expansion; This represents the intermediate features after local structure extraction; For 3×3 depthwise separable convolution operations used to capture local structural details; Indicates 2D dimensional reshaping; The enhanced features are the output of the local enhanced feedforward network level; ECA represents the efficient channel attention function, which is used to adaptively recalibrate the fused features; This means flattening the two-dimensional features back into a one-dimensional sequence; This is the weight matrix of the second-layer linear fully connected layer used for channel dimensionality reduction.

[0069] LeFF's output and input Perform residual connections again to obtain the final output features of the current LeWin Transformer module group. The expression for this process is:

[0070] ;

[0071] in, Representative layer normalization operation; This is a locally enhanced feedforward network.

[0072] Stride-based convolutional downsampling: At the end of each encoding layer, a stride-2 convolution is used to spatially downsample the feature map, doubling the number of channels to provide dimensionality-reduced input features for the next layer. The downsampling process is represented as follows:

[0073] ;

[0074] in, This represents a two-dimensional convolution operation with a stride of 2; This is the encoder's hierarchical index.

[0075] By alternately performing feature extraction and downsampling layer by layer, the network gradually extracts multi-scale scattering context features containing both global distribution and local details. Specifically, in the multi-level encoding process of this invention, as the level index increases, the network sequentially calculates and explicitly outputs intermediate layer features corresponding to different spatial resolution scales. , , ,like Figure 2 As shown. These multi-scale features are related to the aforementioned initial high-dimensional features. Together they constitute the data stream of the network's forward propagation, while being truncated and retained by the bypass to construct the joint contrastive feature space in subsequent training phases.

[0076] (3) Bottleneck layer: The deepest feature output by the encoder The data is fed into the bottleneck layer. The bottleneck layer consists of consecutive LeWinTransformer modules, used to further refine global deep semantic features and scattering context information, outputting bottleneck layer features. And use it as the initial input to the lowest level of the decoder, that is .

[0077] (4) Decoder: The decoder also contains four levels (k=1,2,3,4), which are symmetrical to the encoder structure and are responsible for gradually recovering the image resolution from deep features.

[0078] During the decoding phase, the network progressively upsamples. To avoid the checkerboard effect caused by traditional transposed convolution during feature reconstruction and to improve the operator execution efficiency of edge computing platforms, this invention performs the following operation on the feature fusion of the current decoding level k: At the beginning of each level of the decoder, it first performs upsampling on the features of the previous level. Bilinear interpolation upsampling is performed to double the resolution. Then, a 3×3 convolution is used to adjust the number of channels in the upsampled features, making them comparable to the skip connection features from the encoder. Dimension matching. Finally, the two are concatenated along the channel dimension. The calculation is as follows:

[0079] ;

[0080] in, This is the bottom-up hierarchical index for the decoder; Features of the previous decoding level, For bilinear interpolation smooth upsampling operation, For channel dimensionality reduction, a single-layer 3×3 two-dimensional convolution is used. For the same-scale skip connection of high-frequency features from the encoder (when k=4, call the initial projected features), ), The fusion features after splicing;

[0081] Feature aggregation: combining the concatenated features The input is fed into the LeWin Transformer module group of the current decoding level (the structure is the same as that in the encoder), and global dependency modeling and local detail enhancement are performed through W-MSA and LeFF. The output is the decoding features of the current level. .

[0082] Repeat the above process until the feature map resolution is restored to the H×W size of the original input image.

[0083] (5) Output module: Finally, the high-dimensional feature map reconstructed by the decoder is mapped back to the RGB color space through a convolutional layer (e.g., 3×3 convolution) to generate and output the enhanced underwater image.

[0084] Step S2: Obtain the data to be processed and construct the multimodal input tensor.

[0085] Acquire an underwater RGB image to be processed, which is subject to non-uniform scattering interference. Meanwhile, the corresponding normalized spatial depth map is obtained based on existing binocular vision estimation algorithms (such as SGBM, PSMNet, etc.). Furthermore, the Limited Contrast Adaptive Histogram Equalization (CLAHE) algorithm is used to extract high-frequency edge contrast maps from the original RGB image, serving as a structural prior map. The parameters for the CLAHE algorithm can be set as follows: block size of 8×8 and contrast limit of 2.0.

[0086] The three modal data mentioned above are concatenated along the channel dimension to form a 5-channel multimodal input tensor. : .

[0087] Step S3: Perform forward propagation for image enhancement.

[0088] The constructed multimodal input tensor The trained generator network constructed in step S1 is fed into the network. The data follows the network forward propagation process described in step S1, passing sequentially through the input projection module, the hierarchical feature extraction of the encoder, the global semantic refinement of the bottleneck layer, and the feature reconstruction and fusion of the decoder. Finally, the enhanced underwater image is reconstructed by the output module.

[0089] Example 2: Generator Network Training Method

[0090] This embodiment details how to train the generator network described above using an unpaired dataset to enable it to perform underwater image enhancement. (Refer to...) Figure 1 and Figure 3 The training process includes the following steps:

[0091] Step S1: Prepare the unpaired training dataset.

[0092] Two image domains are constructed: a source domain X, containing underwater images with high turbidity and severe fogging; and a target domain Y, containing clear underwater images with normal colors. Images between the two domains do not need to be paired one-to-one.

[0093] Step S2: Construct the overall training framework.

[0094] The training framework includes a generator G (i.e., the U-shaped Transformer network constructed in Example 1) and a discriminator D. The discriminator D adopts a Markov discriminator (PatchGAN) structure to distinguish between generated images and real, clear images.

[0095] Step S3: Forward propagation and loss calculation.

[0096] A batch of images x and y are sampled from the source domain X and the target domain Y, respectively. The source domain image x and its corresponding depth map and structure prior map are used as multimodal inputs according to step S2 of Example 1 and fed into the generator G to obtain the enhanced image G(x). At the same time, the target domain image y (whose depth map and structure prior map also need to be calculated) can also be fed into the generator G to obtain the output G(y), which is used to calculate the identity loss.

[0097] Then, the following losses are calculated:

[0098] (1) Generative Adversarial Loss: A least-squares generative adversarial network (LSGAN) is used to make training more stable. Generator loss function. With discriminator loss function They are defined as follows:

[0099] ;

[0100] ;

[0101] in, y represents the expected value of the calculation; y~Y represents the reference sample y sampled from the real image domain Y; D is the confidence score output by the discriminator. Represents the source image x sampled from the turbid underwater image domain X; Enhanced image output by the generator.

[0102] (2) Multi-scale Patch NCE Contrast Loss: To significantly reduce the peak memory read / write speeds and computational overhead during the contrast optimization stage, this invention designs a truncated multi-scale feature extraction mechanism in the generator network to force the generated image to maintain structural consistency with the source image under non-pairing settings. A multi-scale feature set participating in the joint contrast space is defined. for:

[0103] ;

[0104] in, The initial features are the output of the input projection module. , , These are the shallow, mid-level, and deep features output by the encoder after three downsampling layers. Features output by the bottleneck layer.

[0105] like Figure 5 As shown, the network computes the feature set during forward propagation. Then directly truncate and return, and then return the feature set. Before feeding the projector H of the Multilayer Perceptron (MLP), this network introduces a patch space sampling mechanism. Specifically, it performs a sampling based on the feature set. In each multi-scale feature map layer, a fixed number of discrete spatial coordinate points (e.g., 256 points per layer) are randomly sampled as anchor points for the sampled image patches. Each spatial coordinate point on the feature map physically corresponds to a local image patch of a specific size in the original underwater image. Based on these sampled coordinates, the feature vector extracted from the enhanced image features output by the generator at this coordinate point is defined as the query sample z; simultaneously, the feature vector extracted from the source domain turbidity input features at strictly identical spatial coordinate points is defined as the positive sample z. This forces the network to anchor itself to the physical geometry of the local region without changing its shape; furthermore, the feature vectors extracted from the turbid input features of the source domain at other random coordinate points on the same feature map are defined as negative samples. Through this tile space sampling operation, the network extracts only representative local feature vectors for subsequent multilayer perceptron mapping.

[0106] Calculate the multi-scale Patch NCE contrastive loss based on the query samples, positive samples, and negative samples:

[0107] ;

[0108] in, For multi-scale Patch NCE contrast loss; This represents the expectation of the sampled data from the source domain; M is the sum of the number of layers extracting features in the joint contrast space (M=5 in this example). This represents the feature layer index in the joint comparison space; Indicates the first The total number of spatial locations in the layer feature map; s is the current spatial location index; Temperature hyperparameters are used to control the smoothness of the distribution (usually set to 0.07). This represents the total number of negative samples.

[0109] Simultaneously, the identity contrast loss is calculated for the target domain image y. The calculation process is exactly the same, except that the input source x is replaced with the target domain image y.

[0110] (3) Overall loss function: The total loss used to optimize the generator is obtained by weighted summation of the above losses. : in, For generator loss, and They are respectively and The weight; in a specific embodiment, it can be set =1.0, =0.5.

[0111] Step S4: Model parameter optimization.

[0112] The parameters of the generator G and discriminator D are alternately optimized using the Adam optimizer (initial learning rate set to 2e-4, β1=0.5, β2=0.999). The generator is updated after each update of the discriminator. Steps S3 and S4 are repeated until the total loss is reached. Converge and save the final generator network model parameters.

[0113] Application example:

[0114] In practical applications, the pre-trained generator network model can be directly loaded, and forward inference can be performed on any acquired underwater RGB image according to steps S2 and S3 in Example 1 to quickly obtain an enhanced, clear image. This method does not require additional parameter optimization or physical model solving for each input image, and is characterized by high efficiency and robustness.

[0115] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A non-paired underwater image enhancement method based on structure-aware multimodal fusion, characterized in that, Includes the following steps: S1: Construct a generator network based on a U-shaped Transformer, which includes an encoder, a bottleneck layer, and a decoder. The encoder receives multimodal input tensors and performs hierarchical encoding through a window-based multi-head self-attention mechanism and a local enhancement feedforward network to extract multi-scale deep features. The decoder upsamples the deep features layer by layer and uses skip connections to stitch together high-frequency features of the same scale from the encoder to reconstruct the enhanced underwater image. S2: The generator network is trained using a pre-built unpaired dataset. During the training process, generative adversarial loss and multi-scale Patch NCE contrastive loss of multi-level feature node sampling are used for joint constraint optimization until the model converges, and the trained generator network is obtained. S3: Acquire the underwater RGB image to be processed, and retrieve the corresponding spatial depth map and structural prior map; stitch and fuse the three modal data of the RGB image, spatial depth map and structural prior map in the channel dimension to construct a multimodal input tensor; S4: Feed the multimodal input tensor into the trained generator network, and output the enhanced underwater image after forward propagation; The computational logic of the locally enhanced feedforward network is as follows: after channel expansion, the features are reshaped into two-dimensional tensors, and local edge details are captured using 3×3 depthwise separable convolutions in the spatial domain. Then, the channel weights are adaptively recalibrated using an efficient channel attention mechanism before outputting. Its transformation expression is: ； ； ； in, These are the features of the intermediate hidden layer after the first step of linear mapping; Z represents a non-linear activation function; Z is a one-dimensional multimodal feature sequence input after processing by a window-based multi-head self-attention mechanism. This is the first-layer linear fully connected weight matrix used for channel expansion; This represents the intermediate features after local structure extraction; For 3×3 depthwise separable convolution operations used to capture local structural details; Indicates 2D dimensional reshaping; The enhanced features are the output of the local enhanced feedforward network level; ECA represents the efficient channel attention function, which is used to adaptively recalibrate the fused features; This means flattening the two-dimensional features back into a one-dimensional sequence; This is the second-layer linear fully connected weight matrix used for channel dimensionality reduction; The multi-scale Patch NCE contrastive loss calculation process includes: (1) Define a multi-scale feature set participating in the joint contrast space in the generator network. , ,in, The initial features are the output of the input projection module. These are shallow features. This is a mid-level feature. For deep features, Features output by the bottleneck layer; (2) For each layer of feature map in the feature set, a fixed number of discrete spatial coordinate points are randomly selected in the spatial dimension as sampling tile anchor points; the feature vector extracted from the enhanced image features output by the generator at the anchor points is defined as the query sample z, and the feature vector extracted from the source domain turbidity input features at the same spatial coordinate points is defined as the positive sample z. The feature vectors extracted from the turbid input features of the source domain at other random coordinate points on the same feature map are defined as negative samples. ; Calculate the multi-scale Patch NCE contrastive loss based on the query sample, positive sample, and negative sample.

2. The unpaired underwater image enhancement method based on structure-aware multimodal fusion according to claim 1, characterized in that, After the encoder of the generator network completes feature extraction at each encoding level, it uses strided convolution with a stride of 2 to perform spatial downsampling and channel dimension expansion. The spatial resolution of the feature map is smoothly halved, and the number of channels is doubled to construct multi-scale hierarchical features.

3. The unpaired underwater image enhancement method based on structure-aware multimodal fusion according to claim 1, characterized in that, The window-based multi-head self-attention mechanism introduces a learnable relative position bias B when calculating attention, and its attention calculation expression is as follows: ； Where Q, K, and V are the query matrix, key matrix, and value matrix, respectively, and Softmax is the normalization exponential function; is the channel dimension of a single attention head; T is the matrix transpose.

4. The unpaired underwater image enhancement method based on structure-aware multimodal fusion according to claim 1, characterized in that, The generator network is further equipped with an input projection module before the encoder, which is used to perform cross-modal feature aggregation and high-dimensional mapping on the multimodal input tensor. The input projection module uses a 3×3 two-dimensional convolutional layer with no bias term and a LeakyReLU activation function with a negative slope set to 0.

2.

5. The unpaired underwater image enhancement method based on structure-aware multimodal fusion according to claim 1, characterized in that, The decoder uses bilinear interpolation to perform layer-by-layer upsampling, and then stitches and fuses the upsampled features with features of the same scale from the encoder to restore the image spatial resolution.

6. The unpaired underwater image enhancement method based on structure-aware multimodal fusion according to claim 1, characterized in that, The generative adversarial loss is constructed using a least-squares generative adversarial network, and its generator loss function is... With discriminator loss function They are defined as follows: ； ； in, y represents the expected value of the calculation; y~Y represents the reference sample y sampled from the real image domain Y; D is the confidence score output by the discriminator. Represents the source image x sampled from the turbid underwater image domain X; Enhanced image output by the generator.

7. The unpaired underwater image enhancement method based on structure-aware multimodal fusion according to claim 1, characterized in that, The multi-scale Patch NCE contrastive loss is calculated as follows: ； in, For multi-scale Patch NCE contrast loss; This represents the expectation of the sampled data from the source domain; M is the total number of layers extracting features from the joint contrast space; This represents the feature layer index in the joint comparison space; Indicates the first The total number of spatial locations in the layer feature map; s is the current spatial location index; Temperature hyperparameters for controlling distribution smoothness; This represents the total number of negative samples.

8. The unpaired underwater image enhancement method based on structure-aware multimodal fusion according to claim 7, characterized in that, Total loss function during training Defined as: ； in, For generator loss, and They are respectively and The weights; The calculation process for the identity contrast loss calculated for the target domain image Y is as follows: Completely identical, except that the input source for the forward propagation is replaced with the target domain image Y.

Images