Non-paired underwater image enhancement method based on multi-scale contrast learning and CLIP cross-modal contrast learning

Abstract

The invention provides a non-paired underwater image enhancement method based on multi-scale contrast learning and CLIP cross-modal contrast learning. The method comprises the following steps: randomly extracting degraded images and non-paired enhanced images thereof from at least two underwater image data sets to construct a training set; extracting a text through a large language model to construct an enhanced image positive sample description and a degraded image negative sample description; configuring a generative adversarial network, wherein a generator comprises: a self-calibration feature enhancement module performing a feature splitting-gating fusion operation; a transform global modeling module of cross-region dependence is established through block self-attention, and a transform global modeling module of cross-region dependence is established through block self-attention; implementing joint contrast learning: calculating multi-scale contrast loss of feature block similarity at a multi-layer feature random sampling position of the generator; constraining CLIP cross-modal comparison loss of semantic alignment of the enhanced image and the positive sample description; optimizing network parameters based on the training set and a loss function; and inputting the underwater degraded image into the optimized generator to obtain an enhanced underwater image.

Description

Technical Field

[0001] The present invention belongs to the technical fields of image processing and computer vision, and specifically relates to an unpaired underwater image enhancement method based on multi-scale contrastive learning and CLIP cross-modal contrastive learning. Background Art

[0002] In the fields of image processing and computer vision, underwater image enhancement technology is of great significance for applications such as marine resource exploration, underwater robot navigation, and ecological monitoring. Due to the complex optical properties of the underwater environment, such as light attenuation, dispersion effects, and scattering by suspended particles, underwater images generally suffer from severe color distortion, reduced contrast, and blurred details, which greatly restrict the reliability of subsequent analysis and applications. Traditional enhancement methods are mostly based on physical models (such as dark channel priors and Retinex theory) or histogram equalization techniques, achieving enhancement by separating the illumination and reflection components or adjusting the global pixel distribution. However, such methods rely on artificial assumptions and parameter tuning, lacking adaptability to complex underwater degradation patterns, and are prone to defects such as color oversaturation, local detail loss, and noise amplification.

[0003] In recent years, deep learning-based underwater image enhancement methods have significantly improved the effectiveness of image enhancement. Deep learning methods mainly construct end-to-end network structures to extract potential degradation patterns from a large number of underwater images. Combined with attention mechanisms, multi-scale feature extraction and other techniques, they effectively suppress problems such as water scattering, color shift and blur, thereby generating clearer, more natural and realistic enhanced images. However, existing underwater image methods still face the following challenges: First, due to the particularity of the scene, it is currently difficult to obtain paired underwater image datasets. Existing deep learning methods usually rely on synthetic datasets for training, which limits the performance of supervised learning. Second, training from a single image modality easily ignores semantic consistency constraints, resulting in deviations between the enhancement results and the real scene. In addition, the multi-scale feature extraction capability is insufficient, making it difficult to simultaneously restore global color balance and local texture details.

[0004] Underwater image enhancement methods based on generative adversarial networks (GANs) can generate enhanced images with higher visual quality through adversarial training mechanisms. Furthermore, contrastive learning has demonstrated strong potential in image processing. By constructing positive and negative sample pairs, it guides the model to learn more discriminative feature representations, promoting the development of unsupervised and weakly supervised methods. Meanwhile, breakthroughs in multimodal semantic alignment achieved by vision-language pre-training models (such as CLIP) have provided new insights into image enhancement tasks. CLIP, trained on large-scale image-text pairs, possesses cross-modal feature mapping capabilities and can theoretically optimize the image enhancement process through semantic guidance.

[0005] While contrastive learning has shown promise in unsupervised representation learning, its application in cross-level feature alignment for underwater image enhancement has not been fully explored. Furthermore, existing research on multimodal semantic alignment has largely focused on general domains, lacking targeted modeling of the unique semantics of underwater scenes, and has not effectively integrated multi-scale contrastive learning to enhance cross-level feature consistency. Summary of the Invention

[0006] To address the shortcomings of existing underwater image enhancement methods, such as reliance on paired degraded-enhanced images (which are difficult to obtain in actual scenes), lack of semantic supervision resulting in semantic deviation between enhancement results and real scenes, and insufficient multi-scale feature extraction capability affecting detail recovery, the present invention provides an unpaired underwater image enhancement method based on multi-scale contrastive learning and CLIP cross-modal contrastive learning.

[0007] This method first randomly extracts degraded images and enhanced images from mainstream unpaired underwater image datasets (such as EUVP and LSUI), and constructs a training set after data augmentation (including random cropping to a uniform size, random application of 8 transformations: keeping the original, vertical flipping, rotating 90 / 180 / 270 degrees and combinations) and standardization to solve the problem of scarce paired data; secondly, the training set images are analyzed through a large language model to generate scene description text covering content and style, and refined to form enhanced image positive sample descriptions and degraded image negative sample descriptions to provide cross-modal semantic supervision for the model; then a generative adversarial network is designed, in which the generator contains a self-calibrated feature enhancement module (enhancing local feature expression through feature splitting-gating fusion operations) and a Transformer global modeling module. Block (establishing cross-region dependencies through block self-attention to capture global context), the discriminator adopts a multi-scale structure to extract features at different levels; further, the model is optimized through joint contrastive learning: the multi-scale contrast loss randomly samples feature blocks at the spatial position of the generator's multi-layer features, calculates their similarity and constrains the alignment of features at the same position and the distinction of features at different positions, thereby enhancing the consistency of multi-scale details; the CLIP cross-modal contrast loss constrains the semantic alignment of the enhancement results with the real scene by calculating the semantic similarity between the enhanced image features and the positive text description; the adversarial loss and the identity mapping loss (constraining the generator's identity mapping ability to the real enhanced image) are combined to collaboratively optimize the model parameters; finally, the underwater degraded image to be enhanced is input into the trained generator, and an enhanced image with natural colors and clear details is output.

[0008] Through the collaborative design of unpaired data driving, cross-modal semantic supervision and multi-scale feature enhancement, the present invention effectively solves the shortcomings of existing methods in data acquisition, semantic consistency and detail recovery, and significantly improves the practicality and effect of underwater image enhancement.

[0009] The technical solution specifically adopted by the present invention to solve the technical problem is:

[0010] An unpaired underwater image enhancement method based on multi-scale contrastive learning and CLIP cross-modal contrastive learning, including:

[0011] Randomly extract degraded images and their unpaired enhanced images from at least two underwater image datasets to construct a training set;

[0012] Analyze images using a large language model to generate scene description text, and refine the text to construct descriptions of enhanced image positive samples and degraded image negative samples;

[0013] Configure the generative adversarial network, including the generator and the multi-scale discriminator, where the generator contains:

[0014] A self-calibrated feature enhancement module that performs feature splitting-gated fusion operations;

[0015] And, a Transformer global modeling module that establishes cross-region dependencies through block self-attention;

[0016] Implementing Joint Contrastive Learning:

[0017] Calculating a multi-scale contrast loss of feature patch similarity at the multi-layer feature random sampling positions of the generator;

[0018] CLIP cross-modal contrast loss for semantic alignment of constrained enhanced images and positive sample descriptions;

[0019] Optimizing network parameters based on the training set and the loss function;

[0020] The underwater degraded image is fed into the optimized generator to produce the enhanced underwater image.

[0021] Furthermore, the feature splitting-gating fusion operation includes:

[0022] Split the input feature map into two sub-feature maps;

[0023] Perform global pooling and convolution on the first subgraph to generate spatial weights;

[0024] Perform convolution on the second subgraph and then multiply it with the activated weight map;

[0025] Dual path features after fusion processing.

[0026] Furthermore, the calculating of the feature block similarity at the random sampling position includes:

[0027] Randomly select p spatial locations in the feature map of the generator layer l;

[0028] Extract the feature blocks at the corresponding position, transform them through MLP and perform L2 normalization;

[0029] Construct a p×p similarity matrix to calculate the cross entropy loss.

[0030] Furthermore, the large language model receives the grouped underwater degraded images and enhanced images, and generates description text covering image content and style.

[0031] Furthermore, the self-calibration feature enhancement module compresses the input feature map through a 1×1 convolution channel, and splits the output into two paths: the first path is processed by pooling downsampling and upsampling, and the second path is processed by convolution; the two paths of features are fused through a gating mechanism.

[0032] Furthermore, the Transformer global modeling module converts feature map blocks into token sequences to model cross-block dependencies.

[0033] Furthermore, feature block similarity calculation is performed on multiple layers of features of the generator, and the loss of each layer is averaged.

[0034] Furthermore, the CLIP cross-modal contrast loss calculation includes:

[0035] Calculate the similarity between the enhanced image features and the positive text features and negative text features;

[0036] The semantic alignment loss is calculated by normalizing the probability.

[0037] Furthermore, the loss function is a weighted sum of multi-scale contrast loss, CLIP cross-modal contrast loss, adversarial loss and identity mapping loss.

[0038] And, an underwater image enhancement system comprising:

[0039] A training set construction module is used to randomly extract degraded images and their unpaired enhanced images from at least two underwater image datasets to construct a training set;

[0040] The semantic supervision module includes a large language model interface for analyzing images to generate scene description text and refine positive and negative sample descriptions;

[0041] The adversarial training engine is configured with a generative adversarial network consisting of a generator and a multi-scale discriminator. The generator includes:

[0042] A self-calibrated feature enhancement module that performs feature splitting-gating fusion operations;

[0043] Transformer global modeling module, which establishes cross-region dependencies through block self-attention;

[0044] Multimodal contrastive learning unit for:

[0045] Calculate the feature block similarity at the random sampling position of the generator's multi-layer features;

[0046] Constrained enhanced images are semantically aligned with positive sample descriptions;

[0047] Parameter optimizer, which updates network parameters based on the training set and loss function;

[0048] The enhancement execution module inputs the underwater degraded image into the optimized generator and outputs the enhanced underwater image.

[0049] And, a computer device includes a memory, a processor and a computer program stored in the memory, and the processor implements the above method when executing the computer program.

[0050] A non-transitory computer-readable storage medium stores a computer program, which implements the method described above when executed by a processor.

[0051] Compared with the prior art, the present invention and its preferred embodiments have at least the following beneficial effects:

[0052] First, to address the core pain point of the underwater image enhancement field, which has long relied on paired degraded-enhanced images (which are difficult to obtain on a large scale in actual scenarios), the present invention breaks through the strong dependence on paired data through a non-paired data-driven training set construction method (such as extracting degraded and enhanced images from mainstream non-paired datasets such as EUVP and LSUI), significantly improving the scalability and practicality of the method in practical applications.

[0053] Secondly, to address the problem that traditional methods lack semantic supervision, resulting in semantic deviation between the enhancement results and the real scene, this invention introduces a large language model to generate and refine cross-modal positive and negative text pairs (positive sample descriptions of enhanced images and negative sample descriptions of degraded images), and combines the CLIP cross-modal contrast loss constraint to semantically align the enhanced image with the positive text, effectively avoiding the defect of "only optimizing visual quality but semantic inconsistency" and improving the semantic consistency between the enhancement results and the real scene.

[0054] Third, to address the technical bottleneck of insufficient multi-scale feature extraction capabilities that affects detail recovery, the present invention collaboratively designs the self-calibration feature enhancement module (feature splitting-gated fusion operation) and the Transformer global modeling module (block self-attention to establish cross-regional dependencies) in the generator. This not only enhances the detailed expression of local features (the self-calibration module suppresses redundant information through spatial weight gating), but also captures global contextual dependencies (the Transformer module models cross-regional associations through block self-attention), significantly improving the accuracy of underwater image color correction and detail restoration.

[0055] Fourthly, the design of the joint contrastive learning loss function (weighted collaboration of multi-scale contrastive loss, CLIP cross-modal contrastive loss, adversarial loss and identity mapping loss) achieves multi-dimensional optimization from local feature consistency, global semantic alignment to image authenticity and model stability, avoiding the performance bias caused by a single loss function and further improving the comprehensive enhancement effect of the model.

[0056] Fifth, the refined design of data augmentation strategies (such as randomly applying eight transformations: preserving the original, vertical flipping, rotating 90 / 180 / 270 degrees and combinations) and multi-scale discriminators (residual connections that fuse features from different levels) provide the model with more robust training signals by increasing the diversity of training data and enhancing the comprehensiveness of feature expression, further consolidating the generalization ability and stability of the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0058] Figure 1 It is a flow chart of the implementation method of the embodiment of the present invention.

[0059] Figure 2 It is a structural diagram of the overall training process in an embodiment of the present invention.

[0060] Figure 3 4 is a framework diagram of the underwater image enhancement generator network in an embodiment of the present invention.

[0061] Figure 4 4 is a structural diagram of the CLIP comparative learning framework in an embodiment of the present invention. DETAILED DESCRIPTION

[0062] In order to make the features and advantages of the present invention more clearly understood, the following embodiments are given for detailed description:

[0063] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meanings as those commonly understood by those skilled in the art to which this application belongs.

[0064] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0065] In response to the defects and shortcomings of the existing technology, an embodiment of the present invention proposes an unpaired underwater image enhancement method based on multi-scale contrastive learning and CLIP cross-modal contrastive learning. By constructing an unpaired training set and using a large language model to generate text descriptions, and innovatively introducing CLIP-driven cross-modal contrast loss, the consistency between the image and the semantic description is constrained and enhanced. At the same time, multi-scale contrastive learning is designed on the designed encoder-bottleneck layer architecture, and multi-scale feature optimization is achieved by combining a multi-scale discriminator. Without the need for paired datasets and supervision, this method significantly improves the color fidelity, detail clarity and semantic rationality of underwater images, providing an efficient solution for the intelligent processing of complex underwater scenes.

[0066] This solution achieves its goals through the following core designs: First, to address the scarcity of paired samples in underwater image datasets, an unpaired training set is constructed and a large language model is used to generate positive and negative text description pairs; second, a multi-scale contrastive loss based on the encoder-bottleneck layer is designed to improve the network's feature extraction capabilities at multiple scales; finally, a cross-modal contrastive loss based on CLIP is introduced, and a large language model is used to generate positive and negative text description pairs to provide cross-modal semantic supervision for the network, thereby improving the semantic consistency and visual quality of the enhanced images.

[0067] The specific implementation of the scheme includes: randomly extracting a certain number of unpaired underwater images and enhanced images from two mainstream underwater image datasets to construct an unpaired underwater image dataset for training; randomly extracting some unpaired underwater images and enhanced images from the training set, using a large language model to generate corresponding underwater scene descriptions, and constructing positive and negative text pairs; designing an underwater image enhancement model based on a generative adversarial network (GAN), including a generator and a discriminator; designing a loss function to guide the parameter optimization of the generator and discriminator designed by S3, wherein the generator adopts a multi-loss joint optimization strategy, including multi-scale contrast loss, CLIP cross-modal contrast loss, adversarial loss and identity mapping loss, and the discriminator adopts adversarial loss; using the unpaired underwater image dataset constructed by S1, the underwater image enhancement model described in S3 is trained with the optimization objective function designed by S4 to optimize the network parameters; the image to be tested is input into the underwater image enhancement generator trained in step S5, and the enhanced underwater image is output.

[0068] like Figure 1-Figure 4 As shown, it is presented as the following steps:

[0069] Step S1: randomly extract a certain number of unpaired underwater images and enhanced images from two mainstream underwater image datasets to construct an unpaired underwater image dataset for training;

[0070] Step S2: Randomly extract some unpaired underwater images and enhanced images from the training set, use the large language model to generate corresponding underwater scene descriptions, and construct positive and negative text pairs;

[0071] Step S3: designing an underwater image enhancement model based on a generative adversarial network (GAN), including a generator and a discriminator;

[0072] Step S4: Design a loss function to guide the parameter optimization of the generator and discriminator designed in S3. The generator adopts a multi-loss joint optimization strategy, including multi-scale contrast loss, CLIP cross-modal contrast loss, adversarial loss and identity mapping loss, and the discriminator adopts adversarial loss;

[0073] Step S5: using the unpaired underwater image dataset constructed in S1, training the underwater image enhancement model described in S3 with the optimization objective function designed in S4, and optimizing the network parameters;

[0074] Step S6: input the image to be tested into the underwater image enhancement generator trained in step S5, and output the enhanced underwater image.

[0075] As a preferred solution of this embodiment, step S1 includes the following steps:

[0076] Step S11: Randomly select N pairs of unpaired underwater images and enhanced images from two currently mainstream underwater image datasets, EUVP and LSUI, respectively, to construct an unpaired underwater image dataset containing 2N pairs of unpaired underwater images and enhanced images, where N is the number of image pairs selected from each underwater image dataset;

[0077] Step S12: Scale each image to a uniform size, and randomly crop the image to The size of the image is guaranteed to be consistent in the final size. are the height and width of the cropped image;

[0078] Step S13: For each cropped image, randomly apply one of the following eight data augmentation methods: original image, vertical flip, 90-degree rotation, 90-degree rotation followed by vertical flip, 180-degree rotation, 180-degree rotation followed by vertical flip, 270-degree rotation, and 270-degree rotation followed by vertical flip. Then, normalize the RGB channels of all images. The resulting unpaired underwater images and augmented images serve as the unpaired underwater image dataset.

[0079] As a preferred solution of this embodiment, step S2 includes the following steps:

[0080] Step S21: randomly select M pairs of unpaired underwater images and enhanced images from the training data set, where M is the number of selected image pairs, and then divide the M pairs of unpaired underwater images and enhanced images into several groups on average;

[0081] Step S22: Construct the input of the large language model. The input content mainly includes two parts: the first part is 2n underwater images after average grouping, including n underwater images and n enhanced images; the second part is the prompt text, which serves as the task instruction description, telling the model that the current specific task is to analyze the input images and generate a text description for each image, including the image style and image content;

[0082] Step S23: Input the content of step S22 into the large language model doubao, and use its visual understanding ability to generate a text description for each image, covering the image content and style, and generate 2n text descriptions for 2n unpaired images;

[0083] Step S24: Input the image groups into the large language model bean bag in sequence according to the grouping in step S21, repeat step S23 several times to obtain several groups of underwater image text descriptions, and integrate all text descriptions into a text file;

[0084] Step S25, use the large language model bean bag to analyze and refine the text file obtained in step S24. First, construct the input of the large language model. The input content mainly includes two parts: the first part is the text file described in step S24, and the second part is the prompt text, which serves as a task instruction description, telling the model that the current specific task is to analyze the input text and perform text extraction on the description of underwater images and enhanced images in the text to generate positive and negative sample pairs of underwater image descriptions; then, input the content described in step S25 into the large language model, and the model will screen, analyze and refine the text, and finally generate several pairs of positive and negative sample pairs of underwater image descriptions, where the i-th pair is recorded as and , , is the number of positive and negative sample pairs.

[0085] As a preferred solution of this embodiment, step S3 includes the following steps:

[0086] Step S31, designing a generator network, the network consisting of an encoder based on a self-calibrated convolutional block, a bottleneck layer based on a Swin-Transformer, and a decoder based on a self-calibrated convolutional block;

[0087] Step S32: Design a discriminator network, use a multi-scale discriminator, and perform adversarial training on the generated enhanced underwater image and the real underwater image.

[0088] As a preferred solution of this embodiment, step S31 includes the following steps:

[0089] Step S311, design a self-calibrated convolution (SCConv) block, which will be used as a multi-scale feature enhancement module and embedded in the encoder and decoder. The self-calibrated convolution block includes the following steps: for a given feature map ( 、 is the height and width of the feature map, C is the number of channels), through Convolutional layers and activation functions Compress the number of input channels to half to obtain the feature map ; The feature map Downsampling is performed through global average pooling, and then Convolutional layer to obtain feature maps At the same time, the feature map Directly pass Convolutional layer, to achieve basic feature extraction, get features ; The feature map After upsampling and feature map Add, then activate with Sigmoid, and add Multiply to get the feature map , the feature map and feature maps Passing respectively Convolutional layer and LeakyReLU activation function, and splicing along the channel dimension to obtain feature out, out is then input After the convolution layer and the original feature map Add and activate through LeakyReLU to get the feature , the specific formula is as follows:

[0090]

[0091]

[0092]

[0093]

[0094]

[0095]

[0096] in, express Convolutional layers, express Convolutional layers, is the activation function, represents the global average pooling operation, is the Sigmoid function, represents element-wise multiplication, Represents the F.interpolate upsampling operation, Represents a concatenation operation along the channel dimension.

[0097] Step S312: Design an encoder. The encoder consists of three downsampling blocks, each of which consists of a convolutional layer, an instance normalization layer, an activation function, and a self-calibration convolution block. The encoder includes the following steps: After a use Convolutional layer extracts initial feature map , perform instance normalization and activate with ReLU activation function, input the features into the self-calibration convolution block designed in step S311 to obtain the feature map ,Will enter Convolutional layer, instance normalization layer and ReLU activation function to obtain feature map , and then the feature map The self-calibrated convolution block designed by S31 further extracts information and obtains the feature map Repeat again, enter Convolutional layer, instance normalization layer and ReLU activation function to obtain feature map , and then the feature map After the self-calibration convolution block, the information is further extracted to obtain the feature map , the specific formula is as follows:

[0098]

[0099]

[0100]

[0101]

[0102]

[0103]

[0104] in, express Convolutional layers, express Convolutional layers, represents the instance normalization operation, is the activation function, Self-calibration convolution block designed for step S31.

[0105] Step S313: Design the bottleneck layer and transform the feature map The input consists of a convolution kernel of size , the step length is The block coding layer consists of convolutional layers, Will follow Size is divided into blocks, i.e. , the channel dimension of each block is ; Then flatten the features of these blocks and map them into token sequences These tokens enter the Swin Transformer block for calculation. Swin Transformer uses the Window Attention mechanism to calculate local features and implements cross-window information interaction through the sliding window mechanism to ensure that global features are modeled at different scales. The features obtained after Swin Transformer processing are , will be converted back to the feature map , the specific formula is as follows:

[0106]

[0107]

[0108]

[0109] in, represents the block coding layer operation, represents the Swin Transformer block, Represents a tensor reshape operation.

[0110] Step S314: Design a decoder. The decoder uses three upsampling blocks combined with skip connections to gradually restore high-resolution features. The upsampling block consists of a convolutional layer, instance normalization, activation function, and self-calibration convolution block. The decoder includes the following steps: and After splicing along the channel dimension, pass through Transposed convolution, instance normalization layer and ReLU activation function are used for upsampling to obtain feature maps , feature map Input the self-calibration convolution block designed in step S311 to further extract information and obtain the feature map .Will and After splicing along the channel dimension, pass through Transposed convolution, instance normalization layer and ReLU activation function are used for upsampling to obtain feature maps , feature map After the self-calibration convolution block, the information is further extracted to obtain the feature map Finally, the feature map and After splicing along the channel dimension, input with reflection filling Convolutional layer and Tanh normalization to generate enhanced images , the specific formula is as follows:

[0111]

[0112]

[0113]

[0114]

[0115]

[0116] in, Represents a reflection fill Convolutional layers, express Transposed convolutional layer, represents the instance normalization operation, is the activation function, The self-calibration convolution block designed for step S31, represents the splicing operation along the channel dimension, is the normalization function.

[0117] As a preferred solution of this embodiment, step S32 includes the following steps:

[0118] Step S321: For a given input image , enter it Convolutional layer, followed by activation function LeakyReLU activation, extracting feature maps , and the feature map enter Convolutional layer, extract and Features of the same dimension ,Will and Add to get the feature map ; The feature map enter The convolution layer and LeakeyReLU activation function are downsampled to obtain , while directly inputting the image enter Convolutional layer, get features ,Will and Add to get the feature map ; The feature map enter Convolutional layer and LeakeyReLU activation function , while directly inputting the image enter Convolutional layer, get features ,Will and Add to get the feature map , the specific formula is as follows:

[0119]

[0120]

[0121]

[0122]

[0123]

[0124]

[0125]

[0126]

[0127]

[0128] in, express Convolutional layers, express Convolutional layers, express Convolutional layers, is the activation function;

[0129] Step S322: enter Convolution layer, the output channel is 1, generates the discrimination result, and obtains the result by Tanh normalization to [-1,1] , which is convenient for calculating the adversarial loss. The specific formula is as follows:

[0130]

[0131] in, express Convolutional layers, is the normalization function.

[0132] As a preferred solution of this embodiment, step S4 includes the following steps:

[0133] Step S41: Design the generator loss. The goal of the generator is to generate images that can deceive the discriminator, so that the discriminator's prediction value is close to 1. The generator loss is composed of CLIP cross-modal contrast loss, multi-scale contrast loss, adversarial loss, and identity mapping loss. The specific formula is as follows:

[0134]

[0135] in, represents the CLIP cross-modal contrast loss, represents the multi-scale contrast loss, Represents resistance to loss, represents the identity mapping loss, 、 、 、 Represent the weight coefficients of each loss respectively.

[0136] Step S42: Design the discriminator loss. The discriminator network is trained using the mean square error loss in the form of the least squares generative adversarial network (LS-GAN). , the discriminator should give a prediction value as close to 1 as possible, and for the generated enhanced image , the discriminator should give a prediction value as close to 0 as possible. Calculate the loss of the real enhanced image and the generated enhanced image separately, and take the average of the two to balance the training. The specific formula is as follows:

[0137]

[0138] in, Indicates the calculation of expected value, represents the discriminator network designed in step S32, whose output is the authenticity probability, i.e. .

[0139] As a preferred solution of this embodiment, step S41 includes the following steps:

[0140] Step S411: Design a multi-scale contrast loss. and the generated enhanced image Input the generator network G designed in step S3 respectively to obtain the feature map output by the specified layer of the generator and (B is the batch size, 、 is the height and width of the feature map, C is the number of channels, Indicates that the extracted Then, the shape of the transformed feature map is and ,in is the dimension of the flattened space. Flattened spatial dimensions Randomly select p feature blocks at p index positions from Select p feature blocks corresponding to the index position, respectively recorded as and In order to further extract more discriminative feature information, the sampled feature block is input into the multi-layer perceptron (MLP), transformed through the fully connected layer, and the output feature vector is L2 normalized to obtain the final feature vector and Then, the similarity between the feature vectors is calculated. Specifically, the feature pairs of each pair of feature maps are and The similarity logits is calculated by dot product and divided by the temperature parameter to get the similarity , Reflect the The original features and The similarity of the enhanced features can finally be obtained by To calculate the contrast loss, for the An enhanced feature, The original features are used as positive examples, that is, the diagonal elements As the positive score, the remaining off-diagonal elements As the negative score. Then, the contrastive loss is calculated using the cross-entropy loss function. This loss encourages matching of features at the same location (high positive score) while distinguishing features at different locations (low negative score). Finally, the contrastive losses calculated from all feature maps of the specified layer are averaged and returned as the final loss. The specific formula is as follows:

[0141]

[0142]

[0143]

[0144]

[0145]

[0146]

[0147]

[0148]

[0149]

[0150] in, Represents the extraction generator Layer characteristics, represents a tensor reshape operation, Indicated in the feature Select the index position The feature block, represents the multilayer perceptron operation, represents the length of the vector, represents the temperature parameter, used to scale the similarity score, represents the logarithm operation based on the natural logarithm base e. represents the exponential operation based on the natural logarithm base e. Indicates calculation of Contrastive loss of layer features.

[0151] Step S412: Design CLIP cross-modal contrast loss. Input the image encoder of the CLIP model to obtain the feature vector (D is the feature dimension generated by the CLIP image encoder). and Input the text encoder of the CLIP model respectively to obtain the feature vector and Next, calculate the eigenvector and the negative sample text feature vectors and Specifically, the enhanced image is obtained by performing a dot product operation on the feature vector and dividing it by the temperature parameter. Similarity scores with positive and negative sample pairs and , perform Softmax normalization on the similarity scores to obtain the matching probability distribution between the image and the text. Finally, CLIP contrastive learning loss is calculated through contrastive learning. The loss function encourages the image to match the positive sample text description while reducing the similarity with the negative sample text description. The specific formula is as follows:

[0152]

[0153]

[0154]

[0155]

[0156]

[0157]

[0158]

[0159]

[0160] in, is the image encoder of the CLIP model, is the text encoder of the CLIP model, represents the temperature parameter, used to scale the similarity score, represents the exponential operation based on the natural logarithm base e. , is the number of positive and negative sample pairs;

[0161] Step S413: Design adversarial loss. The goal of the generator is to generate enhanced images that can deceive the discriminator. , so that the discriminator's predicted value is close to 1. The loss function also adopts the LS-GAN form, and the specific formula is as follows:

[0162]

[0163] in, Indicates the calculation of expected value, represents the discriminator network designed in step S33, whose output is the authenticity probability, i.e. .

[0164] Step S414: Design the identity mapping loss. For the input real enhanced image , the identity mapping loss is used to constrain the generator to remain as unchanged as possible when processing the real image of the target domain. The specific formula is as follows:

[0165]

[0166] in, represents sampling real images from the data distribution of the target domain, The output of the generator, represents the L1 norm, which is used to measure the pixel-wise difference between the generated image and the original input.

[0167] As a preferred solution of this embodiment, step S5 includes the following steps:

[0168] Step S51: randomly divide the training data set obtained in step S1 into several batches;

[0169] Step S52: Input underwater image , after the generator network in step S3, the loss is calculated using the loss function designed in step S4;

[0170] Step S53: Calculate the gradient of the network parameters using the back propagation method according to the loss, and update the network parameters using the Adam optimization method;

[0171] Step S54: Repeat steps S51 to S53 in batches to obtain a trained generator.

[0172] In the above scheme of this embodiment, first, an innovative unpaired underwater training set and an automated text supervision mechanism are constructed. To address the problem of scarce underwater paired data, the present invention extracts unpaired samples from mainstream datasets and uses a large language model to automatically generate cross-modal positive and negative text description pairs, breaking through the traditional supervised learning's reliance on paired data while enhancing the rationality of the generated image through semantic constraints. Secondly, a multi-objective joint optimization strategy is designed to achieve synergistic enhancement of quality and semantics. Through the joint optimization of multi-scale contrast loss (randomly sampling feature blocks to achieve cross-level semantic alignment), CLIP cross-modal contrast loss (constraining image-text semantic consistency), adversarial loss, and identity mapping loss, the color fidelity, detail clarity, and semantic rationality of the enhanced image are simultaneously improved without the need for paired supervision, overcoming the limitations of traditional methods that rely on artificial assumptions or single modality constraints. Finally, the applicability of complex underwater scenes is significantly improved. Compared to existing methods based on physical models or single GANs, this invention, through cross-modal semantic guidance and multi-scale feature fusion, can effectively restore biological outlines and suppress noise amplification in extreme underwater environments such as strong scattering and low contrast, providing highly reliable visual data support for tasks such as underwater detection and ecological monitoring. In summary, through cross-modal semantic alignment, multi-scale feature optimization, and joint loss design, this invention systematically solves the problems of color distortion, detail loss, and semantic deviation in underwater image enhancement, providing an efficient and robust technical solution for intelligent image processing in complex underwater environments.

[0173] Based on the same inventive concept, the present invention also provides a computer device, which includes: one or more processors and a memory for storing one or more computer programs; the program includes program instructions, and the processor is used to execute the program instructions stored in the memory. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is used to implement one or more instructions, specifically for loading and executing one or more instructions in a computer storage medium to implement the above method.

[0174] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium having a computer program stored thereon, which, when executed by a processor, performs the above-described method. The storage medium may be any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0175] It should be noted that, unless otherwise defined, the technical or scientific terms used in the present invention should have the usual meanings understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0176] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other manner. Any person skilled in the art may utilize the above-disclosed technical content to modify or modify the present invention into equivalent embodiments. However, any simple modifications, equivalent variations, and modifications to the above embodiments that do not depart from the technical content of the present invention and are based on the technical essence of the present invention remain within the scope of protection of the present invention.

[0177] The present invention is not limited to the above-mentioned optimal implementation mode. Anyone can derive various other forms of unpaired underwater image enhancement methods based on multi-scale contrastive learning and CLIP cross-modal contrastive learning under the inspiration of the present invention. All equivalent changes and modifications made within the scope of the patent application of the present invention should fall within the scope of the present invention.

Claims

1. An unpaired underwater image enhancement method based on multi-scale contrastive learning and CLIP cross-modal contrastive learning, characterized by: include: Randomly extract degraded images and their unpaired enhanced images from at least two underwater image datasets to construct a training set; Analyze images using a large language model to generate scene description text, and refine the text to construct descriptions of enhanced image positive samples and degraded image negative samples; Configure the generative adversarial network, including the generator and the multi-scale discriminator, where the generator contains: A self-calibrated feature enhancement module that performs feature splitting-gated fusion operations; And, a Transformer global modeling module that establishes cross-region dependencies through block self-attention; Implementing Joint Contrastive Learning: Calculating a multi-scale contrast loss of feature patch similarity at the multi-layer feature random sampling positions of the generator; CLIP cross-modal contrast loss for semantic alignment of constrained enhanced images and positive sample descriptions; Optimizing network parameters based on the training set and the loss function; The underwater degraded image is fed into the optimized generator to produce the enhanced underwater image.

2. The unpaired underwater image enhancement method based on multi-scale contrastive learning and CLIP cross-modal contrastive learning according to claim 1, characterized in that: The feature splitting-gating fusion operation includes: Split the input feature map into two sub-feature maps; Perform global pooling and convolution on the first subgraph to generate spatial weights; Perform convolution on the second subgraph and then multiply it with the activated weight map; Dual path features after fusion processing.

3. The unpaired underwater image enhancement method based on multi-scale contrastive learning and CLIP cross-modal contrastive learning according to claim 1, characterized in that: The random sampling position calculation of feature block similarity includes: Randomly select p spatial locations in the feature map of the generator layer l; Extract the feature blocks at the corresponding position, transform them through MLP and perform L2 normalization; Construct a p×p similarity matrix to calculate the cross entropy loss.

4. The unpaired underwater image enhancement method based on multi-scale contrastive learning and CLIP cross-modal contrastive learning according to claim 1, characterized in that: The large language model receives grouped underwater degraded images and enhanced images and generates description text covering image content and style.

5. The unpaired underwater image enhancement method based on multi-scale contrastive learning and CLIP cross-modal contrastive learning according to claim 1, characterized in that: The self-calibration feature enhancement module compresses the input feature map through a 1×1 convolution channel, and the output is split into two paths: the first path is processed by pooling downsampling and upsampling, and the second path is processed by convolution; The two features are fused through a gating mechanism.

6. The unpaired underwater image enhancement method based on multi-scale contrastive learning and CLIP cross-modal contrastive learning according to claim 1, characterized in that: The Transformer global modeling module converts feature map blocks into token sequences to model cross-block dependencies.

7. The unpaired underwater image enhancement method based on multi-scale contrastive learning and CLIP cross-modal contrastive learning according to claim 3, characterized in that: Feature block similarity calculation is performed on multiple layers of features of the generator, and the loss of each layer is averaged.

8. The unpaired underwater image enhancement method based on multi-scale contrastive learning and CLIP cross-modal contrastive learning according to claim 1, characterized in that: The CLIP cross-modal contrast loss calculation includes: Calculate the similarity between the enhanced image features and the positive text features and negative text features; The semantic alignment loss is calculated by normalizing the probability.

9. The unpaired underwater image enhancement method based on multi-scale contrastive learning and CLIP cross-modal contrastive learning according to claim 1, characterized in that: The loss function is a weighted sum of multi-scale contrast loss, CLIP cross-modal contrast loss, adversarial loss, and identity mapping loss.

10. An underwater image enhancement system, characterized in that: include: A training set construction module is used to randomly extract degraded images and their unpaired enhanced images from at least two underwater image datasets to construct a training set; The semantic supervision module includes a large language model interface for analyzing images to generate scene description text and refine positive and negative sample descriptions; The adversarial training engine is configured with a generative adversarial network consisting of a generator and a multi-scale discriminator, where the generator include: A self-calibrated feature enhancement module that performs feature splitting-gating fusion operations; Transformer global modeling module, which establishes cross-region dependencies through block self-attention; Multimodal contrastive learning unit for: Calculate the feature block similarity at the random sampling position of the generator's multi-layer features; Constrained enhanced images are semantically aligned with positive sample descriptions; Parameter optimizer, which updates network parameters based on the training set and loss function; The enhancement execution module inputs the underwater degraded image into the optimized generator and outputs the enhanced underwater image.

Images