Underwater image enhancement method and system based on vision-text fusion

Abstract

The invention belongs to the technical field of underwater image enhancement, and relates to an underwater image enhancement method and system based on vision-text fusion, and the method comprises the steps: constructing an enhancement network based on vision-text fusion; the enhancement network comprises a generator used for processing an image to generate an enhanced image and a discriminator used for judging whether the image is a real image or the generator generates the image, and the generator comprises a high-speed image generation diffusion model and a text encoder based on a contrast language-image pre-training model; an encoder, a U-net module and a decoder in the high-speed image generation diffusion model are connected in sequence, and a text encoder generates text embedding used for adjusting the U-net module; and inputting the underwater image to be enhanced into the enhancement network to obtain an underwater enhanced image. According to the method, vision-text fusion is realized by utilizing the reasoning ability and powerful priori knowledge of the high-speed image generation diffusion model, so that efficient processing and zero sample generalization are achieved, the stability is high, and the visual quality is ensured.

Description

Technical Field

[0001] The present application belongs to the field of underwater image enhancement technology, and specifically relates to an underwater image enhancement method and system based on vision-text fusion. Background Art

[0002] Underwater vision systems play an irreplaceable role in underwater robotics, ocean observation, resource exploration, ecological monitoring, and other fields. However, due to the selective absorption and scattering of light by water, underwater images commonly suffer from degradation phenomena such as color distortion (predominantly blue-green bands), contrast loss, blurred details, and increased noise, severely limiting the accuracy of subsequent visual tasks. To improve underwater image quality, the industry has proposed a large number of underwater image enhancement methods. Existing underwater image enhancement methods can be roughly divided into three categories: methods based on physical models, methods based on non-physical models, and methods based on deep learning.

[0003] Physics-based methods degrade underwater images by addressing the problem of light propagation in water. These methods study the propagation characteristics of light in water and construct mathematical models to describe the scattering, absorption, and attenuation behavior of the water medium. These methods use parameters of the underwater scene to estimate the light propagation path and attenuation characteristics, then perform inverse calculations to restore the true color and detail of the degraded image. Under specific conditions (laboratory conditions or scenes with uniform water quality), these methods can effectively restore the true color and structure of degraded images. However, these methods often rely on simplified assumptions about the underwater scene, and their parameters are difficult to accurately estimate in practice. Consequently, the enhanced results are prone to color shifting and detail loss, making them less adaptable to diverse and complex scenes and limiting their generalization capabilities.

[0004] Methods based on non-physical models ignore the physical processes of underwater imaging and transform the enhancement task into a general image processing problem. These methods do not focus on the physical degradation mechanisms of underwater images, but instead leverage existing image processing techniques to improve visual quality by optimizing pixel intensity values. These methods primarily enhance underwater images by adjusting color, contrast, and texture details, without relying on complex physical modeling, resulting in more natural and pleasing visual results. However, because they do not consider the underlying mechanisms of underwater imaging, their adaptability to different scenarios is often limited, and they can easily lead to over-enhancement (such as local oversaturation and halos) or under-enhancement (such as greenish casts and blurred details).

[0005] Deep learning-based methods leverage the powerful representational capabilities of neural networks to learn feature distributions from large-scale underwater image datasets, enabling accurate reconstruction and quality improvement of degraded images. These methods formulate the enhancement task as an end-to-end nonlinear mapping, modeling the complex degradation process through network training to generate enhanced images with superior visual quality. However, these methods are limited by the scarcity of paired data consisting of real underwater images and their corresponding real non-underwater images (reference images). Furthermore, their reliance on manually selected reference images further limits optimal performance. The lack of high-quality paired datasets hinders effective model training, which in turn limits their generalization ability to diverse underwater scenarios. Summary of the Invention

[0006] In response to the above-mentioned problems in the prior art, such as over-enhancement or under-enhancement of underwater images and scarcity of paired data, this application provides an underwater image enhancement method and system based on visual-text fusion. The method uses the differentiable underwater color difference between natural images and underwater images to reduce color deviation and improve visual quality; uses the reasoning ability and powerful prior knowledge of the high-speed image generation diffusion model to achieve visual-text fusion, thereby achieving efficient processing and zero-sample generalization, strong stability, and ensuring visual quality; overcomes the defects caused by the scarcity of paired data through an unpaired training strategy; and can achieve effective underwater image enhancement by only using text prompts and any natural image under water conditions as a reference, overcoming the defects caused by relying on manually selected reference images.

[0007] In order to achieve the above objectives, in a first aspect, the present application provides an underwater image enhancement method based on visual-text fusion, the steps of which are as follows: Network construction steps: constructing an enhancement network based on visual-text fusion; the enhancement network includes a generator and a discriminator, the generator is used to process the image to generate an enhanced image, and the discriminator is used to determine whether the image is a real image or an image generated by the generator; the generator includes a high-speed image generation diffusion model and a text encoder based on a contrastive language-image pre-trained model, the encoder, U-net module, and decoder in the high-speed image generation diffusion model are connected in sequence, and the text encoder generates a text embedding for adjustment by the U-net module; Image enhancement step: inputting the underwater image to be enhanced into the enhancement network to obtain the underwater enhanced image.

[0008] In some embodiments, the generator and the discriminator are each provided with two; the method for constructing an enhanced network based on visual-text fusion is: Image acquisition steps: Acquire any underwater image and any natural image without water ; Image processing steps: underwater images Input the first generator , prompt with the first text Enter the first generator as a condition Generate enhanced images ; natural images Enter the second generator , with the second text prompt Enter the second generator as a condition Generate virtual underwater images ; The first text prompt Indicates that this is a waterless image. The second text prompt Indicates that this is an underwater image; Image reconstruction step: enhance the image Enter the second generator , with the second text prompt Enter the second generator as a condition Get the reconstructed underwater image ; The virtual underwater image Input the first generator , prompt with the first text Enter the first generator as a condition Get reconstructed natural image ; Wavelet transform step: Enhance the image by discrete wavelet transform and virtual underwater images Decomposed into low-frequency components that retain image color distribution and overall structural information and high-frequency components that capture image details and edge textures; Constraint step: Constrain the low-frequency component by the color loss function , constraining the high-frequency components through the detail loss function , the adversarial loss function is used to distinguish the real image from the generated image, the cycle consistency function is used to constrain the consistency between the image and the reconstructed image, and the identity loss function is used to constrain the matching degree between the generator input and output until the set constraints are met.

[0009] In some embodiments, the total loss function of the generator is expressed as:

[0010] Where, is the total loss function of the generator, To balance the weight coefficient of the color loss function, is the color loss function, To balance the weight coefficient of the detail loss function, is the detail loss function, To balance the weight coefficient of the adversarial loss function, To counter the loss function, is the weight coefficient of the balanced cycle consistency loss function, is the cycle consistency loss function, is the weight coefficient to balance the identity loss function, is the identity loss function; The color loss function is expressed as:

[0011] in:

[0012]

[0013] Where, is a maximum function that selects the larger value from two inputs, is the sum of the average similarities of the soft-assigned color histograms between the red-green channel, red-blue channel, and green-blue channel of the natural image, is the average similarity of the soft-assigned color histogram between the red and green channels of natural images, is the average similarity of the soft-assigned color histogram between the red and blue channels of natural images, is the average similarity of the soft-assigned color histogram between the green and blue channels of natural images; is the sum of the average similarities of the soft-assigned color histograms between the red-green channel, red-blue channel, and green-blue channel of the underwater image, is the average similarity of the soft-assigned color histogram between the red and green channels of the underwater image, is the average similarity of the soft-assigned color histogram between the red and blue channels of the underwater image, is the average similarity of the soft-assigned color histogram between the green and blue channels of the underwater image; The detail loss function is expressed as:

[0014] in:

[0015] Where, 、 、 are weight coefficients, which are used to balance the contributions of enhanced image detail loss, virtual underwater image loss, and total variation regularization; is the predefined target value for the enhanced image, is a predefined target value of a virtual underwater image; is the total number of pixels in the high-frequency component; is the total variation regularization term; The adversarial loss function is expressed as:

[0016] Where, is the first discriminator, is the second discriminator; The cycle consistency loss function is expressed as:

[0017] in:

[0018] Where, represents the reconstruction loss function, is the processed image, is the target image, To balance the difference loss The weight coefficient of To balance learning perception of image patch similarity loss The weight coefficient of To balance the structural similarity index loss The weight coefficient of The identity loss function is expressed as:

[0019] The total loss function of the discriminator is expressed as: .

[0020] In some embodiments, a skip connection is established between the encoder and the decoder via a zero convolutional layer.

[0021] In some embodiments, the generator also includes a low-rank adaptation module, which is respectively connected to the encoder, U-net module, and decoder, so as to keep all the original weights of the high-speed image generation diffusion model frozen during the fine-tuning process, and only update the low-rank adaptation module parameters, the initial convolution layer of the U-net module, and the jump connection convolution layer in the encoder and decoder.

[0022] In some embodiments, the discriminator includes a contrastive language-image pre-trained model, which serves as a frozen backbone network to extract high-dimensional feature representations from each image input to the discriminator.

[0023] In some embodiments, the discriminator includes a contrastive language-image pre-trained model, which serves as a frozen backbone network to extract high-dimensional feature representations from each image input to the discriminator.

[0024] In some embodiments, the discriminator further comprises: A convolutional downsampling unit, comprising two consecutive convolutional downsampling modules, wherein the input of the first convolutional downsampling module is connected to the output of the contrastive language-image pre-trained model; A feedforward module, whose input is connected to the output of the second convolution downsampling module, and is used to perform linear mapping and cross-channel fusion on the features output by the second convolution downsampling module; The output layer, whose input is connected to the output of the feedforward module, is used to map the features output by the feedforward module to the discriminator scores.

[0025] In some embodiments, the convolution downsampling module includes a first convolution layer, a first activation function layer, a fuzzy pooling layer, and a second convolution layer connected in sequence, wherein the first convolution layer is used for channel projection, the first activation function layer is used to introduce nonlinearity and anti-gradient disappearance, the fuzzy pooling layer is used for anti-aliasing downsampling, and the second convolution layer is used to reduce spatial resolution; The feedforward module includes a fully connected layer and a second activation function layer connected in sequence, the fully connected layer is used to perform linear mapping, and the second activation function layer is used to introduce nonlinearity and resist gradient disappearance.

[0026] In a second aspect of the present application, a system for underwater image enhancement based on visual-text fusion is provided, which is used to implement the underwater image enhancement method based on visual-text fusion described in the first aspect of the present application, comprising: Model building module, used to build an enhanced network based on visual-text fusion; The image enhancement module inputs the underwater image to be enhanced into the enhancement network to obtain an underwater enhanced image.

[0027] In some embodiments, the model building module includes: Image acquisition module, which acquires any underwater image and any natural image under water conditions; The image processing module inputs the underwater image into a first generator and generates an enhanced image by inputting the first text prompt as a condition; inputs the natural image into a second generator and generates a virtual underwater image by inputting the second text prompt as a condition; An image reconstruction module inputs the enhanced image into a second generator, and inputs the second generator with the second text prompt as a condition to obtain a reconstructed underwater image; and inputs the virtual underwater image into the first generator, and inputs the first text prompt as a condition to obtain a reconstructed natural image; Wavelet transform module, which decomposes the enhanced image and virtual underwater image into low-frequency components that retain the image color distribution and overall structure information and high-frequency components that capture image details and edge textures through discrete wavelet transform; The constraint module constrains the low-frequency components through the color loss function, constrains the high-frequency components through the detail loss function, distinguishes the real image from the generated image through the adversarial loss function, constrains the consistency of the image and the reconstructed image through the cycle consistency function, and constrains the matching degree of the generator input and output through the identity loss function until the set constraints are met.

[0028] Compared with the prior art, the advantages and positive effects of this application are: (1) The underwater image enhancement method and system based on visual-text fusion provided in this application, by constructing an enhancement network based on visual-text fusion, utilizes the reasoning ability of the high-speed image generation diffusion model and powerful prior knowledge to realize visual-text fusion, thereby achieving efficient processing and zero-sample generalization, strong stability, ensuring visual quality, and taking differentiable underwater color differences and detail intensity regularization as clues to guide low-frequency color restoration and high-frequency detail optimization, effectively generating enhanced images with both natural color appearance and fine texture reconstruction.

[0029] (2) The underwater image enhancement method and system based on visual-text fusion provided in this application, and the enhancement network constructed, on the one hand, utilize the differentiable underwater color difference between natural images and underwater images to reduce color deviation and improve visual quality; on the other hand, it overcomes the defects caused by the scarcity of paired data through the unpaired training strategy.

[0030] (3) The underwater image enhancement method and system based on visual-text fusion provided in this application can achieve effective underwater image enhancement by only using text prompts and any natural image under water-free conditions as a reference, overcoming the defects caused by relying on manual selection of reference images. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a flowchart of the underwater image enhancement method based on vision-text fusion described in an embodiment of the present application; Figure 2 This is a structural block diagram of the enhanced network described in an embodiment of the present application; Figure 3 This is a structural block diagram of the generator described in an embodiment of the present application; Figure 4 A flowchart of a method for constructing an enhanced network based on visual-text fusion as described in an embodiment of the present application; Figure 5 A flowchart of a method for constructing an enhanced network based on visual-text fusion as described in an embodiment of the present application; Figure 6 This is a structural block diagram of the discriminator described in an embodiment of the present application; Figure 7 This is a structural block diagram of the convolution downsampling module described in an embodiment of the present application; Figure 8 This is a structural block diagram of the feedforward module described in an embodiment of the present application; Figure 9 The natural image described in the embodiment of this application; Figure 10 A hard-assigned color histogram of the natural image described in the embodiment of the present application; Figure 11 The underwater image described in the embodiment of the present application; Figure 12 A hard-assigned color histogram of the underwater image described in the embodiment of the present application; Figure 13 A soft-assigned color histogram of the natural image described in the embodiment of the present application; Figure 14 A soft-assigned color histogram of the underwater image described in an embodiment of the present application; Figure 15 This is a structural block diagram of the underwater image enhancement system based on vision-text fusion described in an embodiment of the present application; Figure 16 Schematic diagram of underwater image enhancement results using the underwater image enhancement system based on vision-text fusion described in this application and existing image enhancement methods; Figure 17 This is a schematic diagram of the results of underwater image detail enhancement using the underwater image enhancement system based on vision-text fusion described in this application and existing image enhancement methods.

[0032] In the figure, 1. Generator, 101. Text encoder, 102. Encoder, 103. U-net module, 104. Decoder, 105. Zero convolution layer, 106. Low-rank adaptation module, 107. Input image, 108. Generated image, 2. Discriminator, 21. Contrastive language-image pre-training model, 22. Convolution downsampling unit, 221. First convolution downsampling module, 222. Second convolution downsampling module, 2201. First convolution layer, 2202. First activation function layer, 2203. Fuzzy pooling layer, 2204. Second convolution layer, 23. Feedforward module, 231. Fully connected layer, 232. Second activation function layer, 24. Output layer, 3. Model building module, 301. Image acquisition module, 302. Image processing module, 303. Image reconstruction module, 304. Wavelet transform module, 305. Constraint module, 4. Image enhancement module. DETAILED DESCRIPTION

[0033] The present application is described in detail below by way of exemplary embodiments, but it should be understood that elements, structures, and features in one embodiment may also be beneficially combined in other embodiments without further description.

[0034] In response to the problems of over-enhancement or under-enhancement, and scarcity of paired data in existing underwater image enhancement methods, the present application provides an underwater image enhancement method and system based on visual-text fusion. It uses the differentiable underwater color difference between natural images and underwater images to reduce color deviation and improve visual quality. It uses the reasoning ability and powerful prior knowledge of the high-speed image generation diffusion model to achieve visual-text fusion, thereby achieving efficient processing and zero-sample generalization. It overcomes the defects caused by the scarcity of paired data through an unpaired training strategy. It can achieve effective underwater image enhancement only by using text prompts and any natural image under water conditions as a reference, overcoming the defects caused by relying on manually selected reference images. The following is a detailed description of the underwater image enhancement method and system based on visual-text fusion provided by the present application in conjunction with the accompanying drawings.

[0035] See also Figures 1 to 3 The first embodiment of the present application provides an underwater image enhancement method based on visual-text fusion, the steps of which are: S1. Network construction step: construct an enhancement network based on visual-text fusion; the enhancement network includes a generator 1 and a discriminator 2, the generator 1 is used to process the image to generate an enhanced image, and the discriminator 2 is used to determine whether the image is a real image or an image generated by the generator; the generator 1 includes a high-speed image generation diffusion model and a text encoder 101 based on a contrastive language-image pre-training model, the encoder 102, U-net module 103, and decoder 104 in the high-speed image generation diffusion model are connected in sequence, and the text encoder 101 generates text embedding for adjustment by the U-net module 103.

[0036] See also Figure 4 、 Figure 5 , the method for constructing an enhanced network based on visual-text fusion is: S11, image acquisition step: acquiring any underwater image and any natural image without water ; S12, image processing step: underwater image Input the first generator , prompt with the first text Enter the first generator as a condition Generate enhanced images ; natural images Enter the second generator , with the second text prompt Enter the second generator as a condition Generate virtual underwater images ; The first text prompt Indicates that this is a waterless image. The second text prompt Indicates that this is an underwater image; S13, image reconstruction step: enhance the image Enter the second generator , with the second text prompt Enter the second generator as a condition Get the reconstructed underwater image ; The virtual underwater image Input the first generator , prompt with the first text Enter the first generator as a condition Get reconstructed natural image ; S14, wavelet transform step: enhance the image by discrete wavelet transform Decompose into low-frequency components that retain image color distribution and overall structural information and high-frequency components that capture image details and edge textures ; The virtual underwater image is transformed by discrete wavelet transform Decompose into low-frequency components that retain image color distribution and overall structural information and high-frequency components that capture image details and edge textures ; S15, constraint step: constraining the low-frequency component by a color loss function , constraining the high-frequency components through the detail loss function , the adversarial loss function is used to distinguish the real image from the generated image, the cycle consistency function is used to constrain the consistency between the image and the reconstructed image, and the identity loss function is used to constrain the matching degree between the generator input and output until the set constraints are met.

[0037] It should be noted that the order of the image reconstruction step S13 and the wavelet transform step S14 can be changed, that is, S13, wavelet transform step: the enhanced image is transformed by discrete wavelet transform Decompose into low-frequency components that retain image color distribution and overall structural information and high-frequency components that capture image details and edge textures ; The virtual underwater image is transformed by discrete wavelet transform Decompose into low-frequency components that retain image color distribution and overall structural information and high-frequency components that capture image details and edge textures ; S14, image reconstruction step: enhance the image Enter the second generator , with the second text prompt Enter the second generator as a condition Get the reconstructed underwater image ; The virtual underwater image Input the first generator , prompt with the first text Enter the first generator as a condition Get reconstructed natural image The image reconstruction step S13 and the wavelet transform step S14 may also be performed simultaneously.

[0038] It should be noted that after the enhancement network is trained, only the generator is needed to complete the underwater image enhancement task. Specifically, given an underwater image and a text prompt, the generator will output an enhanced underwater image (i.e., an underwater enhanced image).

[0039] In one embodiment of the present application, the total loss function of the generator is expressed as:

[0040] Where, is the total loss function of the generator, To balance the weight coefficient of the color loss function, is the color loss function, To balance the weight coefficient of the detail loss function, is the detail loss function, To balance the weight coefficient of the adversarial loss function, To counter the loss function, is the weight coefficient of the balanced cycle consistency loss function, is the cycle consistency loss function, is the weight coefficient to balance the identity loss function, is the identity loss function.

[0041] Specifically, the color loss function is expressed as:

[0042] in:

[0043]

[0044] Where, is a maximum function that selects the larger value from two inputs, is the sum of the average similarities of the soft-assigned color histograms between the red-green channel, red-blue channel, and green-blue channel of the natural image, is the average similarity of the soft-assigned color histogram between the red and green channels of natural images, is the average similarity of the soft-assigned color histogram between the red and blue channels of natural images, is the average similarity of the soft-assigned color histogram between the green and blue channels of natural images; is the sum of the average similarities of the soft-assigned color histograms between the red-green channel, red-blue channel, and green-blue channel of the underwater image, is the average similarity of the soft-assigned color histogram between the red and green channels of the underwater image, is the average similarity of the soft-assigned color histogram between the red and blue channels of the underwater image, The average similarity of the soft-assigned color histogram between the green and blue channels of the underwater image.

[0045] It should be noted that in the RGB color space, an image consists of three channels: red, green, and blue. Figure 9 Natural images typically exhibit a certain degree of hard distribution between any two color channels in their color histograms (see Figure 10 ) similarity, that is, dividing the intensity range of each channel into a fixed space and counting the number of pixels in each interval, the hard-assigned histogram similarity between any two channels of natural images is high. However, see Figure 11 Hard-assigned color histogram of the underwater image shown (see Figure 12 ), the hard-assigned histogram similarity between any two channels of underwater images is low, which shows that the phenomenon of high hard-assigned color histogram similarity between any two channels does not apply to underwater images. To quantitatively analyze this phenomenon, we first construct a hard-assigned color histogram for each channel and then calculate the color channel as follows and color channels Hard-assigned color histogram similarity between :

[0046] Where, For color channels The hard-assigned color histogram of For color channels The hard-assigned color histogram of is the average value over all histogram bins, For color channels The hard-assigned color histogram of For color channels The hard-assigned color histogram of is the average value over all histogram bins, Indicates the sum of all histogram bins.

[0047] The above formula is used to calculate the average similarity of the hard-assigned color histograms between the red-green channel, red-blue channel, and green-blue channel in natural images and underwater images. It is found that the average similarity of the hard-assigned color square maps of these channels in natural images is significantly higher than that in underwater images. This phenomenon is called underwater color difference. Based on this result, the underwater color difference index is constructed. , used to quantitatively evaluate images Underwater color differences are defined as follows:

[0048] Where, is the average similarity of the hard-assigned color histogram between the red and green channels, is the average similarity of the hard-assigned color histogram between the red and blue channels, Hard-assigned color histogram average similarity between green and blue channels.

[0049] It should also be noted that although the underwater color difference metric can effectively quantify the color inconsistency between underwater images and natural images, it is not suitable as a loss function in network training. The main reason is that it involves non-differentiable operations such as discrete binning and counting, which hinder the gradient backpropagation required for the optimization process. To enable its application to the enhanced network in this application, this application constructs a differentiable underwater color difference metric, using a soft-assigned color histogram and introducing a Gaussian kernel function to ensure that each pixel value makes a continuous contribution to all histogram bins.

[0050] Specifically, Indicates the pixel values, Indicates the The center of the soft-assigned color histogram interval, then the pixel For the first The soft assigned weights of the soft assigned color histogram bins for:

[0051] Where, is the total number of pixels, is the total number of intervals, The smoothness assigned to the control.

[0052] No. Soft-assigned color histogram of intervals It is normalized by summing the soft assigned weights of all pixels and dividing by the total weight of all bins.

[0053]

[0054] Where, are all intervals in the normalization process, For the The soft-assigned weights of the soft-assigned color histogram bins, A small constant set for numerical stability. Full soft-assigned color histogram Expressed as:

[0055] According to the complete soft distribution color histogram Calculate the formula to get the natural image (see Figure 9 )’s soft assigned color histogram is as follows Figure 13 As shown, the underwater image is obtained (see Figure 11 )’s soft assigned color histogram is as follows Figure 14 As shown. Figure 13 、 Figure 14 It can be seen that there is a certain degree of soft-assigned color histogram similarity between any two color channels of natural images, but this is not the case with underwater images. In order to further quantitatively analyze this linearity, based on the hard-assigned color histogram similarity calculation formula and the complete soft-assigned color histogram Calculation formula, calculate the color channel according to the following calculation formula and color channels Soft-assigned color histogram similarity between :

[0056] Where, For color channels The soft-assigned color histogram of For color channels The average value of the soft assigned color histogram over all histogram bins, For color channels The soft-assigned color histogram of For color channels The average of the soft assigned color histogram of over all histogram bins.

[0057] Specifically, according to the soft-assigned color histogram similarity The calculation formula is to calculate the average similarity of the soft-assigned color histogram between the red-green channel, red-blue channel and green-blue channel of 10 natural image data sets, which are respectively denoted as 、 、 , and the average similarity of the soft-assigned color histograms between the red-green channel, red-blue channel and green-blue channel of the 10 underwater image datasets are respectively denoted as 、 、 , the calculation results are shown in Table 1.

[0058] Table 1

[0059] As shown in Table 1, the average similarity of the soft-assigned color histograms of natural images is 、 、 , and always maintain a high level. In comparison, the average similarities of the soft-assigned color histograms of underwater images are 、 、 , significantly lower. These results show that both soft-assignment and hard-assignment color histogram similarity can effectively capture the difference between natural images and underwater images. Unlike hard-assignment color histogram similarity, which relies on discrete binning and non-differentiable computational operations, soft-assignment color histogram similarity distributes the contribution of each pixel to all histogram bins in a continuous and differentiable manner through a Gaussian function. Due to this differentiable property, the observed color difference between natural images and underwater images quantified by soft-assignment color histogram similarity is called differentiable underwater color difference. Based on this result, a differentiable underwater color difference index is constructed. , used to quantitatively evaluate images The differentiable underwater color difference is defined as follows:

[0060] Where, is the average similarity of the soft-assigned color histogram between the red and green channels, is the average similarity of the soft-assigned color histogram between the red and blue channels, is the average similarity of the soft-assigned color histogram between the green and blue channels.

[0061] When enhancing underwater images to resemble natural images, the soft-assigned color histogram similarity needs to be increased to match the natural image's similarity level. Conversely, when degrading natural images to simulate the underwater image's appearance, the soft-assigned color histogram similarity needs to be reduced to match the underwater image's similarity level. This application constructs the color loss function based on the average similarity of the soft-assigned color histograms of natural and underwater images. This color loss function, designed in this application, can guide the optimization of low-frequency color restoration and effectively generate enhanced images with the appearance of natural images.

[0062] Specifically, the detail loss function is expressed as:

[0063] in:

[0064] Where, 、 、 are weight coefficients, which are used to balance the contributions of enhanced image detail loss, virtual underwater image loss, and total variation regularization; is the predefined target value for the enhanced image, is a predefined target value of a virtual underwater image; is the total number of pixels in the high-frequency component; is the total variation regularization term.

[0065] It should be noted that, in view of the fact that high-frequency components capture the fine details and edge textures of the image, the detail loss function described in this application designs detail intensity regularization to constrain the high-frequency components. For enhanced images, detail intensity regularization promotes the improvement of detail intensity to a predefined target value to achieve moderate detail enhancement. In order to suppress excessive noise, total variation regularization is introduced to ensure that the enhanced image presents natural and delicate details without introducing artificial textures or noise artifacts. For virtual underwater images, detail intensity regularization can cause the detail intensity to decrease to a predefined target value. Since the goal in this case is to suppress details, total variation regularization is not applied. The detail loss function designed by this application can guide the optimization of high-frequency details and effectively generate enhanced images with fine texture reconstruction.

[0066] Specifically, the adversarial loss function is expressed as:

[0067] Where, is the first discriminator, is the second discriminator.

[0068] This application uses an adversarial loss function to ensure that the discriminator cannot distinguish between generated images and real images. This application constructs two discriminators: the first discriminator is used to distinguish between enhanced images and natural images, and the second discriminator is used to distinguish between virtual underwater images and underwater images. The adversarial loss function designed by this application can make the first discriminator unable to distinguish between enhanced images and natural images, and the second discriminator unable to distinguish between virtual underwater images and underwater images.

[0069] Specifically, the cycle consistency loss function is expressed as:

[0070] in:

[0071] Where, represents the reconstruction loss function, is the processed image, is the target image, To balance the difference loss The weight coefficient of To balance learning perception of image patch similarity loss The weight coefficient of To balance the structural similarity index loss The weight coefficient of .

[0072] This application uses the cycle consistency loss function, the difference loss Emphasis on pixel-level point-by-point accuracy helps preserve overall color fidelity and fine details. Learning perceptual image patch similarity loss For high-level semantics and perceptual quality, it effectively solves the problem of similar pixel values ​​but different visual perceptions. Structural similarity index loss Constraining local structure and texture consistency ensures that the generated image maintains a coherent structure in the local area. These three losses together form a balanced constraint on image quality.

[0073] This application uses a cycle consistency loss function to ensure that the image can be mapped back to its original appearance after continuous transformation. The cycle consistency loss function designed by this application enforces consistency between the forward and reverse mappings, thereby improving the stability of the reinforcement network training process and the reliability of the generated results.

[0074] Specifically, the identity loss function is expressed as:

[0075] This application introduces an identity loss function to ensure that the color and content of images that already belong to the target domain are preserved. Specifically, when the input image is from the target domain, this function forces the generator to output a result that closely matches the input, helping to avoid unnecessary modifications and color shifts.

[0076] The total loss function of the discriminator is expressed as: .

[0077] In one embodiment of the present application, see Figure 3 , a skip connection is established between the encoder 102 and the decoder 104 through a zero convolution layer 105. By establishing a skip connection through the zero convolution layer, these skip paths enable the key spatial features of the input image to be retained in the output, thereby effectively retaining the detail information of the image.

[0078] In one embodiment of the present application, see Figure 3 The generator 1 further includes a low-rank adaptation module 106, which is connected to the encoder 102, U-net module 103, and decoder 104, respectively, so as to keep all the original weights of the high-speed image generation diffusion model frozen during the fine-tuning process, and only update the parameters of the low-rank adaptation module 106, the initial convolutional layer of the U-net module 103, and the skip-connected convolutional layers in the encoder 102 and decoder 104. The low-rank adaptation module enables efficient adaptation with minimal overhead. This selective training strategy effectively retains the prior knowledge embedded in the enhanced network while requiring only a small number of trainable parameters.

[0079] In one embodiment of the present application, see Figure 6 The discriminator 2 includes a contrastive language-image pre-trained model 21, which serves as a frozen backbone network to extract high-dimensional feature representations from each image input to the discriminator 2. Using the contrastive language-image pre-trained model as the backbone network allows high-quality, semantically rich, high-dimensional feature representations to be obtained without retraining, reducing overall training costs.

[0080] In some embodiments, see Figure 6 , the discriminator 2 further includes: The convolution downsampling unit 22 includes two consecutive convolution downsampling modules, where the input of the first convolution downsampling module 221 is connected to the output of the contrastive language-image pre-trained model 21; A feedforward module 23, whose input is connected to the output of the second convolution downsampling module 222, is used to perform linear mapping and cross-channel fusion on the features output by the second convolution downsampling module 222; The output layer 24 has an input connected to the output of the feedforward module 23 and is used to map the features output by the feedforward module 23 into discriminator scores.

[0081] The convolutional downsampling unit and the backbone network form a "coarse-fine" structure, which retains global semantics while introducing local texture information, enhancing the discriminator's sensitivity to subtle forgery traces.

[0082] In one embodiment of the present application, see Figure 7 The convolutional downsampling module includes a first convolutional layer 2201, a first activation function layer 2202, a fuzzy pooling layer 2203, and a second convolutional layer 2204, which are connected in sequence. The first convolutional layer 2201 is used for channel projection, the first activation function layer 2202 is used to introduce nonlinearity and combat vanishing gradients, the fuzzy pooling layer 2203 is used for anti-aliasing downsampling, and the second convolutional layer 2204 is used to reduce spatial resolution. Using two convolutional downsampling modules with the same structure can simplify the training process and reduce the number of parameters.

[0083] In one embodiment of the present application, see Figure 8 The feedforward module 23 includes a fully connected layer 231 and a second activation function layer 232 connected in sequence. The fully connected layer 231 is used to perform linear mapping, and the second activation function layer 232 is used to introduce nonlinearity (enhance the network's ability to fit complex decision-making boundaries) and resist gradient disappearance.

[0084] S2, image enhancement step: inputting the underwater image to be enhanced into the enhancement network to obtain an underwater enhanced image.

[0085] See also Figure 15 The second embodiment of the present application provides an underwater image enhancement system based on visual-text fusion, which is used to implement the underwater image enhancement method based on visual-text fusion described in the first aspect of the present application, including: Model building module 3, used to build an enhanced network based on visual-text fusion; The image enhancement module 4 inputs the underwater image to be enhanced into the enhancement network to obtain an underwater enhanced image.

[0086] In some embodiments, see Figure 15 , the model building module 3 includes: An image acquisition module 301 is used to acquire any underwater image and any natural image under an anhydrous condition; The image processing module 302 inputs the underwater image into a first generator and conditions the first text prompt to generate an enhanced image; inputs the natural image into a second generator and conditions the second text prompt to generate a virtual underwater image; The image reconstruction module 303 inputs the enhanced image into the second generator, and inputs the second text prompt as a condition to obtain a reconstructed underwater image; inputs the virtual underwater image into the first generator, and inputs the first text prompt as a condition to obtain a reconstructed natural image; The wavelet transform module 304 decomposes the enhanced image and the virtual underwater image into a low-frequency component that retains the image color distribution and overall structure information and a high-frequency component that captures the image details and edge texture, respectively, through discrete wavelet transform; The constraint module 305 constrains the low-frequency component through the color loss function, constrains the high-frequency component through the detail loss function, distinguishes the real image from the generated image through the adversarial loss function, constrains the consistency of the image and the reconstructed image through the cycle consistency function, and constrains the matching degree of the generator input and output through the identity loss function until the set constraint conditions are met.

[0087] The following, combined with specific examples, compares the aforementioned underwater image enhancement method and system based on visual-text fusion (hereinafter referred to as the present application) with nine other methods to verify its effectiveness. The other nine methods are GIFM, C3HLM, CRUHL, MSPE, MFCP, CBAF, PUGAN, LANET, and RLUIE. GIFM, C3HLM, and CRUHL are categorized as physical model-based methods; MSPE, MFCP, and CBAF are categorized as non-physical model methods; and PUGAN, LANET, and RLUIE are categorized as deep learning methods.

[0088] Example: The commonly used Underwater Image Enhancement Benchmark (UIEB) dataset is used as a verification dataset. The dataset contains 890 real underwater images collected under various environmental conditions.

[0089] The quality of underwater images was assessed using six metrics: UIQM, CCF, URanker, AG, EI, and BCRA. UIQM combines three sub-metrics: image color, sharpness, and contrast, to comprehensively characterize the perceptual quality of an image. CCF is a weighted sum of color, contrast, and haze. URanker utilizes an efficient convolutional-attention Transformer architecture to provide reliable visual quality ranking for underwater images enhanced by various methods. Higher scores for UIQM, CCF, and URanker indicate better visual quality of the enhanced image. AG reflects image clarity, while EI quantifies the strength of image edge structure. Higher scores for these two metrics indicate better visual quality. BCRA assesses the improvement in detail before and after image enhancement. It consists of two complementary components: one measures the improvement in edge visibility, and the other quantifies the increase in the gradient amplitude of edge pixels. Higher BCRA scores indicate better image enhancement in terms of edge fidelity and detail preservation.

[0090] The comprehensive enhancement performance of various methods is evaluated and compared on the UIEB dataset. Figure 16 As shown, Figure 16 In the figure, (a) corresponds to the original image, (b) corresponds to GIFM, (c) corresponds to C3HLM, (d) corresponds to CRUHL, (e) corresponds to MSPE, (f) corresponds to MFCP, (g) corresponds to CBAF, (h) corresponds to PUGAN, (i) corresponds to LANET, (j) corresponds to RLUIE, and (k) corresponds to this application. For image Img1, GIFM, MSPE, PUGAN, LANET, and RLUIE are all unable to effectively eliminate color deviation. While C3HLM, CRUHL, MFCP, and CBAF mitigate color deviation to some extent, significant residual color cast remains in the background. Furthermore, these methods have limited effectiveness in improving image contrast, and MFCP introduces significant loss of structural details.

[0091] For image Img2, GIFM, CRUHL, MSPE, PUGAN, LANET, and RLUIE fail to sufficiently improve contrast or eliminate color bias, with CRUHL even introducing an additional red tint. While C3HLM, MFCP, and CBAF correct color bias to some extent, they fail to effectively enhance contrast and fail to fully recover detailed structural information.

[0092] For image Img3, although GIFM, C3HLM, CRUHL, MSPE, MFCP, CBAF, PUGAN, LANET and RLUIE correct the color deviation to a certain extent, they all fail to completely remove the blue deviation in the background.

[0093] In contrast, the present application can successfully eliminate color deviation, enhance image contrast, and enrich image details.

[0094] Figure 17 A comparative analysis of the detail enhancement results of this application and different methods is given. Figure 17 In the figure, (a) corresponds to the original image, (b) corresponds to GIFM, (c) corresponds to C3HLM, (d) corresponds to CRUHL, (e) corresponds to MSPE, (f) corresponds to MFCP, (g) corresponds to CBAF, (h) corresponds to PUGAN, (i) corresponds to LANET, (j) corresponds to RLUIE, and (k) corresponds to this application. Within the entire image area, this application demonstrates excellent ability in addressing common underwater degradation issues, including color deviation, low contrast, and blurred details. In addition, in the local area highlighted in the red box, this application produces clearer structures and richer textures compared to other methods.

[0095] Quantitative evaluation results were calculated on the UIEB dataset, as shown in Table 2, where bold indicates the best performance and blue indicates the second-best performance. Our application achieved the best performance across all six evaluation metrics, including UIQM, CCF, AG, EI, BCRA, and URanker. This demonstrates the effectiveness of our application in enhancing underwater imagery captured in a wide range of real-world scenarios.

[0096] Table 2

[0097] The above embodiments are used to explain the present application rather than to limit the present application. Any modifications and changes made to the present application within the spirit of the present application and the protection scope of the claims shall fall within the protection scope of the present application.

Claims

1. An underwater image enhancement method based on vision-text fusion, characterized in that: The steps are: Network construction steps: constructing an enhancement network based on visual-text fusion; the enhancement network includes a generator and a discriminator, the generator is used to process the image to generate an enhanced image, and the discriminator is used to determine whether the image is a real image or an image generated by the generator; the generator includes a high-speed image generation diffusion model and a text encoder based on a contrastive language-image pre-trained model, the encoder, U-net module, and decoder in the high-speed image generation diffusion model are connected in sequence, and the text encoder generates a text embedding for adjustment by the U-net module; Image enhancement step: inputting the underwater image to be enhanced into the enhancement network to obtain the underwater enhanced image.

2. The underwater image enhancement method based on vision-text fusion according to claim 1, characterized in that: The generator and the discriminator are both provided with two; the method for constructing an enhanced network based on visual-text fusion is: Image acquisition steps: Acquire any underwater image and any natural image without water ; Image processing steps: underwater images Input the first generator , prompt with the first text Enter the first generator as a condition Generate enhanced images ; natural images Enter the second generator , with the second text prompt Enter the second generator as a condition Generate virtual underwater images ; The first text prompt Indicates that this is a waterless image. The second text prompt Indicates that this is an underwater image; Image reconstruction step: enhance the image Enter the second generator , with the second text prompt Enter the second generator as a condition Get the reconstructed underwater image ; The virtual underwater image Input the first generator , prompt with the first text Enter the first generator as a condition Get reconstructed natural image ; Wavelet transform step: Enhance the image by discrete wavelet transform and virtual underwater images Decomposed into low-frequency components that retain image color distribution and overall structural information and high-frequency components that capture image details and edge textures; Constraint steps: constrain the low-frequency component through the color loss function, constrain the high-frequency component through the detail loss function, distinguish the real image from the generated image through the adversarial loss function, constrain the consistency of the image and the reconstructed image through the cycle consistency function, and constrain the matching degree of the generator input and output through the identity loss function until the set constraints are met.

3. The underwater image enhancement method based on vision-text fusion according to claim 2, characterized in that: The total loss function of the generator is expressed as: Where, is the total loss function of the generator, To balance the weight coefficient of the color loss function, is the color loss function, To balance the weight coefficient of the detail loss function, is the detail loss function, To balance the weight coefficient of the adversarial loss function, To counter the loss function, is the weight coefficient of the balanced cycle consistency loss function, is the cycle consistency loss function, is the weight coefficient to balance the identity loss function, is the identity loss function; The color loss function is expressed as: in: Where, is a maximum function that selects the larger value from two inputs, is the sum of the average similarities of the soft-assigned color histograms between the red-green channel, red-blue channel, and green-blue channel of the natural image, is the average similarity of the soft-assigned color histogram between the red and green channels of natural images, is the average similarity of the soft-assigned color histogram between the red and blue channels of natural images, is the average similarity of the soft-assigned color histogram between the green and blue channels of natural images; is the sum of the average similarities of the soft-assigned color histograms between the red-green channel, red-blue channel, and green-blue channel of the underwater image, is the average similarity of the soft-assigned color histogram between the red and green channels of the underwater image, is the average similarity of the soft-assigned color histogram between the red and blue channels of the underwater image, is the average similarity of the soft-assigned color histogram between the green and blue channels of the underwater image; The detail loss function is expressed as: in: Where, 、 、 are weight coefficients, which are used to balance the contributions of enhanced image detail loss, virtual underwater image loss, and total variation regularization; is the predefined target value for the enhanced image, is a predefined target value of a virtual underwater image; is the total number of pixels in the high-frequency component; is the total variation regularization term; The adversarial loss function is expressed as: Where, is the first discriminator, is the second discriminator; The cycle consistency loss function is expressed as: in: Where, represents the reconstruction loss function, is the processed image, is the target image, To balance the difference loss The weight coefficient of To balance learning perception of image patch similarity loss The weight coefficient of To balance the structural similarity index loss The weight coefficient of The identity loss function is expressed as: The total loss function of the discriminator is expressed as: 。 4. The underwater image enhancement method based on vision-text fusion according to claim 1, characterized in that: A skip connection is established between the encoder and the decoder via a zero convolution layer.

5. The underwater image enhancement method based on vision-text fusion according to claim 4, characterized in that: The generator also includes a low-rank adaptation module, which is respectively connected to the encoder, U-net module, and decoder to keep all the original weights of the high-speed image generation diffusion model frozen during the fine-tuning process, and only update the low-rank adaptation module parameters, the initial convolutional layer of the U-net module, and the jump connection convolutional layer in the encoder and decoder.

6. The underwater image enhancement method based on vision-text fusion according to claim 1, characterized in that: The discriminator includes a contrastive language-image pre-trained model, which serves as a frozen backbone network to extract high-dimensional feature representations from each image input to the discriminator.

7. The underwater image enhancement method based on vision-text fusion according to claim 6, characterized in that: The discriminator further includes: A convolutional downsampling unit, comprising two consecutive convolutional downsampling modules, where the input of the first convolutional downsampling module is connected to the output of the contrastive language-image pre-trained model; A feedforward module, whose input is connected to the output of the second convolution downsampling module, and is used to perform linear mapping and cross-channel fusion on the features output by the second convolution downsampling module; The output layer, whose input is connected to the output of the feedforward module, is used to map the features output by the feedforward module to the discriminator scores.

8. The underwater image enhancement method based on vision-text fusion according to claim 7, characterized in that: The convolution downsampling module includes a first convolution layer, a first activation function layer, a fuzzy pooling layer, and a second convolution layer connected in sequence, wherein the first convolution layer is used for channel projection, the activation function layer is used to introduce nonlinearity and anti-gradient disappearance, the fuzzy pooling layer is used for anti-aliasing downsampling, and the second convolution layer is used to reduce spatial resolution; The feedforward module includes a fully connected layer and a second activation function layer connected in sequence, the fully connected layer is used to perform linear mapping, and the second activation function layer is used to introduce nonlinearity and resist gradient disappearance.

9. An underwater image enhancement system based on vision-text fusion, used to implement the underwater image enhancement method based on vision-text fusion according to any one of claims 1 to 8, characterized in that: include: Model building module, used to build an enhanced network based on visual-text fusion; The image enhancement module inputs the underwater image to be enhanced into the enhancement network to obtain an underwater enhanced image.

10. The underwater image enhancement system based on vision-text fusion according to claim 9, characterized in that: The model building module includes: Image acquisition module, which acquires any underwater image and any natural image under water conditions; The image processing module inputs the underwater image into a first generator and generates an enhanced image by inputting the first text prompt as a condition; inputs the natural image into a second generator and generates a virtual underwater image by inputting the second text prompt as a condition; An image reconstruction module inputs the enhanced image into a second generator, and inputs the second generator with the second text prompt as a condition to obtain a reconstructed underwater image; and inputs the virtual underwater image into the first generator, and inputs the first text prompt as a condition to obtain a reconstructed natural image; Wavelet transform module, which decomposes the enhanced image and virtual underwater image into low-frequency components that retain the image color distribution and overall structure information and high-frequency components that capture image details and edge textures through discrete wavelet transform; The constraint module constrains the low-frequency components through the color loss function, constrains the high-frequency components through the detail loss function, distinguishes the real image from the generated image through the adversarial loss function, constrains the consistency of the image and the reconstructed image through the cycle consistency function, and constrains the matching degree of the generator input and output through the identity loss function until the set constraints are met.

Images