Self-supervised based underwater image enhancement method and device and computer storage medium

Abstract

The application discloses a self-supervision-based underwater image enhancement method and device and a computer storage medium, relates to the technical field of image enhancement, and comprises the following steps: establishing a self-supervision-based underwater image enhancement network model; the model comprises a Fourier convolution image enhancement module for enhancing the underwater image by using Fourier convolution, a convolution network depth estimation module for generating a depth map by using an encoder-decoder structure, and a self-supervision network imaging module for generating a pseudo-label image for self-supervision training by using an image imaging model on the enhanced image and the depth map; the self-supervision-based underwater image enhancement network model is established by using the pseudo-label image for self-supervision training; an underwater image to be enhanced is acquired; and the underwater image to be enhanced is input into the trained self-supervision-based underwater image enhancement network model to obtain an enhanced underwater image. The underwater image is enhanced by using the self-supervision method, and therefore, a complex artificial labeling data set is not needed, and manpower and material resources are saved.

Description

Technical Field

[0001] This invention relates to the field of image enhancement technology, and in particular to a self-supervised underwater image enhancement method, apparatus, and computer storage medium. Background Technology

[0002] In recent years, underwater vision has received increasing attention in fields such as ocean exploration, deep-sea fishing, and marine life monitoring. The quality of underwater images has a significant impact on underwater operations. For example, vision-dependent tasks such as underwater target detection require high-quality underwater images; low-quality images can severely reduce the efficiency and accuracy of these tasks. However, factors such as the refraction and absorption of light in the underwater environment, suspended particles in the water, and color distortion can affect the quality of visual data, leading to noise, distortion, and color aberrations in underwater images. Therefore, underwater image enhancement is of great importance.

[0003] In existing technologies, underwater image enhancement typically employs fully supervised deep learning-based methods. Supervised learning trains an optimal model using existing training samples (i.e., known data and their corresponding outputs). This model is then used to map all new data samples to the corresponding output results, and a simple judgment is made on the output results to achieve classification. This type of method generally suppresses noise in the enhancement results effectively, achieving good image enhancement effects.

[0004] However, fully supervised underwater image enhancement methods often struggle to be implemented because they require labeled data during training, and acquiring labeled data is expensive and difficult. Summary of the Invention

[0005] In view of this, the present invention provides a self-supervised underwater image enhancement method, apparatus and computer storage medium. Underwater image enhancement by self-supervised method can eliminate the need for complex manually labeled datasets, saving manpower and resources, and can achieve the same or even better image enhancement effect as fully supervised method.

[0006] Therefore, the present invention provides the following technical solution:

[0007] On one hand, the present invention provides a self-supervised underwater image enhancement method, the method comprising:

[0008] A self-supervised underwater image enhancement network model is established. This model includes a Fourier convolutional image enhancement module, a convolutional network depth estimation module, and a self-supervised network imaging module. The Fourier convolutional image enhancement module uses Fourier convolution to enhance underwater images. The convolutional network depth estimation module uses an encoder-decoder structure to generate depth maps. The self-supervised network imaging module uses the image imaging model IFM to generate self-supervised trained pseudo-label images from the enhanced images generated by the Fourier convolutional image enhancement module and the depth maps generated by the convolutional network depth estimation module.

[0009] A self-supervised underwater image enhancement network model was established using the pseudo-labeled images through self-supervised training.

[0010] Acquire underwater images to be enhanced;

[0011] The underwater image to be enhanced is input into a trained self-supervised underwater image enhancement network model to obtain the enhanced underwater image.

[0012] Furthermore, the self-supervised underwater image enhancement network model established using the pseudo-labeled images through self-supervised training includes:

[0013] Acquire realistic RGB underwater images;

[0014] The Fourier convolution image enhancement module is used to enhance the real RGB underwater image to obtain the enhanced image of the real RGB underwater image;

[0015] The depth map of the real RGB underwater image is generated using a convolutional network depth estimation module;

[0016] The enhanced image and the depth map are input into the self-supervised network imaging network module, and the IFM is used to generate a pseudo-label image similar to the real RGB underwater image according to the image generation mechanism.

[0017] A loss function is constructed using the similarity between the real RGB underwater image and the pseudo-labeled image. The gradient of the loss function and the parameters of the self-supervised underwater image enhancement network model are then updated to obtain the trained self-supervised underwater image enhancement network model.

[0018] Furthermore, the self-supervised underwater image enhancement network model established using the pseudo-labeled images through self-supervised training includes:

[0019] Acquire realistic RGB underwater images;

[0020] The real RGB underwater image is enhanced using a Fourier convolution image enhancement module to obtain an enhanced image of the real RGB underwater image.

[0021] The depth map of the real RGB underwater image is generated using a convolutional network depth estimation module;

[0022] Obtain the background light prior;

[0023] The enhanced image, the depth map, and the background light prior are input into the self-supervised network imaging network module, and the IFM is used to generate a pseudo-label image similar to the real RGB underwater image according to the image generation mechanism.

[0024] A loss function is constructed using the similarity between the real RGB underwater image and the pseudo-labeled image. The gradient of the loss function and the parameters of the self-supervised underwater image enhancement network model are then updated to obtain the trained self-supervised underwater image enhancement network model.

[0025] Furthermore, the Fourier convolution image enhancement module uses Fourier convolution to enhance underwater images, including:

[0026] Real RGB underwater images are transformed from the time domain to the frequency domain using Fourier transform;

[0027] Determine a global filter K, and multiply the global filter K by the image in the frequency domain space to obtain the frequency domain space value of the enhanced image;

[0028] The image is transformed from the frequency domain to the time domain by inverse Fourier transform, and the output is the enhanced image.

[0029] Furthermore, the convolutional network depth estimation module generates depth maps using an encoder-decoder architecture, including:

[0030] The encoder of the convolutional network depth estimation module is input into a real RGB underwater image. Features are extracted through convolution and downsampling. Then, the decoder upsamples the low-dimensional feature map to the same dimension as the real RGB underwater image and outputs the depth result map.

[0031] Furthermore, the loss function is:

[0032] L=αL MS-SSIM +(1-α)L MSE ;

[0033] Among them, L MSE For mean squared variance loss, L MS-SSIM This represents the multi-scale structural similarity loss; α is the weight value.

[0034] In another aspect, the present invention also provides a self-supervised underwater image enhancement device, the device comprising:

[0035] A network model building unit is used to construct a self-supervised underwater image enhancement network model. This model includes a Fourier convolutional image enhancement module, a convolutional network depth estimation module, and a self-supervised network imaging module. The Fourier convolutional image enhancement module uses Fourier convolution to enhance underwater images. The convolutional network depth estimation module generates depth maps using an encoder-decoder structure. The self-supervised network imaging module uses the image imaging model IFM to generate self-supervised trained pseudo-label images from the enhanced images generated by the Fourier convolutional image enhancement module and the depth maps generated by the convolutional network depth estimation module.

[0036] The network model training unit is used to self-supervisedly train the self-supervised underwater image enhancement network model constructed by the network model building unit using the pseudo-labeled images;

[0037] The raw image acquisition unit is used to acquire underwater images to be enhanced.

[0038] The underwater image enhancement unit inputs the underwater image to be enhanced acquired by the original image acquisition unit into the self-supervised underwater image enhancement network model trained by the network model training unit to obtain the enhanced underwater image.

[0039] Furthermore, the network model training unit includes:

[0040] The image acquisition subunit is used to acquire real RGB underwater images;

[0041] An image enhancement subunit is used to enhance the real RGB underwater image acquired by the image acquisition subunit using the Fourier convolution image enhancement module;

[0042] The depth map acquisition subunit is used to generate a depth map of the real RGB underwater image acquired by the image acquisition subunit using the convolutional network depth estimation module;

[0043] The pseudo-label generation subunit is used to input the enhanced image generated by the image enhancement subunit and the depth map generated by the depth map acquisition subunit into the self-supervised network imaging network module, and use IFM to generate a pseudo-label image similar to the real RGB underwater image acquired by the image acquisition subunit according to the image generation mechanism.

[0044] The parameter update subunit is used to construct a loss function based on the similarity between the real RGB underwater image obtained by the image acquisition subunit and the image generated by the pseudo-label generation subunit, update the gradient of the loss function and the parameters of the self-supervised underwater image enhancement network model, and obtain the trained self-supervised underwater image enhancement network model.

[0045] Furthermore, the network model training unit includes:

[0046] The image acquisition subunit is used to acquire real RGB underwater images;

[0047] An image enhancement subunit is used to enhance the real RGB underwater image acquired by the image acquisition subunit using the Fourier convolution image enhancement module;

[0048] The depth map acquisition subunit is used to generate a depth map of the real RGB underwater image acquired by the image acquisition subunit using the convolutional network depth estimation module;

[0049] Background light prior subunit, used to obtain background light prior;

[0050] The pseudo-label generation subunit is used to input the enhanced image generated by the image enhancement subunit, the depth map generated by the depth map acquisition subunit, and the background light prior obtained by the background light prior subunit into the self-supervised network imaging network module, and use IFM to generate a pseudo-label image similar to the real RGB underwater image obtained by the image acquisition subunit according to the image generation mechanism.

[0051] The parameter update subunit is used to construct a loss function based on the similarity between the real RGB underwater image obtained by the image acquisition subunit and the image generated by the pseudo-label generation subunit, update the gradient of the loss function and the parameters of the self-supervised underwater image enhancement network model, and obtain the trained self-supervised underwater image enhancement network model.

[0052] In another aspect, the present invention also provides a computer-readable storage medium storing a computer instruction set, which, when executed by a processor, implements the above-described self-supervised underwater image enhancement method.

[0053] Advantages and positive effects of the present invention:

[0054] In this invention, a newly proposed self-supervised underwater image enhancement network model is used for underwater image enhancement. This self-supervised underwater image enhancement network model combines Fourier convolution, encoder-decoder structure, and also utilizes the image imaging model IFM to generate self-supervised training pseudo-labeled images. Then, the similarity between the pseudo-labeled images and the original images is used to construct a loss function, thereby achieving self-supervised training of the self-supervised underwater image enhancement network model. By using a self-supervised method to enhance underwater images, complex manually labeled datasets are not required, saving a significant amount of manpower and resources.

[0055] This invention employs self-supervised learning using unlabeled data. Specifically, it acquires feature representations by learning the objective function of a pretext task. This pretext task can be a prediction task, a generative task, or a contrastive learning task, and the supervision information for the pretext task comes from the data itself. Because the self-supervised signal is derived from the representation of the data itself, a pretext task that learns the data distribution and feature representation exceptionally well allows the self-supervised network to learn parameters that better represent the data features, thus achieving better results—attaining the same image enhancement effect as fully supervised methods, or even surpassing them. The self-supervised underwater image enhancement method in this invention uses an Image Formation Model (IFM) to construct a self-supervised signal. Since the IFM can better represent the data representation, the network can better learn the data's representational information, making it more suitable for image enhancement tasks. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 This is a flowchart of a self-supervised underwater image enhancement method according to an embodiment of the present invention;

[0058] Figure 2 This is a structural diagram of a self-supervised underwater image enhancement model according to an embodiment of the present invention;

[0059] Figure 3 This is a flowchart of a training method for a self-supervised underwater image enhancement model according to an embodiment of the present invention;

[0060] Figure 4 This is a flowchart illustrating how background light priors are obtained through the analysis of a large number of underwater images in an embodiment of the present invention.

[0061] Figure 5 This is a schematic diagram of the structure of a self-supervised underwater image enhancement device according to an embodiment of the present invention;

[0062] Figure 6 This is a schematic diagram of the structure of a network model training unit in an embodiment of the present invention. Detailed Implementation

[0063] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0064] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0065] To address the problems of underwater image distortion and color aberration, this invention proposes a self-supervised underwater image enhancement method. For example... Figure 1 The diagram illustrates a flowchart of a self-supervised underwater image enhancement method according to an embodiment of the present invention. The method includes:

[0066] S1. Establish a self-supervised underwater image enhancement network model;

[0067] like Figure 2As shown, the self-supervised underwater image enhancement network model includes a Fourier convolutional image enhancement module, a convolutional network depth estimation module, and a self-supervised network imaging module. The Fourier convolutional image enhancement module uses Fourier convolution to enhance the underwater image. The original underwater image is transformed from the time domain to the frequency domain using a Fourier transform (FFT). A learnable global filter K is introduced into the frequency domain. The learned filter K is multiplied by the image in the frequency domain to obtain the frequency domain value of the enhanced image. Then, the image is transformed from the frequency domain to the time domain using an inverse Fourier transform (IFFT), and the output is the enhanced image. The convolutional network depth estimation module is an encoder-decoder structure. The original underwater image is input to the encoder, which extracts features through convolution and downsampling. Then, the decoder upsamples the low-dimensional feature map to the same dimension as the original underwater image and outputs the depth map. The self-supervised network imaging module uses IFM to generate pseudo-label images similar to the original underwater images for self-supervised training, based on the image generation mechanism. These pseudo-label images serve as supervision signals for training the self-supervised underwater image enhancement network model.

[0068] Specifically, a real RGB underwater image X is input into both the Fourier convolutional image enhancement module and the convolutional network depth estimation module. The Fourier convolutional image enhancement module first transforms the input real RGB underwater image X from the time domain to the frequency domain using a two-dimensional Fourier transform (2DFFT). Then, a learnable global filter K modulates the value of the real underwater image in the frequency domain. After modulation, a two-dimensional inverse Fourier transform (2DIFFT) transforms the image back from the frequency domain to the time domain, and finally outputs the enhanced underwater image. The convolutional network depth estimation module processes the input real RGB underwater image X using an encoder-decoder structure. The encoder extracts features from the input image and downsamples them, while the decoder upsamples the feature map extracted by the encoder module. The depth map is estimated using the features extracted by the encoder, and the low-dimensional image is reshaped to the same size as the input image, finally outputting the depth map. The enhanced image and depth map output from the Fourier convolutional image enhancement module and the convolutional network depth estimation module are used to construct an image J(x) similar to the input real RGB underwater image X using IFM. This image J(x) is used as a pseudo-label for the self-supervised training of the self-supervised underwater image enhancement network model. After obtaining the fake image J(x), it can be compared with the real RGB underwater image X to construct a loss function. Simultaneously, the gradient of the loss function and the parameters of the self-supervised underwater image enhancement network model are updated, optimizing the self-supervised underwater image enhancement network model for underwater image enhancement and depth map estimation tasks.

[0069] S2. The self-supervised underwater image enhancement network model established through training;

[0070] Specifically, such as Figure 3 As shown, the training process of the above self-supervised underwater image enhancement network model specifically includes the following steps:

[0071] S21. Obtain a real RGB underwater image;

[0072] The real RGB underwater image was used as training data for a self-supervised underwater image enhancement network model.

[0073] S22. Use the Fourier convolution image enhancement module to enhance the real RGB underwater image obtained in S21;

[0074] The Discrete Fourier Transform (DFT) is an important method in digital signal processing. Given an input image x∈C, M×N Two-dimensional discrete Fourier transform f(x)∈C M×N It can be represented as:

[0075]

[0076] In the formula, x represents the input image, M and N represent the height and width dimensions of the input image x, u and v are the frequency values ​​of the input image in the horizontal and vertical dimensions, respectively, and f(x) represents the frequency domain signal of the image after Fourier transform.

[0077] Meanwhile, the Discrete Fourier Transform (DFTx) can also convert the image from the frequency domain back to the time domain through the Inverse Discrete Fourier Transform (IDFTf(x)), and the input signal x satisfies conjugate symmetry.

[0078]

[0079] In its specific implementation, S22 includes the following steps:

[0080] (1) The real RGB underwater image obtained by S1 is transformed from the time domain to the frequency domain through Fourier transform (FFT);

[0081] Specifically, the input signal x is transformed into the frequency domain using a two-dimensional Fourier transform along the spatial dimension:

[0082] X=f(x∈C) M×N×D (3)

[0083] (2) Determine the global filter K (a multidimensional matrix with the same size as the image), and multiply the global filter K by the image in the frequency domain space to obtain the frequency domain space value of the enhanced image;

[0084] Here, a learnable global filter K∈C is used.M×N×D To change signal x:

[0085]

[0086] In the above formula, ⊙ represents element-wise matrix multiplication, and K is a learnable global filter (a multidimensional matrix with the same size as the image) introduced into the frequency domain space. Essentially, a global filter K is learned. Because K has the same dimension as the input signal, it can represent any filter in the frequency domain.

[0087] (3) The image is transformed from the frequency domain space to the time domain space by inverse Fourier transform (IFFT), and the output result is the enhanced image.

[0088] Will The system is transformed back to the time domain using Inverse Fourier Transform (IFFT), and the parameters in the global filter are updated using the Adam optimizer and stochastic gradient descent module provided by the PyTorch deep learning framework.

[0089]

[0090] In the formula f -1 This represents the Inverse Fourier Transform (IFFT).

[0091] By updating the parameters through multiple iterations, the parameters of the global filter K are trained to be optimally suited for the image enhancement task. After transforming the input signal to the frequency domain using FFT, the signal is modulated by the global filter K, similar to convolution in the time domain, but with fewer parameters involved in multiplying matrix elements in the frequency domain compared to convolution in the time domain.

[0092]

[0093] S23. Use the convolutional network depth estimation module to generate a depth map of the real RGB underwater image obtained in S21;

[0094] The convolutional network depth estimation module is an encoder-decoder structure. The real RGB underwater image obtained by S21 is input into the encoder of the convolutional network depth estimation module. After convolution and downsampling, features are extracted. Then, the decoder upsamples the low-dimensional feature map to the same dimension as the original real RGB underwater image and outputs the depth result map.

[0095] It is understood that the execution order of S22 and S23 can be: S22 is executed first, then S23; S23 is executed first, then S22; or S22 and S23 are executed in parallel. This invention does not restrict the execution order of these two steps.

[0096] S24, the enhanced image generated by S22 and the depth map generated by S23 are input to the self-supervised network imaging network module, and the IFM is used to generate a pseudo-label image similar to the real RGB underwater image obtained in S21 according to the image generation mechanism.

[0097] The generated pseudo-label image J(x) is used to train a self-supervised underwater image enhancement network model, and the IFM model takes the following form:

[0098] J(x)=I(x)t(x)+B(1-t(x)) (6)

[0099] t(x)=e -βd (7)

[0100] In the above formula, J(x) represents the fitted underwater image obtained through IFM, which is the pseudo-labeled image used to train the self-supervised underwater image enhancement network model, I(x) represents the true radiance of the underwater scene, t(x) represents the transmission map, B represents the underwater ambient light, β represents the light attenuation rate underwater, and d represents the scene depth.

[0101] In another embodiment, when generating pseudo-labeled images J(x) using the IFM model, a background light prior B obtained from statistical analysis of a large dataset can be used for pseudo-label generation. The background light prior is as follows: Figure 4 As shown, pseudo-label images are generated to be more similar to real underwater images, ensuring that the difference between the IFM-generated images and real underwater images is minimized.

[0102] S25. Construct a loss function using the similarity between the real RGB underwater image obtained in S21 and the image generated in S24. Update the gradient of the loss function and the parameters of the self-supervised underwater image enhancement network model to obtain the trained self-supervised underwater image enhancement network model.

[0103] After obtaining the forged image J(x), it can be compared with the real RGB underwater image X to perform a loss function. This involves comparing X and J(x) by similarity to construct the loss function, updating the gradient of the loss function, and updating the parameters (weights and bias) of the self-supervised underwater image enhancement network model. This optimizes the self-supervised underwater image enhancement network model to be suitable for underwater image enhancement and depth map estimation tasks.

[0104] During training, mean squared error (MSE) and multi-scale structural similarity (MS-SSIM) loss are used to update parameters. After obtaining the pseudo-label image J(x) output by the IFM module, J(x) and the real RGB underwater image X are substituted into the MSE and MS-SSIM loss functions. In each iteration, the gradient of the combined MSE and MS-SSIM loss is calculated. The parameters of the self-supervised imaging network model are optimized using the Adam optimizer to find the optimal parameters suitable for the task. The formulas for the MSE and MS-SSIM loss functions are as follows:

[0105]

[0106]

[0107] Therefore, the total loss function is as follows:

[0108] L=αL MS-SSIM +(1-α)L MSE (10)

[0109] Where α is the weight value, preferably 0.8; L MSE For mean squared variance loss, L MS-SSIM For multi-scale structural similarity loss, k represents the center pixel of the image patch, p, g represent 13×13 image patches, where p∈J(X), g∈X, μ p ,μ g Let J(X) and X represent the average values, respectively, and σ represent the average values ​​of X and X. p and σ g Let J(X) and X represent the standard deviations, respectively, and σ represent the standard deviations of X and X. pg Let c1 = 0.02 and c2 = 0.03 represent the covariances of J(X) and X.

[0110] S3. Acquire the underwater image to be enhanced;

[0111] S4. Input the acquired underwater image to be enhanced into the trained self-supervised underwater image enhancement network model to obtain the enhanced underwater image.

[0112] In this embodiment of the invention, underwater image enhancement using a self-supervised method can eliminate the need for complex manually labeled datasets, saving manpower and resources, and can achieve the same effect as, or even surpass, fully supervised methods.

[0113] Corresponding to the self-supervised underwater image enhancement method in the above embodiments, the present invention also provides a self-supervised underwater image enhancement device, such as... Figure 5 As shown, the device includes:

[0114] The network model building unit 100 is used to build a self-supervised underwater image enhancement network model. The self-supervised underwater image enhancement network model includes a Fourier convolution image enhancement module, a convolutional network depth estimation module, and a self-supervised network imaging module. Among them, the Fourier convolution image enhancement module uses Fourier convolution to enhance underwater images; the convolutional network depth estimation module uses an encoder-decoder structure to generate depth maps; and the self-supervised network imaging module is used to generate self-supervised training pseudo-label images from the enhanced images and depth maps generated by the first two modules using IFM.

[0115] The network model training unit 200 is used to train a self-supervised underwater image enhancement network model using pseudo-labeled images.

[0116] The original image acquisition unit 300 is used to acquire underwater images to be enhanced.

[0117] The underwater image enhancement unit 400 inputs the underwater image to be enhanced acquired by the original image acquisition unit 300 into the self-supervised underwater image enhancement network model trained by the network model training unit 200 to obtain the enhanced underwater image.

[0118] Specifically, the network model training unit 200 includes:

[0119] Image acquisition subunit 201 is used to acquire real RGB underwater images;

[0120] The real RGB underwater image was used as training data for a self-supervised underwater image enhancement network model.

[0121] Image enhancement subunit 202 is used to enhance the real RGB underwater image acquired by image acquisition subunit 201 using the Fourier convolution image enhancement module; specifically, it is used to: transform the real RGB underwater image acquired by image acquisition subunit 201 from the time domain space to the frequency domain space through Fourier transform (FFT); determine the global filter K, multiply the global filter K with the image in the frequency domain space to obtain the frequency domain space value of the enhanced image; transform the image from the frequency domain space to the time domain space through inverse Fourier transform (IFFT), and the output result is the enhanced image.

[0122] The depth map acquisition subunit 203 is used to generate a depth map of the real RGB underwater image acquired by the image acquisition subunit 201 using the convolutional network depth estimation module;

[0123] The convolutional network depth estimation module is an encoder-decoder structure. The real RGB underwater image acquired by the image acquisition subunit 201 is input into the encoder of the convolutional network depth estimation module. After convolution and downsampling, features are extracted. Then, the decoder upsamples the low-dimensional feature map to the same dimension as the original real RGB underwater image and outputs the depth result map.

[0124] The pseudo-label generation subunit 204 is used to input the enhanced image generated by the image enhancement subunit 202 and the depth map generated by the depth map acquisition subunit 203 into the self-supervised network imaging network module, and use IFM to generate a pseudo-label image similar to the real RGB underwater image acquired by the image acquisition subunit 201 according to the image generation mechanism.

[0125] The parameter update subunit 205 is used to construct a loss function based on the similarity between the real RGB underwater image obtained by the image acquisition subunit 201 and the image generated by the pseudo-label generation subunit 204, update the gradient of the loss function and the parameters of the self-supervised underwater image enhancement network model, and obtain the trained self-supervised underwater image enhancement network model.

[0126] After obtaining the forged image J(x), it can be compared with the real RGB underwater image X to perform a loss function. This involves comparing X and J(x) by similarity to construct the loss function, updating the gradient of the loss function and updating the parameters of the self-supervised underwater image enhancement network model, thus optimizing the self-supervised underwater image enhancement network model for underwater image enhancement and depth map estimation tasks.

[0127] During training, mean squared error (MSE) and multi-scale structural similarity (MS-SSIM) loss are used to update the parameters of the self-supervised imaging network model. After obtaining the pseudo-label image J(x) output by the IFM module, J(x) and the real RGB underwater image X are substituted into the MSE and MS-SSIM loss functions. In each iteration, the gradient of the combined MSE and MS-SSIM loss is calculated, and the parameters of the self-supervised imaging network model are optimized using the Adam optimizer to find the optimal parameters suitable for the task. The formulas for the MSE and MS-SSIM loss functions are as follows:

[0128]

[0129]

[0130] Therefore, the total loss function is as follows:

[0131] L=αL MS-SSIM +(1-α)L MSE ;

[0132] Where α is the weight value, preferably 0.8; L MSE For mean squared variance loss, L MS-SSIM For multi-scale structural similarity loss, k represents the center pixel of the image patch, p, g represent 13×13 image patches, where p∈J(X), g∈X, μ p ,μ g Let J(X) and X represent the average values, respectively, and σ represent the average values ​​of X and X. p and σ g Let J(X) and X represent the standard deviations, respectively, and σ represent the standard deviations of X and X. pg Let c1 = 0.02 and c2 = 0.03 represent the covariances of J(X) and X.

[0133] In another embodiment, such as Figure 6 As shown, the network model training unit 200 includes the aforementioned sub-units, as well as a background light prior sub-unit 206.

[0134] Background light prior subunit 206 is used to obtain background light prior;

[0135] Accordingly, when the pseudo-label generation subunit 204 uses IFM to generate the pseudo-label image J(x), it adds the background light prior obtained by the background light prior subunit 206 to generate a pseudo-label image that is more similar to the real underwater image, so as to ensure that the difference between the pseudo-label image J(x) generated by IFM and the real underwater image is minimized.

[0136] The underwater image enhancement device based on self-supervised supervision in this embodiment of the invention is described simply because it corresponds to the underwater image enhancement method based on self-supervised supervision in the above embodiment. For related similarities, please refer to the description of the underwater image enhancement method based on self-supervised supervision in the above embodiment, which will not be described in detail here.

[0137] This invention also discloses a computer-readable storage medium storing a computer instruction set, which, when executed by a processor, implements the self-supervised underwater image enhancement method provided in any of the embodiments above.

[0138] In the several embodiments provided by this invention, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection can be through some interfaces; the indirect coupling or communication connection of units or modules can be electrical or other forms.

[0139] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0140] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0141] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A self-supervised underwater image enhancement method, characterized in that, The method includes: A self-supervised underwater image enhancement network model is established. This model includes a Fourier convolutional image enhancement module, a convolutional network depth estimation module, and a self-supervised network imaging module. The Fourier convolutional image enhancement module uses Fourier convolution to enhance underwater images. The convolutional network depth estimation module generates depth maps using an encoder-decoder structure. The self-supervised network imaging module uses the image imaging model IFM to generate self-supervised trained pseudo-label images from the enhanced images generated by the Fourier convolutional image enhancement module and the depth maps generated by the convolutional network depth estimation module. A self-supervised underwater image enhancement network model was established using the pseudo-labeled images through self-supervised training. Acquire underwater images to be enhanced; The underwater image to be enhanced is input into a trained self-supervised underwater image enhancement network model to obtain the enhanced underwater image. The self-supervised underwater image enhancement network model established using the pseudo-labeled images through self-supervised training includes: Acquire realistic RGB underwater images; The real RGB underwater image is enhanced using a Fourier convolution image enhancement module to obtain an enhanced image of the real RGB underwater image. The depth map of the real RGB underwater image is generated using a convolutional network depth estimation module; Obtain the background light prior; The enhanced image, the depth map, and the background light prior are input into the self-supervised network imaging network module, and the IFM is used to generate a pseudo-label image similar to the real RGB underwater image according to the image generation mechanism. A loss function is constructed using the similarity between the real RGB underwater image and the pseudo-labeled image. The gradient of the loss function and the parameters of the self-supervised underwater image enhancement network model are then updated to obtain the trained self-supervised underwater image enhancement network model.

2. The underwater image enhancement method based on self-supervised method according to claim 1, characterized in that, The Fourier convolution image enhancement module uses Fourier convolution to enhance underwater images, including: Real RGB underwater images are transformed from the time domain to the frequency domain using Fourier transform; Determine a global filter K, and multiply the global filter K by the image in the frequency domain space to obtain the frequency domain space value of the enhanced image; The image is transformed from the frequency domain to the time domain by inverse Fourier transform, and the output is the enhanced image.

3. The underwater image enhancement method based on self-supervised method according to claim 1, characterized in that, The convolutional network depth estimation module uses an encoder-decoder architecture to generate depth maps, including: The encoder of the convolutional network depth estimation module is input into a real RGB underwater image. Features are extracted through convolution and downsampling. Then, the decoder upsamples the low-dimensional feature map to the same dimension as the real RGB underwater image and outputs the depth result map.

4. The underwater image enhancement method based on self-supervised method according to claim 1, characterized in that, The loss function is: ； in, For mean squared variance loss, For multi-scale structural similarity loss; This is the weight value.

5. A self-supervised underwater image enhancement device, characterized in that, The device includes: A network model building unit is used to construct a self-supervised underwater image enhancement network model. This model includes a Fourier convolutional image enhancement module, a convolutional network depth estimation module, and a self-supervised network imaging module. The Fourier convolutional image enhancement module uses Fourier convolution to enhance underwater images. The convolutional network depth estimation module generates depth maps using an encoder-decoder structure. The self-supervised network imaging module uses the image imaging model IFM to generate self-supervised trained pseudo-label images from the enhanced images generated by the Fourier convolutional image enhancement module and the depth maps generated by the convolutional network depth estimation module. The network model training unit is used to self-supervisedly train the self-supervised underwater image enhancement network model constructed by the network model building unit using the pseudo-labeled images; The original image acquisition unit is used to acquire the underwater image to be enhanced; the underwater image enhancement unit inputs the underwater image to be enhanced acquired by the original image acquisition unit into the self-supervised underwater image enhancement network model trained by the network model training unit to obtain the enhanced underwater image. The network model training unit includes: The image acquisition subunit is used to acquire real RGB underwater images; An image enhancement subunit is used to enhance the real RGB underwater image acquired by the image acquisition subunit using the Fourier convolution image enhancement module; The depth map acquisition subunit is used to generate a depth map of the real RGB underwater image acquired by the image acquisition subunit using the convolutional network depth estimation module; Background light prior subunit, used to obtain background light prior; The pseudo-label generation subunit is used to input the enhanced image generated by the image enhancement subunit, the depth map generated by the depth map acquisition subunit, and the background light prior obtained by the background light prior subunit into the self-supervised network imaging network module, and use IFM to generate a pseudo-label image similar to the real RGB underwater image obtained by the image acquisition subunit according to the image generation mechanism. The parameter update subunit is used to construct a loss function based on the similarity between the real RGB underwater image obtained by the image acquisition subunit and the image generated by the pseudo-label generation subunit, update the gradient of the loss function and the parameters of the self-supervised underwater image enhancement network model, and obtain the trained self-supervised underwater image enhancement network model.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a set of computer instructions, which, when executed by a processor, implement the self-supervised underwater image enhancement method as described in any one of claims 1 to 4.

Images