A method for repairing underwater degraded images, a storage medium and an electronic device

Abstract

This invention provides a method for restoring degraded underwater images, comprising: S1: inputting a degraded underwater image; S2: constructing an unsupervised deep learning image reconstruction network based on a physical model; S3: reconstructing the degraded image according to the underwater image imaging physical model and the network output, and constructing a consistency verification module for the reconstruction network; S4: constructing a fidelity image network isotropic loss and an underwater imaging isotropic loss according to image isomorphism theory; S5: constructing a fidelity image restoration image isomorphism verification module according to image isomorphism; S6: constructing a noise loss using an unbiased estimation method based on the noise distribution of the underwater degraded image, and correcting the bias of the reconstruction loss; S7: constructing a color fidelity verification module by compensating the blue and green channels with less color decay to the red channel according to the gray-world assumption; S8: training the network according to the constraints of S3-S7 until the loss converges, obtaining a fidelity image estimation module for restoring the degraded image. Storage media and electronic devices are also provided.

Description

Technical Field

[0001] This application relates to the technical field of underwater image restoration, and more particularly to a method, storage medium, and electronic device for restoring degraded underwater images. Background Technology

[0002] Underwater imaging is crucial in developing ocean-oriented services such as marine ecological assessment, fishing net inspection, automated feeding systems, and underwater autonomous navigation. The quality of underwater imaging is susceptible to challenges such as insufficient natural light and water turbidity, which can lead to color shifts, blurring, and reduced contrast. In particular, short-wavelength light experiences significant attenuation in underwater environments, resulting in underwater images exhibiting predominantly green or blue hues. Therefore, underwater image restoration becomes especially important to generate high-quality images for subsequent visual tasks such as object detection and biological assessment.

[0003] Traditional underwater image restoration methods rely on simulating physical processes to reduce blur and enhance image contrast. Common underwater imaging physics models propose decomposing degraded images into three components: the transmission map, scene radiation, and global background illumination. However, due to the inherently rapid changes in the underwater environment, light propagation in seawater can deviate from physics models with fixed parameters, potentially leading to distortion, especially under complex lighting conditions.

[0004] By utilizing physical models, deep neural networks can effectively reconstruct distorted images and eliminate color shifts. Current deep models use supervised learning, relying on pairing degraded and sharp images for training. However, accurately modeling degradation processes and noise in underwater scenes remains challenging, as unpredictable differences exist between synthetic and real data, making supervised learning prone to bias and overfitting. Summary of the Invention

[0005] To address the technical problems mentioned in the background, the present invention provides a method, storage medium, and electronic device for restoring degraded underwater images.

[0006] This invention provides a method for restoring degraded underwater images, comprising the following steps:

[0007] S1: Input underwater degradation image;

[0008] S2: Construct an unsupervised deep learning image reconstruction network based on a physical model;

[0009] S3: Construct a reconstruction network consistency verification module based on the underwater image imaging physical model and the network output described in S2 to reconstruct the degraded image;

[0010] S4: Based on the theory of image equivariance, construct the fidelity image network equivariance loss and the underwater imaging equivariance loss;

[0011] S5: Based on image isomorphism, construct a fidelity image recovery image isomorphism verification module;

[0012] S6: Based on the noise distribution of the underwater degraded image, construct the noise loss using an unbiased estimation method to correct the bias of the reconstruction loss;

[0013] S7: Based on the grayscale world hypothesis, compensate the red channel with the less weakened blue and green channel information to build a color fidelity verification module.

[0014] S8: Train the network described in S2 according to the constraints of S3-S7 until the loss converges, and obtain the S2 fidelity image estimation module for repairing the degraded image.

[0015] In some of these embodiments: the unsupervised deep learning image reconstruction network based on the physical model described in S2 consists of a transport graph estimation network, a fidelity graph estimation network, and a background light estimation module;

[0016] The network construction formula is: ;in, For the reconstructed degraded image, To ensure accurate estimation of the network, To estimate the network for the transmission graph, and Let be the learnable parameters of the fidelity graph prediction network and , respectively. For the global background light of the image, The input is a degraded image.

[0017] In some embodiments: S3 specifically involves: based on the underwater image imaging formula, using mean square error to correct the error between the reconstructed image and the input image, maintaining the consistency verification module of the reconstruction network; the specific optimization objective function is... ;in, This represents the reconstruction loss, and the optimization objective is to minimize the reconstruction loss, i.e., the reconstructed image. It should be the same as the input degraded image. Maintain consistency This represents the L2 norm.

[0018] In some embodiments: S4 specifically involves: based on the constraint that the result of image restoration after performing translation and rotation transformations on an image should be consistent with the result obtained after performing image restoration first and then performing the same translation and rotation operations, a fidelity image network with variable loss is constructed. ,in The loss on the right side of the equation represents the uniform loss of the high-fidelity image network. This indicates that a translation or rotation transformation is performed on the image. Indicates the first Sub-translation or rotation transformation; uniform loss of underwater imaging physical network; uniform loss of underwater imaging physical network in The loss of the imaging physical network is represented by the isovariance loss. Based on the isovariance consistency of the fidelity map and the isovariance constraint of the underwater imaging legend network, an isovariance consistency module is constructed, and the L2 norm is used to ensure isovariance consistency.

[0019] Based on the aforementioned fidelity map equivariance consistency and underwater imaging legend network equivariance constraints, an equivariance consistency module is constructed, and the L2 norm is used to guarantee equivariance consistency.

[0020] In some embodiments: the result of restoration based on the input degraded image in S5 should have the same properties as the result of restoration based on the input mixed restoration image.

[0021] Construct the image isomorphism verification module for the fidelity image restoration;

[0022] The hybrid image is a randomized mixture of the restored image and the original degraded image, with the ratio taken from a uniform probability distribution of [0,1] each time.

[0023] In some embodiments: in S6, an unbiased estimation reconstruction consistency constraint is constructed based on Gaussian noise under natural light illumination:

[0024] , in The variance of Gaussian noise ranges from 0.01 to 0.1. The number of pixels in a single image. The weight parameter is set to 0.02;

[0025] Under artificial light illumination, unbiased estimation and reconstruction consistency constraints are constructed based on Poisson noise:

[0026] , The scale used to control Poisson noise The value of is consistent with the unbiased estimate constructed using Gaussian noise, and the vector It follows a Bernoulli distribution, taking the value 0 or 1 with a 50% probability. Indicates taking Transpose of;

[0027] Under mixed illumination including natural and artificial light, the unbiased estimation reconstruction consistency constraint based on Gaussian-Poisson noise is as follows:

[0028] , The right side of the equation Represents the Hadamard product of a vector or matrix. Let I be the variance of the Gaussian noise, and let I be an identity matrix with all values ​​on the main diagonal being 1 and all values ​​elsewhere being 0. Let I be a vector. and It follows the Bernoulli distribution. The scale used to control Poisson noise is set to either 0 or 1 with a 50% probability, and the weighting coefficients are... The value is 0.02.

[0029] In some embodiments: In S7, based on the grayscale of the image generated from the natural scene light field, it is directly assumed that the color distribution histogram of the RGB three-channel image should theoretically satisfy a distribution with a mean of 0.5, and a color fidelity loss for training is constructed:

[0030] ; in, Indicates a loss of color fidelity. Indicates color channel These are the red, green, and blue channels. Represents the L2 norm. This represents the mean.

[0031] In some embodiments, the specific method of S8 is as follows:

[0032] Acquire degraded underwater images in RGB format;

[0033] Degraded underwater images are cropped into small image blocks with a width and height of 256*256 pixels. The cropping regions are generated using a random function.

[0034] The cropped image patches are input into an unsupervised reconstruction network for learning. The loss function of the unsupervised learning policy is defined as:

[0035] ;

[0036] The first term of the loss function For unsupervised reconstruction loss of specific noise, the loss weights are... The coefficient is 1, the second term For equivariant verification loss, the loss weight coefficient is... The third item is 0.001. For isomorphic verification loss, the loss weight coefficients are... The fourth item is 1. The loss weighting factor is for color fidelity loss. 0.01; The learnable parameters of the network are represented by the AdamW learning optimizer. The network training iterations are 200, the initial learning rate is 5e-4, and the learning rate is reduced to 0.5 times the current learning rate every 50 training iterations.

[0037] The present invention also provides a computer-readable storage medium storing instructions for performing the above-described method for restoring underwater degraded images.

[0038] The present invention provides an electronic device comprising the aforementioned computer-readable storage medium.

[0039] This application includes the following beneficial effects:

[0040] This invention proposes an unsupervised underwater image inpainting method based on image isomorphism and unbiased estimation. Compared with existing underwater image inpainting methods, the main advantage of this invention is that the trained image inpainting module can realistically restore degraded underwater images without relying on labeled data. By constructing an image isomorphism verification module, a noise prior-based reconstruction consistency module, an image isomorphism verification module, and a color fidelity module, the visual fidelity of the image inpainting is improved, thereby removing noise interference from degraded underwater images and correcting color deviations, thus enhancing the visual effect of underwater images. Attached Figure Description

[0041] Figure 1 A flowchart of an unsupervised underwater image inpainting method based on image isovariability and unbiased estimation is provided for an embodiment of the present invention.

[0042] Figure 2 This is a flowchart of the network training process provided in an embodiment of the present invention.

[0043] Figure 3 This is a network inference flowchart provided in an embodiment of the present invention.

[0044] Figure 4 These are image restoration comparison images provided in embodiments of the present invention.

[0045] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0047] Reference Figure 1-5 This embodiment provides a method for restoring degraded underwater images, including the following steps:

[0048] S1: Input underwater degradation image;

[0049] S2: Construct an unsupervised deep learning image reconstruction network based on a physical model;

[0050] S3: Construct a reconstruction network consistency verification module based on the underwater image imaging physical model and the network output described in S2 to reconstruct the degraded image;

[0051] S4: Based on the theory of image equivariance, construct the fidelity image network equivariance loss and the underwater imaging equivariance loss;

[0052] S5: Based on image isomorphism, construct a fidelity image recovery image isomorphism verification module;

[0053] S6: Based on the noise distribution of the underwater degraded image, construct the noise loss using an unbiased estimation method to correct the bias of the reconstruction loss;

[0054] S7: Based on the grayscale world hypothesis, compensate the red channel with the less weakened blue and green channel information to build a color fidelity verification module.

[0055] S8: Train the network described in S2 according to the constraints of S3-S7 until the loss converges, and obtain the S2 fidelity image estimation module for repairing the degraded image.

[0056] In this embodiment, S2 specifically includes:

[0057] The unsupervised deep learning image reconstruction network based on the physical model consists of a transmission graph estimation network, a fidelity graph estimation network, and a background light estimation module.

[0058] The network construction formula is: ;in, For the reconstructed degraded image, To ensure accurate estimation of the network, To estimate the network for the transmission graph, and Let be the learnable parameters of the fidelity graph prediction network and , respectively. For the global background light of the image, The input is a degraded image.

[0059] In this embodiment, the specific method of S3 is as follows:

[0060] Based on the underwater image imaging formula, mean square error is used to correct the error between the reconstructed image and the input image, maintaining the consistency verification module of the reconstruction network. The specific optimization objective function is as follows: ;in, This represents the reconstruction loss, and the optimization objective is to minimize the reconstruction loss, i.e., the reconstructed image. It should be the same as the input degraded image. Maintain consistency This represents the L2 norm.

[0061] In this embodiment, the specific method of S4 is as follows:

[0062] Based on the constraint that the result of image inpainting after translation and rotation should be consistent with the result obtained after image inpainting followed by the same translation and rotation operations, a fidelity-preserving image network with variable loss is constructed:

[0063] ;

[0064] in The loss on the right side of the equation represents the uniform loss of the high-fidelity image network. This indicates that a translation or rotation transformation is performed on the image. Indicates the first Subsequent translation or rotation transformations; and equivariant loss in underwater imaging physical networks; and equivariant loss in underwater imaging physical networks:

[0065] ; in The variability loss of the imaging physical network is represented by the variability consistency of the fidelity image and the variability constraints of the underwater imaging legend network. A variability consistency module is constructed, and the L2 norm is used to guarantee variability consistency.

[0066] Based on the aforementioned fidelity image equivariance consistency and underwater imaging legend network equivariance constraints, an equivariance consistency module is constructed, and the L2 norm is used to guarantee equivariance consistency. Image equivariance operations include translating in any direction by a distance equal to the image's own width and height, and stitching any portion exceeding the image size into blank areas; rotation operations have angles of {90°, 180°, 270°}. Each time an image transformation is performed, one operation is selected equally probably from translation and rotation.

[0067] In this embodiment, S5 specifically includes:

[0068] The results of restoration based on the input degraded image should maintain the same properties as those of restoration based on the input mixed image:

[0069] Construct the image isomorphism verification module for the fidelity image restoration;

[0070] in, The hybrid image is a randomized mixture of the restored image and the original degraded image, with the ratio taken from a uniform probability distribution of [0,1] each time.

[0071] Based on the unsupervised reconstruction network, and according to the unbiased denoising method, Gaussian noise, Poisson noise, and Gaussian-Poisson mixed noise are used as prior noise knowledge to further refine the reconstruction consistency constraint. In this embodiment, S6 specifically includes:

[0072] Under natural light, the noise in degraded images is closer to Gaussian noise. Unbiased estimation and reconstruction consistency constraints are constructed based on Gaussian noise.

[0073] , in The variance of Gaussian noise ranges from 0.01 to 0.1. The number of pixels in a single image. The weight parameter is set to 0.02 for the vector. It follows a Bernoulli distribution, taking the value 0 or 1 with a 50% probability. Indicates taking Transpose of;

[0074] In turbid waters, suspended particulate matter can produce snowflake-like white noise during imaging against an artificial light background. This white noise removal can be based on a Poisson noise prior, using an unbiased estimate to reconstruct consistency constraints based on the Poisson noise.

[0075] The scale used to control Poisson noise The value of is consistent with the unbiased estimate constructed using Gaussian noise, and the vector It follows a Bernoulli distribution, taking the value 0 or 1 with a 50% probability.

[0076] Under mixed illumination including natural and artificial light, using priors containing both Gaussian and Poisson noise, a consensus constraint is reconstructed based on an unbiased estimate using Gaussian-Poisson noise:

[0077] , The right side of the equation Represents the Hadamard product of a vector or matrix. Let I be the variance of the Gaussian noise, and let I be an identity matrix with all values ​​on the main diagonal being 1 and all values ​​elsewhere being 0. Let I be a vector. and It follows the Bernoulli distribution. The scale used to control Poisson noise is set to either 0 or 1 with a 50% probability, and the weighting coefficients are... The value is 0.02.

[0078] In this embodiment, S7 specifically includes:

[0079] Based on the direct assumption of grayscale generated from the light field of a natural scene, the color distribution histogram of the RGB three-channel image should theoretically satisfy a distribution with a mean of 0.5. A color fidelity loss function is then constructed for training.

[0080] ;

[0081] in, Represents the L2 norm. The above formula constrains the color distribution of the RGB three-channel image to a mean of 0.5, guiding the unsupervised network to use information from the blue and green channels to compensate for information from the red channel, thereby improving the color fidelity of the image.

[0082] The CNN reconstruction network consists of an image fidelity network, a transport network, and a background light estimation module. The image fidelity module has the same network design as the transport map module, containing five convolutional groups. Each convolutional group includes a mirror completion module, a convolutional module with a kernel of 3 and a stride of 1, a ReLU activation function, and a LayerNorm normalization module. The first convolutional group has 3 input channels, and the last convolutional group has 3 output channels. The remaining convolutional groups have 64 input and 64 output channels each. After five convolutional groups, based on the pixel value constraints of the transport and fidelity maps, a Sigmoid activation function is used to limit the output to between 0 and 1. The background light estimation module uses Gaussian blur.

[0083] In this embodiment, the specific method of S8 is as follows:

[0084] Acquire degraded underwater images in RGB format;

[0085] Degraded underwater images are cropped into small image blocks with a width and height of 256*256 pixels. The cropping regions are generated using a random function.

[0086] The cropped image patches are input into an unsupervised reconstruction network for learning. The loss function of the unsupervised learning policy is defined as:

[0087] ;

[0088] The first term of the loss function For unsupervised reconstruction loss of specific noise, the loss weights are... The coefficient is 1, the second term For equivariant verification loss, the loss weight coefficient is... The third item is 0.001. For isomorphic verification loss, the loss weight coefficients are... The fourth item is 1. The loss weighting factor is for color fidelity loss. 0.01; The learnable parameters of the network are represented by the AdamW learning optimizer. The network training iterations are 200, the initial learning rate is 5e-4, and the learning rate is reduced to 0.5 times the current learning rate every 50 training iterations.

[0089] like Figure 4 As shown, from left to right, the images are the original degraded image, existing methods such as Waternet, Semi-UIR, and USUIR, the unsupervised image inpainting method proposed in this embodiment, and a reference image. The unsupervised image inpainting method proposed in this embodiment has higher color fidelity.

[0090] like Figure 5 As shown, the electronic device 500 includes one or more processors 501 and one or more memories 502, wherein the memory 502 stores instructions that are loaded and executed by the processor 501 to implement the above-described unsupervised underwater image inpainting steps based on image isovariance and unbiased estimation methods.

[0091] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A method for restoring degraded underwater images, characterized in that, Includes the following steps: S1: Input underwater degradation image; S2: Construct an unsupervised deep learning image reconstruction network based on a physical model; S3: Construct a reconstruction network consistency verification module based on the underwater image imaging physical model and network output to reconstruct the degraded image; S4: Based on the theory of image equivariance, construct the fidelity image network equivariance loss and the underwater imaging equivariance loss; S5: Based on image isomorphism, construct a fidelity image recovery image isomorphism verification module; S6: Based on the noise distribution of the underwater degraded image, construct the noise loss using an unbiased estimation method to correct the bias of the reconstruction loss; S7: Based on the grayscale world hypothesis, compensate the red channel with the blue and green channels that have less color decay to build a color fidelity verification module; S8: Train the network according to the constraints of S3-S7 until the loss converges, and obtain the fidelity image estimation module for image restoration; The unsupervised deep learning image reconstruction network based on the physical model described in S2 consists of a transmission graph estimation network, a fidelity graph estimation network, and a background light estimation module. The network construction formula is: ; in, For the reconstructed degraded image, To preserve the fidelity of the image, the network is estimated. To estimate the network for the transmission graph, and These are the learnable parameters for the fidelity graph prediction network and the transmission graph estimation network, respectively. For the global background light of the image, The input is a degraded image; The specific method of S3 is as follows: Based on the underwater image imaging formula, the mean square error is used to correct the error between the reconstructed image and the input image, maintaining the consistency verification module of the reconstruction network. The specific optimization objective function is... ; in, This represents the reconstruction loss, and the optimization objective is to minimize the reconstruction loss, i.e., the reconstructed image. It should be the same as the input degraded image. Maintain consistency Represents the L2 norm; The specific method of S4 is as follows: Based on the constraint that the result of image inpainting after performing translation and rotation transformations on an image should be consistent with the result obtained after performing image inpainting first and then performing the same translation and rotation operations, a fidelity image network with equal-variable loss is constructed. ,in The loss on the right side of the equation represents the uniform loss of the high-fidelity image network. This indicates that a translation or rotation transformation is performed on the image. Indicates the first Sub-translation or rotation transformation; variable loss in underwater imaging physical network. ,in The imaging physical network is represented by the isovariant loss. Based on the above-mentioned fidelity map isovariant consistency and underwater imaging legend network isovariant constraints, an isovariant consistency module is constructed, and the L2 norm is used to ensure isovariant consistency. In S5, the result of restoration based on the input degraded image should maintain the same properties as the result of restoration based on the input mixed restoration image, that is... Construct the image isomorphism verification module for the fidelity image restoration; The mixed image is a randomized mixture of the restored image and the original degraded image, with the ratio taking values ​​from a uniform probability distribution of [0,1] each time. In step S6, unbiased estimation reconstruction consistency constraints are constructed based on Gaussian noise under natural light illumination: ， in The variance of Gaussian noise ranges from 0.01 to 0.

1. The number of pixels in a single image. The weight parameter is set to 0.02 for the vector. It follows a Bernoulli distribution, taking the value 0 or 1 with a 50% probability. Indicates taking Transpose of; Under artificial light illumination, unbiased estimation and reconstruction consistency constraints are constructed based on Poisson noise: The scale used to control Poisson noise The value of is consistent with the unbiased estimate constructed using Gaussian noise, and the vector It is a Bernoulli distribution, taking the value 0 or 1 with a 50% probability, and ⊙ represents the Hadamard product of a vector or matrix; Under mixed illumination including natural and artificial light, the unbiased estimation reconstruction consistency constraint based on Gaussian-Poisson noise is as follows: The circle on the right side of the equation represents the Hadamard product of a vector or matrix. Let I be the variance of the Gaussian noise, and let I be an identity matrix with all values ​​on the main diagonal being 1 and all values ​​elsewhere being 0. Let I be a vector. and Following a Bernoulli distribution, the values ​​are 0 or 1 with a 50% probability, and the weighting coefficients are... The value is 0.02; In S7, based on the direct assumptions made about the grayscale of the image generated from the light field of a natural scene, the color distribution histogram of the RGB three-channel image should theoretically satisfy a distribution with a mean of 0.5, and a color fidelity loss function is constructed for training: ； in, Indicates a loss of color fidelity. Indicates color channel These are the red, green, and blue channels. Represents the L2 norm. This represents the mean.

2. The method for restoring degraded underwater images according to claim 1, characterized in that, The specific method for S8 is as follows: Acquire degraded underwater images in RGB format; Degraded underwater images are cropped into small image blocks with a width and height of 256*256 pixels. The cropping regions are generated using a random function. The cropped image patches are input into an unsupervised reconstruction network for learning. The loss function of the unsupervised learning policy is defined as: The first term of the loss function For the unsupervised reconstruction loss corresponding to noise, the loss weights are... The coefficient is 1, the second term For equivariant verification loss, the loss weight coefficient is... The third item is 0.

001. For isomorphic verification loss, the loss weight coefficients are... The fourth item is 1. The loss weighting factor is for color fidelity loss. 0.01; The learnable parameters of the network are represented by the AdamW learning optimizer. The network training iterations are 200, the initial learning rate is 5e-4, and the learning rate is reduced to 0.5 times the current learning rate every 50 training iterations.

3. A computer-readable storage medium, characterized in that, Store instructions for performing the method for restoring degraded underwater images as described in any one of claims 1-2.

4. An electronic device, characterized in that, Includes the computer-readable storage medium as described in claim 3.

Images