Underwater degraded image restoration method, storage medium and electronic equipment

Abstract

The invention provides a method for repairing an underwater degraded image. The method comprises the following steps: S1, inputting the underwater degraded image; s2, constructing an unsupervised deep learning image reconstruction network based on a physical model; s3, constructing a reconstruction network consistency verification module according to an underwater image imaging physical model and the network output reconstruction degraded image; s4, according to an image equivariant theory, constructing a fidelity image network equivariant loss and an underwater imaging equivariant loss; s5, according to the image isomorphism, constructing a fidelity image to restore an image isomorphism verification module; s6, according to the noise distribution of the underwater degraded image, constructing noise loss by using an unbiased estimation method, and correcting the deviation of reconstruction loss; s7, according to a gray world hypothesis, compensating the information of the blue channel and the green channel with relatively small color weakness to the red channel, and constructing a color fidelity verification module; and S8, carrying out constraint training on the network according to the steps S3-S7 until loss convergence, and obtaining a fidelity image estimation module for repairing the degraded image. The invention further provides a storage medium and electronic equipment.

Description

Technical Field

[0001] The present application relates to the technical field of underwater image restoration, and in particular to a restoration method, storage medium and electronic device for underwater degraded images. Background Art

[0002] Underwater imaging is crucial in the development of ocean-oriented services such as marine ecological assessment, fishing net inspection, automatic feeding systems, and underwater autonomous navigation. The quality of underwater imaging is easily affected by challenges such as insufficient natural light and water turbidity, which can cause color shift, blurring, and reduced contrast. In particular, short-wavelength light suffers from significant attenuation in underwater environments, causing underwater images to exhibit a dominant green or blue hue. Therefore, underwater image restoration becomes particularly important in order to generate high-quality images in subsequent vision tasks such as object detection and biological assessment.

[0003] Traditional underwater image restoration methods rely on simulating physical processes to reduce blur and enhance image contrast. Commonly used physical models for underwater imaging propose to decompose the degraded image into three components, namely, transmission map, scene radiance, and global background light. However, due to the inherent rapid changes in the underwater environment, the propagation of light in seawater may deviate from the fixed-parameter physical model, which may lead to distortions, especially under complex lighting conditions.

[0004] By leveraging physical models, deep neural networks can effectively reconstruct distorted images and eliminate color shifts. Existing deep models use supervised learning models and rely on pairing degraded and clear images for training. However, accurately modeling the degradation process and noise in underwater scenes remains challenging, as there are unpredictable differences between synthetic and real data, so supervised learning is prone to bias and overfitting problems. Summary of the invention

[0005] In order to solve the technical problem in the background, the present invention provides a method for repairing an underwater degraded image, a storage medium and an electronic device.

[0006] The present invention provides a method for repairing an underwater degraded image, comprising the following steps:

[0007] S1: input underwater degraded image;

[0008] S2: Construct an unsupervised deep learning image reconstruction network based on physical models;

[0009] S3: reconstruct the degraded image according to the underwater image imaging physical model and the network output described in S2, and build a reconstruction network consistency verification module;

[0010] S4: Based on the image equivariant theory, we construct the equivariant loss of the fidelity image network and the equivariant loss of underwater imaging;

[0011] S5: Based on the image isomorphism, a fidelity image restoration image isomorphism verification module is constructed;

[0012] S6: According to the noise distribution of underwater degraded images, the noise loss is constructed using an unbiased estimation method to correct the deviation of the reconstruction loss;

[0013] S7: Based on the grayscale world hypothesis, the blue channel and green channel information with less color attenuation are compensated to the red channel to build a color fidelity verification module;

[0014] S8: Train the network described in S2 according to the constraints of S3-S7 until the loss converges, and obtain the S2 fidelity image estimation module for repairing the degraded image.

[0015] In some of the embodiments: the physical model-based unsupervised deep learning image reconstruction network described in S2 is composed of a transmission map estimation network, a fidelity map estimation network, and a background light estimation module;

[0016] The network construction formula is: Among them, I θ is the reconstructed degraded image, S θ To estimate the network faithfully, T θ For the transmission graph estimation network, θ S and θ T are the learnable parameters of the fidelity image prediction network and , A is the global background light of the image, x d is the degraded image of the input.

[0017] In some embodiments: the specific method of S3 is: according to the underwater image imaging formula, the error between the reconstructed image and the input image is corrected using the mean square error, and the consistency check module of the reconstruction network is maintained. The specific optimization objective function is: Among them, L DC (·) represents the reconstruction loss. The optimization goal is to minimize the reconstruction loss, that is, the reconstructed image I should be consistent with the input degraded image x d To keep it consistent, ||·|| represents the L2 norm.

[0018] In some of the embodiments: the specific method of S4 is: according to the constraint that the result of image restoration after image translation and rotation transformation should be consistent with the result obtained after image restoration and then performing the same translation and rotation operations, construct a fidelity image network equivariant loss Where L SEL (·) represents the equivariant loss of the fidelity image network, and T on the right side of the equation g Indicates a translation or rotation transformation of the image, Gn represents the nth translation or rotation transformation; and the equivariant loss of the underwater imaging physical network; and the equivariant loss of the underwater imaging physical network Where L OEL (·) represents the equivariant loss of the imaging physical network. According to the equivariant consistency of the fidelity graph and the equivariant constraints of the underwater imaging legend network, an equivariant consistency module is constructed, and the L2 norm is used to ensure the equivariant consistency.

[0019] According to the above-mentioned equivariant consistency of the fidelity graph and the equivariant constraints of the underwater imaging legend network, an equivariant consistency module is constructed, and the L2 norm is used to ensure equivariant consistency.

[0020] In some of the embodiments: the result of restoration based on the input degraded image in S5 should have the same properties as the result of restoration based on the input mixed restoration image:

[0021] Constructing the image isomorphism verification module for restoring the fidelity image;

[0022] The mixed image is a mixture of the repaired image and the original degraded image in a random ratio, and the ratio is taken from a uniform probability distribution of [0,1] each time.

[0023] In some embodiments: in S6, an unbiased estimation and reconstruction consistency constraint is constructed based on Gaussian noise under natural light illumination:

[0024]

[0025] Where σ is the variance of Gaussian noise, ranging from 0.01 to 0.1, m is the number of pixels in a single image, and τ is the weight parameter, which is set to 0.02;

[0026] Under artificial light, an unbiased estimate based on Poisson noise is constructed to reconstruct the consistency constraint: γ>0 is used to control the scale of Poisson noise. The value of τ is consistent with the unbiased estimate constructed with Gaussian noise. The vector b is a Bernoulli distribution, with a probability of 50% to take the value of 0 or 1. T It means taking the transpose of b;

[0027] Under mixed lighting including natural light and artificial light, the unbiased estimate based on Gaussian-Poisson noise is used to reconstruct the consistency constraint:

[0028] The ⊙ on the right side of the equation represents the hadamard product of the vector or matrix, σ is the variance of the Gaussian noise, I is a unit matrix with all 1s on the main diagonal and all 0s on the rest, vectors b and c are Bernoulli distributions, γ>0 is used to control the scale of the Poisson noise, taking the values ​​of 0 or 1 with a probability of 50%, and the weight coefficient τ is 0.02.

[0029] In some of the embodiments: in S7, the grayscale of the image is directly assumed according to the natural scene light field, and the color distribution histogram of the RGB three-channel image should satisfy the distribution theory with a mean of 0.5, and the color fidelity loss for training is constructed:

[0030] Among them, L C represents the color fidelity loss, c represents the color channel Ω∈{R,G,B} which are red, green and blue channels respectively, ||·|| represents the L2 norm, and u||·|| represents the mean.

[0031] In some embodiments: the specific method of S8 is:

[0032] Collect degraded underwater images in RGB format;

[0033] The degraded underwater image is cropped into small blocks of 256*256 pixels in width and height, and the cropped area is generated using a random function;

[0034] The cropped small image is input into the unsupervised reconstruction network for learning. The unsupervised learning strategy loss function is defined as:

[0035]

[0036] The first term of the loss function is L SURE is the unsupervised reconstruction loss of the specific noise, and the loss weight α S The coefficient is 1, and the second term L EI is the equivariant verification loss, and the loss weight coefficient α E is 0.001, the third term L H is the isomorphism verification loss, and the loss weight coefficient α H is 1, the fourth term L C is the color fidelity loss, and the loss weight coefficient is α C 0.01; the learnable parameters of the network are expressed as , the learning optimizer uses AdamW, the network training iterations are 200, the initial learning rate is 5e-4, and the learning rate is reduced to 0.5 times the current learning rate every 50 training rounds.

[0037] The present invention also provides a computer-readable storage medium storing instructions for executing the above-mentioned method for repairing underwater degraded images.

[0038] The present invention provides an electronic device, comprising the above-mentioned computer-readable storage medium

[0039] This application includes the following benefits:

[0040] The present invention proposes an unsupervised underwater image restoration method based on image equivariance and unbiased estimation method. Compared with the existing underwater image restoration method, the main advantage of the present invention is that the image restoration module obtained through training can truly restore the degraded underwater image without relying on labeled data. By constructing an image equivariance verification module, a reconstruction consistency module based on noise prior, an image isomorphism verification module and a color fidelity module, the visual fidelity of image restoration is improved, thereby removing the noise interference of the underwater degraded image and correcting the color deviation of the degraded image, thereby improving the visual effect of the underwater image. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A flowchart of an unsupervised underwater image restoration based on image equivariance and unbiased estimation method provided by an embodiment of the present invention;

[0042] Figure 2 It is a network training flow chart provided by an embodiment of the present invention.

[0043] Figure 3 It is a network reasoning flow chart provided by an embodiment of the present invention.

[0044] Figure 4 An image restoration comparison diagram is provided by an embodiment of the present invention.

[0045] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] Reference Figure 1-5 This embodiment provides a method for restoring an underwater degraded image, comprising the following steps:

[0048] S1: input underwater degraded image;

[0049] S2: Construct an unsupervised deep learning image reconstruction network based on physical models;

[0050] S3: reconstruct the degraded image according to the underwater image imaging physical model and the network output described in S2, and build a reconstruction network consistency verification module;

[0051] S4: Based on the image equivariant theory, we construct the equivariant loss of the fidelity image network and the equivariant loss of underwater imaging;

[0052] S5: Based on the image isomorphism, a fidelity image restoration image isomorphism verification module is constructed;

[0053] S6: According to the noise distribution of underwater degraded images, the noise loss is constructed using an unbiased estimation method to correct the deviation of the reconstruction loss;

[0054] S7: Based on the grayscale world hypothesis, the blue channel and green channel information with less color attenuation are compensated to the red channel to build a color fidelity verification module;

[0055] S8: Train the network described in S2 according to the constraints of S3-S7 until the loss converges, and obtain the S2 fidelity image estimation module for repairing the degraded image.

[0056] In this embodiment, S2 specifically includes:

[0057] The unsupervised deep learning image reconstruction network based on the physical model consists of a transmission image estimation network, a fidelity image estimation network, and a background light estimation module;

[0058] The network construction formula is: Among them, I θ is the reconstructed degraded image, S θ To estimate the network faithfully, T θ For the transmission graph estimation network, θ S and θ T are the learnable parameters of the fidelity image prediction network and , A is the global background light of the image, x d is the degraded image of the input.

[0059] In this embodiment, the specific method of S3 is:

[0060] According to the underwater image imaging formula, the mean square error is used to correct the error between the reconstructed image and the input image, and the consistency check module of the reconstruction network is maintained. The specific optimization objective function is: Among them, L DC (·) represents the reconstruction loss. The optimization goal is to minimize the reconstruction loss, that is, the reconstructed image I should be consistent with the input degraded image x d To keep it consistent, ||·|| represents the L2 norm.

[0061] In this embodiment, the specific method of S4 is:

[0062] According to the constraint that the result of image restoration after translation and rotation transformation should be consistent with the result obtained by first performing image restoration and then performing the same translation and rotation operations, the equivariant loss of the fidelity image network is constructed:

[0063]

[0064] Where L SEL (·) represents the equivariant loss of the fidelity image network, and T on the right side of the equation g Indicates a translation or rotation transformation of the image, G n represents the nth translation or rotation transformation; and the equivariant loss of the underwater imaging physical network; and the equivariant loss of the underwater imaging physical network:

[0065] Where L OEL (·) represents the equivariant loss of the imaging physical network. According to the equivariant consistency of the above-mentioned fidelity graph and the equivariant constraints of the underwater imaging legend network, an equivariant consistency module is constructed, and the L2 norm is used to ensure equivariant consistency. ;

[0066] According to the equivariant consistency of the above-mentioned fidelity map and the equivariant constraints of the underwater imaging legend network, an equivariant consistency module is constructed, and the L2 norm is used to ensure equivariant consistency. The equivariant operation of the image includes translating the width and height of the image in any direction by a distance equal to the width and height of the image itself, and splicing the part exceeding the image size to the blank area; the angle of the rotation operation is {90°, 180°, 270 degrees}. Each time the image is transformed, one operation is selected from the translation and rotation operations with equal probability.

[0067] In this embodiment, S5 specifically includes:

[0068] The result of the restoration based on the input degraded image should have the same properties as the result of the restoration based on the input mixed restoration image:

[0069] Constructing the image isomorphism verification module for restoring the fidelity image;

[0070] The mixup(·) mixed image is a mixture of the repaired image and the original degraded image in a random ratio, and the ratio is taken from a uniform probability distribution in [0,1] each time.

[0071] On the basis of the unsupervised reconstruction network, according to the unbiased denoising method, Gaussian noise, Poisson noise and Gaussian-Poisson mixed noise are used as prior noise knowledge to further correct the reconstruction consistency constraint. In this embodiment, S6 specifically includes:

[0072] Under natural light, the noise points of the degraded image are closer to Gaussian noise. Based on Gaussian noise, an unbiased estimation and reconstruction consistency constraint is constructed:

[0073]

[0074] The variance of σ Gaussian noise ranges from 0.01 to 0.1, m is the number of pixels in a single image, τ is the weight parameter, which is set to 0.02, and vector b is a Bernoulli distribution, which takes the value of 0 or 1 with a probability of 50%. T It means taking the transpose of b;

[0075] There are suspended particles in turbid waters. In the imaging process with artificial light as the background, snowflake-like white noise will be generated. The removal of this white noise can be based on the Poisson noise prior and the unbiased estimation and reconstruction consistency constraints can be constructed based on the Poisson noise:

[0076] γ>0 is used to control the scale of Poisson noise, the value of τ is consistent with the unbiased estimate constructed by Gaussian noise, and the vector b is a Bernoulli distribution, taking the value of 0 or 1 with a probability of 50%;

[0077] Under mixed lighting including natural light and artificial light, using a prior containing both Gaussian noise and Poisson noise, an unbiased estimate based on Gaussian-Poisson noise is constructed to reconstruct the consistency constraint:

[0078] The ⊙ on the right side of the equation represents the hadamard product of a vector or matrix, σ is the variance of Gaussian noise, I is a unit matrix with all 1s on the main diagonal and all 0s on the rest, vectors b and c are Bernoulli distributions, γ>0 is used to control the scale of Poisson noise, taking values ​​of 0 or 1 with a probability of 50%, and the weight coefficient τ is 0.02.

[0079] In this embodiment, S7 specifically includes:

[0080] According to the direct assumption of grayscale of natural scene light field generated images, the color distribution histogram of RGB three-channel images should satisfy the distribution theory with a mean of 0.5, and the color fidelity loss for training is constructed:

[0081]

[0082] Among them, ||·|| represents the L2 norm, u||·|| represents the mean. The above formula can constrain the mean of the color distribution of the RGB three-channel image to be 0.5, guide the unsupervised network to use the information of the blue and green channels to compensate for the information of the red channel, and improve the color fidelity of the image.

[0083] The CNN reconstruction network consists of an image fidelity network, a transmission network and a background light estimation module. The image fidelity module has the same network design as the transmission image module, and contains five convolution groups, each of which contains a mirror completion module, a convolution module with a convolution kernel of 3 and a step size of 1, a ReLU activation function, and a LayerNorm normalization module. The number of input channels of the first convolution group and the number of output channels of the last convolution group are 3, and the number of convolution input and output channels of the remaining convolution groups is 64. After five convolution groups, according to the value constraints of the transmission image and the fidelity image pixels, the Sigmoid activation function is used to limit the output result to between 0 and 1. The background light estimation module is obtained using Gaussian blur.

[0084] In this embodiment, the specific method of S8 is:

[0085] Collect degraded underwater images in RGB format;

[0086] The degraded underwater image is cropped into small blocks of 256*256 pixels in width and height, and the cropped area is generated using a random function;

[0087] The cropped small image is input into the unsupervised reconstruction network for learning. The unsupervised learning strategy loss function is defined as:

[0088]

[0089] The first term of the loss function is L SURE is the unsupervised reconstruction loss of the specific noise, and the loss weight α S The coefficient is 1, and the second term L EI is the equivariant verification loss, and the loss weight coefficient α E is 0.001, the third term L H is the isomorphism verification loss, and the loss weight coefficient α H is 1, the fourth term L C is the color fidelity loss, and the loss weight coefficient is α C 0.01; the learnable parameters of the network are expressed as , the learning optimizer uses AdamW, the network training iterations are 200, the initial learning rate is 5e-4, and the learning rate is reduced to 0.5 times the current learning rate every 50 training rounds.

[0090] like Figure 4 As shown, from left to right are the original degraded image, the existing methods Waternet, Semi-UIR, USUIR, the unsupervised image restoration method proposed in this embodiment, and the reference image. The unsupervised image restoration method proposed in this embodiment has higher color fidelity.

[0091] like Figure 5As shown, the electronic device 500 includes one or more processors 501 and one or more memories 502, wherein the memory 502 stores instructions, and the instructions are loaded and executed by the processor 501 to implement the above-mentioned unsupervised underwater image restoration steps based on image equivariance and unbiased estimation method.

[0092] The above are all preferred embodiments of the present application, and the protection scope of the present application is not limited thereto. Therefore, any equivalent changes made according to the structure, shape, and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for restoring an underwater degraded image, characterized in that: The following steps are involved: S1: input underwater degraded image; S2: Construct an unsupervised deep learning image reconstruction network based on physical models; S3: Reconstruct the degraded image according to the underwater image imaging physical model and network output, and build a reconstruction network consistency verification module; S4: Based on the image equivariant theory, we construct the equivariant loss of the fidelity image network and the equivariant loss of underwater imaging; S5: Based on the image isomorphism, a fidelity image restoration image isomorphism verification module is constructed; S6: According to the noise distribution of underwater degraded images, the noise loss is constructed using an unbiased estimation method to correct the deviation of the reconstruction loss; S7: Based on the grayscale world hypothesis, the blue channel and green channel information with less color attenuation are compensated to the red channel to build a color fidelity verification module; S8: Train the network according to S3-S7 constraints until the loss converges, and obtain a fidelity image estimation module for repairing degraded images.

2. The method for restoring an underwater degraded image according to claim 1, characterized in that: The physical model-based unsupervised deep learning image reconstruction network described in S2 is composed of a transmission map estimation network, a fidelity map estimation network, and a background light estimation module; The network construction formula is: Among them, I θ is the reconstructed degraded image, S θ To estimate the network faithfully, T θ For the transmission graph estimation network, θ S and θ T are the learnable parameters of the fidelity image prediction network and respectively, A is the global background light of the image, x d is the degraded image of the input.

3. The method for restoring an underwater degraded image according to claim 2, characterized in that: The specific method of S3 is: according to the underwater image imaging formula, the mean square error is used to correct the error between the reconstructed image and the input image, and the consistency check module of the reconstruction network is maintained. The specific optimization objective function is: Among them, L DC (·) represents the reconstruction loss. The optimization goal is to minimize the reconstruction loss, that is, the reconstructed image I should be consistent with the input degraded image x d To keep it consistent, ||·|| represents the L2 norm.

4. The method for repairing underwater degraded images according to claim 3, characterized in that: The specific method of S4 is: according to the constraint that the result of image restoration after translation and rotation transformation should be consistent with the result obtained after image restoration and then the same translation and rotation operations, a fidelity image network equivariant loss is constructed. Where L SEL (·) represents the equivariant loss of the fidelity image network, and T on the right side of the equation g Indicates a translation or rotation transformation of the image, G n represents the nth translation or rotation transformation; and the equivariant loss of the underwater imaging physical network Where L OEL (·) represents the equivariant loss of the imaging physical network. According to the equivariant consistency of the fidelity graph and the equivariant constraints of the underwater imaging legend network, an equivariant consistency module is constructed, and the L2 norm is used to ensure the equivariant consistency.

5. The method for restoring an underwater degraded image according to claim 4, characterized in that: The result of restoration based on the input degraded image in S5 should be consistent with the result of restoration based on the input mixed restoration image, that is, Constructing the image isomorphism verification module for restoring the fidelity image; The mixed image is a mixture of the repaired image and the original degraded image in a random ratio, and the ratio is taken from a uniform probability distribution of [0,1] each time.

6. The method for repairing underwater degraded images according to claim 5, characterized in that: In S6, an unbiased estimation and reconstruction consistency constraint is constructed based on Gaussian noise under natural light: The variance of σ Gaussian noise ranges from 0.01 to 0.1, m is the number of pixels in a single image, τ is the weight parameter, which is set to 0.02, and vector b is a Bernoulli distribution, which takes the value of 0 or 1 with a probability of 50%. T It means taking the transpose of b; Under artificial light, an unbiased estimate based on Poisson noise is constructed to reconstruct the consistency constraint: γ>0 is used to control the scale of Poisson noise, the value of τ is consistent with the unbiased estimate constructed by Gaussian noise, the vector b is a Bernoulli distribution, taking the value of 0 or 1 with a probability of 50%, and ⊙ represents the hadamard product of a vector or matrix; Under mixed lighting including natural light and artificial light, the unbiased estimate based on Gaussian-Poisson noise is used to reconstruct the consistency constraint: The ⊙ on the right side of the equation represents the hadamard product of the vector or matrix, σ is the variance of Gaussian noise, I is a unit matrix with all 1s on the main diagonal and all 0s on the rest, vectors b and c are Bernoulli distributed, taking values ​​of 0 or 1 with a probability of 50%, and the weight coefficient τ is 0.

02.

7. The method for restoring an underwater degraded image according to claim 6, characterized in that: In S7, the grayscale of the image generated by the natural scene light field is directly assumed. The color distribution histogram of the RGB three-channel image should satisfy the distribution theory with a mean of 0.5, and the color fidelity loss for training is constructed: Among them, L C represents the color fidelity loss, c represents the color channel Ω∈{R,G,B} which are red, green and blue channels respectively, ||·|| represents the L2 norm, and u||·|| represents the mean.

8. The method for restoring an underwater degraded image according to claim 7, characterized in that: The specific method of S8 is: Collect degraded underwater images in RGB format; The degraded underwater image is cropped into small blocks of 256*256 pixels in width and height, and the cropped area is generated using a random function; The cropped small image is input into the unsupervised reconstruction network for learning. The unsupervised learning strategy loss function is defined as: The first term of the loss function is L SURE is the unsupervised reconstruction loss of the specific noise, and the loss weight α S The coefficient is 1, and the second term L EI is the equivariant verification loss, and the loss weight coefficient α E is 0.001, the third term L H is the isomorphism verification loss, and the loss weight coefficient α H is 1, the fourth term L C is the color fidelity loss, and the loss weight coefficient is α C 0.01; the learnable parameters of the network are expressed as , the learning optimizer uses AdamW, the network training iterations are 200, the initial learning rate is 5e-4, and the learning rate is reduced to 0.5 times the current learning rate every 50 training rounds.

9. A computer-readable storage medium, characterized in that: The device stores instructions for executing the method for restoring an underwater degraded image as claimed in any one of claims 1 to 8.

10. An electronic device, characterized in that: Includes the computer-readable storage medium of claim 9.

Images