Image restoration methods, model training methods, chips, media, software products, GPU servers

Abstract

This invention relates to the field of GPU-based image processors in artificial intelligence hardware platforms, and particularly to an image inpainting method, a model training method, a chip, a medium, a program product, and a GPU server. The image inpainting model training method of this invention includes: Step A, forward noise addition, including: adding noise to the original image I... raw And the reference image I after repair ref The process involves: performing image synthesis; progressively introducing Gaussian noise into the synthesized image in N steps; step B, reverse denoising training, including: M steps of denoising training, wherein for the m-th denoising training step, it includes: noise estimation using a noise estimation neural network; noise removal; wherein the loss function L of the noise estimation neural network includes: the difference L1 between the mean of the estimated noise obtained in the denoising training steps and the mean of the true noise. This invention can adapt to imaging conditions in specific environments, and the denoising effect is greatly improved.

Description

Technical Field

[0001] This invention relates to the field of GPU branch image processors in artificial intelligence hardware platforms, and in particular to an image restoration method, a model training method, a chip, a medium, a program product, and a GPU server. Background Technology

[0002] Underwater environmental information collection, especially in optical imaging, presents a series of unique and complex challenges. These challenges primarily stem from the drastically different physical characteristics of the underwater world compared to terrestrial imaging environments. Furthermore, particles and plankton in the water scatter passing light, altering the direction of light captured during imaging. This leads to image blurring, loss of detail, reduced contrast, and chromatic aberration further exacerbating color distortion. These problems are even more severe in deep-sea environments such as the ocean or deep lakes. Since sunlight struggles to penetrate deep water, these areas are almost entirely dark. In such environments, visible light primarily comes from specific bioluminescent phenomena, such as those of fluorescent jellyfish and certain deep-sea fish. However, the illumination provided by these bioluminescent organisms is extremely limited, far from sufficient to illuminate the surrounding environment, resulting in extremely low visibility in deep-sea imaging. This extremely low illumination, coupled with the influence of water's physical properties on light, severely limits the application of underwater imaging technology, making the acquisition of high-quality underwater images exceptionally difficult.

[0003] Underwater image enhancement technology is a key technique for improving the visual quality of images. These techniques aim to process images degraded by the unique optical properties of underwater environments (such as light absorption, scattering, and dispersion) to improve image usability and information content. In traditional underwater image enhancement methods, methods based on underwater physical models occupy an important position. Although these methods are theoretically sound, they face significant challenges in practical applications. These methods rely on accurate estimations of parameters such as underwater depth and light propagation coefficients, which are often difficult to obtain precisely in real-world environments. Furthermore, due to the complexity and variability of the marine environment, methods based on simplified physical models may not fully capture all the characteristics of underwater imaging, resulting in a lack of robustness in the enhancement results and sometimes even producing unnatural visual effects. Summary of the Invention

[0004] I. Technical problems to be solved

[0005] The present invention aims to at least partially solve one of the above-mentioned technical problems.

[0006] II. Technical Solution

[0007] The first aspect of this invention provides a model training method for image inpainting. The model training method includes:

[0008] Step A, forward noise addition, includes: adding noise to the original image I raw And the reference image I after repair ref The composite image is obtained by performing the synthesis. I 0: I0 = f(I raw I ref f is the synthesis function; Gaussian noise is gradually introduced into the synthesized image I0 in N steps to obtain the noisy image I. N N≥100;

[0009] Step B, reverse denoising training, includes: M denoising training steps, wherein the m-th denoising training step includes: performing noise estimation using a noise estimation neural network. Among them, F θ For a neural network that estimates noise, X m For the noisy image in the current step; I ref For reference image, The noise is estimated from the regression output of the noise estimation neural network; noise removal is then performed to obtain the denoised image. Where m = MM-1, ..., 1, M ≥ 100, M ≤ N, X M =I N If the M-step denoising training is not completed, X m-1 As input for the next denoising training step; after the M-step denoising training step is completed, the denoised image X0 is obtained; wherein, in the denoising training step, the noise estimation neural network is trained by backpropagation, and the loss function L of the noise estimation neural network includes: the difference L1 between the mean of the estimated noise obtained in the denoising training step and the mean of the real noise.

[0010] A second aspect of the present invention provides an image restoration method. The image restoration method includes:

[0011] Obtain the image to be repaired;

[0012] The image to be repaired is combined with the Gaussian noise image to obtain a composite image;

[0013] Denoising the synthesized image includes an S-step denoising process.

[0014] The denoising step s-th step includes: using a noise estimation neural network to estimate noise. Among them, F θ To obtain the noise estimation neural network through the above model training method, X s The noisy image in the current step; The noise is estimated from the regression output of the noise estimation neural network; noise removal is then performed to obtain the denoised image. Where s = S, S-1, ..., 1, S≥100, and X is the value of X before the denoising process is completed. s+1 As input for the next denoising process, after the denoising process is completed, the denoised image X0 is obtained.

[0015] A third aspect of the present invention provides a computer chip. This computer chip is configured to implement: the image inpainting model training method as described above; or, the image inpainting method as described above.

[0016] A fourth aspect of the present invention provides a computer-readable storage medium. The computer-readable storage medium stores a computer program that, when executed by a processor, implements: the image inpainting model training method as described above; or, an image inpainting method.

[0017] A fifth aspect of the present invention provides a computer program product. The computer program product includes: a computer program, wherein, when executed by a processor, the computer program implements: the image inpainting model training method as described above; or, an image inpainting method.

[0018] A sixth aspect of the present invention provides a GPU server. The GPU server includes: a processor; a memory storing a computer program thereon; wherein, when the processor executes the computer program, it implements: the image inpainting model training method as described above; or, an image inpainting method.

[0019] III. Beneficial Effects

[0020] As can be seen from the above technical solution, the present invention has at least one of the following beneficial effects compared to the prior art:

[0021] 1. A diffusion model is introduced, which optimizes the model parameters to adapt to the imaging conditions of a specific environment through a supervised learning strategy, thereby greatly improving the denoising effect.

[0022] 2. In the reverse denoising step, the color correction module uses a deep learning network to accurately correct the color of each pixel, ensuring that the final output image fully reflects the colors of the original scene. Attached Figure Description

[0023] Figure 1 This is a flowchart of the image restoration model training method according to an embodiment of the present invention.

[0024] Figure 2A for Figure 1 The diagram illustrates the image changes during forward noise addition and backward noise reduction training in the image restoration model training method shown.

[0025] Figure 2B for Figure 1The diagram illustrates the backward denoising training in the image restoration model training method shown.

[0026] Figure 2C for Figure 1 The diagram illustrates noise estimation during backward denoising training in the image inpainting model training method.

[0027] Figure 3 for Figure 2B The diagram shows the color correction steps in the backward denoising training.

[0028] Figure 4 This image compares the quality of image denoising using different methods under different conditions on the UIEB underwater image enhancement dataset.

[0029] Figure 5 This is a schematic diagram of a computer chip according to an embodiment of the present invention.

[0030] Figure 6 This is a schematic diagram of a GPU server according to an embodiment of the present invention. Detailed Implementation

[0031] This invention proposes a model training method and an image inpainting method. It introduces a diffusion model and optimizes the model parameters to adapt to the imaging conditions of a specific environment through a supervised learning strategy. Based on this, this invention also designs a color correction module, which uses a deep learning network to accurately correct the color of each pixel, ensuring that the final output image fully reflects the colors of the original scene.

[0032] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0033] The first aspect of this invention provides a model training method for image restoration. Figure 1 This is a flowchart of the image restoration model training method according to an embodiment of the present invention. Figure 2A for Figure 1 The diagram illustrates the image changes during forward noise addition and backward noise reduction training in the image restoration model training method shown. Figure 2B for Figure 1 The diagram illustrates the backward denoising training in the image restoration model training method shown. Figure 2C for Figure 1 The diagram illustrates noise estimation during backward denoising training in the image inpainting model training method.

[0034] like Figure 1 , Figure 2A , Figure 2B , Figure 2C As shown, the image restoration model training method of this invention includes:

[0035] Step A, forward noise addition, includes:

[0036] Sub-step A0, Image Composition: Combine the original image I raw And the reference image I after repair ref The composite image I0 is obtained by performing synthesis: I0 = f((I raw I ref ), where f is the composition function;

[0037] Sub-steps A1 to A N Image noise is added by introducing Gaussian noise into the synthesized image in N steps to obtain a noisy image iN, where N≥100;

[0038] Step B, reverse denoising training, includes:

[0039] Sub-steps B1 to B M The M-th step of the denoising training process includes:

[0040] Noise estimation is performed using a noise estimation neural network. Among them, F θ For a neural network that estimates noise, X m For the noisy image in the current step; I ref For reference image, To estimate the noise in the regression output of a neural network for noise estimation;

[0041] Noise removal yields denoised images:

[0042] Where m = M, M-1, ..., 1, M ≥ 100, M ≤ N, X M =I N Before the reverse denoising training is completed, X m-1 This serves as the input for the next denoising training step; after the M-step denoising training process, the denoised image X0 is obtained.

[0043] In the denoising training step, the noise estimation neural network is trained through backpropagation. The loss function L of the noise estimation neural network includes the difference L1 between the mean of the estimated noise obtained in the denoising training step and the mean of the real noise.

[0044] The following provides a detailed description of the model training method for image restoration in this embodiment.

[0045] In this invention, in sub-step A0, the original image and the reference image are obtained through one of the following methods: ① Obtaining the original image from an underwater image enhancement training dataset such as UIEB or EUVP. raw and reference image I ref ②Own original image I raw The image image I is obtained after processing by the image enhancement algorithm. ref Among them, image enhancement algorithms are, for example, generative adversarial networks or processing algorithms based on convolutional neural networks.

[0046] In this embodiment, in sub-step A0, data input (InputData) is performed by transferring the raw images (RawImages) I. raw And reference images (Conditional Images) ref Linear synthesis is performed to obtain the synthesized image I0; I0 = I raw +αI ref α is a linear coefficient: 0.5 < α < 2. Preferably, α = 1.

[0047] Those skilled in the art should understand that the above image synthesis methods are merely examples. In other embodiments of the present invention, linear coefficients can be adjusted, or other nonlinear synthesis methods existing in the prior art, such as quadratic, cubic polynomial, and weight distribution-based image synthesis, can be used to achieve the present invention and are all within the protection scope of the present invention.

[0048] In this embodiment, in the step of introducing Gaussian noise in the nth step of synthesizing the image, Gaussian noise is introduced linearly, I n =I n-1 +N n N n Let n be the Gaussian noise introduced in step n, where n = 1, 2, ..., N.

[0049] In particular, for the first noise introduction step, I1 = I0 + N1, where I0 = I raw +I ref I raw N is the original image; N1 is the noise introduced in the first noise introduction step.

[0050] Wherein, noise N n The noise is randomly sampled from a Gaussian-distributed noise signal by an algorithm. Due to the randomness of the algorithm, the distribution of the noise added each time will differ. For details on the noise acquisition process, please refer to relevant existing technical descriptions; they will not be repeated here.

[0051] Similarly, those skilled in the art should understand that the above methods of introducing Gaussian noise are merely examples. In other embodiments of the present invention, the linear coefficient can be adjusted, or other nonlinear methods can be used to introduce Gaussian noise. All of these methods can achieve the present invention and are within the scope of protection of the present invention.

[0052] In this embodiment, N = 5000, but the present invention is not limited thereto. In other embodiments of the present invention, the present invention can be implemented as long as N ≥ 100, preferably N ≥ 2000.

[0053] This concludes the introduction to forward noise addition in this invention. As can be seen, in forward noise addition, since the noise addition process can be viewed as a linear transformation, the model does not need to learn any parameters. Furthermore, the final generated noisy image I... N If there are no corresponding original image features, they are sent to the inverse denoising training module for processing.

[0054] As mentioned earlier, the reverse denoising training includes M denoising training steps. Specifically, the m-th denoising training step includes:

[0055] Noise estimation is performed using a noise estimation neural network. Among them, F θ For a neural network that estimates noise, X m For the noisy image in the current step; I ref For reference image, To estimate the noise in the regression output of a neural network for noise estimation;

[0056] Noise removal yields the denoised image:

[0057] Where m = M, M-1, ..., 1, M ≥ 100, X M =I N ,

[0058] like Figure 2B and Figure 2CAs shown, since the noise addition process is treated as a linear transformation, the added noise in the noisy image can be gradually estimated during backward denoising training. The network regression output of the noise estimation neural network at each step is the noise added in the current step during forward denoising, compared to the previous step. During training (TrainingAddNoiseProcess / NoiseEstimation Network), the noise estimation neural network primarily learns the ability to predict noise from each step, gradually refining the denoising model through backpropagation by calculating the distance (loss) between the predicted noise and the actual noise. The following aspects require special explanation regarding backward denoising training:

[0059] (1) Types of noise estimation neural networks

[0060] In the reverse denoising training of this embodiment, U was used Net Neural networks. The following discusses U... Net The neural network will be explained in detail. Net The basic structure of a neural network includes an encoder, a decoder, and the jump paths connecting the two. The encoder extracts features from the input image through downsampling and convolutional layers, progressively reducing the image size and increasing the channel depth. These features then undergo a series of processing and upsampling operations, gradually restoring the original image size in the decoder.

[0061] The main task of the encoder is to extract useful features from the input data. It consists of multiple alternating convolutional and pooling layers, each layer reducing spatial dimensionality while increasing feature dimensionality. Specifically, the input image first passes through convolutional layers to extract features, and then is downsampled through pooling layers. This structure allows the network to capture multi-scale features from shallow to deep layers.

[0062] The decoder's structure is the opposite of the encoder's, primarily using upsampling and convolution operations to restore the encoder's output features to the original image size. In the denoising diffusion model, the decoder's goal is to predict noise and generate a clean target image. Upsampling can be achieved through deconvolution, which effectively transforms low-resolution feature maps into high-resolution outputs.

[0063] Skip connections directly connect the encoder's feature maps to the corresponding decoder layers. This design allows the network to consider both low-level detail and high-level semantic information simultaneously, thereby improving the quality of image generation. In diffusion models, these connections help the model more effectively utilize features from different levels for accurate noise prediction and removal.

[0064] About U NetFurther information on neural networks can be found in the following two papers:

[0065] 【1】Ho J,JainA,Abbeel P.Denoising diffusion probabilistic models[J].Advances in neural informationprocessingsystems,2020,33:6840-6851.

[0066] 【2】U-Net: Ronneberger O, Fischer P, Brox TU-net: Convolutional networks for biomedical image segmentation[C] / / Medicalimagecomputingandcomputer-assistedintervention–MICCAI2015:18thinternational conference,Munich,Germany,October 5-9,2015,proceedings,partIII 18.Springer InternationalPublishing,2015:234-241.

[0067] Since this embodiment only uses U, which is already available in the prior art. Net Neural networks, therefore the focus is on providing a detailed description of the content related to this invention, U Net The details of neural networks themselves will not be discussed in detail here.

[0068] Those skilled in the art should understand that, U Net The neural network is just an example. In other embodiments of the present invention, other types of neural networks can also be used, such as end-to-end CNN networks, ResNet, codec neural networks, variational autoencoders. Any neural network that meets the condition of having a codec structure can be applied to the present invention, and will not be elaborated here.

[0069] (2) Input of the noise estimation neural network

[0070] For the noise estimation neural network F θ For the m-th reverse denoising step, the input includes: (1) the noisy image X of the current step. m (2) Refer to image I ref (3) Time-coded value m; its output Let be the estimated mean of the noise signal distribution added at time m-1. During training, the network primarily learns the ability to extract noise from each step, i.e., the ability of a progressively denoising model.

[0071] It is particularly important to note that in existing technologies, the input to the noise estimation neural network is: the original image I raw In this invention, such as Figure 2C As shown, the reference image I is after noise reduction processing. ref Compared with the original image I raw The combination of I in In this scenario, the color distribution of the original image can be corrected simultaneously while removing the added Gaussian noise. Additionally, the encoded time step embedding needs to be input so that the network explicitly defines the forward and backward denoising steps (time).

[0072] (3) Loss function of noise estimation neural network

[0073] For U Net The loss function (Loss) L of the neural network for noise estimation includes: the difference L1 between the estimated noise mean obtained in the m-th denoising training step and the true noise mean (represented by KL divergence).

[0074] For technical terms related to image processing, such as noise mean and noise mean difference, please refer to the relevant explanations in existing technologies; further explanation will not be provided here.

[0075] Regarding the mean of true noise, it can be divided into the following two cases:

[0076] ① When M=N, the N steps of introducing Gaussian noise correspond to the N steps of denoising training, and the mean of the true noise is the mean of the Gaussian noise introduced in the corresponding steps of introducing Gaussian noise during the forward noise addition process.

[0077] like Figure 2A As shown, in this embodiment, M = N; M ≥ 2000. The steps in the forward noise addition and backward noise reduction training correspond one-to-one. Therefore, the mean value of the actual noise in each backward noise reduction training exactly corresponds to the mean value of the Gaussian noise introduced in the corresponding step of introducing Gaussian noise in the forward noise addition process. This makes the calculation simpler and the debugging more convenient.

[0078] ② When M < N, the first M steps of introducing Gaussian noise out of the N steps of introducing Gaussian noise are taken and correspond to the M denoising training steps. The mean of the true noise is the mean of the Gaussian noise introduced in the corresponding steps of introducing Gaussian noise in the forward denoising process.

[0079] In this embodiment, M = N, U Net The neural network performs backpropagation based on the aforementioned loss function L1, thereby adjusting the connection weights of each layer of its network to complete the denoising training in the m-th step.

[0080] In this embodiment, the KL divergence represents the difference between the estimated noise mean and the true noise mean. However, those skilled in the art should understand that other representations can be used to represent the difference between the estimated noise mean and the true noise mean, and the present invention can be achieved in the same way.

[0081] (4) Training process of noise estimation neural network

[0082] like Figure 2B As shown, after the noise estimation neural network completes its prediction, the predicted noise is subtracted from the current step. At this point, a denoising training step is complete. Next, the noise estimation neural network will change from 'm' to 'ml'. The forward process remains controllable and linear. The degradation of underwater images primarily relies on the initial input to constrain the entire process of adding and removing noise. Based on this, noisy images can be gradually removed using a distribution estimator. During image denoising, because the input training data are paired (including a reference image and the image to be processed), the training process can be considered as being constrained by the reference image. Given a training output image y, we generate a noisy sample y. e And train a neural network f θ Given a X and a noisy sample, perform regression denoising. This process can be represented as:

[0083] loss = E (x，y)，∈，γ ||f θ (x, y0)-∈||

[0084] y0 is the initial input to the image. f θ The output of represents the estimated image mean obtained by the estimator, and this value is constrained by the reference image. Here, x is the reference image used to train the network. In this optimization process, the estimator does not simply use information from the current and initial input images to estimate the mean distribution over previous steps, as in traditional DDPM. Instead, the reference image is used as conditional information and embedded into the estimator's input vector. This process can be represented as:

[0085]

[0086] In the diffusion denoising process, since the diffusion noise is added randomly, the output image obtained by simply estimating and correcting the noise will have a certain color difference. This color difference is mostly due to the cumulative bias in noise estimation during the reverse denoising process. Therefore, how to handle this cumulative error is a key aspect of the algorithm. To eliminate this accumulated color noise interference, this invention also proposes a pixel-level color correction module based on a channel fusion mechanism to correct the resulting color deviation.

[0087] In this embodiment, the denoising training steps from step 2501 to step 5000 each include:

[0088] Noise estimation is performed using a noise estimation neural network.

[0089] Noise removal

[0090] Color correction: The color correction module is used to correct color deviations.

[0091] Where m = M2, M2-1, ..., 1, X is the value of X before the reverse denoising training is completed. m-1 As input for the next denoising training step; after the reverse denoising training is completed, the denoised image X0 is obtained;

[0092] In the denoising / color correction step, the noise estimation neural network is trained through backpropagation. The backpropagation loss L is: L = L1 + L2, where L1 can be referred to the previous explanation, and L2 is the difference between the color-corrected image and the reference image I. ref The distance between them. The distance can be Euclidean distance, chessboard distance, L2 norm distance, KL divergence, etc.

[0093] Figure 3 for Figure 2B The diagram illustrates the color correction step in the backward denoising training. Figure 3 As shown, in this embodiment, the main function of the color correction step (ColorCastCorrectionModule) is to correct the color deviation generated during diffusion. This step is an image reconstruction step with an encoder-decoder structure. In this module, data processing is first performed to extract the L channel (Lchannel) in the LAB color mode and the R channel (RChannel), G channel (GChannel), and B channel (BChannel) in the RGB color mode of the input image. Then, it is fed into the encoder for compression and dimensionality reduction. The encoder is mainly responsible for compressing the image, extracting color features, and encoding them. The decoder is mainly responsible for restoring the image and correcting the colors.

[0094] Regarding the color correction steps, the following aspects require further explanation:

[0095] (1) Input of color correction steps

[0096] For the color correction step, the input is the L channel in the LAB color mode of the noise-removed image, and the R, G, and B channels in the RGB color mode.

[0097] In the image enhancement tasks involved in this algorithm, using the LAB color model has many advantages. First, the LAB color model consists of a luminance component (L) and two chrominance components (a and b), where luminance represents the brightness of the image and chrominance represents color information. Because luminance and chrominance are represented separately, they allow for more flexible processing.

[0098] The LAB model is better at handling the effects of lighting changes. Because its luminance channel (L) is insensitive to lighting changes, this characteristic can be used to perform operations such as illumination equalization to further improve image quality.

[0099] The module's inputs include the RGB and L channels from the Lab channels. The Lab color model consists of a luminance component L and two chrominance components ab, where L represents the image's luminance, and chrominance represents color information. Because luminance and chrominance are represented separately, they can be handled more flexibly, allowing independent operation on each channel of the Lab model to achieve different image enhancement goals.

[0100] (2) Color Compression Encoder

[0101] Please refer to Figure 3 The color compression encoding takes the L channel, R channel, G channel, and B channel as input, and its specific processing includes: CRP*3 steps, CA steps, and DBC steps.

[0102] The CRP*3 module (convolution-ReLU-pooling module) further includes: a convolution module, a ReLU module, and a pooling module. These three modules are used to perform downsampling convolution processing on the 4-channel image to remove unnecessary texture information and retain color features; non-linear activation is applied to the downsampling convolution image to enhance the expressive power of features; and pooling is applied to the non-linearly activated image to obtain the color-compressed image.

[0103] The CA module (Channel Attention Module) is used to extract attention from the multi-channel image output by the convolution-ReLU-pooling module to find important parts in different channels.

[0104] The DBC module is used to further compress and extract features from the weighted feature map output from the channel attention module. It performs dual-branch convolution on the same feature map using convolution kernels of different scales, and then merges the results of the two branches to enhance the detailed representation of the feature map.

[0105] (3) Deconvolution reconstruction decoding step (Transposed convolution reconstruction decoder)

[0106] Deconvolutional reconstruction decoding is used to reconstruct key color information by taking the feature map output from color compression encoding as input. For example... Figure 3 As shown, in the transposed convolutional reconstruction decoder part, upsampling is performed using trilinear interpolation (TI) and two transposed convolutions (TCR). Simultaneously, an attention-related model in the encoder is connected to the TCR part of the decoder via skip links for attention fusion. The channel attention results are further used to enhance the weights of favorable channels during reconstruction, thereby improving the recovery performance.

[0107] Specifically, in the convolutional reconstruction decoder, the input is the encoded output of the color compression encoder; trilinear interpolation (TI) is used to perform preliminary restoration and reconstruction of the image, and then two deconvolution modules are used to restore the image details and color expression based on the channel weight details provided by channel attention.

[0108] In the color correction step, the decoder outputs a corrected image, which is compared with the reference image (I). ref The L2 distance (the distance between the currently corrected image and the reference image) is calculated and then fed into the NoiseEstimationNetwork. This loss is used in reverse denoising along with the mean estimation loss L1 of the NoiseEstimationNetwork until denoising is complete. This will be explained in detail in the Loss Function section.

[0109] Regarding the color correction steps, the following aspects also need to be explained:

[0110] (1) Color correction is the preferred step.

[0111] In this embodiment, a color correction step is used to correct the accumulated noise reduction error. However, this is only a preferred embodiment of the present invention, which has a better image restoration effect. In other embodiments of the present invention, the color correction step may not be included. Although the effect will be slightly inferior to this embodiment, the present invention can still be achieved.

[0112] (2) The color correction module is an incomplete neural network.

[0113] It should be noted that the color correction step is a separate neural network, but it cannot be completely and independently reversed and trained. After obtaining the corrected image, it relies on a noise estimation neural network for reverse denoising and training.

[0114] (3) When to introduce the color correction step

[0115] In this embodiment, color correction is incorporated into the denoising training step. During the initial denoising training, the cumulative deviation is small, so color correction is not necessary. Instead, it is introduced in subsequent steps when the cumulative deviation becomes larger. Preferably...

[0116]

[0117] Specifically, in this embodiment, M = 5000. In the first 2500 steps of the denoising training, no color correction is performed; that is, only the denoising process is included. Starting from step 2501, color correction is added to the denoising training steps from 2501 to 5000, meaning it includes both a denoising step and a color correction step.

[0118] Those skilled in the art should understand that the timing of introducing the color correction step and the number of cycles can be adjusted according to the needs of the actual scenario, and are not limited to the scope given in this embodiment.

[0119] This concludes the introduction of the image restoration model training method according to the embodiments of the present invention.

[0120] Based on the above-described image inpainting model training method, a second aspect of the present invention also provides an image inpainting method. In an exemplary embodiment of the present invention, the image inpainting method includes:

[0121] Obtain the image to be repaired;

[0122] The image to be repaired is combined with the Gaussian noise image to obtain a composite image;

[0123] Denoising the synthesized image includes: an S-step denoising step, which further includes: an S1-step denoising step; and an S2-step denoising / color correction step, where S1 + S2 = S.

[0124] ① In the noise reduction step S1, For the s-th denoising step, it includes:

[0125] Noise estimation is performed using a noise estimation neural network. Among them, F θ To obtain the noise estimation neural network using the model training method described above, X s The noisy image in the current step; To estimate the noise in the regression output of a neural network for noise estimation;

[0126] Noise removal

[0127] Where s = S, S-1, ..., S-S1, X is the value of X before the denoising process is completed. s-1 As input for the next step of the denoising process.

[0128] ② In the noise reduction / color correction step S2, the noise reduction / color correction step s-th step includes:

[0129] Noise estimation is performed using a noise estimation neural network. Among them, X s The noisy image in the current step; To estimate the noise in the regression output of a neural network for noise estimation;

[0130] Noise removal

[0131] Color correction is performed using the color correction module;

[0132] Where s = S2, S2-1, ..., 1. Before the reverse denoising process is completed, X... s-1 As input for the next denoising process; after the denoising process is completed, the denoised image X0 is obtained.

[0133] To verify the image restoration effect of the image restoration method in this embodiment, the applicant conducted a comparative experiment. In the comparative experiment, a GPU server (GPU: NVIDIA RTX A6000, operating system: Ubuntu) was used to train and test the algorithm proposed in this invention. When evaluating the dataset, three main underwater image shooting scenarios that are prone to problems were selected: underwater color cast scenarios, underwater hazy scenarios, and underwater low-light scenarios. Underwater color cast scenarios can be further divided into (blue color cast and green color cast). The UIEB890

[10] dataset was used as the experimental object. On the UIEB dataset, the method of this invention was quantitatively compared with the existing deep learning enhancement methods, and the results are shown in Table 1.

[0134] Table 1 shows the results of a quantitative comparison between the method of the present invention and existing deep learning enhancement methods.

[0135]

[0136] In Table 1, Fusion, UWCNN, MFFN, UIEC2Net, UColor, PUIE-Net, BRUE, and NU2Net are existing underwater image enhancement methods; Raw represents the original image; Water-Diff(ours) represents the method of this invention; and UCIQE, UIQM, CCF, and FDUM are evaluation metrics. Figure 1 As shown, the MFFN method has the highest UCIQE index, followed by Water-Diff. The Water-Diff method has the highest UIQM index, followed by BRUE. The BRUE method has the highest CCF index, followed by Water-Diff. The FDUM method has the highest BRUE index, followed by UIEC2Net.

[0137] Figure 4 This image compares the quality of image denoising using different methods under different conditions on the UIEB dataset. Figure 4 As shown:

[0138] ①The image enhancement method of Ancuti et al. can improve image contrast, but it is not effective in removing haze and enhancing low light.

[0139] The image enhancement method of Ancuti et al. can be found in the following literature: Ancuti C O, Ancuti C, DeVleeschouwer C, et al. Color balance and fusion for underwater image enhancement[J].IEEE Transactions on image processing, 2017, 27(1):379-393.

[0140] ② In blue and yellow scenes, UGAN and FUNIE do not recover colors well. UIEC^2-Net can handle color cast better, but it cannot enhance background contours in hazy scenes. GLCHE can effectively remove haze, but it cannot effectively compensate for light in low-light scenes.

[0141] The UGAN method can be found in the following literature: Fabbri C, Islam MJ, Sattar J. Enhancing underwater imagery using generative adversarial networks [C] / / 2018 IEEE international conference on robotics and automation (ICRA). IEEE, 2018: 7159-7165.

[0142] The FUnIE method can be found in the following literature: Islam MJ, Xia Y, Sattar J. Fast underwater image enhancement for improved visual perception[J]. IEEE Robotics and Automation Letters, 2020, 5(2): 3227-3234.

[0143] ③MFFN improves contrast and image outline, but produces a yellowish tint.

[0144] MFFN can be found in the following literature: Chen R, Cai Z, Cao W. MFFN: An underwater sensing sceneimage enhancement method based on multiscale feature fusion network [J]. IEEE Transactions onGeoscience andRemoteSensing, 2021, 60: 1-12.

[0145] ④ The MTUR method improves image contrast, but its color restoration and dehazing effects are poor.

[0146] The MTUR method can be found in the following literature: Yan K, Liang L, Zheng Z, et al. Medium transmission map matters for learning to restore real-world underwater images[J]. Applied Sciences, 2022, 12(11): 5420.

[0147] ⑤ Water-Net is better at restoring image details, but it is not as effective as other methods in low-light enhancement.

[0148] The Water-Net method can be found in the following literature: C. Li et al., "An Underwater Image EnhancementBenchmark Dataset and Beyond", in IEEE Transactions on Image Processing, vol.29, pp.4376-4389, 2020, doi:10.1109 / TIP.2019.2955241.

[0149] ⑥ Compared with the above methods, the method of the present invention, namely UIESC in the figure, can effectively remove color cast and blur, while adjusting brightness and enhancing image texture to obtain better visual quality.

[0150] In summary, underwater images based on imaging suffer from a certain degree of degradation due to wavelength absorption, scattering, and other factors. Repairing this degradation is challenging. This invention combines a diffusion model and a correction network. The diffusion model is used to reconstruct the image distribution, and a color correction module based on a channel fusion mechanism is utilized. Experimental results show that the method of this invention can effectively solve the underwater image degradation problem and has strong robustness. This provides a reliable reference framework for future underwater image processing technologies and has significant practical implications for improving image analysis capabilities in fields such as underwater autonomous robots, underwater archaeology, and biodiversity research.

[0151] Those skilled in the art should understand that although this embodiment uses underwater images as an example for verification, the present invention can also be applied to other image enhancement scenarios, such as foggy scenes and old photo restoration, which are also within the scope of protection of the present invention.

[0152] This concludes the description of the image restoration method according to the embodiments of the present invention.

[0153] Based on the above-mentioned image restoration model training method and image restoration method, the third aspect of the present invention provides a computer chip. Figure 5 This is a schematic diagram of a computer chip according to an embodiment of the present invention. Figure 5 As shown, the computer chip in this embodiment of the invention is configured to implement: the image restoration model training method as described in the above embodiment; or, the image restoration method as described in the above embodiment.

[0154] Based on the above-described image inpainting model training method and image inpainting method, a fourth aspect of the present invention provides a computer-readable storage medium. The computer-readable storage medium stores a computer program that, when executed by a processor, implements: the image inpainting model training method as described in the above embodiments; or the image inpainting method as described in the above embodiments.

[0155] Based on the above-mentioned image restoration model training method and image restoration method, the fifth aspect of the present invention provides a GPU server. Figure 6 This is a schematic diagram of a GPU server according to an embodiment of the present invention. The GPU server includes: a processor; and a memory storing a computer program thereon; wherein, when the processor executes the computer program, it implements: the image inpainting model training method as described above; or the image inpainting method as described above.

[0156] Based on the above-described image restoration model training method and image restoration method, a sixth aspect of the present invention provides a computer program product. The computer program product includes: a computer program, wherein, when executed by a processor, the computer program implements: the image restoration model training method as described in the above embodiments; or the image restoration method as described in the above embodiments.

[0157] This concludes the description of the various embodiments of the present invention. Based on the above description, those skilled in the art should have a clear understanding of the present invention.

[0158] It should be noted that, unless explicitly stated otherwise, the numerical parameters in the specification and claims of this invention may be approximate values ​​and can be changed according to the content of this invention. Specifically, all figures in the specification and claims indicating the content of components, reaction conditions, etc., should be understood to be modified by the term "about" in all cases, which may vary by ±10% in some embodiments.

[0159] The present invention can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such a program implementing the present invention can be stored on a computer-readable medium or can take the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0160] This invention can be implemented using hardware comprising several different components and a suitably programmed computer. Various component embodiments of the invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. The physical implementation of the hardware structure includes, but is not limited to, physical devices, including, but not limited to, transistors, memristors, DNA computers, microcontrollers, microprocessors, or digital signal processors (DSPs). Furthermore, this invention is not directed to any particular programming language. It should be understood that the contents of this invention can be implemented using various programming languages; the description of specific languages ​​herein is for the purpose of disclosing the best mode of implementation of the invention.

[0161] Those skilled in the art will understand that in the claims and specification of this invention, the word "comprising" does not exclude the presence of elements (or steps) not listed in the claims. The word "a" or "an" preceding an element (or step) does not exclude the presence of a plurality of such elements (or steps).

[0162] For certain implementation methods, if they are not key aspects of this invention and are well-known to those skilled in the art, they are not described in detail in the accompanying drawings or text due to space limitations. In such cases, they can be understood by referring to relevant prior art.

[0163] Furthermore, the above embodiments are provided only to enable the invention to meet legal requirements, and the invention can be implemented in many different forms and should not be construed as limited to the embodiments set forth herein.

[0164] Similarly, it should be understood that, for the sake of brevity, in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof. However, this method of invention should not be construed as reflecting an intention that the claimed invention requires more features than expressly recited in each claim. Rather, as reflected in the claims, the various inventive aspects consist of fewer than all the features of the preceding single embodiment. Furthermore, embodiments may be used in combination with each other or with other embodiments based on design and reliability considerations; that is, technical features from different embodiments can be freely combined to form more embodiments. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the invention.

[0165] The above specific embodiments have provided a detailed description of the purpose, technical means, and beneficial effects of the present invention. It should be understood that the purpose of the detailed description is to enable those skilled in the art to better understand the present invention, and it is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for training an image restoration model, characterized in that, include: Step A, forward noise addition, includes: Original image And the reference image after repair The composite image is obtained by performing the synthesis. : =f( , ), where f is the composition function; In the composite image The process involves introducing Gaussian noise step by step in N steps to obtain a noisy image I. N N≥100; Step B, reverse denoising training, includes: M-step denoising training steps, wherein the M-step denoising training steps include: M1-step denoising training steps; and M2-step denoising / color correction steps, M1+M2=M. The m-th denoising training step includes: using a noise estimation neural network to estimate noise. ,in, For noise estimation neural networks, The noisy image in the current step; For reference image, The noise is estimated from the regression output of the noise estimation neural network; noise removal is then performed to obtain the denoised image. Where m = M, M-1, ..., M-M1+1, M ≥ 100, M ≤ N, X M =I N If the denoising training step in step M1 is not completed, As the input for the next denoising training step; after the denoising training step of step M1 is completed, the input for the denoising / color correction step is obtained; wherein, in the denoising training step, the noise estimation neural network is trained by backpropagation, and the loss function L of the noise estimation neural network includes: the difference L1 between the mean of the estimated noise obtained in the denoising training step and the mean of the real noise. The denoising / color correction step m includes: using a noise estimation neural network to estimate noise. Noise removal yields the denoised image: The color correction module, a deep learning neural network, is used to correct color deviations in the denoised image. Here, m = M2, M2-1, ..., 1. This is done before the reverse denoising training is complete. As input for the next denoising training step; after the reverse denoising training is completed, the denoised image X0 is obtained; wherein, in the denoising / color correction step, the noise estimation neural network is trained through backpropagation, and the loss L of the backpropagation is: L=L1+L2, where L2 is the difference between the color-corrected image and the reference image. The distance between them; The color correction module includes: data processing, extracting the L channel in the LAB color mode and the R, G, and B channels in the RGB color mode of the noise-removed image; color compression encoding, taking the L, R, G, and B channels as input and outputting a feature map; and convolutional reconstruction decoding, taking the feature map output by the color compression encoding as input and reconstructing key color information.

2. The model training method according to claim 1, characterized in that, In the forward noise addition, the original image and the reference image are obtained through one of the following methods: obtaining the original image in UIEB or EUVP. and reference image Original images The reference image is obtained after processing by the image enhancement algorithm. ; And / or, in the reverse denoising training, during the denoising training step: When M=N, the N steps of introducing Gaussian noise correspond to the N denoising training steps, and the mean of the real noise is the mean of the Gaussian noise introduced in the corresponding Gaussian noise introduction step during the forward noise addition process. Alternatively, when M < N, the first M steps of introducing Gaussian noise out of the N steps of introducing Gaussian noise are respectively matched with the M denoising training steps, and the mean of the real noise is the mean of the Gaussian noise introduced in the corresponding steps of introducing Gaussian noise in the forward noise addition process.

3. The model training method according to claim 2, characterized in that, In step A Original image and reference image Linear synthesis is performed to obtain the synthesized image. ; ; Linear coefficients: 2; And / or, in the step of introducing Gaussian noise in the nth step on the synthesized image, Gaussian noise is introduced linearly. , Let n be the Gaussian noise introduced in step n, where n = 1, 2, ..., N.

4. The model training method according to claim 3, characterized in that, N≥2000； And / or, M=N; And / or, the noise estimation neural network is U Net network; And / or, =1.

5. The model training method according to claim 4, characterized in that, The U Net The network includes an encoder, a decoder, and hop paths connecting the two; where: The encoder is used to extract useful features from the input data and reduce the spatial dimension. It consists of multiple convolutional layers and pooling layers that appear alternately. In each layer, the input image first passes through the convolutional layer to extract features, and then passes through the pooling layer for downsampling. This structure enables the network to capture multi-scale features from shallow to deep layers. The decoder has the opposite structure to the encoder, restoring the encoder's output features to the original image size through upsampling and convolution operations; Skip connections directly connect the encoder's feature map to the corresponding decoder layer.

6. The model training method according to claim 1, characterized in that, Color compression encoding includes: a convolution-ReLU-pooling module for downsampling convolution of a 4-channel image; non-linear activation of the downsampling convolution image; pooling of the non-linearly activated image to obtain a multi-channel image after color compression; a channel attention module for attention extraction of the multi-channel image output by the convolution-ReLU-pooling module, finding important parts in different channels and obtaining weighted feature maps; and a DBC module for further feature compression and extraction of the weighted feature maps output by the channel attention module, performing dual-branch convolution on the same feature map using convolution kernels of different scales, and then merging the results of the two branches to enhance the detailed representation of the feature map. Convolutional reconstruction decoding includes: a TI module, used to perform preliminary upsampling on the feature map to restore preliminary image details; and a TCR module, used to refine the feature map obtained by the TI module upsampling, add details and restore normal color distribution, while receiving channel weight information provided by the channel attention module, and reconstructing image colors according to different channel relationships to obtain a color-corrected image.

7. The model training method according to claim 1, characterized in that, Where M≥2000, the ; And / or, the distance is Euclidean distance, chessboard distance, L2 distance, or KL divergence.

8. An image restoration method, characterized in that, include: Obtain the image to be repaired; The image to be repaired is combined with the Gaussian noise image to obtain a composite image; Denoising the synthesized image includes: The S-step denoising process includes: Noise estimation is performed using a noise estimation neural network. ,in, To obtain a noise estimation neural network via the model training method according to any one of claims 1 to 7, The noisy image in the current step; To estimate the noise in the regression output of a neural network for noise estimation; Noise removal ; Where s = S, S-1, ..., 1, S ≥ 100, assuming the denoising process is not yet complete. This serves as the input for the next denoising process; after the denoising process is completed, the denoised image is obtained. .

9. The image restoration method according to claim 8, characterized in that, The S-step denoising step includes: The noise reduction step in step S1; The denoising / color correction step S2, S1+S2=M, where the denoising / color correction step s-th includes: Noise estimation is performed using a noise estimation neural network. ,in, The noisy image in the current step; To estimate the noise in the regression output of a neural network for noise estimation; Noise removal yields the denoised image: ; The color correction module is used to correct the color of the denoised image; Where s = S2, S2-1, ..., 1, assuming the denoising process is not yet complete. This serves as the input for the next denoising process; after the denoising process is completed, the denoised image is obtained. ; Wherein, the noise estimation neural network is obtained by the model training method of any one of claims 1 to 7, and the color correction module is the color correction module of any one of claims 1 to 7.

10. The image restoration method according to claim 9, characterized in that, The image to be repaired is an underwater image; And / or, wherein S≥2000, wherein .

11. A computer chip, characterized in that, The computer chip is configured to implement: a model training method for image restoration as described in any one of claims 1 to 7; or, an image restoration method as described in claim 8 or 9.

12. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements: a model training method for image restoration as described in any one of claims 1 to 7; or, an image restoration method as described in claim 8 or 9.

13. A computer program product, characterized in that, include: A computer program, wherein when executed by a processor, the computer program implements: the model training method for image inpainting as described in any one of claims 1 to 7; Alternatively, the image restoration method as described in claim 8 or 9.

14. A GPU server, characterized in that, include: processor; A memory on which computer programs are stored; When the processor executes the computer program, it implements: the image restoration model training method as described in any one of claims 1 to 7; or, the image restoration method as described in claim 8 or 9.

Images