Image reconstruction model training method and image reconstruction method

Abstract

The invention provides an image reconstruction model training and image reconstruction method, and belongs to the technical field of image processing. According to the method, an image training set is constructed, the image training set comprises noise image subsets and non-noise image subsets, and the noise image subsets and the non-noise image subsets are in one-to-one correspondence, so that coding can be performed according to noise images and corresponding non-noise images, and prior features can be obtained; and on the basis of the prior features, the to-be-trained reconstruction model is trained based on the prior feature corresponding to any noise image in the image training set and the image feature corresponding to the noise image, so that the target reconstruction model is obtained, and the image reconstruction quality is improved to a certain extent.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technologies, and particularly relates to an image reconstruction model training and an image reconstruction method. Background Art

[0002] Image super-resolution is to reconstruct a low-resolution image into a high-resolution image; inverse restoration is to restore an image affected by rain, such as raindrops and rain streaks, to display a clean background.

[0003] When an image needs to be reconstructed into a high-resolution image and inversely restored at the same time, in related technologies, inverse restoration and high-resolution reconstruction are usually performed sequentially. However, the above method of image processing has the problem of low accuracy in inverse restoration. Summary of the Invention

[0004] The present disclosure provides an image reconstruction model training and an image reconstruction method.

[0005] The first aspect embodiment of the present disclosure provides an image reconstruction model training method, including:

[0006] Construct an image training set, where the image training set includes a noise image subset and a non-noise image subset; the noise image subset and the non-noise image subset correspond one by one;

[0007] Encode any noise image in the noise image subset and the non-noise image corresponding to the noise image to obtain a prior feature;

[0008] Train a to-be-trained reconstruction model based on the prior feature corresponding to any noise image in the image training set and the image feature corresponding to the noise image to obtain a target reconstruction model.

[0009] In the embodiment of the present disclosure, the training of the to-be-trained reconstruction model based on the prior feature corresponding to any noise image in the image training set, the first image feature corresponding to the noise image, and the second image feature corresponding to the non-noise image to obtain a target reconstruction model includes:

[0010] Perform medium removal reconstruction based on the prior feature and the first image feature to obtain a medium-free reconstruction feature;

[0011] Perform edge reconstruction based on the prior feature and the medium-free reconstruction feature to obtain a target edge feature;

[0012] Fuse the medium-free reconstruction feature, the target edge feature, and the prior feature to obtain a reconstructed image;

[0013] Determine the loss value corresponding to the training of the to-be-trained reconstruction model based on the reconstructed image and the second image feature;

[0014] Optimize the to-be-trained reconstruction model based on the relationship between the loss value and a preset standard, and output a target reconstruction model.

[0015] In an embodiment of the present disclosure, the performing medium removal reconstruction based on the prior feature and the first image feature to obtain a medium-free reconstruction feature includes:

[0016] After splitting the prior feature, divide it into a first sub-prior feature and a second sub-prior feature;

[0017] Fuse the first sub-prior feature with the first image feature to obtain a first modulated feature; fuse the second sub-prior feature with the first image feature to obtain a second modulated feature;

[0018] Perform guided filtering based on the first modulated feature, the second modulated feature, and the prior feature to obtain a medium-free feature;

[0019] Perform feature reconstruction based on the medium-free feature and the prior feature to obtain the medium-free reconstruction feature.

[0020] In an embodiment of the present disclosure, the performing guided filtering based on the first modulated feature, the second modulated feature, and the prior feature to obtain a medium-free feature includes:

[0021] Perform mean filtering on the first modulated feature to obtain a first filtered modulated feature; and perform mean filtering on the prior feature to obtain a filtered prior feature;

[0022] Extract local relationship features from the first filtered modulated feature and the filtered prior feature to obtain local relationship features;

[0023] Perform mean filtering on the local relationship features to obtain a first learning coefficient and a second learning coefficient;

[0024] Fuse the first learning coefficient, the second learning coefficient, and the second modulated feature to obtain the medium-free feature.

[0025] In an embodiment of the present disclosure, the performing edge reconstruction based on the prior feature and the medium-free reconstruction feature to obtain a target edge feature includes:

[0026] Fuse the first sub-prior feature with the medium-free reconstruction feature to obtain a third modulated feature; fuse the second sub-prior feature with the third modulated feature to obtain a stage feature;

[0027] Extract the edge feature of the prior feature;

[0028] Perform feature reconstruction based on the edge feature and the stage feature to obtain a reconstructed edge feature;

[0029] Compensate the reconstructed edge feature to obtain a target edge feature.

[0030] In an embodiment of the present disclosure, the extracting the edge feature of the prior feature includes:

[0031] Convert the prior feature into a frequency domain feature;

[0032] Extract the high-frequency feature in the frequency domain feature;

[0033] Convert the high-frequency feature into a spatial domain feature, and use the spatial domain feature as the edge feature.

[0034] In an embodiment of the present disclosure, the performing feature fusion on the medium-free reconstruction feature, the target edge feature, and the prior feature to obtain a reconstructed image includes:

[0035] Multiply the medium-free reconstruction feature, the target edge feature, and the prior feature to obtain a fusion feature;

[0036] Perform image reconstruction based on the fusion feature to obtain the reconstructed image.

[0037] In an embodiment of the present disclosure, the prior feature includes a first prior feature and a second prior feature, and the second prior feature is a diffusion feature of the first prior feature; the target reconstruction model includes a first target reconstruction model and a second target reconstruction model,

[0038] The training the reconstruction model to be trained based on the prior feature corresponding to any noise image in the image training set, the first image feature corresponding to the noise image, and the second image feature corresponding to the non-noise image to obtain a target reconstruction model includes:

[0039] Train a first reconstruction model to be trained based on the first prior feature corresponding to any noise image in the image training set, the first image feature corresponding to the noise image, and the second image feature corresponding to the non-noise image to obtain the first target reconstruction model;

[0040] Train a first reconstruction model to be trained based on the second prior feature corresponding to any noise image in the image training set, the first image feature corresponding to the noise image, and the second image feature corresponding to the non-noise image to obtain the second target reconstruction model.

[0041] In an embodiment of the present disclosure, the second prior feature is obtained by the following method:

[0042] Add pre-acquired noise to the first prior feature to obtain a noisy prior feature;

[0043] Extract the conditional feature of the noisy image;

[0044] Guided by the conditional feature, denoise the noisy prior feature to obtain the second prior feature.

[0045] An embodiment of the second aspect of the present disclosure provides an image reconstruction method, the method includes:

[0046] Obtain an image to be reconstructed and noise;

[0047] Input the image to be reconstructed and the noise into a target reconstruction model to obtain a target reconstructed image, where the target reconstruction model is trained by the method described in the first aspect or any optional implementation manner of the first aspect.

[0048] In an embodiment of the present disclosure, the target reconstruction model includes a first target reconstruction model and a second target reconstruction model. The step of inputting the image to be reconstructed and the noise into the target reconstruction model to obtain a target reconstructed image includes:

[0049] Input the image feature of the image to be reconstructed and the noise into the first target reconstruction model to obtain a target diffusion prior feature;

[0050] Input the image feature of the image to be reconstructed and the target diffusion prior feature into the second target reconstruction model to obtain the target reconstructed image.

[0051] An embodiment of the third aspect of the present disclosure provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. The processor runs the computer program to implement the methods described in the first aspect or any optional implementation manner of the first aspect, the second aspect, and any optional implementation manner of the second aspect.

[0052] An embodiment of the fourth aspect of the present disclosure provides a computer-readable storage medium, on which a computer program is stored. The program is executed by a processor to implement the methods described in the first aspect and any optional implementation manner of the first aspect, the second aspect, and any optional implementation manner of the second aspect.

[0053] The technical solutions provided in the embodiments of the present disclosure have at least the following technical effects or advantages:

[0054] Construct an image training set. Since the image training set includes a noise image subset and a non-noise image subset, and the noise image subset and the non-noise image subset are in one-to-one correspondence, prior features can be obtained by encoding according to the noise images and the corresponding non-noise images. Based on the prior features, a to-be-trained reconstruction model is trained using the prior features corresponding to any noise image in the image training set and the image features corresponding to the noise images, and a target reconstruction model is obtained, which improves the quality of image reconstruction to a certain extent.

[0055] Additional aspects and advantages of the present disclosure will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] By reading the following detailed description of the preferred embodiments, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present disclosure. Also, throughout the drawings, the same reference numerals are used to represent the same components.

[0057] In the drawings:

[0058] Figure 1 shows a flowchart of a method for training an image reconstruction model provided by an embodiment of the present disclosure;

[0059] Figure 2 shows a schematic diagram of first prior feature extraction in a method for training an image reconstruction model provided by an embodiment of the present disclosure;

[0060] Figure 3 shows a schematic diagram of second prior feature extraction in a method for training an image reconstruction model provided by an embodiment of the present disclosure;

[0061] Figure 4 shows a schematic diagram of image reconstruction in a method for training an image reconstruction model provided by an embodiment of the present disclosure;

[0062] Figure 5 shows a schematic diagram of medium removal in a method for training an image reconstruction model provided by an embodiment of the present disclosure;

[0063] Figure 6 shows a schematic diagram of guided filtering in a method for training an image reconstruction model provided by an embodiment of the present disclosure;

[0064] Figure 7 shows a schematic diagram of edge feature extraction in a method for training an image reconstruction model provided by an embodiment of the present disclosure;

[0065] Figure 8Shows a flowchart of an image reconstruction method provided by an embodiment of the present disclosure;

[0066] Figure 9 Shows a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure;

[0067] Figure 10 Shows a schematic diagram of a storage medium provided by an embodiment of the present disclosure. Detailed implementation manners

[0068] Hereinafter, the exemplary embodiments of the present disclosure will be described in more detail with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be completely conveyed to those skilled in the art.

[0069] It should be noted that unless otherwise specified, the technical terms or scientific terms used in the present disclosure should have the ordinary meanings understood by those skilled in the art to which the present disclosure pertains.

[0070] The embodiment of the present disclosure proposes an image reconstruction model training method, as Figure 1 Shown is an image reconstruction model training method provided by an embodiment of the present disclosure, including the following steps:

[0071] In step S11, an image training set is constructed.

[0072] Among them, the image training set includes a noisy image subset and a non-noisy image subset; the noisy image subset and the non-noisy image subset correspond one by one.

[0073] Exemplarily, the noisy images in the image training set correspond to the non-noisy images. According to the actual application scenario of the image reconstruction model in the embodiment of the present disclosure, the image training set can be constructed according to the type of the application scenario. Specifically, if the image reconstruction model is applied to low-resolution images and the images are affected by weather, the noisy images in the image training set can all be images affected by weather, and the non-noisy images can be noisy images corresponding to rainy days, foggy days, etc.

[0074] It should be noted that for the scenarios corresponding to the same weather type, the noisy images should also be of the same type. For example, for the rainy day scenario, the noisy images are all images affected by rain.

[0075] In the process of image reconstruction, it is usually necessary to compensate with high-resolution features corresponding to non-noise images to obtain a relatively accurate reconstructed image. However, in actual applications, there are no high-resolution features for compensation. Therefore, in the embodiments of the present disclosure, by analyzing the prior features of the image, when subsequently using the image reconstruction model of the embodiments of the present disclosure for image reconstruction, only the noise image is needed to obtain the corresponding prior features.

[0076] In step S12, any noise image in the noise image subset and the non-noise image corresponding to the noise image are encoded to obtain prior features.

[0077] Exemplarily, before encoding, the noise image and the non-noise image can also be preprocessed to ensure that all images have the same size for subsequent processing. Specifically, normalization can be used to normalize the pixel values to the range of [0, 1] or [-1, 1], and data diversity can be increased by means such as rotation and flipping.

[0078] As Figure 2 shown, the process of extracting prior features can map the image features corresponding to the noise image and the non-noise image to the latent space to obtain the corresponding prior features.

[0079] Specifically, features need to be extracted from the noise image and the corresponding non-noise image. This can be achieved by traditional feature extraction methods such as convolution. The extracted features may need to be transformed to better capture the image content and noise information. For example, a dual-stream Unet can be used as a noise predictor, and the dual-stream encoder is used to extract the features of the noise image and the prior image respectively, and the difference is used to enhance the representation of the noise features, thereby improving the denoising accuracy. The extracted and transformed features need to be encoded to obtain prior features. This can be achieved by an encoder as Figure 2 shown. The encoded features can be further mapped to the latent space to obtain prior features.

[0080] In some embodiments, the prior features include a first prior feature and a second prior feature, and the second prior feature is the diffusion feature of the first prior feature; the target reconstruction model includes a first target reconstruction model and a second target reconstruction model, where the first prior feature is used for the extraction of the prior features corresponding to the non-noise image during the training process of the image reconstruction model in the embodiments of the present disclosure. The second prior feature is used for the training of the extraction of the prior features corresponding to the absence of the non-noise image in the embodiments of the present disclosure, so that when subsequently using the trained image reconstruction model for image reconstruction, only the noise image is needed to achieve image reconstruction.

[0081] In step S13, based on the prior feature corresponding to any noisy image in the image training set and the image feature corresponding to the noisy image, train the reconstruction model to be trained to obtain the target reconstruction model.

[0082] Exemplarily, fuse the prior feature and the image feature. This can be achieved by methods such as simple concatenation, weighted summation, or using more complex attention mechanisms.

[0083] Design a reconstruction model whose input is the fused feature and output is the restored clean image. The reconstruction model can be a CNN-based architecture, such as the generator part of U-Net, GAN (Generative Adversarial Network), or other architectures suitable for image-to-image conversion.

[0084] Select one or more loss functions to measure the difference between the reconstructed image and the true clean image. Commonly used loss functions include mean squared error (MSE), loss related to peak signal-to-noise ratio (PSNR), adversarial loss (if using GAN), perceptual loss (based on the distance in the feature space of a pre-trained network), etc.

[0085] Use the training data (noisy images, non-noisy images, and their corresponding features and prior features) to train the reconstruction model. During the training process, update the weights of the model through the backpropagation algorithm to minimize the loss function. Optimizers such as Adam, SGD, etc. can be used for weight update. Evaluate the performance of the model on the validation set, and adjust the model architecture, loss function, training strategy, etc. according to the evaluation results. Quantitative metrics (such as PSNR, SSIM) and qualitative analysis (such as visualizing the reconstruction results) can be used to evaluate the model. When the model reaches satisfactory performance on the validation set, stop training and save the model at this time as the target reconstruction model.

[0086] In some embodiments, the prior feature includes a first prior feature and a second prior feature, and the second prior feature is the diffusion feature of the first prior feature; the target reconstruction model includes a first target reconstruction model and a second target reconstruction model, where the first prior feature is used for extracting the prior feature corresponding to the case where there is a non-noisy image during the training process of the image reconstruction model in the embodiments of the present disclosure. And the second prior feature is used for the training of extracting the prior feature corresponding to the case where there is no non-noisy image in the embodiments of the present disclosure, so that when using the trained image reconstruction model for image reconstruction later, only the noisy image is needed to achieve image reconstruction.

[0087] Among them, the second prior feature is obtained through the following method: add pre-acquired noise to the first prior feature to obtain a noisy prior feature; extract the conditional feature of the noisy image; guided by the conditional feature, denoise the noisy prior feature to obtain the second prior feature.

[0088] Exemplarily, as Figure 3 shown, on the basis of obtaining the first prior feature, after adding preset Gaussian noise to the first prior feature, denoising is performed to obtain the second prior feature, and the denoising process is guided according to the conditions generated from the noisy image. In Figure 3 the encoder E1 is the same as the encoder E1 in Figure 2 . When the first prior feature is known, when determining the second prior feature, the corresponding first prior feature P can be directly obtained, that is, Figure 3 the parameters of the encoder E1 shown in are unchanged.

[0089] Specifically, Gaussian noise is added using the following formula:

[0090]

[0091] where P t is the noise prior at time step t, N is the Gaussian distribution, α t = 1 - β t , α t is the proportionality factor controlling the noise variance, I is the identity matrix, and β t is the scaling factor controlling the noise variance and can be set artificially.

[0092] For the denoising process, according to the Markov chain, the reverse process from P t to P t-1 can be expressed as:

[0093]

[0094] where is the Gaussian noise intensity at time t, where the Gaussian noise intensity is represented by the variance; ε is the noise in P t . In the denoising stage, the encoder E in Figure 3 is also used to encode the input noisy image and output the condition C and P 2 as the conditions for the denoising network to predict the noise: t

[0095]

[0096] where ε t is the estimated noise ε at each step. Through the above T - time iteration of sampling, the second prior feature

[0097] The above step S13 can also be implemented in the following manner: Based on the first prior feature corresponding to any noise image in the image training set, the first image feature corresponding to the noise image, and the second image feature corresponding to the non-noise image, train the first to-be-trained reconstruction model to obtain the first target reconstruction model; Based on the second prior feature corresponding to any noise image in the image training set, the first image feature corresponding to the noise image, and the second image feature corresponding to the non-noise image, train the first to-be-trained reconstruction model to obtain the second target reconstruction model.

[0098] Exemplarily, in some embodiments, the training of the to-be-reconstructed model is divided into two stages of training, namely the training of the first to-be-trained reconstruction model and the second to-be-trained reconstruction model. Among them, the prior feature of the first to-be-trained reconstruction model is the first prior feature; the prior feature of the second to-be-trained reconstruction model is the second prior feature. The remaining input and output parameters are the same. After the first to-be-trained reconstruction model is trained according to the images in the image training set to obtain the first target reconstruction model, then the second to-be-trained reconstruction model is trained according to the images in the image training set.

[0099] Specifically, in the embodiments of the present disclosure, the loss function of the first target reconstruction model can be:

[0100] L S1 =‖I Rec -I GT ‖ 1

[0101] Wherein, I Rec1 is the reconstructed image obtained according to the first prior feature, and I GT is the non-noise image.

[0102] The loss function of the second target reconstruction model can be:

[0103]

[0104] Wherein, I Rec2 is the reconstructed image obtained according to the second prior feature, is the second prior feature, and P is the first prior feature.

[0105] In the actual training and optimization process, the channel count can be set to 64, N is 12, and the time step for determining the second prior feature is set to 4. In the training stage, the Adam optimizer is used: β 1 =0.9, β 2 =0.99. The initial learning rate is set to 0.0002, and the training batch size is 16.

[0106] In some embodiments, the above step S13 can be implemented in the following manner: perform medium-removing reconstruction based on prior features and first image features to obtain medium-free reconstruction features; perform edge reconstruction based on prior features and medium-free reconstruction features to obtain target edge features; fuse the medium-free reconstruction features, target edge features, and prior features to obtain a reconstructed image; determine the loss value corresponding to the to-be-trained reconstruction model based on the reconstructed image and second image features; optimize the to-be-trained reconstruction model based on the relationship between the loss value and a preset standard, and output the target reconstruction model.

[0107] Exemplarily, remove the medium influence (such as haze, rain blur, etc.) in the image to obtain image features without medium influence. Specifically, use prior features and first image features (features directly extracted from the noisy image) as inputs. Design a medium-removing reconstruction network, which can be an architecture based on a convolutional neural network, such as DehazeNet, underwater image enhancement network, etc. Through the trained medium-removing reconstruction network, output medium-free reconstruction features.

[0108] Recover clear edge information from the medium-removed image features. Specifically, prior features and the medium-free reconstruction features obtained in the previous step can be used as inputs. Design an edge reconstruction network, which can be a deep learning model based on edge detection, or a method combining traditional edge detection algorithms (such as Canny, Sobel) and deep learning feature extraction. Through the trained edge reconstruction network, output target edge features.

[0109] Finally, fuse the medium-free reconstruction features, target edge features, and prior features to obtain a rich image representation for reconstructing the image. Specifically, design a feature fusion module, which can be simple concatenation, weighted summation, or more complex attention mechanisms, recurrent neural networks, etc. Input the above three features into the feature fusion module, and output the fused features, that is, the feature representation of the reconstructed image.

[0110] An image reconstruction network, which can be an architecture based on a generative adversarial network (GAN), or a simple convolutional neural network (CNN) for upsampling and image generation. Input the fused features into the image reconstruction network, and output the reconstructed image.

[0111] Compare the second image feature (feature extracted from the non-noisy image) with the reconstructed image. Losses related to mean squared error (MSE), peak signal-to-noise ratio (PSNR), structural similarity (SSIM), or a combination of adversarial loss, perceptual loss, etc. can be utilized to calculate the loss value. Use an optimizer (such as Adam, SGD) and the loss value to update the weights of the model. During training, monitor the change in the loss value and perform model validation and adjustment as needed. When the loss value converges to a satisfactory level, stop training and save the model as the target reconstruction model.

[0112] Among them, in some embodiments, the process of the above target reconstruction model obtaining the reconstructed target image can also be achieved through a process as Figure 4 shown. Specifically, after obtaining the first prior feature, perform medium removal based on the noisy image and the first prior feature P to obtain the medium-free reconstruction feature F MR ; next, utilize the first prior feature P and the medium-free reconstruction feature F MR to perform texture compensation to obtain the target edge feature F TR ; finally, after feature fusion, perform upsampling to obtain the reconstructed image.

[0113] The medium-free reconstruction feature can also be obtained in the following manner: After splitting the prior feature, divide it into a first sub-prior feature and a second sub-prior feature; fuse the first sub-prior feature with the first image feature to obtain a first modulated feature; fuse the second sub-prior feature with the first image feature to obtain a second modulated feature; perform guided filtering based on the first modulated feature, the second modulated feature, and the prior feature to obtain a medium-free feature; perform feature reconstruction based on the medium-free feature and the prior feature to obtain the medium-free reconstruction feature.

[0114] Exemplarily, taking the first prior feature P as an example, introduce the process of the medium-free reconstruction feature, guided filtering, and subsequent edge reconstruction.

[0115] As Figure 5 shown, the first prior feature P ∈ R 4C , and the first image feature F in ∈ R H×W×C . Split the first prior feature P to obtain a set of vectors {α, β 1 , …, β n} ∈ R C×1×1 , where α is the first sub-prior feature and β i is the first sub-prior feature, 1 < i < n.

[0116] Next is the feature multiplication feature fusion of α and F in to obtain the first modulated feature F′ ∈ R H×W×C , and β iWith F in Perform feature multiplication feature fusion to obtain the second modulation feature F″ ∈ R H×W×C . The first modulation feature F′ ∈ R H×W×C , the second modulation feature F″ ∈ R H×W×C and the first prior feature P are input into the guided filter module as shown in Figure 6 to obtain the medium-free feature F G ∈ R H×W×C . Next, using cross self-attention, the medium-free reconstruction feature F G is reconstructed based on the medium-free feature F MR and the prior feature P. Among them, before reconstruction using cross self-attention, the medium-free feature F G can also be embedded into the global feature (KV obtained by feature splitting after 1*1 convolution of F G , Q obtained by first convolving P and then changing the dimension), projected into the vectors {Q, K, V} ∈ R HW×C and then reconstructed.

[0117] Further, the medium-free feature obtained by the guided filter can also be determined in the following way: perform mean filtering on the first modulation feature to obtain the first filtered modulation feature; and perform mean filtering on the prior feature to obtain the filtered prior feature; extract local relationship features from the first filtered modulation feature and the filtered prior feature to obtain local relationship features; perform mean filtering on the local relationship features to obtain the first learning coefficient and the second learning coefficient; fuse the first learning coefficient, the second learning coefficient and the second modulation feature to obtain the medium-free feature.

[0118] Exemplarily, as shown in Figure 6 , mean filtering on the first modulation feature F′ and the first prior feature P (here the prior feature takes the first prior feature as an example) can be obtained by the following formula:

[0119]

[0120] where, is the first filtered modulation feature, is the filtered prior feature.

[0121] Input the first filtered modulation feature and the filtered prior feature into the local linear model to obtain local relationship features. Specifically, the following formula is used for local relationship feature extraction.

[0122]

[0123] Among them, the coefficients A and B represent the local linear relationship between the input and the output. A quantifies the influence of the quantization input on the output, and B captures the low-frequency details. ∑P refers to the local features captured within the specified window, while ∑PF′ refers to the relationship between the prior feature P and the first modulation feature, and the guidance filter is improved according to the guidance.

[0124] To maintain smooth consistency, mean filtering is applied to generate the first learning coefficient A′ and the second learning coefficient B′ of the learning coefficients, and then they are fused with the second filtered modulation feature F″ (feature splicing + convolution dimensionality reduction) to generate the medium-free feature F. G :

[0125]

[0126] Among them, in some embodiments, the target edge feature can also be obtained in the following manner: fusing the first sub-prior feature with the medium-free reconstruction feature to obtain the third modulation feature; fusing the second sub-prior feature with the third modulation feature to obtain the stage feature; extracting the edge feature of the prior feature; performing feature reconstruction based on the edge feature and the stage feature to obtain the reconstructed edge feature; compensating the reconstructed edge feature to obtain the target edge feature.

[0127] Exemplarily, as Figure 7 shown, the first sub-prior feature α is fused with the medium-free reconstruction feature F MR to obtain the third modulation feature F1. Next, the third modulation feature F 1 is fused with the second sub-prior feature β i to obtain the stage feature F2. The specific fusion method is the same as the fusion method of the aforementioned first modulation feature and the second modulation feature, and will not be elaborated here.

[0128] Edge detection algorithms such as Sobel, Canny, or learning-based methods (such as the Hough transform or deep learning models) can be used to extract edge information from the prior feature. Using reconstruction algorithms such as sparse coding, dictionary learning, or deep learning models (such as autoencoders or generative adversarial networks), feature reconstruction is performed by combining the edge feature and the stage feature to obtain the reconstructed edge feature. According to a specific compensation strategy or algorithm, the reconstructed edge feature is adjusted or optimized to obtain the target edge feature. Error correction, contrast enhancement, or other image enhancement techniques may be involved.

[0129] In some embodiments, among them, extracting the edge feature of the prior feature includes: converting the prior feature into a frequency-domain feature; extracting the high-frequency feature in the frequency-domain feature; converting the high-frequency feature into a spatial-domain feature, and taking the spatial-domain feature as the edge feature.

[0130] Exemplarily, when extracting edge features, a high-pass filter can be applied to extract the high-frequency components of the first prior feature P. Specifically, a one-dimensional discrete cosine transform (DCT) can be performed along the channels to transform the first prior feature P from the spatial domain to the frequency domain. After filtering the high-frequency components, the spatial domain features are restored through an inverse one-dimensional discrete cosine transform (IDCT) to extract the edge feature F edge .

[0131] After obtaining the medium-free reconstruction feature, the target edge feature, and the prior feature, feature fusion is performed to obtain the reconstructed image. Specifically, the medium-free reconstruction feature, the target edge feature, and the prior feature can be multiplied in terms of features to obtain the fused feature; image reconstruction is performed based on the fused feature to obtain the reconstructed image.

[0132] Through the image reconstruction model training method of the embodiments of the present application, an image training set is constructed. Since the image training set includes a noise image subset and a non-noise image subset, and the noise image subset and the non-noise image subset are in one-to-one correspondence, prior features can be obtained by encoding according to the noise images and the corresponding non-noise images; based on the prior features, the to-be-trained reconstruction model is trained based on the prior features corresponding to any noise image in the image training set and the image features corresponding to the noise images, and the target reconstruction model is obtained, which improves the quality of image reconstruction to a certain extent.

[0133] Corresponding to the above embodiments, the embodiments of the present disclosure further provide an image reconstruction method, which is an application method of the image reconstruction model obtained by the image reconstruction model training method as described above, as Figure 1 shown, and the method includes: Figure 8 In step S81, the image to be reconstructed and the noise are acquired;

[0134] In step S82, the image to be reconstructed and the noise are input into the target reconstruction model to obtain the target reconstructed image, where the target reconstruction model is trained by the above embodiments.

[0135] Exemplarily, the image to be reconstructed is a noise image, which can be a rain image affected by rain, an image affected by foggy weather, etc. The noise can be Gaussian noise, salt-and-pepper noise, Poisson noise, and impulse noise, etc. The embodiments of the present disclosure do not limit the type of noise, and those skilled in the art can determine it according to the actual situation.

[0136] After acquiring the image to be reconstructed and the noise, the image to be reconstructed and the noise are input into the foregoing as

[0137] shown Figure 1Using the target reconstruction model trained by the illustrated embodiment to reconstruct the target image. Specifically, the target reconstruction model includes a first target reconstruction model and a second target reconstruction model. Inputting the image to be reconstructed and the noise into the target reconstruction model to obtain the target reconstructed image includes: inputting the image features of the image to be reconstructed and the noise into the first target reconstruction model to obtain the target diffusion prior features; and inputting the image features of the image to be reconstructed and the target diffusion prior features into the second target reconstruction model to obtain the target reconstructed image.

[0138] Through the image reconstruction method of the embodiments of the present application, the image to be reconstructed and the noise are obtained; by using the target reconstruction model, the target diffusion prior features of the image to be reconstructed can be obtained according to the image to be reconstructed and the noise; on this basis, further reconstruction is performed according to the target diffusion prior features and the image to be reconstructed to obtain the target reconstructed image. An image that is noise-free and high-resolution and is affected by both weather and low-resolution degradation during reconstruction is realized.

[0139] Corresponding to the implementation manner of the above method, the embodiments of the present disclosure further provide an image reconstruction model training device for executing the image reconstruction model training method of any of the above Figure 1 illustrated embodiments. The device for training the image reconstruction model includes:

[0140] A training set construction module for constructing an image training set, where the image training set includes a noise image subset and a non-noise image subset; the noise image subset and the non-noise image subset correspond one-to-one;

[0141] An encoding module for encoding any noise image in the noise image subset and the non-noise image corresponding to the noise image to obtain prior features;

[0142] A training module for training the reconstruction model to be trained based on the prior features corresponding to any noise image in the image training set and the image features corresponding to the noise image to obtain the target reconstruction model.

[0143] The image reconstruction model training device provided by the above embodiments of the present disclosure and the image reconstruction model training method provided by the embodiments of the present disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run, or implemented by the application programs stored therein.

[0144] Corresponding to the implementation manner of the above method, the embodiments of the present disclosure further provide an image reconstruction device for executing the image reconstruction method of any of the above Figure 8 illustrated embodiments. The image reconstruction device includes:

[0145] An acquisition module for acquiring the image to be reconstructed and the noise;

[0146] A reconstruction module, configured to input the image to be reconstructed and the noise into a target reconstruction model to obtain a target reconstructed image, where the target reconstruction model is trained according to the foregoing embodiments.

[0147] The image reconstruction apparatus provided by the foregoing embodiments of the present disclosure and the image reconstruction method provided by the embodiments of the present disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run, or implemented by the application programs stored therein.

[0148] The present disclosure also provides an electronic device for executing the foregoing method. Please refer to Figure 9 , which shows a schematic diagram of an electronic device provided by some embodiments of the present disclosure. As Figure 9 shown, the electronic device includes: a processor 900, a memory 901, a bus 902, and a communication interface 903. The processor 900, the communication interface 903, and the memory 901 are connected through the bus 902. A computer program that can run on the processor 900 is stored in the memory 901. When the processor 900 runs the computer program, it executes the method provided by any of the foregoing Figure 1 or Figure 8 illustrated embodiments.

[0149] Among them, the memory 901 may include a high-speed random access memory (Random Access Memory, RAM), and may also include a non-volatile memory, such as at least one disk memory. Through at least one communication interface 903 (which can be wired or wireless), a communication connection is established between this system network element and at least one other network element, and the Internet, a wide area network, a local area network, a metropolitan area network, etc. can be used.

[0150] The bus 902 may be an ISA bus, a PCI bus, an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. Among them, the memory 901 is used to store a program. After receiving an execution instruction, the processor 900 executes the program, and the method disclosed in any of the foregoing Figure 1 or Figure 8 illustrated embodiments can be applied to the processor 900 or implemented by the processor 900.

[0151] The processor 900 may be an integrated circuit chip with the ability to process signals. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in the processor 900 or the instructions in the form of software. The above-mentioned processor 900 may be a general-purpose processor, including a central processing unit (CPU for short), a network processor (NP for short), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute each method, step and logic block diagram disclosed in the embodiments of the present disclosure. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present disclosure can be directly embodied as being executed by the hardware decoding processor, or executed by a combination of the hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory 901, and the processor 900 reads the information in the memory 901 and combines its hardware to complete the steps of the above method.

[0152] The electronic device provided by the embodiments of the present disclosure and the method provided by the embodiments of the present disclosure are based on the same inventive concept and have the same beneficial effects as the method adopted, run or implemented by it.

[0153] The present disclosure also provides a computer-readable storage medium corresponding to the method provided by the foregoing embodiment. Please refer to Figure 10 which shows that the computer-readable storage medium is an optical disc 30, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it will execute the method provided by any of the foregoing embodiments.

[0154] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here one by one.

[0155] The computer-readable storage medium provided by the above embodiments of the present disclosure and the method provided by the embodiments of the present disclosure are based on the same inventive concept and have the same beneficial effects as the method adopted, run or implemented by the application program stored in it.

[0156] It should be noted that:

[0157] In the specification provided herein, a large number of specific details are set forth. However, it will be understood that embodiments of the present disclosure may be practiced without these specific details. In some instances, well-known structures and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0158] Similarly, it should be understood that in order to streamline the present disclosure and assist in understanding one or more of the various inventive aspects, in the foregoing description of the exemplary embodiments of the present disclosure, various features of the present disclosure are sometimes grouped together in a single embodiment, figure, or description thereof. However, the disclosed method should not be construed as reflecting an intention that: the claimed present disclosure requires more features than are expressly recited in each embodiment. The inventive aspects lie in less than all the features of the single embodiment(s) disclosed previously. Accordingly, the implementation of the following specific embodiments is hereby expressly incorporated into that specific embodiment, where each embodiment stands on its own as a separate embodiment of the present disclosure.

[0159] Furthermore, those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not others, the combination of features of different embodiments is meant to be within the scope of the present disclosure and forms different embodiments.

[0160] The above is only a preferred specific implementation manner of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present disclosure should be covered by the protection scope of the present disclosure.

Claims

1. A method for training an image reconstruction model, characterized in that: The method comprises: Constructing an image training set, wherein the image training set includes a noise image subset and a non-noise image subset; the noise image subset and the non-noise image subset correspond one to one; Encoding any noise image in the noise image subset and a non-noise image corresponding to the noise image to obtain a priori features; Based on the prior features corresponding to any noise image in the image training set and the image features corresponding to the noise image, the reconstruction model to be trained is trained to obtain a target reconstruction model.

2. The method according to claim 1, characterized in that The step of training the reconstruction model to be trained based on the prior features corresponding to any noise image in the image training set, the first image features corresponding to the noise image, and the second image features corresponding to the non-noise image to obtain the target reconstruction model includes: Perform medium-free reconstruction based on the priori feature and the first image feature to obtain a medium-free reconstruction feature; Perform edge reconstruction based on the priori features and the medium-free reconstruction features to obtain target edge features; Performing feature fusion on the medium-free reconstruction feature, the target edge feature and the prior feature to obtain a reconstructed image; Based on the reconstructed image and the second image features, determining a loss value corresponding to the training reconstruction model to be trained; Based on the relationship between the loss value and the preset standard, the reconstruction model to be trained is optimized and the target reconstruction model is output.

3. The method according to claim 2, characterized in that The performing medium-free reconstruction based on the priori feature and the first image feature to obtain a medium-free reconstruction feature includes: After performing feature splitting on the prior feature, the prior feature is divided into a first sub-prior feature and a second sub-prior feature; The first sub-prior feature is fused with the first image feature to obtain a first modulation feature; the second sub-prior feature is fused with the first image feature to obtain a second modulation feature; Perform guided filtering based on the first modulation feature, the second modulation feature and the priori feature to obtain a medium-free feature; Feature reconstruction is performed based on the medium-free feature and the priori feature to obtain the medium-free reconstructed feature.

4. The method according to claim 3, characterized in that The step of performing guided filtering based on the first modulation feature, the second modulation feature, and the priori feature to obtain a medium-free feature includes: Performing mean filtering on the first modulation feature to obtain a first filtered modulation feature; and performing mean filtering on the priori feature to obtain a filtered priori feature; Performing local relationship feature extraction on the first filtering modulation feature and the filtering priori feature to obtain a local relationship feature; Perform mean filtering on the local relationship feature to obtain a first learning coefficient and a second learning coefficient; The first learning coefficient, the second learning coefficient and the second modulation feature are fused to obtain the medium-free feature.

5. The method according to claim 3, characterized in that: The edge reconstruction is performed based on the priori feature and the medium-free reconstruction feature to obtain the target edge feature, including: The first sub-prior feature is fused with the medium-free reconstruction feature to obtain a third modulation feature; the second sub-prior feature is fused with the third modulation feature to obtain a stage feature; Extracting edge features of the prior features; Perform feature reconstruction based on the edge feature and the stage feature to obtain a reconstructed edge feature; The reconstructed edge feature is compensated to obtain a target edge feature.

6. The method according to claim 5, characterized in that The step of extracting edge features of the prior features comprises: Converting the prior features into frequency domain features; Extracting high-frequency features from the frequency domain features; The high-frequency features are converted into spatial domain features, and the spatial domain features are used as the edge features.

7. The method according to claim 2, characterized in that The step of fusing the medium-free reconstruction feature, the target edge feature and the priori feature to obtain a reconstructed image includes: Multiplying the medium-free reconstruction feature, the target edge feature and the prior feature to obtain a fusion feature; Image reconstruction is performed based on the fusion features to obtain the reconstructed image.

8. The method according to any one of claims 1 to 7, characterized in that: The a priori feature includes a first a priori feature and a second a priori feature, wherein the second a priori feature is a diffusion feature of the first a priori feature; The target reconstruction model includes a first target reconstruction model and a second target reconstruction model. The step of training the reconstruction model to be trained based on the prior features corresponding to any noise image in the image training set, the first image features corresponding to the noise image, and the second image features corresponding to the non-noise image to obtain the target reconstruction model includes: Based on a first priori feature corresponding to any noise image in the image training set, a first image feature corresponding to the noise image, and a second image feature corresponding to the non-noise image, training a first reconstruction model to be trained to obtain the first target reconstruction model; Based on the second priori feature corresponding to any noise image in the image training set, the first image feature corresponding to the noise image, and the second image feature corresponding to the non-noise image, the first reconstruction model to be trained is trained to obtain the second target reconstruction model.

9. The method according to claim 8, characterized in that The second prior feature is obtained by: Adding pre-acquired noise to the first prior feature to obtain a noise prior feature; Extracting conditional features of the noise image; Guided by the conditional feature, the noise prior feature is denoised to obtain the second prior feature.

10. An image reconstruction method, characterized in that: The method comprises: Obtaining an image to be reconstructed and noise; The image to be reconstructed and the noise are input into a target reconstruction model to obtain a target reconstructed image, wherein the target reconstruction model is trained by the method described in any one of claims 1 to 9.

11. The method according to claim 10, characterized in that The target reconstruction model includes a first target reconstruction model and a second target reconstruction model, and the image to be reconstructed and the noise are input into the target reconstruction model to obtain a target reconstructed image, including: Inputting the image features of the image to be reconstructed and the noise into the first target reconstruction model to obtain target diffusion prior features; The image features of the image to be reconstructed and the target diffusion prior features are input into the second target reconstruction model to obtain the target reconstructed image.

Images