Image reconstruction model training and image reconstruction methods

Abstract

This disclosure proposes an image reconstruction model training and image reconstruction method, belonging to the field of image processing technology. The method constructs an image training set. Since the image training set includes a subset of noisy images and a subset of non-noisy images, and the noisy image subset and the non-noisy image subset correspond one-to-one, prior features can be obtained by encoding based on the noisy images and their corresponding non-noisy images. Based on the prior features, and using the prior features corresponding to any noisy image in the image training set and the image features corresponding to the noisy image, the reconstruction model to be trained is trained to obtain the target reconstruction model, which improves the quality of image reconstruction to a certain extent.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, specifically to an image reconstruction model training and image reconstruction method. Background Technology

[0002] Image super-resolution is the process of reconstructing a low-resolution image into a high-resolution image; reverse restoration is the process of restoring an image affected by rain, such as raindrops and rain streaks, to reveal a clean background.

[0003] When an image needs to be reconstructed into a high-resolution image and then reversed, related techniques typically employ a sequential approach of reverse reconstruction and high-resolution reconstruction. However, this image processing method suffers from the problem of low accuracy in reverse reconstruction. Summary of the Invention

[0004] This disclosure proposes an image reconstruction model training method and an image reconstruction method.

[0005] The first aspect of this disclosure proposes an image reconstruction model training method, including:

[0006] Construct an image training set, which includes a noisy image subset and a non-noisy image subset; the noisy image subset and the non-noisy image subset correspond one-to-one.

[0007] Encode any noisy image in the subset of noisy images and the corresponding non-noisy image to obtain prior features;

[0008] Based on the prior features corresponding to any noisy image in the image training set and the image features corresponding to the noisy image, the reconstruction model to be trained is trained to obtain the target reconstruction model.

[0009] In this embodiment of the disclosure, the step of training the reconstruction model to obtain the target reconstruction model based on the prior features corresponding to any noisy image in the image training set, the first image features corresponding to the noisy image, and the second image features corresponding to the non-noisy image includes:

[0010] Based on the prior features and the first image features, media-free reconstruction is performed to obtain media-free reconstruction features;

[0011] Edge reconstruction is performed based on the prior features and the medium-free reconstruction features to obtain the target edge features;

[0012] The reconstructed image is obtained by fusing the media-free reconstruction features, the target edge features, and the prior features.

[0013] Based on the reconstructed image and the features of the second image, determine the loss value corresponding to the training reconstruction model;

[0014] Based on the relationship between the loss value and the preset standard, the reconstruction model to be trained is optimized, and the target reconstruction model is output.

[0015] In this embodiment of the disclosure, the step of performing media-free reconstruction based on the prior features and the first image features to obtain media-free reconstructed features includes:

[0016] After performing feature splitting, the prior features are divided into a first sub-prior feature and a second sub-prior feature.

[0017] The first sub-prior feature is fused with the first image feature to obtain the first modulation feature; the second sub-prior feature is fused with the first image feature to obtain the second modulation feature.

[0018] Guided filtering is performed based on the first modulation feature, the second modulation feature, and the prior feature to obtain a medium-free feature;

[0019] Based on the medium-free features and the prior features, feature reconstruction is performed to obtain the medium-free reconstructed features.

[0020] In this embodiment of the disclosure, the step of performing guided filtering based on the first modulation feature, the second modulation feature, and the prior feature to obtain a medium-free feature includes:

[0021] The first modulation feature is subjected to mean filtering to obtain the first filtered modulation feature; and the prior feature is subjected to mean filtering to obtain the filtered prior feature.

[0022] Local relation features are extracted from the first filter modulation features and the filter prior features to obtain local relation features;

[0023] The local relation features are subjected to mean filtering to obtain the first learning coefficient and the second learning coefficient;

[0024] The first learning coefficient, the second learning coefficient, and the second modulation feature are fused to obtain the medium-free feature.

[0025] In this embodiment of the disclosure, the step of performing edge reconstruction based on the prior features and the medium-free reconstruction features to obtain target edge features includes:

[0026] The first sub-prior feature is fused with the medium-free reconstruction feature to obtain the third modulation feature; the second sub-prior feature is fused with the third modulation feature to obtain the stage feature;

[0027] Extract the edge features of the prior features;

[0028] Based on the edge features and the stage features, feature reconstruction is performed to obtain the reconstructed edge features;

[0029] The reconstructed edge features are compensated to obtain the target edge features.

[0030] In this embodiment of the disclosure, the extraction of edge features of the prior features includes:

[0031] The prior features are converted into frequency domain features;

[0032] Extract the high-frequency features from the frequency domain features;

[0033] The high-frequency features are converted into spatial domain features, and the spatial domain features are used as the edge features.

[0034] In this embodiment of the disclosure, the feature fusion of the medium-free reconstruction features, the target edge features, and the prior features to obtain the reconstructed image includes:

[0035] The media-free reconstruction features, the target edge features, and the prior features are multiplied together to obtain the fused features;

[0036] The image is reconstructed based on the fusion features to obtain the reconstructed image.

[0037] In this embodiment of the disclosure, the prior features include a first prior feature and a second prior feature, wherein the second prior feature is a diffusion feature of the first prior feature; the target reconstruction model includes a first target reconstruction model and a second target reconstruction model.

[0038] The step of training a reconstruction model based on prior features corresponding to any noisy image in the image training set, first image features corresponding to the noisy image, and second image features corresponding to the non-noisy image to obtain the target reconstruction model includes:

[0039] Based on the first prior feature corresponding to any noisy image in the image training set, the first image feature corresponding to the noisy image, and the second image feature corresponding to the non-noisy image, a first reconstruction model to be trained is trained to obtain the first target reconstruction model.

[0040] Based on the second prior feature corresponding to any noisy image in the image training set, the first image feature corresponding to the noisy image, and the second image feature corresponding to the non-noisy image, a first reconstruction model to be trained is trained to obtain the second target reconstruction model.

[0041] In this embodiment of the disclosure, the second prior feature is obtained in the following manner:

[0042] Add pre-acquired noise to the first prior feature to obtain the noisy prior feature;

[0043] Extract the conditional features of the noisy image;

[0044] Guided by the conditional features, the noise prior features are denoised to obtain the second prior features.

[0045] An embodiment of the second aspect of this disclosure provides an image reconstruction method, the method comprising:

[0046] Obtain the image to be reconstructed and the noise;

[0047] The image to be reconstructed and the noise are input into the target reconstruction model to obtain the target reconstruction image, wherein the target reconstruction model is trained by the method described in the first aspect or any optional embodiment of the first aspect.

[0048] In this embodiment of the disclosure, the target reconstruction model includes a first target reconstruction model and a second target reconstruction model. The step of inputting the image to be reconstructed and the noise into the target reconstruction model to obtain the target reconstructed image includes:

[0049] The image features of the image to be reconstructed and the noise are input into the first target reconstruction model to obtain the target diffusion prior features;

[0050] The image features of the image to be reconstructed and the target diffusion prior features are input into the second target reconstruction model to obtain the target reconstruction image.

[0051] An embodiment of the third aspect of this disclosure provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the methods described in the first aspect or any optional embodiment of the first aspect, the second aspect, and any optional embodiment of the second aspect.

[0052] An embodiment of the fourth aspect of this disclosure provides a computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the methods described in the first aspect and any optional embodiments of the first aspect, the second aspect and any optional embodiments of the second aspect.

[0053] The technical solutions provided in this disclosure have at least the following technical effects or advantages:

[0054] An image training set is constructed. Since the image training set includes a subset of noisy images and a subset of non-noisy images, and the subset of noisy images and the subset of non-noisy images correspond one-to-one, prior features can be obtained by encoding based on the noisy images and the corresponding non-noisy images. Based on the prior features, the reconstruction model to be trained is trained based on the prior features corresponding to any noisy image in the image training set and the image features corresponding to the noisy image, thus obtaining the target reconstruction model, which improves the quality of image reconstruction to a certain extent.

[0055] Additional aspects and advantages of this disclosure will be set forth in part in the description which follows, and in part will be obvious from the description or may be learned by practice of this disclosure. Attached Figure Description

[0056] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this disclosure. Furthermore, the same reference numerals denote the same parts throughout the drawings.

[0057] In the attached diagram:

[0058] Figure 1 A flowchart of an image reconstruction model training method provided in an embodiment of this disclosure is shown;

[0059] Figure 2 This diagram illustrates the extraction of the first prior feature in an image reconstruction model training method provided in an embodiment of the present disclosure.

[0060] Figure 3 This diagram illustrates the extraction of a second prior feature in an image reconstruction model training method provided in an embodiment of the present disclosure.

[0061] Figure 4 A schematic diagram of image reconstruction in an image reconstruction model training method provided in an embodiment of this disclosure is shown;

[0062] Figure 5 This diagram illustrates media removal in an image reconstruction model training method provided in an embodiment of the present disclosure;

[0063] Figure 6 A schematic diagram of guided filtering in an image reconstruction model training method provided in an embodiment of this disclosure is shown;

[0064] Figure 7 This diagram illustrates edge feature extraction in an image reconstruction model training method provided in an embodiment of the present disclosure;

[0065] Figure 8A flowchart of an image reconstruction method provided in one embodiment of this disclosure is shown;

[0066] Figure 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure is shown;

[0067] Figure 10 A schematic diagram of a storage medium provided according to an embodiment of the present disclosure is shown. Detailed Implementation

[0068] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0069] It should be noted that, unless otherwise stated, the technical or scientific terms used in this disclosure shall have the ordinary meaning as understood by one of ordinary skill in the art to which this disclosure pertains.

[0070] This disclosure proposes an image reconstruction model training method, such as... Figure 1 The image shown is an embodiment of an image reconstruction model training method provided by this disclosure, including the following steps:

[0071] In step S11, an image training set is constructed.

[0072] The image training set includes a noisy image subset and a non-noisy image subset; the noisy image subset and the non-noisy image subset correspond one-to-one.

[0073] For example, noisy images in the image training set correspond to non-noisy images. Depending on the actual application scenario of the image reconstruction model in this embodiment, the image training set can be constructed according to the type of application scenario. Specifically, if the image reconstruction model is applied to low-resolution images or images affected by weather, the noisy images in the image training set can all be images affected by weather, while the non-noisy images can be noisy images corresponding to rainy days, noisy images corresponding to heavy fog, etc.

[0074] It is important to note that for scenes corresponding to the same weather type, the noisy images should also be of the same type. For example, for rainy scenes, the noisy images should all be images affected by rain.

[0075] In image reconstruction, high-resolution features corresponding to the non-noisy image are usually required for compensation to obtain a reconstructed image with high accuracy. However, in practical applications, high-resolution features are often unavailable for compensation. Therefore, this embodiment analyzes the prior features of the image, so that when using the image reconstruction model of this embodiment for subsequent image reconstruction, only the noisy image is needed to obtain the corresponding prior features.

[0076] In step S12, any noisy image in the subset of noisy images and the corresponding non-noisy image are encoded to obtain prior features.

[0077] For example, before encoding, noisy and non-noisy images can be preprocessed to ensure that all images have the same size for subsequent processing. Specifically, normalization can be used to normalize pixel values ​​to the range of [0,1] or [-1,1], and data diversity can be increased through rotation, flipping, etc.

[0078] like Figure 2 As shown, the process of extracting prior features can map the image features corresponding to both noisy and non-noisy images to the latent space to obtain the corresponding prior features.

[0079] Specifically, features need to be extracted from the noisy image and its corresponding non-noisy image. This can be achieved using traditional feature extraction methods such as convolution. The extracted features may need to be transformed to better capture image content and noise information. For example, a dual-stream UNet can be used as a noise predictor, utilizing a dual-stream encoder to extract features from the noisy image and the prior image separately, enhancing the representation of noise features through differences, thereby improving denoising accuracy. The extracted and transformed features need to be encoded to obtain prior features. This can be achieved through methods such as... Figure 2 The encoder implementation is shown. The encoded features can be further mapped to the latent space to obtain prior features.

[0080] In some embodiments, the prior features include a first prior feature and a second prior feature, wherein the second prior feature is a diffusion feature of the first prior feature; the target reconstruction model includes a first target reconstruction model and a second target reconstruction model, wherein the first prior feature is used to extract the prior features corresponding to the existence of a non-noise image during the training process of the image reconstruction model in this embodiment. The second prior feature is used for training the prior feature extraction corresponding to the absence of a non-noise image in this embodiment, so that when performing image reconstruction using the trained image reconstruction model, only a noisy image is needed to achieve image reconstruction.

[0081] In step S13, the target reconstruction model is obtained by training the reconstruction model based on the prior features corresponding to any noisy image in the image training set and the image features corresponding to the noisy image.

[0082] For example, prior features and image features can be fused. This can be achieved through simple concatenation, weighted summation, or more complex attention mechanisms.

[0083] Design a reconstruction model whose input is the fused features and whose output is the restored clean image. The reconstruction model can be based on a CNN architecture, such as U-Net, the generator part of a GAN (Generative Adversarial Network), or other architectures suitable for image-to-image translation.

[0084] Choose one or more loss functions to measure the difference between the reconstructed image and the real clean image. Commonly used loss functions include mean squared error (MSE), peak signal-to-noise ratio (PSNR) related loss, adversarial loss (if using GAN), perceptual loss (based on distance in the feature space of the pre-trained network), etc.

[0085] The reconstruction model is trained using training data (noisy images, non-noisy images, and corresponding features and prior features). During training, the model weights are updated using the backpropagation algorithm to minimize the loss function. Optimizers such as Adam and SGD can be used for weight updates. The model's performance is evaluated on a validation set, and the model architecture, loss function, and training strategy are adjusted based on the evaluation results. Quantitative metrics (such as PSNR and SSIM) and qualitative analyses (such as visualizing the reconstruction results) can be used to evaluate the model. When the model achieves satisfactory performance on the validation set, training is stopped, and the current model is saved as the target reconstruction model.

[0086] In some embodiments, the prior features include a first prior feature and a second prior feature, wherein the second prior feature is a diffusion feature of the first prior feature; the target reconstruction model includes a first target reconstruction model and a second target reconstruction model, wherein the first prior feature is used to extract the prior features corresponding to the existence of a non-noise image during the training process of the image reconstruction model in this embodiment. The second prior feature is used for training the prior feature extraction corresponding to the absence of a non-noise image in this embodiment, so that when performing image reconstruction using the trained image reconstruction model, only a noisy image is needed to achieve image reconstruction.

[0087] The second prior feature is obtained by adding pre-acquired noise to the first prior feature to obtain the noise prior feature; extracting the conditional features of the noisy image; and denoising the noise prior feature based on the conditional features to obtain the second prior feature.

[0088] For example, such as Figure 3 As shown, based on the first prior feature, preset Gaussian noise is added to the first prior feature for denoising, resulting in the second prior feature. The denoising process is guided by the conditions generated from the noisy image. Figure 3 Encoder E1 and Figure 2 The encoder E1 in the code is the same encoder. Given that the first prior feature is known, when determining the second prior feature, the corresponding first prior feature P can be directly obtained, that is... Figure 3 The parameters of encoder E1 shown in the diagram remain unchanged.

[0089] Specifically, Gaussian noise is added using the following formula:

[0090]

[0091] Among them, P t Let N be a Gaussian distribution with a noise prior of time step t, and α be a noise prior of time step t. t =1-β t , α t It is the scaling factor that controls the noise variance, I is the identity matrix, and β is the scaling factor. t The scaling factor for controlling the noise variance can be set manually.

[0092] The denoising process can follow a Markov chain, starting from P t To P t-1 The reverse process can be described as:

[0093]

[0094] in Let be the Gaussian noise intensity at time t, where the Gaussian noise intensity is represented by variance; ε is the Gaussian noise intensity at time t. t Noise in the noise. In the noise reduction stage, methods such as... Figure 3 The encoder E2 in the code encodes the input noisy image and outputs conditions C and P. t As a condition for the denoising network to predict noise:

[0095]

[0096] Where ε t This represents the estimated noise ε at each step. Through T iterations of the above sampling, the second prior feature can be generated.

[0097] Step S13 above can also be implemented in the following way: based on the first prior feature corresponding to any noisy image in the image training set, the first image feature corresponding to the noisy image, and the second image feature corresponding to the non-noisy image, train a first reconstruction model to be trained to obtain a first target reconstruction model; based on the second prior feature corresponding to any noisy image in the image training set, the first image feature corresponding to the noisy image, and the second image feature corresponding to the non-noisy image, train the first reconstruction model to be trained to obtain a second target reconstruction model.

[0098] For example, in some embodiments, the training of the model to be reconstructed is divided into two stages: training a first reconstructed model to be trained and training a second reconstructed model to be trained. The prior features of the first reconstructed model to be trained are first prior features; the prior features of the second reconstructed model to be trained are second prior features. All other input and output parameters are the same. After the first reconstructed model to be trained completes training based on images in the image training set to obtain the first target reconstruction model, the second reconstructed model to be trained is then trained based on images in the image training set.

[0099] Specifically, in this embodiment of the disclosure, the loss function of the first target reconstruction model may be:

[0100] L S1 =||I Rec -I GT ||1

[0101] Among them, I Rec1 For the reconstructed image obtained based on the first prior features, I GT This is a non-noise image.

[0102] The loss function for the second-objective reconstruction model can be:

[0103]

[0104] Among them, I Rec2 The reconstructed image is obtained based on the second prior feature. P is the second prior feature, and P is the first prior feature.

[0105] In the actual training and optimization process, the channel count can be set to 64, N to 12, and the time step for determining the second prior feature can be set to 4. During the training phase, the Adam optimizer is used: β1 = 0.9, β2 = 0.99. The initial learning rate is set to 0.0002, and the batch size for training is 16.

[0106] In some embodiments, step S13 above can be implemented in the following ways: performing media-free reconstruction based on prior features and first image features to obtain media-free reconstruction features; performing edge reconstruction based on prior features and media-free reconstruction features to obtain target edge features; fusing media-free reconstruction features, target edge features, and prior features to obtain a reconstructed image; determining the loss value corresponding to the training reconstruction model based on the reconstructed image and second image features; optimizing the training reconstruction model based on the relationship between the loss value and a preset standard, and outputting the target reconstruction model.

[0107] For example, the effects of media in an image (such as haze, rain blur, etc.) are removed to obtain media-free image features. Specifically, prior features and first image features (features extracted directly from the noisy image) are used as input. A media-free reconstruction network is designed, which can be based on a convolutional neural network architecture, such as DehazeNet, underwater image enhancement networks, etc. The trained media-free reconstruction network outputs media-free reconstructed features.

[0108] Sharp edge information is recovered from the features of the demediated image. Specifically, prior features and the demediated reconstructed features obtained in the previous step can be used as input. An edge reconstruction network is designed, which can be a deep learning model based on edge detection, or a combination of traditional edge detection algorithms (such as Canny and Sobel) and deep learning feature extraction methods. The trained edge reconstruction network outputs the target edge features.

[0109] Finally, the media-free reconstruction features, target edge features, and prior features are fused to obtain a rich image representation for image reconstruction. Specifically, a feature fusion module is designed, which can be a simple concatenation or weighted summation, or a more complex attention mechanism, recurrent neural network, etc. The three features mentioned above are input into the feature fusion module, and the fused features are output, which is the feature representation of the reconstructed image.

[0110] An image reconstruction network can be based on a Generative Adversarial Network (GAN) architecture, or a simple Convolutional Neural Network (CNN) for upsampling and image generation. The fused features are input into the image reconstruction network, and the output is a reconstructed image.

[0111] The reconstructed image is compared using second image features (features extracted from the non-noise image). Losses can be calculated using mean squared error (MSE), peak signal-to-noise ratio (PSNR), structural similarity (SSIM) related losses, or a combination of adversarial loss, perceptual loss, etc. The model weights are updated using an optimizer (such as Adam or SGD) and the loss value. During training, changes in the loss value are monitored, and the model is validated and adjusted as needed. Training stops when the loss value converges to a satisfactory level, and the model is saved as the target reconstruction model.

[0112] In some embodiments, the process by which the above-mentioned target reconstruction model obtains the reconstructed target image can also be achieved through methods such as... Figure 4 The process is implemented as shown. Specifically, after obtaining the first prior feature, media removal is performed based on the noisy image and the first prior feature P, and the media-free feature F is reconstructed. MR Next, the first prior feature P and the medium-free reconstruction feature F are used. MR Perform texture compensation to obtain the target edge features F TR Finally, after feature fusion, upsampling is performed to obtain the reconstructed image.

[0113] The medium-free reconstruction features can also be obtained in the following ways: after feature splitting of the prior features, they are divided into a first sub-priority feature and a second sub-priority feature; the first sub-priority feature is fused with the first image feature to obtain a first modulation feature; the second sub-priority feature is fused with the first image feature to obtain a second modulation feature; guided filtering is performed based on the first modulation feature, the second modulation feature, and the prior features to obtain the medium-free features; feature reconstruction is performed based on the medium-free features and the prior features to obtain the medium-free reconstruction features.

[0114] For example, the process of media-free reconstruction features, guided filtering, and subsequent edge reconstruction is introduced using the first prior feature P as an example.

[0115] like Figure 5 As shown, the first prior feature P∈R 4C First image feature F in ∈R H×W×C The first prior feature P is split into a set of vectors {α,β1,…,β}. n}∈R C×1×1 Where α is the first sub-prior feature, β i Let i be the first sub-prior feature, and 1 < i < n.

[0116] Next are α and F in The first modulation feature F′∈R is obtained by performing feature multiplication and feature fusion. H×W×C , and β i With Fin The second modulation feature F″∈R is obtained by performing feature multiplication and feature fusion. H×W×C The first modulation feature F′∈R H×W×C The second modulation feature F″∈R H×W×C And the first prior feature P is input to... Figure 6 The guided filtering module shown obtains the medium-free characteristic F. G ∈R H×W×C Next, leveraging cross-self-attention, based on the mediumless feature F... G The media-free reconstruction feature F was reconstructed from the prior feature P. MR Before using cross-self-attention for reconstruction, the medium-free feature F can also be... G Embedded into global features (F G The KV vector is obtained by feature splitting after 1x1 convolution, and the Q vector is obtained by convolution and dimensionality transformation of P. These vectors are then projected onto the vector {Q,K,V}∈R. HW×C Reconstruction will then take place.

[0117] Furthermore, the medium-free features obtained by guided filtering can also be determined in the following ways: mean filtering is applied to the first modulation feature to obtain the first filtered modulation feature; mean filtering is applied to the prior feature to obtain the filtered prior feature; local relation features are extracted from the first filtered modulation feature and the filtered prior feature to obtain the local relation feature; mean filtering is applied to the local relation feature to obtain the first learning coefficient and the second learning coefficient; the first learning coefficient, the second learning coefficient, and the second modulation feature are fused to obtain the medium-free features.

[0118] For example, such as Figure 6 As shown, mean filtering of the first modulation feature F′ and the first prior feature P (taking the first prior feature as an example) can be obtained by the following formula:

[0119]

[0120] in, This is the first filter modulation feature. These are the prior features for filtering.

[0121] The first filter modulation features and filter prior features are input into the local linear model to obtain local relation features. Specifically, the following formula is used to extract local relation features.

[0122]

[0123] Here, coefficients A and B represent the local linear relationship between the input and output. A is the effect of the quantized input on the output, and B is the capture of low-frequency details. ∑P refers to the local features captured within a specified window, while ∑PF′ refers to the relationship between the prior feature P and the first modulation feature, which is used to improve the guided filter.

[0124] To maintain smooth consistency, mean filtering is applied to generate learning coefficients, specifically the first learning coefficient A′ and the second learning coefficient B′. These are then fused with the second filtered modulation feature F″ (feature concatenation + convolutional dimensionality reduction) to generate the medium-free feature F. G :

[0125]

[0126] In some embodiments, the target edge features can also be obtained by: fusing the first sub-prior features with the medium-free reconstruction features to obtain the third modulation features; fusing the second sub-prior features with the third modulation features to obtain the stage features; extracting the edge features of the prior features; performing feature reconstruction based on the edge features and the stage features to obtain the reconstructed edge features; and compensating the reconstructed edge features to obtain the target edge features.

[0127] For example, such as Figure 7 As shown, the first sub-prior feature α and the media-free reconstruction feature F MR The third modulation feature F1 is obtained by fusion. Then, the third modulation feature F1 is fused with the second sub-prior feature β. i The fusion process yields stage feature F2. The specific fusion method is the same as the fusion method of the first modulation feature and the second modulation feature mentioned above, and will not be repeated here.

[0128] Edge detection algorithms, such as Sobel, Canny, or learning-based methods (such as Hough transform or deep learning models), can be used to extract edge information from prior features. Reconstruction algorithms, such as sparse coding, dictionary learning, or deep learning models (such as autoencoders or generative adversarial networks), are then used to combine edge features and stage features for feature reconstruction, resulting in reconstructed edge features. Based on specific compensation strategies or algorithms, the reconstructed edge features are adjusted or optimized to obtain the target edge features. This may involve error correction, contrast enhancement, or other image enhancement techniques.

[0129] In some embodiments, extracting edge features of prior features includes: converting prior features into frequency domain features; extracting high-frequency features from the frequency domain features; converting high-frequency features into spatial domain features, and using the spatial domain features as edge features.

[0130] For example, when extracting edge features, a high-pass filter can be applied to extract the high-frequency components of the first prior feature P. Specifically, a one-dimensional discrete cosine transform (DCT) can be performed along the channel to transform the first prior feature P from the spatial domain to the frequency domain. After filtering the high-frequency components, the spatial domain features are recovered through a one-dimensional inverse discrete cosine transform (IDCT) to extract the edge features F. edge .

[0131] After obtaining the media-free reconstruction features, target edge features, and prior features, feature fusion is performed to obtain the reconstructed image. Specifically, the media-free reconstruction features, target edge features, and prior features can be multiplied to obtain fused features; image reconstruction is then performed based on the fused features to obtain the reconstructed image.

[0132] The image reconstruction model training method of this application constructs an image training set. Since the image training set includes a subset of noisy images and a subset of non-noisy images, and the subset of noisy images and the subset of non-noisy images correspond one-to-one, prior features can be obtained by encoding based on the noisy images and the corresponding non-noisy images. Based on the prior features, the reconstruction model to be trained is trained based on the prior features corresponding to any noisy image in the image training set and the image features corresponding to the noisy image, thereby obtaining the target reconstruction model, which improves the quality of image reconstruction to a certain extent.

[0133] Corresponding to the above embodiments, this disclosure also provides an image reconstruction method, which is as described above. Figure 1 The image reconstruction model training method shown is used to apply the image reconstruction model obtained by the following methods: Figure 8 As shown, the method includes:

[0134] In step S81, the image to be reconstructed and the noise are acquired;

[0135] In step S82, the image to be reconstructed and noise are input into the target reconstruction model to obtain the target reconstruction image, wherein the target reconstruction model is trained by the above embodiment.

[0136] For example, the image to be reconstructed is a noisy image, such as a rain image affected by rain or an image affected by fog. The noise can be Gaussian noise, salt-and-pepper noise, Poisson noise, impulse noise, etc. This disclosure does not limit the type of noise, and those skilled in the art can determine it according to the actual situation.

[0137] After obtaining the image to be reconstructed and the noise, the image to be reconstructed and the noise are input into the aforementioned... Figure 1The illustrated embodiment uses a trained target reconstruction model to reconstruct a target image. Specifically, the target reconstruction model includes a first target reconstruction model and a second target reconstruction model. The image to be reconstructed and noise are input into the target reconstruction model to obtain the target reconstructed image. This includes: inputting the image features of the image to be reconstructed and noise into the first target reconstruction model to obtain target diffusion prior features; and inputting the image features of the image to be reconstructed and the target diffusion prior features into the second target reconstruction model to obtain the target reconstructed image.

[0138] The image reconstruction method of this application acquires the image to be reconstructed and noise; using a target reconstruction model, the target diffusion prior features of the image to be reconstructed can be obtained based on the image to be reconstructed and the noise; based on this, further reconstruction is performed according to the target diffusion prior features and the image to be reconstructed to obtain the target reconstructed image. This achieves the reconstruction of noise-free, high-resolution images simultaneously affected by weather and low-resolution degradation.

[0139] Corresponding to the above implementation methods, this disclosure also provides an image reconstruction model training apparatus for performing the above-described methods. Figure 1 An image reconstruction model training method according to any of the illustrated embodiments, the apparatus for training the image reconstruction model includes:

[0140] A training set construction module is used to construct an image training set, which includes a noisy image subset and a non-noisy image subset; the noisy image subset and the non-noisy image subset correspond one-to-one.

[0141] The encoding module is used to encode any noisy image in the subset of noisy images and the non-noisy image corresponding to the noisy image to obtain prior features;

[0142] The training module is used to train the reconstruction model to be trained based on the prior features corresponding to any noisy image in the image training set and the image features corresponding to the noisy image, so as to obtain the target reconstruction model.

[0143] The image reconstruction model training apparatus and the image reconstruction model training method provided in the above embodiments of this disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0144] Corresponding to the above implementation methods, this disclosure also provides an image reconstruction apparatus for performing the above-described methods. Figure 8 An image reconstruction method according to any of the illustrated embodiments, the image reconstruction apparatus comprising:

[0145] The acquisition module is used to acquire the image to be reconstructed and the noise.

[0146] The reconstruction module is used to input the image to be reconstructed and the noise into the target reconstruction model to obtain the target reconstructed image, wherein the target reconstruction model is trained by the above embodiment.

[0147] The image reconstruction apparatus and the image reconstruction method provided in the above embodiments of this disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0148] This disclosure also provides an electronic device for performing the above-described method. Please refer to... Figure 9 This illustrates a schematic diagram of an electronic device provided by some embodiments of the present disclosure. For example... Figure 9 As shown, the electronic device includes: a processor 900, a memory 901, a bus 902, and a communication interface 903. The processor 900, the communication interface 903, and the memory 901 are connected via the bus 902. The memory 901 stores a computer program that can run on the processor 900. When the processor 900 runs the computer program, it executes the aforementioned provisions of this disclosure. Figure 1 or Figure 8 The method provided by any of the illustrated embodiments.

[0149] The memory 901 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 903 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.

[0150] Bus 902 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. Memory 901 is used to store programs, and the processor 900 executes the programs after receiving execution instructions. Figure 1 or Figure 8 The illustrated method can be applied to or implemented by the processor 900.

[0151] The processor 900 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed through integrated logic circuits in the hardware of the processor 900 or instructions in software form. The processor 900 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this disclosure can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 901. Processor 900 reads the information in memory 901 and, in conjunction with its hardware, completes the steps of the above method.

[0152] The electronic devices and methods provided in this disclosure are based on the same inventive concept and have the same beneficial effects as the methods they employ, operate, or implement.

[0153] This disclosure also provides a computer-readable storage medium corresponding to the method provided in the foregoing embodiments. Please refer to... Figure 10 The computer-readable storage medium shown is an optical disc 30, on which a computer program (i.e., a program product) is stored, which, when run by a processor, executes the methods provided in any of the foregoing embodiments.

[0154] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0155] The computer-readable storage medium provided in the above embodiments of this disclosure and the methods provided in the embodiments of this disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0156] It should be noted that:

[0157] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this disclosure may be practiced without these specific details. In some instances, well-known structures and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0158] Similarly, it should be understood that, in order to simplify this disclosure and aid in understanding one or more of the various inventive aspects, in the foregoing description of exemplary embodiments of this disclosure, various features of this disclosure are sometimes grouped together in a single embodiment, figure, or description thereof. However, this approach to disclosure should not be construed as reflecting a schematic diagram in which the claimed disclosure requires more features than are explicitly stated in each embodiment. The inventive aspect lies in fewer than all features of a single foregoing disclosed embodiment. Therefore, the implementation following the detailed description is hereby explicitly incorporated into this detailed description, wherein each embodiment itself serves as a separate embodiment of this disclosure.

[0159] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of this disclosure and form different embodiments.

[0160] The above description is merely a preferred embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for training an image reconstruction model, characterized in that, The method includes: Construct an image training set, which includes a noisy image subset and a non-noisy image subset; the noisy image subset and the non-noisy image subset correspond one-to-one. Encode any noisy image in the subset of noisy images and the corresponding non-noisy image to obtain prior features; Based on the prior features corresponding to any noisy image in the image training set, the first image features corresponding to the noisy image, and the second image features corresponding to the non-noisy image, the reconstruction model to be trained is trained to obtain the target reconstruction model: The step of training the reconstruction model based on the prior features corresponding to any noisy image in the image training set, the first image features corresponding to the noisy image, and the second image features corresponding to the non-noisy image to obtain the target reconstruction model includes: Based on the prior features and the first image features, media-free reconstruction is performed to obtain media-free reconstruction features; Edge reconstruction is performed based on the prior features and the medium-free reconstruction features to obtain the target edge features; The reconstructed image is obtained by fusing the media-free reconstruction features, the target edge features, and the prior features. Based on the reconstructed image and the features of the second image, the loss value corresponding to the reconstruction model to be trained is determined; Based on the relationship between the loss value and the preset standard, the reconstruction model to be trained is optimized, and the target reconstruction model is output. The prior features include a first prior feature and a second prior feature, wherein the second prior feature is a diffusion feature of the first prior feature; the target reconstruction model includes a first target reconstruction model and a second target reconstruction model. The step of training a reconstruction model based on prior features corresponding to any noisy image in the image training set, first image features corresponding to the noisy image, and second image features corresponding to the non-noisy image to obtain the target reconstruction model includes: Based on the first prior feature corresponding to any noisy image in the image training set, the first image feature corresponding to the noisy image, and the second image feature corresponding to the non-noisy image, a first reconstruction model to be trained is trained to obtain the first target reconstruction model. Based on the second prior feature corresponding to any noisy image in the image training set, the first image feature corresponding to the noisy image, and the second image feature corresponding to the non-noisy image, a first reconstruction model to be trained is trained to obtain the second target reconstruction model. The second prior feature is obtained in the following way: Add pre-acquired noise to the first prior feature to obtain the noisy prior feature; Extract the conditional features of the noisy image; Guided by the conditional features, the noise prior features are denoised to obtain the second prior features.

2. The method according to claim 1, characterized in that, The process of performing media-free reconstruction based on the prior features and the first image features to obtain media-free reconstruction features includes: After performing feature splitting, the prior features are divided into a first sub-prior feature and a second sub-prior feature. The first sub-prior feature is fused with the first image feature to obtain the first modulation feature; the second sub-prior feature is fused with the first image feature to obtain the second modulation feature. Guided filtering is performed based on the first modulation feature, the second modulation feature, and the prior feature to obtain a medium-free feature; Based on the medium-free features and the prior features, feature reconstruction is performed to obtain the medium-free reconstructed features.

3. The method according to claim 2, characterized in that, The guided filtering based on the first modulation feature, the second modulation feature, and the prior feature to obtain the medium-free feature includes: The first modulation feature is subjected to mean filtering to obtain the first filtered modulation feature; and the prior feature is subjected to mean filtering to obtain the filtered prior feature. Local relation features are extracted from the first filter modulation features and the filter prior features to obtain local relation features; The local relation features are subjected to mean filtering to obtain the first learning coefficient and the second learning coefficient; The first learning coefficient, the second learning coefficient, and the second modulation feature are fused to obtain the medium-free feature.

4. The method according to claim 2, characterized in that, The edge reconstruction based on the prior features and the medium-free reconstruction features to obtain the target edge features includes: The first sub-prior feature is fused with the medium-free reconstruction feature to obtain the third modulation feature; the second sub-prior feature is fused with the third modulation feature to obtain the stage feature; Extract the edge features of the prior features; Based on the edge features and the stage features, feature reconstruction is performed to obtain the reconstructed edge features; The reconstructed edge features are compensated to obtain the target edge features.

5. The method according to claim 4, characterized in that, The extraction of edge features from the prior features includes: The prior features are converted into frequency domain features; Extract the high-frequency features from the frequency domain features; The high-frequency features are converted into spatial domain features, and the spatial domain features are used as the edge features.

6. The method according to claim 1, characterized in that, The step of fusing the medium-free reconstruction features, the target edge features, and the prior features to obtain the reconstructed image includes: The media-free reconstruction features, the target edge features, and the prior features are multiplied together to obtain the fused features; The image is reconstructed based on the fusion features to obtain the reconstructed image.

7. An image reconstruction method, characterized in that, The method includes: Obtain the image to be reconstructed and the noise; The image to be reconstructed and the noise are input into the target reconstruction model to obtain the target reconstruction image, wherein the target reconstruction model is trained by the method described in any one of claims 1-6.

8. The method according to claim 7, characterized in that, The target reconstruction model includes a first target reconstruction model and a second target reconstruction model. The step of inputting the image to be reconstructed and the noise into the target reconstruction model to obtain the target reconstructed image includes: The image features of the image to be reconstructed and the noise are input into the first target reconstruction model to obtain the target diffusion prior features; The image features of the image to be reconstructed and the target diffusion prior features are input into the second target reconstruction model to obtain the target reconstruction image.

Images