Image super-resolution reconstruction method and system based on deep learning model

Abstract

The invention discloses an image super-resolution reconstruction method and system based on a deep learning model, and the method comprises the steps: collecting a preset number of high-resolution images, and obtaining a degradation phenomenon which may occur during the imaging of a target imaging device; degenerating the high-resolution image according to a preset rule on the basis of a degeneration phenomenon possibly occurring during imaging of the target imaging equipment to obtain a corresponding low-resolution image; establishing a training data set according to the high-resolution image and the corresponding low-resolution image, constructing a preset neural network and a loss function, and inputting the training data set into the preset neural network for training until the loss tends to be stable to obtain an image super-resolution reconstruction model; and obtaining a to-be-reconstructed low-resolution image, and inputting the to-be-reconstructed low-resolution image into the corresponding image super-resolution reconstruction model to obtain a corresponding reconstructed high-resolution image. The low-resolution image generated by the method is closer to real degradation distribution, so that the quality of super-resolution reconstruction of the image is improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image super-resolution reconstruction method and system based on a deep learning model. Background Technology

[0002] Image super-resolution reconstruction technology aims to restore low-resolution (LR) images to high-resolution (HR) images using algorithms. Its core value lies in improving the spatial resolution and detail of images, thereby meeting the demand for high-quality images in fields such as security monitoring, medical diagnosis, remote sensing observation, and high-definition display. For example, in security monitoring, low-resolution facial or license plate images often fail to meet recognition requirements; super-resolution technology can achieve effective identity verification through detail enhancement.

[0003] With the development of computer vision and deep learning technologies, super-resolution methods based on convolutional neural networks (CNNs), such as SRCNN, have achieved end-to-end pixel mapping learning. By modeling the mapping relationship between high- and low-resolution images through deep networks, they have significantly improved reconstruction accuracy. However, the performance of existing deep learning methods is highly dependent on the quality and quantity of paired training data ("low-resolution - high-resolution"). Ideally, the training data should be strictly matched to the imaging equipment and degradation conditions of the actual application scenario, but obtaining such data in real-world scenarios faces many challenges, becoming a core bottleneck restricting the practical application of super-resolution technology.

[0004] In existing technologies, the generation of low-resolution images often relies on simplified mathematical models (such as fixed kernel downsampling and globally uniform noise addition), ignoring the physical characteristics of real imaging devices. This results in poor generalization ability of the reconstruction model, which in turn leads to poor image super-resolution reconstruction quality. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide an image super-resolution reconstruction method and system based on a deep learning model, which aims to solve the problem of poor image super-resolution reconstruction quality in the prior art.

[0006] This invention is implemented as follows:

[0007] A method for image super-resolution reconstruction based on a deep learning model, the method comprising:

[0008] Acquire a preset number of high-resolution images and obtain the degradation phenomena that may occur when the target imaging device is imaging;

[0009] Based on the degradation phenomenon that may occur when the target imaging device is imaging, the high-resolution image is degraded according to the preset rules to obtain the corresponding low-resolution image.

[0010] A training dataset is established based on high-resolution images and corresponding low-resolution images. A pre-defined neural network and loss function are constructed, and the training dataset is input into the pre-defined neural network for training until the loss tends to stabilize to obtain the image super-resolution reconstruction model.

[0011] Obtain the low-resolution image to be reconstructed, and input the low-resolution image to be reconstructed into the corresponding image super-resolution reconstruction model to obtain the corresponding reconstructed high-resolution image.

[0012] Furthermore, in the aforementioned image super-resolution reconstruction method based on a deep learning model, the step of degrading the high-resolution image to obtain the corresponding low-resolution image according to a preset rule based on the degradation phenomenon that may occur during imaging by the target imaging device includes:

[0013] A physical model is constructed based on the degradation phenomena that may occur when the target imaging device is used for imaging, and the high-resolution image is degraded based on the corresponding physical model.

[0014] For lens blur, a point spread function model based on the principle of light propagation is used;

[0015] For sensor noise, a combination of Poisson noise model and readout noise model is used;

[0016] For atmospheric scattering, an atmospheric scattering model is used.

[0017] Furthermore, in the above-mentioned image super-resolution reconstruction method based on a deep learning model, after the step of degrading the high-resolution image based on the corresponding physical model, the method further includes:

[0018] Using known high-resolution image samples, images with different parameter settings are acquired through actual imaging equipment, and the acquired low-resolution images are compared with low-resolution images generated based on physical models.

[0019] Adjust the parameters in the physical model to minimize the difference between the theoretically generated image and the actual acquired image, ensuring that the physical model can accurately simulate the real degradation process.

[0020] Furthermore, in the above-mentioned image super-resolution reconstruction method based on a deep learning model, the step of degrading the high-resolution image based on the corresponding physical model includes:

[0021] The various physical models are combined to form a degradation model. High-resolution images are input into the degradation model to generate corresponding low-resolution images, thereby constructing image data pairs.

[0022] In the process of generating low-resolution images, some parameters in the degradation model are randomly adjusted to simulate different degrees and types of degradation, thereby increasing the diversity of the data.

[0023] The established degradation model is transformed into a differentiable neural network layer;

[0024] For lens blur described by point spread function, the point spread function model is defined as a convolution kernel. For Poisson noise model and readout noise model, a random number generation layer conforming to Poisson distribution and Gaussian distribution is added to the network and its differentiability is ensured. For Mie scattering or Rayleigh scattering model, a differentiable function layer containing parameters such as scattering coefficient is constructed.

[0025] Using the prepared image data pairs, the differentiable degenerate model is trained iteratively. In each training iteration, the gradient of the loss function with respect to the parameters of the differentiable degenerate model is calculated using the backpropagation algorithm, and the model parameters are updated according to the gradient, so that the loss function gradually decreases.

[0026] After a set number of training rounds, the differentiable degradation model can stably transform high-resolution images into low-resolution images that conform to the degradation distribution of real scenes, thereby using the differentiable degradation model obtained from the physical model to degrade high-resolution images.

[0027] Furthermore, in the above image super-resolution reconstruction method based on a deep learning model, the expression for the point spread function model is:

[0028]

[0029] I low (x,y)=I high (x,y)*h(x,y);

[0030] Where (x,y) are the imaging plane coordinates, σ is the blur radius, and I high (x,y) represents a high-resolution image, * represents the convolution operator, and I low (x,y) represents a low-resolution image;

[0031] The combined expression for the Poisson noise model and the readout noise model is:

[0032] I noisy (x,y)=I blur (x,y)+n poisson (x,y)+n gaussian (x,y);

[0033] n poisson (x,y)~Poisson(α·I blur (x,y));

[0034]

[0035] Where, n poisson (x,y) represents the Poisson noise components, n gaussian (x,y) represents the Gaussian noise components, α is the reciprocal of the quantum efficiency (determined by the sensor hardware), σ is the noise standard deviation, and I blur (x,y) is a high-resolution image input to a combination of a Poisson noise model and a readout noise model, I noisy (x,y) is a low-resolution image output by the combination of the Poisson noise model and the readout noise model;

[0036] The expression for the atmospheric scattering model is:

[0037] I scatter (x,y)=I noisy (x,y)·t(x,y)+A·(1-t(x,y));

[0038] t(x,y)=exp(-β R ·d(x,y)-β M ·d(x,y));

[0039] Where, β R Rayleigh coefficient, β M Let d(x,y) be the Mie coefficient, d(x,y) be the scene depth, A be the ambient light intensity, and I be the ambient light intensity. noisy (x,y) is a low-resolution image output by a combination of the Poisson noise model and the readout noise model.

[0040] Furthermore, in the above image super-resolution reconstruction method based on a deep learning model, the expression for the degradation model is:

[0041] I low (x,y)=(I high (x,y)*h(x,y)+n poisson (x,y)+n gaussian (x,y))·t(x,y)+A·(1-t(x,y));

[0042] Among them, I high (x,y) represents a high-resolution image, * represents the convolution operator, and I low (x,y) represents a low-resolution image, and n poisson (x,y) represents the Poisson noise components, n gaussian (x,y) represents the Gaussian noise component, A represents the ambient light intensity, and t(x,y) represents the transmittance function.

[0043] Furthermore, in the aforementioned image super-resolution reconstruction method based on a deep learning model, different blur kernels are designed for the RGB three channels of the image in the point spread function model:

[0044]

[0045] Where (x,y) are the imaging plane coordinates, and σ(λ) is the blur radius corresponding to the wavelength.

[0046] Another object of the present invention is to provide an image super-resolution reconstruction system based on a deep learning model, the system comprising:

[0047] The acquisition module is used to acquire a preset number of high-resolution images and to acquire the degradation phenomena that may occur when the target imaging device is imaging.

[0048] The degradation module is used to degrade high-resolution images according to preset rules to obtain corresponding low-resolution images based on the degradation phenomena that may occur when the target imaging device is imaging.

[0049] The training module is used to build a training dataset based on high-resolution images and corresponding low-resolution images, and to construct a preset neural network and loss function. The training dataset is input into the preset neural network for training until the loss tends to stabilize to obtain the image super-resolution reconstruction model.

[0050] The reconstruction module is used to acquire the low-resolution image to be reconstructed, and input the low-resolution image to be reconstructed into the corresponding image super-resolution reconstruction model to obtain the corresponding reconstructed high-resolution image.

[0051] Another object of the present invention is to provide a readable storage medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the steps of the method as described in any of the preceding claims.

[0052] Another object of the present invention is to provide an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the steps of any of the methods described above.

[0053] This invention acquires a predetermined number of high-resolution images and identifies potential degradation phenomena that may occur during imaging by the target imaging device. Based on these degradation phenomena, the high-resolution images are degraded according to predetermined rules to obtain corresponding low-resolution images. A training dataset is established using the high-resolution and corresponding low-resolution images, and a predetermined neural network and loss function are constructed. The training dataset is input into the predetermined neural network for training until the loss stabilizes, resulting in an image super-resolution reconstruction model. The low-resolution image to be reconstructed is acquired and input into the corresponding image super-resolution reconstruction model to obtain the corresponding reconstructed high-resolution image. Degradation of the high-resolution image is specifically applied based on potential degradation phenomena during imaging, avoiding the addition of globally uniform noise for image degradation. This solves the problem of poor image super-resolution reconstruction quality in existing technologies. Attached Figure Description

[0054] Figure 1 This is a flowchart of the image super-resolution reconstruction method based on a deep learning model in the first embodiment of the present invention;

[0055] Figure 2 This is a structural block diagram of the image super-resolution reconstruction system based on a deep learning model in the third embodiment of the present invention.

[0056] The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0057] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0058] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or there may be an intervening component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there may be an intervening component. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the specification of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed types.

[0060] The following will describe in detail, with reference to specific embodiments and accompanying drawings, how to solve the problem of unreasonable allocation of computing tasks when reducing the PUE value.

[0061] Example One

[0062] Please see Figure 1 The image super-resolution reconstruction method based on a deep learning model is shown in the first embodiment of the present invention. The method includes steps S10 to S12.

[0063] Step S10: Acquire a preset number of high-resolution images and obtain the degradation phenomena that may occur when the target imaging device is imaging.

[0064] This requires the targeted collection of a certain number of high-resolution images (preset according to the needs of the actual application scenario and the amount of data required for model training, such as hundreds to thousands or even more). These images should cover as much content and scene type as possible in the target application scenario. These images can come from publicly available professional image libraries, be taken by ourselves using high-precision equipment, or be obtained through legal channels for specific fields. This ensures that they have sufficient diversity and representativeness, and can provide rich original materials for the subsequent generation of diverse low-resolution samples.

[0065] Simultaneously, during the acquisition of high-resolution images, it is also necessary to systematically understand and record the various degradation phenomena that may occur during the imaging process of the target imaging device (i.e., the device whose low-resolution images the super-resolution reconstruction model will process in future practical applications, such as a certain model of mobile phone camera, surveillance camera, or camera mounted on a drone). This requires in-depth analysis of the hardware characteristics and imaging principles of the device, such as lens blur caused by the optical characteristics of the lens (including different degrees of blur due to different focal length and aperture settings, as well as non-uniform blur caused by lens aberrations), wavelength-dependent dispersion effects (different wavelengths of light refract at different angles when passing through the lens, resulting in more obvious blurring of specific color channels), and sensor noise that may be caused by the performance parameters of the sensor (such as random noise generated by the sensor under low light conditions, and noise caused by the characteristics of the sensor circuit). In addition to fixed-mode noise, if the device is used for outdoor imaging, it is also necessary to consider the degradation caused by environmental factors such as atmospheric scattering (such as the reduction in image contrast and blurring of details caused by the scattering of light by particles in the atmosphere during propagation under weather conditions such as fog, haze, and rain) and motion blur (image trailing caused by the movement of the device or the subject). By actually testing the imaging effect of the device under different parameter settings and different environmental conditions, the manifestations, degree of impact of various degradation phenomena and related device and environmental parameters are recorded. This provides an accurate basis for the subsequent construction of a physical model that conforms to the real degradation characteristics of the device, ensuring that the low-resolution samples generated later can realistically simulate the image degradation of the device, and thus enable the super-resolution reconstruction model trained based on these samples to have better adaptability and reconstruction effect in practical applications.

[0066] Step S11: Based on the degradation phenomenon that may occur when the target imaging device is imaging, the high-resolution image is degraded according to a preset rule to obtain the corresponding low-resolution image.

[0067] Among them, by simulating the imaging defects of real devices, high-quality original images are transformed into low-quality samples that conform to actual application scenarios, providing training data that is close to reality for the super-resolution model. Specifically, it is necessary to first base the analysis on the degradation characteristics of the target imaging device (such as a certain surveillance camera).

[0068] Next, based on the physical laws and actual manifestations of these degradation phenomena, a set of preset rules should be designed. These rules include both parameterized definitions of a single degradation type (such as the size and direction of the fuzzy kernel, the intensity distribution range of noise, and the value range of the scattering coefficient) and regulations on the combination of multiple degradation types.

[0069] Subsequently, using differentiable degradation models and other technical tools, the collected high-resolution images are degraded one by one according to preset rules. For example, for a high-resolution urban street scene image, the corresponding PSF model is first called to blur it according to the lens parameters of the target surveillance camera. Then, random noise that matches the noise distribution of the camera is added according to the sensor characteristics of the camera in night mode. If the simulated weather is haze, an atmospheric scattering module is further superimposed to adjust the contrast and color of the image. The final low-resolution image not only matches the output of the target device in resolution, but also completely reproduces the typical defects of the image captured by the device in terms of visual features. Through this process, a large number of pairs of "high-resolution original image - low-resolution degraded image" data can be generated in batches. These data not only retain the scene content diversity of the original image, but also carry the real degradation characteristics of the target device. Compared with the low-quality images generated by traditional manual downsampling, it can more effectively train the super-resolution model, enabling it to learn the repair rules for the degradation mode of the target device. Thus, in practical applications, more accurate detail restoration and quality improvement can be achieved for the low-resolution images captured by the device.

[0070] Step S12: Establish a training dataset based on the high-resolution image and the corresponding low-resolution image, and construct a preset neural network and loss function. Input the training dataset into the preset neural network for training until the loss tends to stabilize to obtain the image super-resolution reconstruction model.

[0071] Among them, the data-driven approach enables the neural network to learn the mapping rules from low-resolution images to high-resolution images, and finally obtains a reconstruction model that can stably improve image quality. Specifically, the first step is to organize the previously prepared "high-resolution image-low-resolution image" pairs into a standardized training dataset.

[0072] Next, a pre-defined neural network needs to be constructed. Its structure design should be combined with the requirements of the super-resolution task. If real-time performance is required (such as mobile applications), a lightweight architecture can be selected (such as ESPCN based on CNN, which improves resolution through sub-pixel convolution). If the focus is on reconstruction accuracy and detail restoration, a deeper network can be used (such as RCAN, which focuses on key features through residual channel attention mechanism, or combined with Transformer's SwinIR to capture long-distance dependencies). The input of the network is a low-resolution image, and the output is the corresponding high-resolution reconstruction result. In the middle, multiple feature extraction modules (such as convolutional layers, residual blocks, and attention layers) are used to gradually learn the feature mapping from low resolution to high resolution.

[0073] Simultaneously, a targeted loss function needs to be designed. This function should quantify the difference between the reconstructed result and the real high-resolution image to guide the direction of network optimization. For example, MSE loss (mean squared error) can be used to ensure pixel-level accuracy, making the reconstructed image numerically close to the real value; perceptual loss (based on feature differences extracted by the pre-trained VGG network) can be introduced to improve the semantic and visual consistency of the image, avoiding the reconstruction result being "pixel accurate but visually distorted"; if a GAN architecture is used, adversarial loss will also be added, which uses a discriminator to judge the realism of the reconstructed image, prompting the generator to generate more natural texture details. In addition, TV loss (total variational loss) may be combined to reduce block artifacts in the image. By weighted combination of these loss terms, the reconstruction quality of different dimensions can be balanced.

[0074] Next, the model training phase begins. Low-resolution images from the training set are input into a pre-defined neural network in batches. The network calculates high-resolution reconstructed images through forward propagation. Then, the designed loss function is used to calculate the loss value between the reconstructed image and the real high-resolution image. Subsequently, the loss signal is transmitted from the output layer to the input layer through the backpropagation algorithm to update the parameters of each layer in the network (such as convolutional kernel weights and bias terms) so that the loss value gradually decreases. During the training process, the learning rate is dynamically adjusted (e.g., a larger learning rate is used initially to accelerate convergence, and the learning rate is reduced later for fine optimization). The model performance is tested on the validation set periodically. If the loss on the validation set no longer decreases or begins to increase, it indicates that the model may be overfitting, and the training needs to be terminated through an early stopping strategy.

[0075] When the training iterations reach a point where the loss value tends to stabilize (e.g., the loss fluctuation is less than a preset threshold for several consecutive epochs), it indicates that the network has fully learned the mapping rules from low-resolution images to high-resolution images. The model obtained at this point is a usable image super-resolution reconstruction model. This model can receive new low-resolution images and output high-resolution reconstruction results that conform to the characteristics of real scenes. It also exhibits good generalization ability and detail restoration ability in degraded scenes of target imaging devices.

[0076] Step S13: Obtain the low-resolution image to be reconstructed, and input the low-resolution image to be reconstructed into the corresponding image super-resolution reconstruction model to obtain the corresponding reconstructed high-resolution image.

[0077] Among them, low-resolution images captured by the target imaging device in real-world scenarios are acquired. These images may contain various degradation phenomena common in practical applications, such as nighttime noise and motion blur in surveillance footage, or atmospheric scattering and sensor noise in satellite images. Moreover, these images are often the objects that users actually need to process, such as surveillance faces that need to be cleared.

[0078] Next, a corresponding image super-resolution reconstruction model is selected—"corresponding" here means that the model must match the degradation characteristics of the imaging device and the type of image content. For example, for a low-resolution photo taken by a certain mobile phone, a model trained based on the degradation characteristics of that mobile phone should be selected. Subsequently, the low-resolution image to be reconstructed is preprocessed according to the model requirements, such as adjusting the image size and normalizing the pixel values, so that it conforms to the input format of the model, and then input into the selected super-resolution reconstruction model. The model extracts, analyzes and reconstructs the features of the low-resolution image through its internal neural network structure. Finally, the model outputs a processed high-resolution image that can meet the needs of practical application scenarios.

[0079] In summary, the image super-resolution reconstruction method based on a deep learning model in the above embodiments of the present invention acquires a preset number of high-resolution images and obtains the degradation phenomena that may occur during imaging by the target imaging device; based on the degradation phenomena that may occur during imaging by the target imaging device, the high-resolution images are degraded according to preset rules to obtain corresponding low-resolution images; a training dataset is established based on the high-resolution images and the corresponding low-resolution images, and a preset neural network and loss function are constructed. The training dataset is input into the preset neural network for training until the loss tends to stabilize to obtain an image super-resolution reconstruction model; the low-resolution image to be reconstructed is acquired, and the low-resolution image to be reconstructed is input into the corresponding image super-resolution reconstruction model to obtain the corresponding reconstructed high-resolution image. The high-resolution image is degraded in a targeted manner according to the degradation phenomena that may occur during imaging, avoiding the addition of globally uniform noise for image degradation. This solves the problem of poor image super-resolution reconstruction quality in existing technologies.

[0080] Example Two

[0081] This embodiment also proposes an image super-resolution reconstruction method based on a deep learning model. The difference between the image super-resolution reconstruction method based on a deep learning model proposed in this embodiment and the image super-resolution reconstruction method based on a deep learning model proposed in Embodiment 1 is as follows:

[0082] The step of degrading a high-resolution image to obtain a corresponding low-resolution image based on the degradation phenomenon that may occur during imaging by the target imaging device, according to a preset rule, includes:

[0083] A physical model is constructed based on the degradation phenomena that may occur when the target imaging device is used for imaging, and the high-resolution image is degraded based on the corresponding physical model.

[0084] For lens blur, a point spread function model based on the principle of light propagation is used;

[0085] For sensor noise, a combination of Poisson noise model and readout noise model is used;

[0086] For atmospheric scattering, an atmospheric scattering model is used.

[0087] For example, various physical models are combined to form a degradation model, and a high-resolution image is input into the degradation model to generate a corresponding low-resolution image, thereby constructing an image data pair;

[0088] In the process of generating low-resolution images, some parameters in the degradation model are randomly adjusted to simulate different degrees and types of degradation, thereby increasing the diversity of the data.

[0089] The established degradation model is transformed into a differentiable neural network layer;

[0090] For lens blur described by point spread function, the point spread function model is defined as a convolution kernel. For Poisson noise model and readout noise model, a random number generation layer conforming to Poisson distribution and Gaussian distribution is added to the network and its differentiability is ensured. For Mie scattering or Rayleigh scattering model, a differentiable function layer containing parameters such as scattering coefficient is constructed.

[0091] Using the prepared image data pairs, the differentiable degenerate model is trained iteratively. In each training iteration, the gradient of the loss function with respect to the parameters of the differentiable degenerate model is calculated using the backpropagation algorithm, and the model parameters are updated according to the gradient, so that the loss function gradually decreases.

[0092] After a set number of training rounds, the differentiable degradation model can stably transform high-resolution images into low-resolution images that conform to the degradation distribution of real scenes, thereby using the differentiable degradation model obtained from the physical model to degrade high-resolution images.

[0093] This method integrates physical degradation models such as lens blur, sensor noise, and atmospheric scattering to construct a learnable differential network. This network generates high- and low-resolution data pairs that closely resemble real-world scenes. First, the three physical models decomposed in the early stages are concatenated into a complete degradation model according to the actual imaging process. Lens blur is simulated by performing convolution operations on the high-resolution image using the PSF model to obtain a blurred image. Then, photon noise conforming to a Poisson distribution and circuit readout noise conforming to a Gaussian distribution are superimposed to generate a noisy image. Finally, an atmospheric scattering model that includes transmittance and ambient light is used to attenuate and superimpose with ambient light to obtain the final low-resolution image, thus constructing a "high-resolution image - low-resolution image" data pair.

[0094] Furthermore, during the generation process, key parameters of the degradation model (such as the blur radius and trajectory length of the PSF model, the inverse of the quantum efficiency and the standard deviation of the noise model, and the Rayleigh coefficient and Mie coefficient of the scattering model) will be randomly adjusted to simulate the degradation under different device parameters and environmental conditions, greatly increasing the diversity of data pairs and avoiding overfitting in subsequent model training.

[0095] Next, the degradation model needs to be transformed into a differentiable neural network layer, adapting to the automatic differentiation mechanism of the deep learning framework. For lens blur, the corresponding convolution kernel (such as Gaussian kernel or linear motion kernel) is calculated according to the selected PSF type and encapsulated as a learnable convolutional layer, so that the convolution kernel parameters can be updated with training. For sensor noise, a custom random number generation layer is designed, where the Poisson noise layer generates noise by parameterized random sampling of pixel values, and the Gaussian noise layer generates noise by generating random numbers with zero mean and learnable standard deviation. At the same time, a differentiable approximation calculation method is used to ensure that the noise generation process can participate in backpropagation. For the atmospheric scattering model, a differentiable function layer with learnable parameters is constructed, and the transmittance is calculated by exponential function, and then superimposed with ambient light in a linear combination to ensure that the entire scattering calculation process is differentiable. Then, the model training stage is entered, using the diverse image data pairs generated in the early stage as training samples, and the training is based on "predicting the difference between the low-resolution image and the real low-resolution image (or the reference low-resolution image acquired by the real device)". The target design loss function (such as mean squared error (MSE), structural similarity loss (SSIM), or combining perceptual loss to improve visual consistency) is used. In each training iteration, the gradient of the loss function with respect to all model parameters is calculated layer by layer along the differentiable network layer by using the backpropagation algorithm. Then, the gradient descent optimizer (such as Adam or SGD) is used to adjust the parameter values ​​according to the gradient direction, so that the loss function gradually decreases. After a preset number of training rounds, the parameters of the differentiable degradation model will gradually stabilize. At this time, when any high-resolution image is input, the model can automatically generate a low-resolution image that conforms to the degradation law of the real scene. For example, for the same high-resolution urban street view image, it can generate a low-resolution close-up image of a sunny day with "slight blur + a small amount of noise + weak scattering" as well as a low-resolution distant image of a hazy scene with "strong blur + a lot of noise + strong scattering". Moreover, the degradation features of these low-resolution images (such as the gradual change law of blurred edges, the distribution density of noise, and the color shift caused by scattering) are highly consistent with the images taken by real devices, providing high-quality and diverse training data support for subsequent image super-resolution reconstruction models.

[0096] Specifically, the expression for the point spread function model is:

[0097]

[0098] I low (x,y)=Ihigh (x,y)*h(x,y);

[0099] Where (x,y) are the imaging plane coordinates, σ is the blur radius, and I high (x,y) represents a high-resolution image, * represents the convolution operator, and I low (x,y) represents a low-resolution image;

[0100] The combined expression for the Poisson noise model and the readout noise model is:

[0101] I noisy (x,y)=I blur (x,y)+n poisson (x,y)+n gaussian (x,y);

[0102] n poisson (x,y)~Poisson(α·I blur (x,y));

[0103]

[0104] Where, n poisson (x,y) represents the Poisson noise components, n gaussian (x,y) represents the Gaussian noise components, α is the reciprocal of the quantum efficiency (determined by the sensor hardware), σ is the noise standard deviation, and I blur (x,y) is a high-resolution image input to a combination of a Poisson noise model and a readout noise model, I noisy (x,y) is a low-resolution image output by the combination of the Poisson noise model and the readout noise model;

[0105] The expression for the atmospheric scattering model is:

[0106] I scatter (x,y)=I noisy (x,y)·t(x,y)+A·(1-t(x,y));

[0107] t(x,y)=exp(-β R ·d(x,y)-β M ·d(x,y));

[0108] Where, β R Rayleigh coefficient, β M Let d(x,y) be the Mie coefficient, d(x,y) be the scene depth, A be the ambient light intensity, and I be the ambient light intensity. noisy (x,y) is a low-resolution image output by a combination of the Poisson noise model and the readout noise model.

[0109] The expression for the degradation model is:

[0110] I low (x,y)=(I high (x,y)*h(x,y)+n poisson (x,y)+n gaussian (x,y))·t(x,y)+A·(1-t(x,y));

[0111] Among them, I high (x,y) represents a high-resolution image, * represents the convolution operator, and I low (x,y) represents a low-resolution image, and n poisson (x,y) represents the Poisson noise components, n gaussian (x,y) represents the Gaussian noise component, A represents the ambient light intensity, and t(x,y) represents the transmittance function.

[0112] In the point spread function model, different blur kernels are designed for the RGB channels of the image:

[0113]

[0114] Where (x,y) are the imaging plane coordinates, and σ(λ) is the blur radius corresponding to the wavelength.

[0115] Specifically, this channel differentiation design makes the generated low-resolution image closer to the chromatic aberration characteristics of a real lens, avoiding the non-physical phenomenon of "uniform blur across three channels" caused by the traditional single blur kernel, and providing more realistic training data support for the subsequent super-resolution model to learn and correct color cast and improve detail restoration capabilities.

[0116] In addition, in some optional embodiments of the present invention, the step of degrading the high-resolution image based on the corresponding physical model further includes:

[0117] Using known high-resolution image samples, images with different parameter settings are acquired through actual imaging equipment, and the acquired low-resolution images are compared with low-resolution images generated based on physical models.

[0118] Adjust the parameters in the physical model to minimize the difference between the theoretically generated image and the actual acquired image, ensuring that the physical model can accurately simulate the real degradation process.

[0119] First, it is necessary to prepare known high-resolution image samples. These samples should be diverse, covering different scenes, objects, and texture features, and should reflect the image characteristics of the target application domain well. For example, if optimizing the degradation model of a surveillance camera, the samples can include high-resolution images with different lighting (daytime, nighttime), different distances (close-up, distant view), and different motion states (static objects, dynamic people). Then, using the actual imaging device of the target (i.e., the device whose low-resolution image the subsequent super-resolution reconstruction model will process, such as a certain model of surveillance camera, mobile phone camera, etc.), these known high-resolution image samples are captured under different parameter settings to acquire low-resolution images. The parameter settings here need to cover the commonly used working modes of the device, such as adjusting the camera's focal length (simulating different degrees of out-of-focus blur), aperture size (affecting the amount of light entering, and thus affecting sensor noise), ISO sensitivity (directly related to sensor noise intensity), shutter speed (simulating different degrees of motion blur), etc. At the same time, different environmental parameters (such as light intensity and atmospheric visibility) can also be combined for acquisition to ensure that the acquired real low-resolution images can contain the degradation types and degradation degrees that the device may experience in various actual use scenarios.

[0120] Then, the low-resolution images generated based on physical models (such as the degradation model composed of the lens blur PSF model, sensor noise mixing model, and atmospheric scattering model mentioned earlier) are compared with the low-resolution images of the corresponding high-resolution samples acquired by actual imaging equipment in multiple dimensions. When comparing, we should not only pay attention to the visual differences (such as whether the degree of blur is consistent, whether the noise distribution is similar, and whether the color shift matches), but also measure the size of the difference through quantitative indicators. Commonly used quantitative indicators include mean square error (MSE, which calculates the sum of squares of the differences between corresponding pixel values ​​in two images, the smaller the value, the smaller the difference), structural similarity index (SSIM, which evaluates the similarity of images from three dimensions: brightness, contrast, and structure, the closer the value is to 1, the more similar they are), peak signal-to-noise ratio (PSNR, calculated based on MSE, the higher the value, the closer the image quality is to the real image), etc. Through these indicators, the gap between the images generated by the physical model and the real acquired images can be objectively quantified.

[0121] Finally, adjust the parameters in the physical model based on the comparison results. For example, if the blurriness of the model-generated image is weaker than that of the actual acquired image, and the quantitative indicators show a larger MSE and a lower SSIM, then the blur radius in the lens blur PSF model needs to be increased. If the noise intensity of the model-generated image is lower than that of the actual image, the reciprocal of the quantum efficiency of Poisson noise or the standard deviation of Gaussian noise in the sensor noise mixing model can be appropriately increased. If the fogging effect of the model-generated image in an atmospheric scattering scene does not match that of the actual image, the Rayleigh coefficient, Mie system, or atmospheric ambient light intensity parameters in the atmospheric scattering model can be adjusted. The adjustment process needs to be iterated repeatedly, that is, each adjustment of parameters... After counting, a low-resolution image is regenerated and compared with the real image for evaluation until the visual difference between the theoretically generated image and the actual acquired image is minimized, and all quantitative indicators reach the preset ideal thresholds (such as MSE decreasing to a certain minimum value, SSIM increasing to above 0.9, and PSNR reaching above 30dB). At this point, the parameters of the physical model can match the degradation characteristics of the actual imaging device well, ensuring that the physical model can accurately simulate the real degradation process in the future. This provides high-quality training data that is closer to the actual application scenario for the super-resolution reconstruction model, thereby improving the generalization ability and reconstruction effect of the super-resolution reconstruction model in real scenarios.

[0122] In summary, the image super-resolution reconstruction method based on a deep learning model in the above embodiments of the present invention acquires a preset number of high-resolution images and obtains the degradation phenomena that may occur during imaging by the target imaging device; based on the degradation phenomena that may occur during imaging by the target imaging device, the high-resolution images are degraded according to preset rules to obtain corresponding low-resolution images; a training dataset is established based on the high-resolution images and the corresponding low-resolution images, and a preset neural network and loss function are constructed. The training dataset is input into the preset neural network for training until the loss tends to stabilize to obtain an image super-resolution reconstruction model; the low-resolution image to be reconstructed is acquired, and the low-resolution image to be reconstructed is input into the corresponding image super-resolution reconstruction model to obtain the corresponding reconstructed high-resolution image. The high-resolution image is degraded in a targeted manner according to the degradation phenomena that may occur during imaging, avoiding the addition of globally uniform noise for image degradation. This solves the problem of poor image super-resolution reconstruction quality in existing technologies.

[0123] Example Three

[0124] Please see Figure 2 The image shown is an image super-resolution reconstruction system based on a deep learning model proposed in the third embodiment of the present invention. The system includes:

[0125] The acquisition module 100 is used to acquire a preset number of high-resolution images and to acquire the degradation phenomena that may occur when the target imaging device is imaging.

[0126] The degradation module 200 is used to degrade the high-resolution image according to preset rules to obtain the corresponding low-resolution image based on the degradation phenomenon that may occur when the target imaging device is imaging.

[0127] The training module 300 is used to establish a training dataset based on high-resolution images and corresponding low-resolution images, and to construct a preset neural network and loss function. The training dataset is input into the preset neural network for training until the loss tends to stabilize to obtain the image super-resolution reconstruction model.

[0128] The reconstruction module 400 is used to acquire the low-resolution image to be reconstructed and input the low-resolution image to be reconstructed into the corresponding image super-resolution reconstruction model to obtain the corresponding reconstructed high-resolution image.

[0129] The functions or operation steps implemented by the above modules are largely the same as those in the above method embodiments, and will not be repeated here.

[0130] Example Four

[0131] In another aspect, the present invention provides a readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps of the method described in any one of the above embodiments one to two.

[0132] Example Five

[0133] In another aspect, the present invention provides an electronic device, the electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any one of the methods described in embodiments one to two above.

[0134] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0135] Those skilled in the art will understand that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequential list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can mean any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0136] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0137] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0138] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0139] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. An image super-resolution reconstruction method based on a deep learning model, characterized in that, The method includes: Acquire a preset number of high-resolution images and obtain the degradation phenomena that may occur when the target imaging device is imaging; Based on the degradation phenomenon that may occur when the target imaging device is imaging, the high-resolution image is degraded according to the preset rules to obtain the corresponding low-resolution image. A training dataset is established based on high-resolution images and corresponding low-resolution images. A pre-defined neural network and loss function are constructed, and the training dataset is input into the pre-defined neural network for training until the loss tends to stabilize to obtain the image super-resolution reconstruction model. Obtain the low-resolution image to be reconstructed, and input the low-resolution image to be reconstructed into the corresponding image super-resolution reconstruction model to obtain the corresponding reconstructed high-resolution image.

2. The image super-resolution reconstruction method based on a deep learning model according to claim 1, characterized in that, The step of degrading a high-resolution image to obtain a corresponding low-resolution image based on the degradation phenomenon that may occur during imaging by the target imaging device, according to a preset rule, includes: A physical model is constructed based on the degradation phenomena that may occur when the target imaging device is used for imaging, and the high-resolution image is degraded based on the corresponding physical model. For lens blur, a point spread function model based on the principle of light propagation is used; For sensor noise, a combination of Poisson noise model and readout noise model is used; For atmospheric scattering, an atmospheric scattering model is used.

3. The image super-resolution reconstruction method based on a deep learning model according to claim 2, characterized in that, The step of degrading the high-resolution image based on the corresponding physical model also includes: Using known high-resolution image samples, images with different parameter settings are acquired through actual imaging equipment, and the acquired low-resolution images are compared with low-resolution images generated based on physical models. Adjust the parameters in the physical model to minimize the difference between the theoretically generated image and the actual acquired image, ensuring that the physical model can accurately simulate the real degradation process.

4. The image super-resolution reconstruction method based on a deep learning model according to claim 2, characterized in that, The steps for degrading high-resolution images based on the corresponding physical model include: The various physical models are combined to form a degradation model. High-resolution images are input into the degradation model to generate corresponding low-resolution images, thereby constructing image data pairs. In the process of generating low-resolution images, some parameters in the degradation model are randomly adjusted to simulate different degrees and types of degradation, thereby increasing the diversity of the data. The established degradation model is transformed into a differentiable neural network layer; For lens blur described by point spread function, the point spread function model is defined as a convolution kernel. For Poisson noise model and readout noise model, a random number generation layer conforming to Poisson distribution and Gaussian distribution is added to the network and its differentiability is ensured. For Mie scattering or Rayleigh scattering model, a differentiable function layer containing parameters such as scattering coefficient is constructed. Using the prepared image data pairs, the differentiable degenerate model is trained iteratively. In each training iteration, the gradient of the loss function with respect to the parameters of the differentiable degenerate model is calculated using the backpropagation algorithm, and the model parameters are updated according to the gradient, so that the loss function gradually decreases. After a set number of training rounds, the differentiable degradation model can stably transform high-resolution images into low-resolution images that conform to the degradation distribution of real scenes, thereby using the differentiable degradation model obtained from the physical model to degrade high-resolution images.

5. The image super-resolution reconstruction method based on a deep learning model according to claim 4, characterized in that, The expression for the point spread function model is: I low (x,y)=I high (x,y)*h(x,y); Where (x,y) are the imaging plane coordinates, σ is the blur radius, and I high (x,y) represents a high-resolution image, * represents the convolution operator, and I low (x,y) represents a low-resolution image; The combined expression for the Poisson noise model and the readout noise model is: I noisy (x,y)=I blur (x,y)+n poisson (x,y)+n gaussian (x,y); n poisson (x,y)~Poisson(α·I blur (x,y)); Where, n poisson (x,y) represents the Poisson noise components, n gaussian (x,y) represents the Gaussian noise components, α is the reciprocal of the quantum efficiency (determined by the sensor hardware), σ is the noise standard deviation, and I blur (x,y) is a high-resolution image input to a combination of a Poisson noise model and a readout noise model, I noisy (x,y) is a low-resolution image output by the combination of the Poisson noise model and the readout noise model; The expression for the atmospheric scattering model is: I scatter (x,y)=I noisy (x,y)·t(x,y)+A·(1-t(x,y)); t(x,y)=exp(-β R ·d(x,y)-β M ·d(x,y)); Where, β R Let β be the Rayleigh coefficient. M Let d(x,y) be the Mie coefficient, d(x,y) be the scene depth, A be the ambient light intensity, and I be the ambient light intensity. noisy (x,y) is a low-resolution image output by a combination of the Poisson noise model and the readout noise model.

6. The image super-resolution reconstruction method based on a deep learning model according to claim 5, characterized in that, The expression for the degradation model is: I low (x,y)=(I high (x,y)*h(x,y)+n poisson (x,y)+n gaussian (x,y))·t(x,y)+A·(1-t(x,y)); Among them, I high (x,y) represents a high-resolution image, * represents the convolution operator, and I low (x,y) represents a low-resolution image, and n poisson (x,y) represents the Poisson noise components, n gaussian (x,y) represents the Gaussian noise component, A represents the ambient light intensity, and t(x,y) represents the transmittance function.

7. The image super-resolution reconstruction method based on a deep learning model according to claim 6, characterized in that, In the point spread function model, different blur kernels are designed for the RGB channels of the image: Where (x,y) are the imaging plane coordinates, and σ(λ) is the blur radius corresponding to the wavelength.

8. An image super-resolution reconstruction system based on a deep learning model, characterized in that, The system includes: The acquisition module is used to acquire a preset number of high-resolution images and to acquire the degradation phenomena that may occur when the target imaging device is imaging. The degradation module is used to degrade high-resolution images according to preset rules to obtain corresponding low-resolution images based on the degradation phenomena that may occur when the target imaging device is imaging. The training module is used to build a training dataset based on high-resolution images and corresponding low-resolution images, and to construct a preset neural network and loss function. The training dataset is input into the preset neural network for training until the loss tends to stabilize to obtain the image super-resolution reconstruction model. The reconstruction module is used to acquire the low-resolution image to be reconstructed, and input the low-resolution image to be reconstructed into the corresponding image super-resolution reconstruction model to obtain the corresponding reconstructed high-resolution image.

9. A readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the steps of the method as described in any one of claims 1 to 7.

Images