Image super-resolution reconstruction method, electronic equipment and medium

Abstract

The invention provides an image super-resolution reconstruction method, electronic equipment and a medium. The image super-resolution reconstruction method comprises the steps that a to-be-processed image is received, a reconstruction mode of the to-be-processed image is acquired, the reconstruction mode is a combination of physical constraint and deep learning, different reconstruction modes correspond to different deep learning modes, the physical constraint corresponds to an imaging process of the to-be-processed image, and the deep learning modes correspond to the imaging process of the to-be-processed image. Comprising at least one of noise constraint and fuzzy constraint; and performing super-resolution reconstruction on the to-be-processed image based on the reconstruction mode to generate a target image corresponding to the to-be-processed image. According to the method, the physical constraint can be added into the super-resolution reconstruction of the image, so that the image quality is ensured, the noise influence is reduced, the image detail recovery capability is improved, and the image reconstruction effect is effectively improved.

Description

Technical Field

[0001] This application relates to the field of image reconstruction technology. Specifically, this application relates to an image super-resolution reconstruction method, an electronic device, and a medium. Background Art

[0002] In the big data era, the wide application of high-resolution images in various fields has become increasingly prominent. In the field of social media, high-resolution images help improve users' visual experience; in the medical field, high-resolution images can assist professionals in making more accurate diagnoses and judgments; in the security field, high-resolution images help maintain social security and surveillance. However, due to limitations such as equipment costs and natural environments, the images collected usually have characteristics such as low resolution, poor quality, and blurriness.

[0003] In related technologies, currently, for image super-resolution processing, it is mainly achieved by using a pre-trained super-resolution model. By inputting a low-resolution image into the super-resolution model, the super-resolution model outputs the corresponding high-resolution image. When training this model, usually a low-resolution image is input into the model, and the super-resolution processing prediction result for the low-resolution image is output by the model, and a loss function is constructed using this prediction result and the real high-resolution image, thereby realizing the training of the model. However, this method only relies on data-driven models and ignores the influence of physical factors in the physical imaging process, resulting in uncertainty and artifacts in the reconstruction results and reducing the image reconstruction effect. Summary of the Invention

[0004] In view of the shortcomings of the existing methods, this application proposes an image super-resolution reconstruction method, an electronic device, and a medium, which can solve the problem that the existing image super-resolution processing method ignores the influence of physical factors and results in poor reconstruction effect.

[0005] According to one aspect of the embodiments of this application, the embodiments of this application provide an image super-resolution reconstruction method, and the method includes:

[0006] Receiving an image to be processed, and obtaining the reconstruction method of the image to be processed, where the reconstruction method is a combination of physical constraints and deep learning, and different reconstruction methods correspond to different deep learning methods, and the physical constraints correspond to the imaging process of the image to be processed and include at least one of noise constraint and blur constraint;

[0007] Performing super-resolution reconstruction on the image to be processed based on the reconstruction method to generate a target image corresponding to the image to be processed.

[0008] In a possible implementation, the reconstruction method is a combination of physical constraints and a convolutional neural network. The super-resolution reconstruction of the image to be processed based on the reconstruction method includes:

[0009] Obtain the physical constraint information corresponding to the image to be processed, where the physical constraint information includes noise constraint and blur constraint;

[0010] Add the physical constraint information to the loss function of the convolutional neural network to obtain a first imaging model, and use the first imaging model to perform super-resolution reconstruction on the image to be processed.

[0011] In a possible implementation, the acquisition of the physical constraint information includes:

[0012] Determine the noise existing in the image to be processed according to the imaging process of the image to be processed, where the noise includes at least one of Gaussian noise, Poisson noise, and mixed noise;

[0013] Generate noise constraint information using the noise model corresponding to the noise;

[0014] Determine the blur kernel corresponding to the image to be processed according to the characteristics of the imaging process, and obtain the blur constraint through the blur kernel.

[0015] In a possible implementation, the loss function for adding the physical constraint information is:

[0016] L = λ1L revonstruction + λ2L adversarial + λ3L physical

[0017] where λ1, λ2, and λ3 are the weights of the loss function, L reconstruction is the reconstruction loss, L adversarial is the adversarial loss, L physical is the physical constraint loss, including noise constraint and modulus constraint, where is the reconstructed image of the i-th sample, y i is the real image of the i-th sample, N is the number of samples, and an adversarial network is used to optimize L adversarial . D(x) is the judgment of the discriminator in the adversarial network on the real image x, and G(z) is the image generated by the generator based on the input noise z, is the expectation of the input generator noise z, p z (z) is the distribution function of the noise z, represents the expected value of the real image x under the distribution p data (x) of the real image x, λ4 is the weight related to the fuzzy constraint, λ5 is the weight corresponding to the noise constraint, H(.) represents the fuzzy kernel, and I LR represents the image input to the convolutional neural network, and I HR represents the reconstructed image output by the convolutional neural network for this image, and N1 represents the noise constraint information.

[0018] In a possible implementation, the reconstruction method is a combination of physical constraints and a generative adversarial network. The super-resolution reconstruction of the image to be processed based on the reconstruction method includes:

[0019] Obtain the physical constraint information corresponding to the processed image, and determine the loss function of the generative adversarial network according to the physical constraint information;

[0020] Train the generator and discriminator of the generative adversarial network using the training data corresponding to the image to be processed and the loss function.

[0021] In a possible implementation, the loss function corresponding to the generator is:

[0022] L total = L GAN + λ phy ·L physical

[0023] In the formula, L GAN is the first loss function, L physical is the physical constraint loss, λ phy is the weight of the physical constraint loss, L physical = L noise + L blur , is the predicted noise distribution, N2 is the real noise, E ILR is the expected value of the image input to the generative adversarial network, λ noise is the weight of the noise constraint, λ blur is the weight of the fuzzy constraint, H(.) represents the fuzzy kernel, and I LR represents the image input to the generative adversarial network, and I HR represents the reconstructed image output by the generative adversarial network for this image.

[0024] In a possible implementation, the reconstruction method is a combination of multi-scale learning and physical constraints. The super-resolution reconstruction of the image to be processed based on the reconstruction method includes:

[0025] Obtain the physical constraint information corresponding to the image to be processed, and add the physical constraint information to the convolutional layers of different scales of the multi-scale convolutional neural network to obtain the second imaging model;

[0026] Input the image to be processed into the second imaging model, extract the features of the image to be processed at different scales using the convolutional layers of different scales in the second imaging model, and perform weighted fusion on the features to generate the target image.

[0027] In a possible implementation, the reconstruction method is a combination of self-supervised learning and physical constraints. The super-resolution reconstruction of the image to be processed based on the reconstruction method includes:

[0028] Obtain the training set, physical constraint information, and prior loss corresponding to the image to be processed, construct a loss function based on the physical constraint information and the prior loss, and the prior loss is used to limit the solution space of self-supervised learning;

[0029] Based on the loss function and the training set, train the self-supervised model to obtain a third imaging model, and use the third imaging model to perform super-resolution reconstruction on the image to be processed.

[0030] According to one aspect of the embodiments of the present application, there is provided an electronic device, including a memory, a processor, and a computer program stored on the memory, and the processor executes the computer program to implement the steps of the method as described above.

[0031] According to one aspect of the embodiments of the present application, there is provided a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method as described above are implemented.

[0032] The beneficial technical effects brought by the technical solutions provided in the embodiments of the present application include:

[0033] A method for super-resolution reconstruction of an image provided by the present application has the beneficial effect that it receives an image to be processed, obtains the reconstruction method of the image to be processed, and the reconstruction method is a combination of physical constraints and deep learning. Different reconstruction methods correspond to different deep learning methods, and the physical constraints correspond to the imaging process of the image to be processed; based on the reconstruction method, super-resolution reconstruction is performed on the image to be processed to generate a target image corresponding to the image to be processed. The present application can add physical constraints to the super-resolution reconstruction of the image, thereby reducing the influence of noise while ensuring the image quality, improving the ability to restore image details, and effectively improving the image reconstruction effect.

[0034] The additional aspects and advantages of the present application will be partially given in the following description, and these will become obvious from the following description, or can be understood through the practice of the present application. Description of the Drawings

[0035] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the following description of embodiments in conjunction with the accompanying drawings, where:

[0036] Figure 1 is a flowchart of an image super-resolution reconstruction method provided by an embodiment of the present application;

[0037] Figure 2 is a structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0038] Embodiments of the present application will be described below with reference to the accompanying drawings in the present application. It should be understood that the implementation manners described below in conjunction with the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application, and do not constitute limitations on the technical solutions of the embodiments of the present application.

[0039] Those skilled in the art of the present technology can understand that, unless specifically stated otherwise, the "the" and "this" used herein may also include plural forms. It should be further understood that the term "including" used in the specification of the present application means the presence of the described features, integers, steps, operations, elements, and / or components, but does not exclude the presence of other features, information, data, steps, operations, elements, components, and / or combinations thereof supported by the art of the present technology. It should be understood that when we say an element is "connected" or "coupled" to another element, this element can be directly connected or coupled to the other element, or it can mean that this element and the other element establish a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used here can include wireless connection or wireless coupling. The term "and / or" used here means at least one of the items defined by this term. For example, "A and / or B" can be implemented as "A", or implemented as "B", or implemented as "A and B".

[0040] To make the objectives, technical solutions, and advantages of the present application clearer, the following will describe the implementation manners of the present application in further detail in conjunction with the accompanying drawings.

[0041] An embodiment of the present application provides an image super-resolution reconstruction method, which can be used in mobile phones, computers, servers, the cloud, and other terminals that can obtain an image to be processed and perform super-resolution reconstruction on the image.

[0042] As Figure 1 shown, the image super-resolution reconstruction method of the present application includes:

[0043] S101: Receive the image to be processed and obtain the reconstruction method of the image to be processed.

[0044] Optionally, the image to be processed can be an image with a resolution lower than a preset resolution, and this image can be a medical image, a surveillance image, and other images that need to improve the resolution.

[0045] Optionally, the image to be processed can also be an image detected to have noise (such as Gaussian noise, Poisson noise). During the image processing, it can be detected whether the acquired image has noise. If so, it is determined as the image to be processed.

[0046] Optionally, an image transmitted by an external device can also be received, and it is detected whether an instruction for super-resolution reconstruction is received. If so, the image is determined as the image to be processed.

[0047] Optionally, the reconstruction method is a combination of physical constraints and deep learning. Different reconstruction methods correspond to different deep learning methods, and the physical constraints correspond to the imaging process of the image to be processed. The physical constraints include at least one of noise constraints and blur constraints.

[0048] Optionally, the reconstruction method includes any one of the combination of physical constraints and convolutional neural network, the combination of physical constraints and generative adversarial network, and the combination of multi-scale learning and physical constraints. The reconstruction method of the image to be processed can be determined according to the received reconstruction method selection instruction or information such as the type of the image to be processed and the scene corresponding to the image to be processed.

[0049] Optionally, the physical constraints of the image to be processed can be determined according to the imaging process of the image to be processed. By analyzing the imaging process of the image to be processed, the physical constraints of the image to be processed are determined, and the physical constraints are applied to the super-resolution reconstruction of the image to be processed.

[0050] S102: Perform super-resolution reconstruction on the image to be processed based on the reconstruction method to generate a target image corresponding to the image to be processed.

[0051] Optionally, when the reconstruction method is the combination of physical constraints and convolutional neural network, performing super-resolution reconstruction on the image to be processed based on the reconstruction method includes: obtaining physical constraint information corresponding to the image to be processed, where the physical constraint information includes noise constraints and blur constraints; adding the physical constraint information to the loss function of the convolutional neural network to obtain a first imaging model, and using the first imaging model to perform super-resolution reconstruction on the image to be processed. The blur effect and noise interference in the image to be processed can be evaluated through blur constraints and noise constraints, and the negative impacts of these factors can be reduced. By introducing these physical constraints into the loss function, it can help the model better recover the high-resolution target image, reduce noise interference, and improve the stability of the reconstruction result. The introduction of physical constraint information enables the first imaging model to not only learn features from the data but also follow the physical laws in the imaging process, thereby improving the quality of image reconstruction.

[0052] Optionally, physical constraint information can be obtained through a preset physical constraint model, which may include a fuzzy model, a noise model, and other models capable of generating physical constraint information. Specifically, the expression of the physical constraint model can be:

[0053] I LR = H(I LR ) + N

[0054] where H(·) represents the fuzzy kernel and N represents the noise term. I LR represents the image input into the first imaging model (i.e., the image to be super-resolution reconstructed).

[0055] Optionally, the acquisition of physical constraint information includes: determining the noise existing in the image to be processed according to the imaging process of the image to be processed, where the noise includes at least one of Gaussian noise, Poisson noise, and mixed noise; generating noise constraint information using the noise model corresponding to the noise; determining the fuzzy kernel corresponding to the image to be processed according to the characteristics of the imaging process, and obtaining the blur constraint through the fuzzy kernel.

[0056] Optionally, the corresponding noise model can be selected according to the type of the image to be processed, and the noise constraint information is generated through this noise model. Specifically, for Gaussian noise, the noise constraint information can be Gaussian noise with a mean of 0 and a variance of σ, and the specific value of σ can be determined according to the imaging process of the image to be processed. The quantization error of the sensor for acquiring the image to be processed can be simulated by Poisson noise, and the intensity of Poisson noise is proportional to the pixel value of the image to be processed, and Poisson noise follows the Poisson distribution. The constraint information of the mixed noise can be generated by the weighted fusion of Gaussian noise and Poisson noise, and the calculation formula can be:

[0057] n1 = w1·N(0,σ 2 ) + w2Posson(λI LR ), w1 + w2 = 1, w1 and w2 are weights, N(0,σ 2 ) represents a Gaussian distribution with a mean of 0 and a variance of σ, Posson(λI LR ) represents the Poisson distribution, λ is the gain factor, n1 represents the noise value, and I LR represents the image to be processed.

[0058] Optionally, the blur effect in the imaging process is simulated through the blur constraint, and the blur constraint can be represented by a blur function (i.e., the blur kernel). Among them, the blur kernel includes functions such as uniform blur and motion blur. The generation process of image blur is simulated through the blur function, and the model is guided by the blur loss to correct the blur effect. When the blur function is a uniform blur kernel, the expression is:

[0059]

[0060] Where \(k\) is the size of the blur kernel, and \(\delta(i, j)\) is the Kronecker function (which takes the value of 1 only when \(i, j = 0\)).

[0061] Optionally, the convolutional neural network uses multiple convolutional layers for feature extraction and reconstruction. The design of the convolutional layer can effectively capture the spatial and context information in the image, enabling the network to automatically learn the mapping relationship between the low-resolution image and the high-resolution image. Through multiple convolutional layers, the network can extract different levels of features of the image, thereby effectively restoring the details of the high-resolution image. Moreover, the convolutional neural network adopts a dynamic convolutional kernel design, which can dynamically adjust the shape of the convolutional kernel according to the local features of the input image, solving the problem of insufficient adaptability of traditional fixed convolutional kernels to complex structures.

[0062] Optionally, in order to improve the accuracy of super-resolution reconstruction, the convolutional neural network adopts the method of residual learning and integrates the residuals in multiple layers. By predicting the residuals between the low-resolution input image and the high-resolution output image and stacking multiple-layer residuals, finer details are generated. In the convolutional neural network, residual learning is mainly achieved by designing residual modules. Specifically, the network first performs preliminary feature extraction on the low-resolution image, and then predicts the difference (residual) between the low-resolution image and the high-resolution image. This residual information is weighted and superimposed on the basis of the original low-resolution image, gradually improving the reconstruction result, thereby restoring finer details. Through the iterative training process of the convolutional neural network, the convolutional neural network can gradually improve the quality of the reconstructed image, avoiding the problems of over-smoothing or detail loss. The key of residual learning is that it can make the network easier to train by reducing the traditional gradient vanishing problem, improving the convergence speed and enhancing the accuracy.

[0063] Optionally, the loss function adding physical constraint information is:

[0064] \(L=\lambda_1L_{\text{rec}}+\lambda_2L_{\text{adv}}+\lambda_3L_{\text{phy}}\) reconstruction where \(\lambda_1,\lambda_2,\lambda_3\) are the weights of the loss function, \(L_{\text{rec}}\) adversarial is the reconstruction loss, \(L_{\text{adv}}\) physical

[0065] is the adversarial loss, and \(L_{\text{phy}}\) reconstruction is the physical constraint loss, including noise constraint and modulus constraint. Among them, adversarial \(\hat{y}_i\) physical is the reconstructed image of the \(i\)-th sample, \(y_i\) is the real image of the \(i\)-th sample, \(N\) is the number of samples, and the adversarial network is used to optimize \(L_{\text{adv}}\). i adversarial ​D(x) is the discriminator's judgment on the real image x in the adversarial network, and G(z) is the image generated by the generator based on the input noise z. To represent the expectation of the input generator noise z, p z (z) is the distribution function of the noise z. Denotes the expected value of the real image x under the distribution p data (x) of the real image x. λ4 is the weight related to the blurring constraint, λ5 is the weight corresponding to the noise constraint, H(.) represents the blurring kernel, and I LR represents the image input to the convolutional neural network, and I HR represents the reconstructed image output by the convolutional neural network for this image. N1 represents the noise constraint information. By adding the physical constraint as a regularization term to the loss function through the above loss function, and using this loss function for training during the training process of the convolutional neural network, it is ensured that the first imaging model obtained by training minimizes traditional losses (such as pixel loss, perceptual loss, adversarial loss, etc.) while following the physical laws in the imaging process.

[0066] In one embodiment, the cross-entropy loss can also be used for the training of the convolutional neural network, and during the training process, the backpropagation algorithm can also be used to optimize the convolutional neural network. Specifically, the training process of the convolutional neural network can be as follows:

[0067] 1. Data preprocessing: (1) Obtain the imaging equation corresponding to the training samples, solve this imaging equation using the finite element method, and normalize the boundary measurement values y and the medium distribution x corresponding to the boundary measurement values obtained by the finite element method to form training samples. (2) When the training samples are ECT data, for ECT data, the training samples can be classified according to the flow pattern (full-phase flow, core flow, etc.) by improving the AdaBoost algorithm to ensure the uniformity and consistency of the input data. 2. Parameter initialization of the convolutional neural network: (1) Initialize the weight matrix and set the bias b = 0.1. (2) Set the number of mini-batch training samples m = 128, the learning rate η = 0.99, the moving average decay rate β = 0.99, and the regularization coefficient λ = 0.0001. 3. Forward propagation: The input layer receives the normalized boundary measurement value y′, and extracts features layer by layer through the convolutional layer, pooling layer, and fully connected layer. The activation function uses the ReLU function to enhance the non-linear expression ability. 4. Loss function calculation: Use the cross-entropy loss. Where For network output, λ controls overfitting. 5. Backpropagation and optimization: Update the parameters through the Adam algorithm and combine the moving average model to stabilize the regularization coefficient. During training, continuously optimize the loss function to adjust the parameters and learn the mapping relationship between low-resolution images and high-resolution images. To ensure that the network can not only accurately reconstruct the image but also follow the physical laws.

[0068] Optionally, after generating the target image, the first imaging model can also denoise and enhance the details of the target image. After these processes, perform a quality assessment on the target image. If it is determined that the quality requirements are met according to the assessment results, output the target image. If it is determined that the assessment results do not meet the quality requirements, the target image can be discarded, and the parameters of the first imaging model can be adjusted. Generate a new target image that meets the quality requirements through the adjusted first imaging model.

[0069] Optionally, the detail enhancement process can be performed by a deep learning-based edge enhancement and detail restoration method. And high-frequency noise can be further suppressed and details can be restored through an edge enhancement filter (such as the Sobel operator) and a variational denoising model.

[0070] Optionally, when performing quality assessment, the peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM) can be used to evaluate the target image. These metrics can not only be used for quantitative analysis of image quality but also evaluate the detail restoration effect and structural fidelity of the image. First, perform a quantitative assessment. PSNR measures the pixel-level similarity between the target image and the real image, and the higher the value, the better the noise suppression effect. SSIM measures the structural consistency, and the closer the value is to 1, the stronger the ability to retain details. During the quality assessment process, evaluate the clarity, noise suppression effect, and detail restoration degree of the image by comparing the differences between the generated target image and the real image. The general metrics are PSNR (≥30dB), SSIM (≥0.9), and relative image error (RIE, ≤5%).

[0071] Optionally, when the reconstruction method is a combination of physical constraints and a generative adversarial network, perform super-resolution reconstruction on the image to be processed based on the reconstruction method, including: obtaining the physical constraint information corresponding to the processed image, and determining the loss function of the generative adversarial network according to the physical constraint information; using the training data corresponding to the image to be processed and the loss function to train the generator and discriminator of the generative adversarial network.

[0072] Optionally, the generator of the generative adversarial network is responsible for converting the low-resolution image into a high-resolution image, and the discriminator classifies the generated image to determine whether it is a real image. During training, the continuous optimization of the generator and discriminator is achieved through the adversarial training of the generator and discriminator.

[0073] Optionally, after the training of the generative adversarial network is completed, the image to be processed is input into the generator, and the generator processes it to generate a high-resolution target image. Among them, the generator includes a convolutional layer, a transposed convolutional layer, a batch normalization (BatchNormalization) layer, and an activation layer. And in order to adapt to complex structures, the generator can also use a deformable convolutional kernel (DeformableConv) to replace the fixed convolutional kernel. The discriminator processes the input image through a series of convolutional layers and outputs the probability that the image is "real". The goal of the discriminator is to accurately identify the difference between the image generated by the generator and the real image, and optimize the training of the generator through feedback information.

[0074] Optionally, the loss function of the generative adversarial network introduces physical constraints such as imaging noise, blur, and spatial information constraints to ensure that the generated target image is not only visually realistic but also conforms to the actual imaging physical process of the image. The loss function can be expressed as:

[0075] H is the blur kernel, and σ represents the noise variance.

[0076] Optionally, since the image to be processed is disturbed by noise, the generator needs to be able to effectively denoise when generating high-resolution images. By introducing a noise model (such as additive Gaussian noise), that is, adding Gaussian noise N ∼ N(0,σ 2 ) to simulate the random interference in the imaging process, the input of the generator can be expressed as where z is the input image, is the noise term. The intensity of the noise term can be adjusted through a gating mechanism. A noise layer can be inserted before the first convolutional layer of the generator, and the noise is added element-wise to the feature map of the input image. The formula is: where F is the original feature map, is the generated noise, and σ is the variance of the Gaussian noise. The discriminator not only judges the authenticity of the image but also can evaluate the denoising effect to help the generator optimize the image quality. During the training phase, the noise variance σ can be dynamically adjusted (such as gradually increasing from 0.1 to 0.5) to enhance the robustness of the model.

[0077] Optionally, in order to eliminate the blur effect of the image to be processed, by adding a blur kernel (such as a Gaussian blur kernel) to the generative adversarial network, that is, introducing the blur kernel H in the discriminator to perform a convolution operation on the reconstructed image to simulate the spatial resolution limitation during imaging. The expression can be: where represents the convolution operation. Add a blur consistency loss to the loss function: where, L blur is the blur consistency loss, and λblur is the weight of the blur constraint, H(.) represents the blur kernel, and I HR represents the reconstructed image output by the generative adversarial network for this image. During training, the generator is guided to restore the high-frequency details in the image and reduce the image blurring phenomenon caused by blur. The discriminator's goal is to distinguish the high-frequency details (such as edges and textures) between real images and generated images. The generator is forced to imitate these details through adversarial training, thereby enhancing the reconstruction ability of high-frequency information. The high-frequency detail loss can also be added to the discriminator's loss function, and the formula for this high-frequency detail loss is: where λ is the weight of the high-frequency detail loss, represents the gradient operator, and I HR1 is the real high-resolution image, and G(z) is the high-resolution image output by the generator. This loss term guides the restoration of high-frequency details by enhancing the edge sharpness.

[0078] In one embodiment, the loss function corresponding to the generator is:

[0079] L total = L GAN + λ phy ·L physical

[0080] In the formula, L GAN is the first loss function, L physical is the physical constraint loss, λ phy is the weight of the physical constraint loss, L physical = L noise + L blur , is the predicted noise distribution, N2 is the real noise, and E ILR is the expected value of the image input to the generative adversarial network, λ noise is the weight of the noise constraint, λ blur is the weight of the blur constraint, H(.) represents the blur kernel, and I LR represents the image input to the generative adversarial network, and I HR represents the reconstructed image output by the generative adversarial network for this image, where the high-frequency detail loss can also be added to the physical constraint loss.

[0081] Optionally, the generator and discriminator are trained in a joint optimization manner. The specific training method can be: 1. Data preprocessing: Add noise N and blur kernel H to the high-resolution image I HR to generate a synthetic degraded image The training data input to the generator G is 2. Adversarial Training Loop: During training, the discriminator is updated by minimizing its corresponding loss function to distinguish real images and high-frequency details. The generator is updated by minimizing its corresponding loss function to optimize high-frequency details and physical consistency. 3. Physical Constraint Weight Adjustment: In the initial stage, the weight of the physical constraint is set small (e.g., 10 -4 ), and it gradually increases with the training iterations (e.g., linear scheduling) to balance detail restoration and physical compliance.

[0082] Optionally, during training, the mean squared error or perceptual loss can also be used to obtain the loss function of the generator. The mean squared error formula is: where I HR (i) is the i-th sample of the real high-resolution image, and G(z i ) is the reconstructed image output by the generator for I HR (i), and N is the number of samples. The perceptual loss is: It uses a pre-trained convolutional neural network in the generator to extract features and calculates the Euclidean distance between the generated image output by the generator and the real image in the feature space: where L percep is the Euclidean distance, φ c (·) is the convolutional feature map of the c-th layer, C is the number of convolutional layers, and G(z) is the generated image. At the same time, the generator also needs to satisfy physical constraints, that is, the generated image should conform to physical laws such as noise, blur, and imaging models.

[0083] Optionally, the discriminator can be optimized by forward propagation and backpropagation.

[0084] Optionally, the reconstruction method is a combination of multi-scale learning and physical constraints. Super-resolution reconstruction of the image to be processed is performed based on the reconstruction method, including: obtaining the physical constraint information corresponding to the image to be processed, adding the physical constraint information to the convolutional layers of different scales of the multi-scale convolutional neural network to obtain a second imaging model; inputting the image to be processed into the second imaging model, using the convolutional layers of different scales in the second imaging model to extract the features of the image to be processed at different scales, and weighted-fusing the features to generate the target image.

[0085] Optionally, the multi-scale convolutional neural network can be composed of multiple parallel convolutional neural network branches, and each branch network is responsible for processing images of different scales. Each scale network can extract features at different resolutions and then fuse them to improve the quality of image reconstruction.

[0086] Optionally, the multi-scale convolutional neural network is trained and optimized by combining the loss of each scale with physical constraints. The loss function of this multi-scale convolutional neural network can be expressed as:

[0087]

[0088] wherein, L i is the reconstruction loss of the i-th scale, and the reconstruction loss can be the reconstruction loss of the convolutional neural network in the above text. is the image reconstructed at the i-th scale, L physical is the physical constraint loss, which is the same as before, λ i and α i are the weight parameters corresponding to the i-th scale, and n1 is the number of scales.

[0089] Optionally, the second imaging model obtains the final high-resolution image by weighted fusion of the outputs of multiple scales. This process can not only improve the image quality but also reduce the errors that may occur during the feature extraction of a certain scale. The specific process is as follows: 1. Calculate the signal-to-noise ratio (SNR) of each scale: where SNR(s) is the signal-to-noise ratio of the s-th scale, which is used to measure the ratio of the effective signal intensity to the noise intensity of the reconstructed image at this scale, E[·] is the mathematical expectation, and the image pixels are statistically averaged, I s is the reconstructed image of the s-th scale, I HR is the high-resolution image corresponding to the reconstructed image. 2. Dynamic weight update: The specific value of the weight w s can be obtained by normalizing the signal-to-noise ratio of each scale, and it is ensured that the sum of all weights is 1. For scales with high SNR (such as regions with rich details where the SNR value is greater than a predetermined value), higher weights will be assigned to enhance their contribution to the final result; scales with low SNR (such as blurred regions with more noise where the SNR value is less than the preset value) will be suppressed to reduce their negative impact on the final result. The formula is:

[0090] where w s is the dynamic weight of the s-th scale, and the value range is [0,1], and s′ is the index for traversing all possible scales. 3. By weighted fusion, enhance the dependence on scales with high signal-to-noise ratio and suppress the noise interference of scales with low signal-to-noise ratio. The fusion formula is: where is the finally fused reconstructed image, is the reconstruction result of the s-th scale.

[0091] Optionally, when the reconstruction method is a combination of self-supervised learning and physical constraints, super-resolution reconstruction is performed on the image to be processed based on the reconstruction method, including: obtaining a training set, physical constraint information, and a prior loss corresponding to the image to be processed, constructing a loss function based on the physical constraint information and the prior loss, where the prior loss is used to limit the solution space of self-supervised learning; training a self-supervised model based on the loss function and the training set to obtain a third imaging model, and using the third imaging model to perform super-resolution reconstruction on the image to be processed.

[0092] Optionally, a key advantage of self-supervised learning methods is that they can be trained using unlabeled low-resolution medical images. By constructing self-supervised tasks for images (such as image reconstruction, denoising, artifact removal, etc.), the third imaging model can learn how to recover high-resolution details from low-resolution images.

[0093] Optionally, the loss function corresponding to self-supervised learning can be:

[0094] L self-supervised = L reconstruction + λL physical + μL prior

[0095] where L prior is the prior knowledge loss, which uses domain knowledge to limit the solution space. Its formula is: L prior = μ · Dice(G(z), I HR ), where the Dice coefficient measures the overlap between the generated image and the true anatomical structure; G(z) is the generated image, and λ, μ are regularization coefficients used to balance the weights of different loss terms. During training, pre-training can be used first, and only L prior is optimized to learn the feature representation of low-resolution images. Then, fine-tuning of the third imaging model is performed to optimize the physical constraint loss L physical and the reconstruction loss L reconstruction to improve the high-resolution performance. The functions of the reconstruction loss and the physical constraint loss can be the same as those in the above embodiments.

[0096] The image super-resolution reconstruction method of this application receives the image to be processed, obtains the reconstruction method of the image to be processed. The reconstruction method is a combination of physical constraints and deep learning, and the corresponding deep learning methods are different for different reconstruction methods. The physical constraints correspond to the imaging process of the image to be processed; super-resolution reconstruction is performed on the image to be processed based on the reconstruction method to generate a target image corresponding to the image to be processed. This application can incorporate physical constraints into the super-resolution reconstruction of images, thereby reducing the noise impact while ensuring image quality, improving the ability to recover image details, and effectively enhancing the image reconstruction effect.

[0097] In an alternative embodiment, an electronic device is provided, such as Figure 2 shown, Figure 2 the electronic device 4000 shown includes: a processor 4001 and a memory 4003. Among them, the processor 4001 and the memory 4003 are connected, such as connected through a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, and the transceiver 4004 may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception, etc. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0098] The processor 4001 may be a CPU (Central Processing Unit, central processor), a general-purpose processor, a DSP (Digital Signal Processor, data signal processor), an ASIC (Application Specific Integrated Circuit, application-specific integrated circuit), an FPGA (Field Programmable Gate Array, field programmable gate array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It can implement or execute various exemplary logic blocks, modules and circuits described in connection with the disclosure of the present application. The processor 4001 may also be a combination that implements computing functions, such as a combination including one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0099] The bus 4002 may include a path for transmitting information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect, peripheral component interconnect standard) bus or an EISA (Extended Industry Standard Architecture, extended industry standard architecture) bus, etc. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, only a thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0100] The memory 4003 can be a ROM (ReadOnlyMemory), or other types of static storage devices that can store static information and instructions, a RAM (RandomAccessMemory), or other types of dynamic storage devices that can store information and instructions. It can also be an EEPROM (ElectricallyErasableProgrammableReadOnlyMemory), a CD-ROM (CompactDiscReadOnlyMemory), or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, which is not limited herein.

[0101] The memory 4003 is used to store the computer program for implementing the embodiments of the present application and is controlled by the processor 4001 for execution. The processor 4001 is used to execute the computer program stored in the memory 4003 to implement the steps shown in the foregoing method embodiments.

[0102] Among them, the electronic device can be any kind of electronic product that can perform human-computer interaction with an object. For example, a personal computer, a tablet computer, a smart phone, a personal digital assistant (PersonalDigitalAssistant, PDA), a game console, an Internet Protocol Television (IPTV), a smart wearable device, etc.

[0103] The electronic device may further include a network device and / or an object device. Among them, the network device includes, but is not limited to, a single network server, a server group composed of multiple network servers, or a cloud composed of a large number of hosts or network servers based on cloud computing (CloudComputing).

[0104] The network where the electronic device is located includes, but is not limited to, the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (VirtualPrivateNetwork, VPN), etc.

[0105] The embodiments of the present application provide a computer-readable storage medium that stores a computer program, and when the computer program is executed by a processor, it can implement the steps and corresponding content of the foregoing method embodiments.

[0106] Those skilled in the art of the present application can understand that the various operations, methods, steps, measures, and solutions in the processes discussed in the present application can be alternated, changed, combined, or deleted. Further, other steps, measures, and solutions in the various operations, methods, and processes discussed in the present application can also be alternated, changed, rearranged, decomposed, combined, or deleted. Further, the steps, measures, and solutions in the related art that are the same as those disclosed in the various operations, methods, and processes in the present application can also be alternated, changed, rearranged, decomposed, combined, or deleted.

[0107] In the description of the present application, the directions or positional relationships indicated by the words "center", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. are the exemplary directions or positional relationships based on the drawings, which are for the convenience of describing or simplifying the embodiments of the present application, rather than indicating or implying that the device or component referred to must have a specific orientation or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation to the present application.

[0108] The terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the present application, unless otherwise specified, the meaning of "a plurality" is two or more.

[0109] In the description of the present application, it should be noted that unless otherwise clearly specified and defined, the terms "mounted", "connected", and "coupled" should be understood in a broad sense. For example, it may be a fixed connection, a detachable connection, or an integral connection; it may be directly connected or indirectly connected through an intermediate medium, and it may be the communication inside two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to specific circumstances.

[0110] In the description of this specification, specific features, structures, materials, or characteristics can be combined in a suitable manner in any one or more embodiments or examples.

[0111] The above are only some embodiments of the present application. It should be noted that for those of ordinary skill in the art of the present technology, without departing from the technical concept of the solution of the present application, other similar implementation means based on the technical idea of the present application also belong to the protection scope of the embodiments of the present application.

Claims

1. A method for super-resolution image reconstruction, characterized in that: The method comprises: receiving an image to be processed, and obtaining a reconstruction method of the image to be processed, wherein the reconstruction method is a combination of physical constraints and deep learning, and different reconstruction methods correspond to different deep learning methods, and the physical constraints correspond to the imaging process of the image to be processed, including at least one of noise constraints and fuzzy constraints; The image to be processed is reconstructed with super resolution based on the reconstruction method to generate a target image corresponding to the image to be processed.

2. The image super-resolution reconstruction method according to claim 1, characterized in that: The reconstruction method is a combination of physical constraints and convolutional neural networks, and the super-resolution reconstruction of the image to be processed based on the reconstruction method includes: Acquire physical constraint information corresponding to the image to be processed, wherein the physical constraint information includes noise constraint and fuzzy constraint; The physical constraint information is added to the loss function of the convolutional neural network to obtain a first imaging model, and the first imaging model is used to perform super-resolution reconstruction on the image to be processed.

3. The image super-resolution reconstruction method according to claim 2, characterized in that: The acquisition of the physical constraint information includes: Determining noise existing in the image to be processed according to an imaging process of the image to be processed, wherein the noise includes at least one of Gaussian noise, Poisson noise, and mixed noise; generating noise constraint information using a noise model corresponding to the noise; A blur kernel corresponding to the image to be processed is determined according to the characteristics of the imaging process, and blur constraints are obtained through the blur kernel.

4. The image super-resolution reconstruction method according to claim 3, characterized in that: The loss function with the physical constraint information added is: L=λ1L reconstruction +λ2L adversarial +λ3L physical In the formula, λ1, λ2, λ3 are the weights of the loss function, L reconstruction is the reconstruction loss, L adversarial To combat the loss, L physical is the physical constraint loss, including noise constraint and modulus constraint, where is the reconstructed image of the i-th sample, y i is the real image of the i-th sample, N is the number of samples, and the adversarial network is used to train L adversarial To optimize, D(x) is the judgment of the discriminator on the real image x in the adversarial network, and G(z) is the image generated by the generator based on the input noise z. To represent the expectation of the input generator noise z, p z (z) is the distribution function of noise z, Represents the distribution p of the real image x data (x), the expected value of the real image x, λ4 is the weight associated with the fuzzy constraint, λ5 is the weight corresponding to the noise constraint, H(.) represents the fuzzy kernel, I LR Represents the image input to the convolutional neural network, I HR Represents the reconstructed image output by the convolutional neural network for the image, and N1 represents the noise constraint information.

5. The image super-resolution reconstruction method according to claim 1, characterized in that: The reconstruction method is a combination of physical constraints and a generative adversarial network, and the super-resolution reconstruction of the image to be processed based on the reconstruction method includes: Acquire physical constraint information corresponding to the processed image, and determine a loss function of the generative adversarial network according to the physical constraint information; The generator and the discriminator of the generative adversarial network are trained using the training data corresponding to the image to be processed and the loss function.

6. The image super-resolution reconstruction method according to claim 5, characterized in that: The loss function corresponding to the generator is: L total =L GAN +λ phy ·L physical Where, L GAN is the first loss function, L physical is the physical constraint loss, λ phy is the weight of the physical constraint loss, L physical =L noise +L blur , is the predicted noise distribution, N2 is the real noise, is the expected value of the image input to the generative adversarial network, λ noise is the weight of the noise constraint, λ blur is the weight of the fuzzy constraint, H(.) represents the fuzzy kernel, I LR I represents the image input to the generative adversarial network, HR Represents the reconstructed image output by the generative adversarial network for this image.

7. The image super-resolution reconstruction method according to claim 1, characterized in that: The reconstruction method is a combination of multi-scale learning and physical constraints, and the super-resolution reconstruction of the image to be processed based on the reconstruction method includes: Acquiring physical constraint information corresponding to the image to be processed, and adding the physical constraint information to convolution layers of different scales of a multi-scale convolutional neural network to obtain a second imaging model; The image to be processed is input into the second imaging model, convolutional layers of different scales in the second imaging model are used to extract features of the image to be processed at different scales, and the features are weightedly fused to generate the target image.

8. The image super-resolution reconstruction method according to claim 1, characterized in that: The reconstruction method is a combination of self-supervised learning and physical constraints, and the super-resolution reconstruction of the image to be processed based on the reconstruction method includes: Obtaining a training set, physical constraint information, and a priori loss corresponding to the image to be processed, and constructing a loss function according to the physical constraint information and the a priori loss, wherein the a priori loss is used to limit the solution space of self-supervised learning; Based on the loss function and the training set training, self-supervised model training is performed to obtain a third imaging model, and the third imaging model is used to perform super-resolution reconstruction on the image to be processed.

9. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images