Image super-resolution method and device based on diffusion model and storage medium

Abstract

The present disclosure relates to a diffusion model based image super-resolution method, device and storage medium. The diffusion model based image super-resolution method according to the embodiment of the present disclosure comprises: acquiring a low-resolution image of an object; inputting the low-resolution image and randomly sampled Gaussian noise into a trained super-resolution image generation model, the super-resolution image generation model comprising a differential attention-free upscaling network for processing input features in a differential manner to predict noise and a sampling control unit for controlling a sampling strategy of a diffusion model; iteratively performing denoising processing on the Gaussian noise through the differential attention-free upscaling network and the sampling control unit to obtain a predicted image indicating a residual error between a low-resolution image and a high-resolution image; and adding the low-resolution image and the predicted image to obtain a high-resolution image of the object.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of image processing, and in particular to a diffusion model based image super-resolution method, device and storage medium. BACKGROUND

[0002] Industrial computed tomography (ICT) has become an indispensable tool for non-destructive testing and precision manufacturing, enabling the detection of micron-scale defects in critical components such as turbine blades and battery electrodes.

[0003] To ensure product reliability, modern industrial systems increasingly require high-resolution ICT images, which are achieved in the prior art using hardware enhancements, such as detector sub-pixel shifting techniques in the prior art that mechanically shift the imaging sensor during ICT multiple exposures to synthesize high-resolution (HR) images by interpolation. However, while the resolution is improved by the multiple steps of shifting, the scan time is also lengthened by several times.

[0004] This resolution-efficiency paradox has led to interest in image super-resolution (SR) methods as an economically viable alternative. Diffusion models have demonstrated promising capabilities in synthesizing high-fidelity images, however, existing diffusion models are too slow to process due to heavy computational demands and slow inference speeds, limiting their practical deployment in industrial settings. Furthermore, the fidelity of the reconstructed images obtained by existing diffusion models is not ideal.

[0005] Furthermore, in the context of industrial CT, real-world SR datasets are very scarce. The images in most SR datasets are usually referred to as high-resolution (HR) images, and the corresponding low-resolution (LR) images are obtained by down-sampling the HR, as it is difficult to collect LR-HR image pairs. However, the degradation process in real-world images can be more complex and variable.

[0006] Therefore, it is desirable to have an image super-resolution method that can quickly obtain high-resolution images with high fidelity. Furthermore, it is also desirable to be able to provide a usable real-world dataset. SUMMARY

[0007] To solve the above technical problems, the embodiment of the present disclosure provides an image super-resolution method based on a diffusion model. By introducing an attention-free differential upscaling network that does not require attention, the processing speed can be accelerated and the sensitivity to inherent noise in the HR image can be reduced. By using a sampling control unit that controls the sampling strategy of the diffusion model, the generation stability can be enhanced and the inference time can be significantly reduced. By using an optimized double-scale fidelity loss function, the overall reconstruction fidelity of the image can be improved. Finally, the embodiment of the present disclosure also captures high and low resolution image pairs without registration errors through hardware-level control, providing a usable real-world dataset, especially for image super-resolution in industrial CT scenarios.

[0008] According to the embodiment of the present disclosure, an image super-resolution method based on a diffusion model is provided, comprising:

[0009] obtaining a low-resolution image of an object;

[0010] inputting the low-resolution image and randomly sampled Gaussian noise into a trained super-resolution image generation model, the super-resolution image generation model comprising an attention-free differential upscaling network for processing input features in a differential manner to predict noise and a sampling control unit for controlling the sampling strategy of the diffusion model;

[0011] iteratively performing denoising processing on the Gaussian noise by the attention-free differential upscaling network and the sampling control unit to obtain a predicted image indicating the residual error between the low-resolution image and the high-resolution image;

[0012] adding the low-resolution image and the predicted image to obtain a high-resolution image of the object.

[0013] Preferably, inputting the low-resolution image and randomly sampled Gaussian noise into the trained super-resolution image generation model comprises:

[0014] inputting the low-resolution image and randomly sampled Gaussian noise as input features into the attention-free differential upscaling network of the super-resolution image generation model, the attention-free differential upscaling network comprising a plurality of adaptive differential upscaling modules cascaded;

[0015] extracting feature information of a plurality of scales of the input features by the plurality of adaptive differential upscaling modules, the feature information of the plurality of scales having different sensitivities to noise;

[0016] performing differential upscaling processing on the extracted feature information of the plurality of scales by the plurality of adaptive differential upscaling modules to remove common mode noise in the feature information of the plurality of scales, and integrating the feature information of the plurality of scales to generate output features.

[0017] Preferably, in each adaptive differential amplification module, each adaptive differential amplification module comprises a first differential amplification unit, a time information embedding unit, a feature affine unit, a deep separable convolution unit, a second differential amplification unit and a ReLU activation function unit. The input features are subjected to differential amplification processing by the first differential amplification unit; the features subjected to differential amplification processing are aggregated with the time embedding features provided by the time information embedding unit by the feature affine unit; the aggregated features are subjected to differential amplification processing by the second differential amplification unit; the input features are subjected to convolution processing by the deep separable convolution unit; the output of the deep separable convolution unit and the output of the second differential amplification unit are added and then passed through the ReLU activation function unit to become the output of the adaptive differential amplification module.

[0018] Preferably, in each of the first differential amplification unit and the second differential unit, each of the first differential amplification unit and the second differential unit comprises a first deep separable convolution unit having a kernel of a first scale, a second deep separable convolution unit having a kernel of a second scale larger than the first scale, and a third convolution unit. Feature information of the first scale is extracted from the input features by the first deep separable convolution unit; feature information of the second scale is extracted from the input features by the second deep separable convolution unit; the feature information of the first scale and the feature information of the second scale are fused; and the fused feature information is subjected to convolution processing by the third convolution unit to generate the output of the differential amplification unit.

[0019] Preferably, fusing the feature information of the first scale and the feature information of the second scale further comprises:

[0020] Adaptively determining a fusion coefficient according to the noise size of the input features;

[0021] Multiplying the feature information of the second scale by the fusion coefficient and fusing with the feature information of the first scale.

[0022] Preferably, iteratively performing the denoising processing on the Gaussian noise by the attention-free differential amplification network and the sampling control unit to obtain a predicted image indicating a residual between a low-resolution image and a high-resolution image further comprises:

[0023] Inputting the low-resolution image and the Gaussian noise into the attention-free differential amplification network to generate a predicted noise of a current time step;

[0024] Inputting the predicted noise of the current time step into the sampling control unit to generate a predicted noise image of a next time step;

[0025] inputting the low-resolution image and the predicted noise image of the next time step into the attention-free differential amplification network to generate predicted noise of the next time step, the predicted noise of the next time step being less than the predicted noise of the current time step;

[0026] iteratively performing the denoising processing by the attention-free differential amplification network and the sampling control unit until reaching a final time step;

[0027] when reaching the final time step, generating the predicted noise image generated by the sampling control unit as a predicted image indicating a residual error between the low-resolution image and the high-resolution image.

[0028] Preferably, inputting the predicted noise of the current time step into the sampling control unit to generate the predicted noise image of the next time step comprises:

[0029] inputting the predicted noise of the current time step into the sampling control unit;

[0030] generating the predicted noise image of the next time step based on the predicted noise of the current time step and the predicted noise images of the previous multiple time steps of the sampling control unit.

[0031] Preferably, the super-resolution image generation model is used to generate a high-resolution image of an industrial CT image.

[0032] The training of the super-resolution image generation model comprises:

[0033] obtaining paired low-resolution images and high-resolution images from a pre-prepared industrial CT image dataset, the industrial CT image dataset comprising a plurality of paired low-resolution images and high-resolution images;

[0034] in a diffusion stage, performing diffusion processing on a residual image of the obtained low-resolution image and high-resolution image iteratively using the low-resolution image as a conditional guide, generating noise images corresponding to multiple time steps until the residual image becomes a pure noise signal;

[0035] in a reverse diffusion stage, inputting the low-resolution image and a predicted noise image corresponding to a time step into the attention-free differential amplification network to perform noise prediction processing to obtain the predicted noise of the time step, setting a loss function, and iteratively optimizing model parameters until a stop training condition is met;

[0036] when the training is stopped, the obtained attention-free differential amplification network is taken as a trained attention-free differential amplification network, and the super-resolution image generation model comprising the trained attention-free differential amplification network is taken as a trained super-resolution image generation model.

[0037] Preferably, the preparing the industrial CT image dataset comprises:

[0038] irradiating the object to be detected using an X-ray source of the industrial CT device, and detecting an X-ray signal using a linear array detector of the industrial CT device to obtain a set of low-resolution projection data of the object;

[0039] at each projection angle, moving the linear array detector along an array direction multiple times and obtaining a set of low-resolution projection data of the object at each moving position, wherein each movement has an equal amplitude and is smaller than a single detector pixel pitch;

[0040] fusing the obtained multiple sets of low-resolution projection data to generate high-resolution projection data;

[0041] extracting a set of low-resolution projection data from the obtained multiple sets of low-resolution projection data, and performing a filtered back-projection process on the extracted set of low-resolution projection data to generate a low-resolution CT slice image of the object as a low-resolution image of the object;

[0042] performing a filtered back-projection process on the high-resolution projection data to generate a high-resolution CT slice image of the object as a high-resolution image of the object;

[0043] storing the generated low-resolution image of the object and the high-resolution image of the object as a pair of low-resolution image and high-resolution image;

[0044] repeating the above processes to obtain multiple pairs of low-resolution image and high-resolution image as the industrial CT image dataset.

[0045] Preferably, the loss function L final is set as:

[0046]

[0047] wherein the loss function L final is set as a mask-based loss function L sample* , an Otsu-Guided Dice loss function L OGD and a mask-based L1 loss function Lm, wog is a weighting coefficient of the loss function L OGD , and wm is a weighting coefficient of the loss function Lm.

[0048] According to another embodiment of the present disclosure, there is provided a diffusion model-based image super-resolution device, comprising: a processor; and a memory storing computer readable instructions which, when executed by the processor, cause the processor to perform the above-mentioned method.

[0049] According to another embodiment of the present disclosure, a computer readable storage medium is provided, having stored thereon computer readable instructions which, when executed by a processor, cause the processor to perform the method described above.

[0050] Therefore, according to the image super-resolution method based on the diffusion model according to the embodiments of the present disclosure, by introducing the attention-free differential magnification network which does not require attention, the processing speed can be accelerated and the sensitivity to the inherent noise in the HR image can be reduced, by using the sampling control unit which controls the sampling strategy of the diffusion model, the generation stability can be enhanced and the inference time can be significantly reduced, by using the optimized double-scale fidelity loss function, the overall reconstruction fidelity of the image can be improved. Finally, the embodiments of the present disclosure also capture high and low resolution image pairs without registration errors through hardware-level control, providing a usable real-world dataset, especially suitable for image super-resolution in industrial CT scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0051] The above and other aspects, features, and advantages of certain embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings.

[0052] Figure 1 is a schematic diagram of a diffusion model describing the prior art;

[0053] Figure 2 is a flowchart of an image super-resolution method based on a diffusion model according to an embodiment of the present disclosure;

[0054] Figure 3 is a working schematic diagram of a super-resolution image generation model according to an embodiment of the present disclosure;

[0055] Figure 4 is a schematic diagram of an attention-free differential magnification network according to an embodiment of the present disclosure;

[0056] Figure 5 is a schematic diagram of the training phase of a diffusion model-based framework according to an embodiment of the present disclosure;

[0057] Figure 6A and Figure 6B is a schematic diagram of the generation process of a LR-HR image pair according to an embodiment of the present disclosure;

[0058] Figure 7 is a comparison diagram of the effects according to an embodiment of the present disclosure and the prior art;

[0059] Figure 8 is a schematic diagram of the qualitative results and analysis of the sampling method and the DDIM method on the ICT-SR dataset according to an embodiment of the present disclosure;

[0060] Figure 9is a comparison plot of qualitative results of the method and the DDIM method according to embodiments of the present disclosure under different sampling steps; and

[0061] Figure 10 is a structural block diagram of an apparatus according to embodiments of the present disclosure. DETAILED DESCRIPTION

[0062] Before undertaking the detailed description below, it can be advantageous to set forth definitions of certain words and phrases used throughout this patent document. The terms "include" and "comprise," as well as derivatives thereof, mean inclusion without limitation. The term "or," is used in the inclusive sense of "and / or" unless it is used in the context of exclusivity. The term "control" or "controller" means any device, system or part thereof that controls at least one operation. Such a controller can be implemented in hardware or a combination of hardware and software and / or firmware. The functionality associated with any particular controller can be centralized or distributed, whether locally or remotely. The phrase "at least one of," when used with respect to a list of items, means that one or more of the listed items can be used and that only one item from the list can be needed. For example, "at least one of A, B, and C" includes: A; B; C; A and B; A and C; B and C; and A, B, and C.

[0063] Definitions for other certain words and phrases are provided throughout this patent document. Those of ordinary skill in the art will understand that in many, if not most instances, such definitions apply to prior and future uses of such defined words and phrases.

[0064] Various embodiments of the principles of the present disclosure in the present patent document are described below in connection with the appended drawings, which are meant to be illustrative and not limiting. Those of ordinary skill in the art will understand that the principles of the present disclosure can be implemented in any suitably arranged system or device. In some instances, actions described in the specification can be performed in a different order and still achieve the desired results. Additionally, the processes depicted in the figures do not necessarily require the particular order shown or sequential order to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0065] Industrial computed tomography (ICT) is a necessary technique for non-destructive testing of complex components, but it faces a fundamental trade-off between resolution and efficiency. While learning-based super-resolution (SR) methods offer promising solutions, their practical deployment is hindered by the lack of real-world datasets and computationally efficient algorithms.

[0066] Figure 1 is a schematic diagram depicting a diffusion model of the prior art.

[0067] As Figure 1As shown, most of the U-Net variants employed in the diffusion model in the prior art rely heavily on attention mechanisms, which significantly increases the computational overhead and slows down the inference. Therefore, there is a need for fast but accurate networks to balance model efficiency and reconstruction quality.

[0068] In addition, ICT images inherently suffer from artifacts caused by inconsistencies in detector responses and limitations of reconstruction algorithms. These artifacts cannot be completely eliminated during imaging, resulting in HR images still containing a certain level of noise. When such artifact-polluted HR images are used as supervisory targets, the model tends to learn and replicate these undesirable artifact patterns during training.

[0069] To solve the above technical problems, the embodiments of the present disclosure provide an image super-resolution method based on a diffusion model, which can speed up the processing speed and reduce the sensitivity to inherent noise in high-resolution (HR) images by introducing an attention-free differential amplification network that does not require attention, can enhance the generation stability and significantly reduce the inference time by using a sampling control unit that controls the sampling strategy of the diffusion model, and can improve the overall reconstruction fidelity of the image by using an optimized double-scale fidelity loss function.

[0070] Figure 2 is a flowchart of an image super-resolution method based on a diffusion model according to an embodiment of the present disclosure.

[0071] As Figure 2 shown, the image super-resolution method 200 according to the embodiments of the present disclosure includes:

[0072] Step S201: Obtain a low-resolution image of a subject.

[0073] Step S202: Input the low-resolution image and randomly sampled Gaussian noise into a trained super-resolution image generation model. The super-resolution image generation model includes an attention-free differential amplifier (ADA) network for processing input features in a differential manner to predict noise and a sampling control unit for controlling the sampling strategy of the diffusion model.

[0074] Step S203: Iteratively perform denoising processing on the Gaussian noise by the attention-free differential amplifier network and the sampling control unit to obtain a predicted image indicating the residual error between the low-resolution image and the high-resolution image.

[0075] Step S204: Add the low-resolution image and the predicted image to obtain a high-resolution image of the subject.

[0076] Specifically, in step S201, the acquired low-resolution image is an original image of the object, for example, a low-resolution image LR of the object acquired by industrial CT.

[0077] In step S202, the ADA network of the super-resolution image generation model is pre-trained, and the related network parameters have been fixed when the training is completed.

[0078] The operations of steps S203 and S204 are implemented by the super-resolution image generation model shown in FIG. 3, which will be described below in detail. Figure 3 Figure 3

[0079] Figure 3 is a working schematic diagram of a super-resolution image generation model according to an embodiment of the present disclosure. As shown in Figure 3 , the super-resolution image generation model 300 includes an ADA network 301 and a sampling control unit 302 that controls the sampling strategy of the diffusion model. In one embodiment, the sampling control unit 302 uses, for example, a progressively refined sampling (PRS) strategy which will be described in detail below.

[0080] The trained super-resolution image generation model starts from the random sampling of Gaussian noise , uses the pre-trained ADA network 301 to iteratively denoise the Gaussian noise . As is well known to those skilled in the art, the diffusion model uses several time steps of noise for denoising.

[0081] At the beginning of the operation, the low-resolution image LR and the Gaussian noise are input into the ADA network 301 to generate the predicted noise of the current time step. Then, the predicted noise of the current time step is input into the sampling control unit 302 to generate the predicted noise image x t .

[0082] Then, the predicted noise image x t of the next time step is concatenated with the LR image and fed into the ADA network 301 to output the predicted noise of the next time step , and then the predicted noise is refined by the sampling control unit 302 using the PRS strategy to predict the noise image x t-1 of the next time step.

[0083] By iteratively performing the above operations, when the last time step t=0 is reached, the predicted clean image (i.e., the high-resolution image) is obtained by adding the LR image and the generated predicted noise image , that is, .​​

[0084] The sampling control unit 302 combines historical information to improve the stability and quality of the sampling process. In one embodiment, after the predicted noise of the current time step is input into the sampling control unit 302, the sampling control unit 302 generates the predicted noise image of the next time step based on the predicted noise of the current time step and the predicted noise images of the previous multiple time steps of the sampling control unit 302.

[0085] For example, the sampling control unit 302 is based on a PRS strategy described below.

[0086] In the inverse sampling process of the diffusion model, the selection of the sampling strategy plays a crucial role in the quality and efficiency of image reconstruction. Most existing methods rely on numerical methods to approximate the inverse diffusion trajectory. For example, the widely used deterministic sampling method (DDIM) can be interpreted as solving ordinary differential equations (ODEs) using the Euler method. Although DDIM significantly speeds up inference by skipping sampling, DDIM suffers from problems such as error accumulation and denoising instability when the number of sampling steps is low.

[0087] To address these issues, the embodiments of the present disclosure introduce a progressively refined sampling (PRS) strategy to improve the sampling rate, which improves the denoising process from the perspective of numerical optimization to improve the CT recovery quality and sampling efficiency. This method improves the quality of image reconstruction with low sampling steps, providing a more stable and accurate solution for HR imaging tasks.

[0088] The inverse denoising process of the diffusion model can be written in the form of a differential equation. Taking the sampling method of DDIM as an example, the form of the ordinary differential equation is:

[0089] Equation 1

[0090] The above Equation 1 is equivalent to the numerical solution step of solving ODE, and the present disclosure can re-parameterize it as:

[0091] The above Equation 1 becomes:

[0092] Equation 2

[0093] The PRS strategy according to the embodiments of the present disclosure is applied to the above formula 2 to accelerate the diffusion sampling process. In contrast to the single-step prediction employed by DDIM, the method according to the embodiments of the present disclosure utilizes historical information from multiple previous steps to guide the sampling update. Specifically, the method according to the embodiments of the present disclosure utilizes the sample states from the previous two time steps to estimate the next state , thereby reducing the cumulative error and enhancing the sampling accuracy. The specific process is shown in Algorithm 1:

[0094]

[0095] As a result, compared with the traditional single-step prediction method, the method according to the embodiments of the present disclosure more effectively utilizes the historical information from the previous steps, thereby resulting in a more stable update step and reducing the sensitivity to error accumulation. In addition, the quality of image reconstruction is significantly improved with a small number of step sampling.

[0096] In the following, the attention-free differential amplification network will be described in detail with reference to Figure 4 . Figure 4 is a schematic diagram of an attention-free differential amplification network according to an embodiment of the present disclosure.

[0097] As Figure 4 shown, the attention-free differential amplification network 400 includes a first convolutional layer, four adaptive differential amplification modules (ADAMs) 401 including a cascade, a second convolutional layer, four adaptive differential amplification modules ADAMs 402 including a cascade, and a third convolutional layer. The four ADAMs 401 form an encoder. The four ADAMs 402 form a decoder. The ADAMs 401 and the ADAMs 402 have the same structure.

[0098] In the encoder, the scales of the four ADAMs 401 gradually decrease. In the decoder, the scales of the four ADAMs 402 gradually increase, respectively the same as the scales of the four corresponding ADAMs 401 in the encoder. The ADAM avoids attention blocks while maintaining the ability of the model to distinguish between real structures and artifacts, effectively improving the speed and fidelity.

[0099] As Figure 4 shown in the upper part, a low-resolution image and randomly sampled Gaussian noise are input as input features Fin to the attention-free differential amplification network 400 of the super-resolution image generation model.

[0100] Then, the feature information of multiple scales of the input features Fin is extracted by the ADAMs 401 and the ADAMs 402, which have different sensitivities to noise.

[0101] The ADAM 401 and the ADAM 402 also perform differential amplification processing on the extracted multi-scale feature information to remove common mode noise in the multi-scale feature information and integrate the multi-scale feature information to generate output features Fout.

[0102] As shown in the middle part, Figure 4 Each ADAM includes a first differential amplification block (DA-block) 403, a Time Embedding 404, a FeatureWiseAffine 405, a Deeply Separable Convolution (DSConv) 406, a second differential amplification block (DA-block) 407, and a ReLU activation function unit (ReLU) 408.

[0103] The input features Fin are subjected to differential amplification processing by the DA-block 403. The differential amplification-processed features are aggregated with time embedding features provided by the Time Embedding 404 by the FeatureWiseAffine 405. The aggregated features are subjected to differential amplification processing by the DA-block 407. The input features Fin are subjected to convolution processing by the DSConv 406. The output of the DSConv 406 and the output of the DA-block 407 are added and then passed through the ReLU 408 to become the output of the ADAM.

[0104] The DA-block 403 and the DA-block 404 have the same structure. As shown in the lower part of Figure 4 As shown in the lower part of

[0105] The first-scale feature information is extracted from the input features Fin by the DConv3 409 to generate a feature map A. The second-scale feature information is extracted from the input features Fin by the DConv5 410 to generate a feature map B.

[0106] The first-scale feature information and the second-scale feature information are fused by A- ·B to generate a fused feature map D. These are fusion coefficients, and they are learnable parameters. For example, the fusion coefficients can be adaptively determined based on the noise level of the input features, allowing each scale feature to adaptively adjust its contribution weight according to the input features.

[0107] The third convolutional unit 411 performs convolution processing on the fused feature information (i.e., feature map D) to generate the output Fout of the DA-block.

[0108] Figure 5 This is a schematic diagram of the training phase of a diffusion-based model framework according to embodiments of the present disclosure. Figure 5 As shown, firstly, paired low-resolution (LR) images and high-resolution (HR) images are obtained from a pre-prepared industrial CT image dataset. The industrial CT image dataset includes multiple paired low-resolution (LR) images and high-resolution (HR) images.

[0109] During the diffusion phase, the low-resolution image LR is used as a conditional guide to iteratively perform diffusion processing on the residual images x0 of the acquired low-resolution and high-resolution images, generating noise images x corresponding to multiple time steps t. t until the noisy image becomes a pure noise signal x T The forward process follows a Markov chain and is represented as... , where y represents the LR image used as conditional guidance.

[0110] During the inverse diffusion stage, the low-resolution image LR and the noisy image x corresponding to one time step t are combined. t The input is fed into an attentionless differential amplification (ADA) network to perform noise prediction processing (i.e., denoising). The learning objective is to learn the conditional inverse diffusion process. This process can iteratively denoise and restore a clean image. For example, at each time step t, the low-resolution image LR and the noisy image x are compared. t Cascaded and fed into the ADA network to estimate time t To eliminate noise, set a loss function and iteratively optimize the model parameters until the training stop condition is met.

[0111] ICT images must not only accurately represent the contours and dimensions of objects but also clearly reveal subtle defects, thus demanding high fidelity. However, existing diffusion model methods suffer from insufficient fidelity, and the generated images cannot meet the high-precision requirements of industrial applications. Therefore, the image super-resolution method according to embodiments of this disclosure proposes a novel dual-scale fidelity loss. It imposes stricter constraints on images generated by the diffusion model. The dual-scale fidelity loss function consists of two key components: the Otsu-Guided Dice loss function L...OGD and Mask-based L1 loss function. Loss).

[0112] (1) Otsu-Guided Dice loss function:

[0113] Otsu-Guided Dice loss function L OGD aims to improve the accuracy of structure alignment and enhance image fidelity from a regional perspective. Specifically, for the predicted image , the Otsu threshold T is first calculated. Since the Otsu method is not differentiable, the method according to the embodiments of the present disclosure applies a gradient detachment process to ensure that it does not interfere with gradient calculation. Then, a sigmoid-based smoothing binarization operation is introduced, which is formulated as follows:

[0114] Formula 3

[0115] Formula 4

[0116] where respectively represent the estimated denoised image and the real label (ground truth) image, p and are values processed by the sigmoid function, controls the steepness of the sigmoid function, where a larger makes the output closer to hard binarization, while a smaller results in a smoother transition. This realizes a differentiable soft binarization, allowing the neural network to adjust its predictions during training to better approximate the final binarization result calculated by Otsu. When calculating the loss, the Dice loss is further combined to optimize the model. The Otsu-Guided Dice loss can be represented as follows:

[0117] Formula 5

[0118] where represents the predicted value at pixel (i,j), represents the real label value at pixel (i,j), B is the batch size, is a smoothing term to prevent division by zero error.

[0119] (2) Mask-based L1 loss function:

[0120] The mask-based L1 loss function is designed to directly penalize the pixel-level difference between the denoised output image and the real label image, thereby improving the fidelity of the denoised output image. The loss formula is as follows:

[0121] Formula 6

[0122] wherein, denotes the denoised output image, is the real label clean image, denotes the mask corresponding to .

[0123] The total training loss function also considers the mask-based loss function.

[0124] Formula 7

[0125] wherein, denotes the network-predicted noise, is the real added noise, denotes the mask corresponding to .

[0126] The total training loss function can be summarized as:

[0127] Formula 8

[0128] wherein, is a weighting coefficient, denotes the mask-based loss. By strengthening the supervision in the key areas, the model can generate more accurate, clear and structurally coherent image content in these areas.

[0129] That is, according to the method of the embodiments of the present disclosure, the loss function L final is set as a composite loss function of the mask-based loss loss function L sample* , the Otsu-Guided Dice loss function L OGD and the mask-based L1 loss function Lm.

[0130] When the loss function satisfies the condition, stop training. The attention-free differential magnification network at the time of stopping training is taken as the trained attention-free differential magnification network, and the super-resolution image generation model including the trained attention-free differential magnification network is taken as the trained super-resolution image generation model.

[0131] Below, the specific process of generating the LR-HR image dataset will be described with reference to Figure 6A and Figure 6B .Figure 6A and Figure 6B is a schematic diagram of a generation process of a LR-HR image pair according to embodiments of the present disclosure.

[0132] The dataset according to embodiments of the present disclosure is for example constructed using a high-energy industrial CT system. This high-energy industrial CT system is optimized for penetrating dense industrial parts with steel equivalent thickness up to 240 mm.

[0133] The high-energy industrial CT system comprises four core subsystems:

[0134] X-ray generator: In operation, high-energy X-rays with a focal spot of 2.0 mm are generated, for example using a 9 MeV linear accelerator.

[0135] Precise rotation mechanism: The test object is mounted on a high-precision rotation stage for precise rotation. This is crucial for eliminating geometric distortions in CT image reconstruction.

[0136] Linear array detector: Linear array detector with sub-pixel shifting mechanism. Attenuated X-ray signals can be captured across 360° rotational projections. The sub-pixel shifting mechanism enables super-resolution of images through hardware shifting.

[0137] Mechanical system: Integrates the main components together and controls their relative positions. Active isolation granite base ensures position stability.

[0138] The proposed dataset comprises 152 paired LR-HR slices acquired from three classes of metallic parts including aluminum alloy, titanium alloy, and steel. Each constituent is scanned in 3-5 different axial layers to capture representative structural variations such as internal holes. Each CT slice has a resolution of 2048 x 2048 pixels with 16-bit grayscale depth.

[0139] To focus on the super-resolution related foreground region, a foreground mask is first generated from each CT image. The Otsu thresholding method is used. The mask is then carefully refined by manual correction to ensure accurate isolation of the foreground region. Adaptive cropping is performed based on the resulting mask and the target object size, yielding images ranging from 800 x 800 to 1200 x 1200 pixels. It is worth noting that the final mask is preserved and applied during the subsequent model training and testing process to reinforce the focus on the relevant region of interest.

[0140] The dataset is divided into training, validation, and test sets in a 7:1:2 ratio by a holdout method. For the training set, an overlapping sliding window strategy (stride = 256 pixels) is used to generate 4,000 pairs of 512 x 512 pixel patches. These pairs of patches, along with their corresponding masks, form the core training data used to develop and evaluate the super-resolution model.

[0141] Figure 6A The process of obtaining low resolution images and high resolution projection pairs is shown. As shown in Figure 6A At each projection angle, the array detector undergoes a number of minute shifts along its array direction. Each shift has an equal magnitude and is less than the pixel pitch of the detector. The projection data collected at each discrete position is recorded, and each individual discrete projection data can be used as a low resolution (LR) sinogram, thereby obtaining a number of low resolution sinograms. Then, these sub-pixel shifted low resolution LR datasets are fused to form a high resolution (HR) sinogram. Finally, a low resolution LR slice image is reconstructed from the LR sinogram using a filtered back projection (FBP) algorithm, and a high resolution HR slice image is reconstructed from the HR sinogram. The obtained LR slice image and HR slice image are paired as a low resolution image and a high resolution projection pair.

[0142] Figure 6B The process of obtaining high resolution projection data by array detector shifting is further shown. Figure 6B Taking two minute shifts as an example, let the original pixel size be N, and define the minute shift distance as M = N / 3. By shifting the detector twice, three different sets of LR measurements can be obtained, denoted as LI, L2, and L3. Each is offset by M from the previous position. Subsequently, these sub-pixel shifted datasets are aligned and averaged on a pixel-by-pixel basis to form the output H of the final HR dataset. Because each shift samples a different sub-pixel region, combining the three datasets effectively increases the spatial sampling rate and results in a super-resolution image that exceeds the original resolution of the detector.

[0143] Unlike in the prior art, where a low resolution image is typically obtained by downsampling a high resolution image, thereby forming a paired image pair, according to the present disclosure, by employing precise mechanical control during data acquisition, the registration error between the LR-HR pair is effectively eliminated, thus obtaining a dataset suitable for industrial CT image super-resolution.

[0144] To verify the effectiveness of the method according to the embodiments of the disclosure, comparative experiments were conducted using typical CNN, transformer, GAN and diffusion image restoration methods, including DnCNN, MIRNet, HINet, RESTOMER, SwinIR, SRGAN, ESRGAN, CAL-GAN, SR3, Diwa.

[0145] Figure 7 is a comparison chart of the effects of the embodiments according to the disclosure and the prior art. From the chart, it can be clearly observed that the method of the embodiments of the disclosure has two obvious advantages over other methods. First, for the restoration of internal defects, the method of the embodiments of the disclosure completely preserves the detailed information of the defects, better consistent with the characteristics of human visual perception. In the visualization results of the first two rows, MIRNet and Restormer both show over-smoothed repairs, ignoring many internal defect details. Although CNN-based methods generally cannot restore defect details, SRGAN and CAL-GAN perform better in defect restoration, but it is still difficult to completely restore complex defect structures. In contrast, SR3 and Diwa show excellent defect restoration capability, significantly better than GAN-based methods.

[0146] However, despite these improvements, there are still some defects in the restored images that have not been addressed. In addition, compared with the method of the embodiments of the disclosure, the Diwa method produces over-smoothed repairs. Furthermore, the method of the embodiments of the disclosure performs well in restoring edge information, effectively maintaining the sharpness and structural integrity of the image boundaries. For example, in the visualization results of the last two rows, the edges of the geometric structures restored by DnCNN look over-smoothed, while SR3 produces blurred edges lacking texture details. By comparison, the method of the embodiments of the disclosure not only preserves more texture details, but also restores edge contours with higher sharpness and accuracy.

[0147] Table 1 Quantitative results of prior art methods and the method of the disclosure on the ICT-SR dataset

[0148]

[0149] Table 1 presents a quantitative comparison between the method of the embodiments of the disclosure and several prior art methods on the ICT-SR dataset. The results show that the method of the embodiments of the disclosure achieves excellent performance in terms of LPIPS and FID, significantly better than all the comparison methods. Specifically, compared with CNN-based and transformer-based models, it always produces higher perceptual quality and better fine image detail restoration. In terms of LPIPS, the method of the embodiments of the disclosure improves by 40.7% and 41.5% over MIRNet and SwinIR, respectively.

[0150] Compared to the perceptual optimization based approach, the model of the embodiments of the present disclosure achieves more accurate pixel-wise reconstruction as reflected by the PSNR metric. For example, it achieves 4.5% improvement compared to CAL-GAN, and outperforms the recent diffusion based method DIWA by 2.7%.

[0151] In terms of computational efficiency, the model of the embodiments of the present disclosure achieves a significant reduction in model size, containing 433 million fewer parameters compared to the recent diffusion based method (e.g. Diwa). It also provides faster inference, requiring only 0.975 seconds for a 512512 resolution image which is much faster than the existing diffusion based methods. As shown in Table 1, the computational cost of the method of the embodiments of the present disclosure is 93.219 g flops, demonstrating comparable efficiency to the state-of-the-art diffusion models including SR3 and Diwa.

[0152] To evaluate the effectiveness of the proposed PRS strategy, the embodiments of the present disclosure compare it with the single-step DDIM sampling method which relies only on fixed time steps.

[0153] Figure 8 The qualitative results and analysis of the sampling method and the DDIM method according to the embodiments of the present disclosure on the ICT-SR dataset are shown in Figure 8 As shown, the advantages of PRS become more apparent with a limited number of sampling steps. Specifically, with fewer sampling steps, PRS consistently outperforms DDIM on multiple metrics, demonstrating its superior ability to generate high-quality images under constrained inference conditions. As the number of sampling steps increases, the performance gap between PRS and DDIM gradually narrows.

[0154] Figure 9 The comparison chart of the qualitative results of the method and the DDIM method according to the embodiments of the present disclosure under different sampling steps is Figure 9 Further visual comparisons between PRS and DDIM are shown. Obviously, the method of the present disclosure produces more faithful reconstructed images with fewer structural artifacts and higher texture continuity. The results also show that PRS maintains more stable image quality across different regions, especially in the recovery of fine details.

[0155] Figure 10 The structural block diagram of the apparatus according to the embodiments of the present disclosure is

[0156] As Figure 10As shown, the apparatus 1100 for image super-resolution can include a processor 1110 and a memory 1120. The memory 1120 has stored therein computer readable instructions which, when executed by the processor 1110, cause the processor 1110 to perform one or more steps of the various methods as described above and additional aspects thereof.

[0157] At least one embodiment according to the present disclosure can be a computer readable storage medium. The computer readable storage medium has stored thereon computer instructions which, when executed by a processor, perform one or more steps of the various methods as described above and additional aspects thereof.

[0158] Exemplarily, the non-transitory computer readable storage medium can be any combination of one or more computer readable storage media, for example, one computer readable storage medium contains program code for performing the various methods described above.

[0159] Exemplarily, when the program code is read by a computer, the computer can execute the program code stored in the computer storage medium to perform one or more steps of the various methods described above and additional aspects thereof, for example, according to at least one embodiment of the present disclosure.

[0160] Exemplarily, the computer readable storage medium can include a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM), a portable compact disc read only memory (CD-ROM), a flash memory, and other non-transitory readable storage medium or any combination thereof.

[0161] According to the image super-resolution method based on the diffusion model of the embodiment of the present disclosure, by introducing the attention-free differential magnification network which does not require attention, the processing speed can be accelerated and the sensitivity to the inherent noise in the HR image can be reduced, by using the sampling control unit which controls the sampling strategy of the diffusion model, the generation stability can be enhanced and the inference time can be significantly reduced, by using the optimized double-scale fidelity loss function, the overall reconstruction fidelity of the image can be improved. Finally, the embodiment of the present disclosure also captures the high and low resolution image pairs without registration error through hardware level control, provides a usable real world dataset, and is especially suitable for image super-resolution in industrial CT scenarios.

[0162] It should be noted that the flowcharts and block diagrams in the drawings are illustrations of the possible architectures, functional processes, and operations of systems, methods, and computer program products in accordance with various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or combinations of hardware and software.

[0163] In general, the various example embodiments of the application can be implemented in hardware or special-purpose circuits, software, firmware, logic, or any combination thereof. Some aspects of the application can be implemented in hardware, while other aspects can be implemented in firmware or software which can be executed by a controller, microprocessor or other computing device, although the application is not limited thereto. While various aspects of the application can be illustrated and described as block diagrams, flow charts, or using some other pictorial representation, it is well understood that these forms of representation are used by those skilled in the art to facilitate discussion of the concepts involved.

[0164] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0165] The foregoing is a summary of the present application and is not to be considered as limiting its scope. While several example embodiments of the application have been described, it will be apparent to those of ordinary skill in the art that many modifications are possible without departing from the novel teachings and advantages of the application. The embodiments chosen and described are meant to be illustrative only and are not intended to limit the scope of the application. It is to be understood that the above description is intended to be illustrative and not restrictive. Many embodiments will be apparent to those of skill in the art upon reviewing the above description. The scope of the application should, therefore, be determined not with reference to the above description, but should instead be determined with reference to the appended claims, along with their full scope of equivalents.

Claims

1. A method for image super-resolution based on a diffusion model, characterized in that, Comprising: obtaining a low-resolution image of an object; inputting the low-resolution image and randomly sampled Gaussian noise into a trained super-resolution image generation model, the super-resolution image generation model comprising a non-attentional differential upscaling network for processing input features in a differential manner to predict noise and a sampling control unit for controlling a sampling strategy of a diffusion model, the non-attentional differential upscaling network comprising a plurality of adaptive differential upscaling modules cascaded; iteratively performing denoising processing on the Gaussian noise by the non-attentional differential upscaling network and the sampling control unit to obtain a predicted image indicative of a residual between a low-resolution image and a high-resolution image; and adding the low-resolution image and the predicted image to obtain a high-resolution image of the object, the training of the super-resolution image generation model comprising: obtaining paired low-resolution images and high-resolution images from a pre-prepared image dataset, the image dataset comprising a plurality of paired low-resolution images and high-resolution images; in a diffusion stage, iteratively performing noise adding processing on a residual image of the obtained low-resolution image and high-resolution image using the low-resolution image as a conditional guide, generating noise images corresponding to a plurality of time steps until the residual image becomes a pure noise signal; in a reverse diffusion stage, inputting the low-resolution image and a predicted noise image corresponding to a time step into the non-attentional differential upscaling network to perform noise prediction processing to obtain noise prediction of the time step, setting a loss function, and iteratively optimizing model parameters until a training termination condition is met; and when the training is terminated, obtaining the non-attentional differential upscaling network as a trained non-attentional differential upscaling network, and obtaining the super-resolution image generation model comprising the trained non-attentional differential upscaling network as a trained super-resolution image generation model, The loss function L final is set to:

2. wherein, the loss function L final the mask-based L sample the loss function L sample* the Otsu-Guided Dice loss function L OGD the composite loss function of the mask-based L1 loss function Lm, w og The loss function L OGD The weighting coefficient, w m These are the weighting coefficients of the loss function Lm.

3. The method of claim 1, wherein, inputting the low-resolution image and randomly sampled Gaussian noise into the trained super-resolution image generation model comprising: inputting the low-resolution image and randomly sampled Gaussian noise as input features into the non-attentional differential upscaling network of the super-resolution image generation model; extracting feature information of a plurality of scales of the input features by the plurality of adaptive differential upscaling modules, the feature information of the plurality of scales having different sensitivities to noise; and performing differential upscaling processing on the extracted feature information of the plurality of scales by the plurality of adaptive differential upscaling modules to remove common mode noise in the feature information of the plurality of scales, and integrating the feature information of the plurality of scales to generate output features.

4. The method of claim 2, wherein, Each adaptive differential upscaling module comprises a first differential upscaling unit, a time information embedding unit, a feature affine unit, a depth separable convolution unit, a second differential upscaling unit, and a ReLU activation function unit, performing differential upscaling processing on the input features by the first differential upscaling unit; aggregating the features after the differential upscaling processing with time embedding features provided by the time information embedding unit by the feature affine unit; performing differential upscaling processing on the aggregated features by the second differential upscaling unit; convolve the input feature through a depth separable convolution unit; and add the output of the depth separable convolution unit and the output of the second differential amplification unit through a ReLU activation function unit to become the output of the adaptive differential amplification module.

5. The method of claim 3, wherein, Each of the first differential amplification unit and the second differential amplification unit includes a first depth separable convolution unit having a kernel of a first scale, a second depth separable convolution unit having a kernel of a second scale larger than the first scale, and a third convolution unit, extracting, by the first depth separable convolution unit, feature information of a first scale from the input feature; extracting, by the second depth separable convolution unit, feature information of a second scale from the input feature; performing feature fusion on the feature information of the first scale and the feature information of the second scale; and performing convolution processing on the fused feature information by the third convolution unit to generate the output of the differential amplification unit.

6. The method of claim 4, wherein, The feature fusion on the feature information of the first scale and the feature information of the second scale further includes: adaptively determining a fusion coefficient according to the noise size of the input feature; and multiplying the feature information of the second scale by the fusion coefficient and fusing with the feature information of the first scale.

7. The method of claim 1, wherein, The iterative denoising processing on the Gaussian noise by the attention-free differential amplification network and the sampling control unit to obtain the predicted image indicating the residual between the low-resolution image and the high-resolution image further includes: inputting the low-resolution image and the Gaussian noise into the attention-free differential amplification network to generate a predicted noise of a current time step; inputting the predicted noise of the current time step into the sampling control unit to generate a predicted noise image of a next time step; inputting the low-resolution image and the predicted noise image of the next time step into the attention-free differential amplification network to generate a predicted noise of the next time step, the predicted noise of the next time step being smaller than the predicted noise of the current time step; iteratively performing the denoising processing by the attention-free differential amplification network and the sampling control unit until reaching a last time step; and when reaching the last time step, generating the predicted noise image generated by the sampling control unit as the predicted image indicating the residual between the low-resolution image and the high-resolution image.

8. The method of claim 6, wherein, The inputting the predicted noise of the current time step into the sampling control unit to generate the predicted noise image of the next time step includes: inputting the predicted noise of the current time step into the sampling control unit; and generating the predicted noise image of the next time step based on the predicted noise of the current time step and the predicted noise images of previous time steps of the sampling control unit.

9. The method of claim 1, wherein, The super-resolution image generation model is used to generate a high-resolution image of an industrial CT image, and the image data set includes an industrial CT image data set.

10. The method of claim 8, wherein, Preparing the industrial CT image data set includes: illuminating an object to be detected using an X-ray source of an industrial CT device, and detecting an X-ray signal using a linear array detector of the industrial CT device to obtain a set of low-resolution projection data of the object; at each projection angle, moving the linear array detector along the array direction multiple times and acquiring a set of low-resolution projection data of the object at each moved position, wherein each movement has an equal magnitude and is smaller than a single detector pixel pitch; fusing the acquired multiple sets of low-resolution projection data to generate high-resolution projection data; extracting a set of low-resolution projection data from the acquired multiple sets of low-resolution projection data, and performing a filtered back-projection process on the extracted set of low-resolution projection data to generate a low-resolution CT slice image of the object as a low-resolution image of the object; performing a filtered back-projection process on the high-resolution projection data to generate a high-resolution CT slice image of the object as a high-resolution image of the object; storing the generated low-resolution image of the object and the high-resolution image of the object as a paired low-resolution image and high-resolution image; and repeating the above processes to obtain multiple paired low-resolution images and high-resolution images as an industrial CT image data set.

11. An image super-resolution apparatus based on a diffusion model, characterized by, comprising: a processor; and a memory storing computer readable instructions that, when executed by the processor, cause the processor to perform the method of any one of claims 1 to 9. a computer readable medium having stored thereon computer readable instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, ​

Images