Image enhancement method and device, electronic equipment and storage medium

Abstract

The invention provides an image enhancement method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a to-be-processed source domain image and a to-be-processed noise image, extracting the edge information of the source domain image, inputting the edge information and the noise image into a preset image enhancement diffusion model, and obtaining an enhanced target domain image, according to the method, conversion from the source domain image to the target domain image is achieved, the target domain image is generated by guiding the image enhancement diffusion model through the edge information, details of the source domain image can be reserved, the fidelity and the structural similarity of the enhanced image are greatly improved, and compared with an existing image translation method, the enhanced image with higher quality can be generated.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image enhancement method, device, electronic device and storage medium. Background Art

[0002] In the field of medical imaging, a variety of methods are commonly used to obtain spatially resolved information about organs and tissues in the body. This includes X-ray imaging, CT (computed tomography), MRI (magnetic resonance imaging), and PET (positron emission tomography). Among them, X-ray imaging is often used for modern surgical navigation, formulation of treatment plans, rapid detection and diagnosis of diseases, etc. However, X-ray imaging often has problems such as poor clarity, image distortion, and information confusion.

[0003] Optimizing image quality is an important step before extracting diagnostic information. Especially when using automated image analysis tools, the accuracy and reliability of their results rely on high-quality images. In certain cases, information from the acquired data can be used to generate additional image information without additional inspection. Medical image enhancement can be classified as image-to-image translation, which aims to transfer images from one domain to another while maintaining structural integrity. It is considered a new frontier in the field of medical image analysis, which helps to improve the diagnostic accuracy and performance of downstream imaging tasks and has many potential applications.

[0004] In response to the problem of image distortion, related technologies have proposed image enhancement through diffusion models. However, the application of diffusion models in medical image-to-image translation tasks is currently limited because its forward diffusion process on the source image leads to the loss of structural details, which cannot be fully restored in the reverse denoising process. The geometric and structural integrity of the translated (enhanced) image is particularly important for medical applications such as surgical planning and radiotherapy. The compromise of structural accuracy in image translation may lead to serious consequences, including potential harm to patients or inadequate treatment of diseases. Therefore, how to achieve high-fidelity, geometrically complete image enhancement has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide an image enhancement method, device, electronic device and storage medium to solve the above-mentioned technical problems.

[0006] In one aspect, an image enhancement method is provided, comprising:

[0007] Obtaining a source domain image and a noise image to be processed;

[0008] Extracting edge information of the source domain image;

[0009] The edge information and the noise image are input into a preset image enhancement diffusion model to obtain an enhanced target domain image.

[0010] In one embodiment, the image enhancement diffusion model is a model trained in the following manner:

[0011] Acquire a training sample data set and an initial diffusion model; the training sample data set includes a plurality of training sample image pairs, wherein the training sample image pairs include a source domain training sample image and a corresponding target domain training sample image;

[0012] Training is performed based on the training sample data set and the initial diffusion model, and when a training completion condition is met, the image enhancement diffusion model is obtained.

[0013] In one embodiment, the initial diffusion model includes a non-diffusion network; and obtaining a training sample data set includes:

[0014] Acquire a real sample image in the source domain; input the real sample image in the source domain into the non-diffusion network to generate a pseudo target domain image corresponding to the real sample image in the source domain; use the real sample image in the source domain and the pseudo target domain image as the training sample image pair;

[0015] and / or,

[0016] Acquire a real sample image in the target domain; input the real sample image in the target domain into the non-diffusion network to generate a pseudo source domain image corresponding to the real sample image in the target domain; and use the real sample image in the target domain and the pseudo source domain image as the training sample image pair.

[0017] In one of the embodiments, the source domain real sample image is a two-dimensional X-ray image, and the target domain real sample image is a target two-dimensional image obtained by projecting a three-dimensional CT image.

[0018] In one embodiment, the non-diffusion network is a non-diffusion generative adversarial network, and the step of obtaining a training sample data set includes:

[0019] Input the source domain real sample image into the first non-diffusion generator of the non-diffusion generative adversarial network to obtain the pseudo target domain image; input the target domain real sample image into the second non-diffusion generator of the non-diffusion generative adversarial network to obtain the pseudo source domain image;

[0020] The training based on the training sample data set and the initial diffusion model includes:

[0021] The pseudo target domain image and the target domain real sample image are input into a first non-diffusion discriminator of the non-diffusion generative adversarial network; and the pseudo source domain image and the source domain real sample image are input into a second non-diffusion discriminator of the non-diffusion generative adversarial network.

[0022] In one embodiment, the initial diffusion model includes a reverse diffusion generative adversarial network, and the training based on the training sample data set and the initial diffusion model includes:

[0023] Adding noise to the target domain real sample image to obtain a target domain noise image;

[0024] Extracting source edge information of the pseudo source domain image;

[0025] Inputting the target domain noise image and the source edge information into a first diffusion generator of the back-diffusion generative adversarial network to obtain an enhanced reconstructed target domain image;

[0026] The reconstructed target domain image is input into the first diffusion discriminator of the back-diffusion generative adversarial network.

[0027] In one embodiment, the training based on the training sample data set and the initial diffusion model includes:

[0028] Adding noise to the real sample image in the source domain to obtain a source domain noise image;

[0029] Extracting target edge information of the target domain training sample image;

[0030] Inputting the source domain noise image and the target edge information into a second diffusion generator of the back-diffusion generative adversarial network to obtain an enhanced reconstructed source domain image;

[0031] The reconstructed source domain image is input into the second diffusion discriminator of the back-diffusion generative adversarial network.

[0032] Furthermore, an image enhancement device is provided, comprising:

[0033] An acquisition module, used for acquiring a source domain image and a noise image to be processed;

[0034] An extraction module, used for extracting edge information of the source domain image;

[0035] The processing module is used to input the edge information and the noise image into a preset image enhancement diffusion model to obtain an enhanced target domain image.

[0036] Furthermore, an electronic device is provided, comprising a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to implement any of the above methods.

[0037] Furthermore, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by at least one processor, it implements any of the methods described above.

[0038] The image enhancement method, device, electronic device and storage medium provided by the present application obtain a source domain image and a noise image to be processed, extract edge information of the source domain image, input the edge information and the noise image into a preset image enhancement diffusion model, and obtain an enhanced target domain image, thereby realizing the conversion of the source domain image to the target domain image. The image enhancement diffusion model is guided by the edge information to generate the target domain image, which can retain the details of the source domain image, greatly improve the fidelity and structural similarity of the enhanced image, and can generate a higher quality enhanced image compared with the existing image translation method. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of the process of the image enhancement method provided in Example 1 of the present application;

[0040] Figure 2 A schematic diagram of a training process based on a training sample data set and an initial diffusion model provided in Example 1 of the present application;

[0041] Figure 3 A schematic diagram of a real sample image in the source domain obtained according to the first embodiment of the present application;

[0042] Figure 4 A schematic diagram of a real sample image of a target domain obtained according to the first embodiment of the present application;

[0043] Figure 5 A schematic diagram of extracting edge information provided in Example 1 of the present application;

[0044] Figure 6 A schematic diagram of a non-diffusive generative adversarial network in one mode provided in Example 1 of the present application;

[0045] Figure 7 A schematic diagram of a non-diffusive generative adversarial network in another mode provided in Example 1 of the present application;

[0046] Figure 8-1 A schematic diagram of a portion of the structure of the initial diffusion model provided in Example 1 of the present application;

[0047] Figure 8-2A schematic diagram of another part of the structure of the initial diffusion model provided in Example 1 of the present application;

[0048] Fig. 9 A flowchart of the model training process of the initial diffusion model provided in Example 1 of the present application;

[0049] Fig.10 A flowchart of the test reasoning provided in Example 1 of the present application;

[0050] Fig.11 A schematic diagram of the structure of an image enhancement device provided in Example 2 of the present application;

[0051] Fig.12 This is a schematic diagram of the structure of an electronic device provided in Example 3 of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] This application embodiment provides an image enhancement method, see Figure 1 As shown, the following steps are included:

[0054] S11: Obtain a source domain image and a noise image to be processed.

[0055] S12: Extract edge information of the source domain image.

[0056] S13: Input the edge information and the noise image into a preset image enhancement diffusion model to obtain an enhanced target domain image.

[0057] The above steps are described in detail below.

[0058] It should be noted that the source domain image and the target domain image in the embodiment of the present application are images of the same dimension, for example, both are three-dimensional images, or both are two-dimensional images. Specifically, they can be three-dimensional CT images or two-dimensional X-ray images.

[0059] The difference between a source domain image and a target domain image is that a source domain image is a low-quality image to be enhanced, while a target domain image is an image of higher quality than the source domain image. By using the image enhancement method provided in the embodiment of the present application, a low-quality source domain image can be converted into a high-quality target domain image.

[0060] The image enhancement diffusion model in the embodiment of the present application is a model obtained in advance through model training. For details, see Figure 2 As shown, training can be performed as follows:

[0061] S21: Acquire a training sample data set and an initial diffusion model; the training sample data set includes a plurality of training sample image pairs, and the training sample image pairs include a source domain training sample image and a corresponding target domain training sample image;

[0062] S22: Performing training based on the training sample data set and the initial diffusion model, and obtaining the image enhancement diffusion model when a training completion condition is met.

[0063] The training completion conditions in the embodiments of the present application can be flexibly set by the developer. For example, the training can be stopped when the number of iterations reaches a preset iteration threshold T, or the training can be stopped when the loss value of the global loss function of the initial diffusion model is less than or equal to a preset loss threshold.

[0064] In a first optional implementation, the target domain training sample image is an image that is truly paired with the source domain training sample image. Taking the source domain image as a two-dimensional X-ray image as an example, during the training process, an X-ray machine is used to collect images of the target object to obtain a two-dimensional X-ray image, and the image is used as the source domain training sample image. The target object is obtained by CT imaging technology to obtain a three-dimensional CT image of a specific angle corresponding to the target object. The three-dimensional CT image needs to have an anatomical structure matching the two-dimensional X-ray image. The target two-dimensional image obtained by projecting the three-dimensional CT image is used as the target domain training sample image that is truly paired with the source domain training sample image.

[0065] In this embodiment, the target domain training sample image is an image that is truly paired with the source domain training sample image and can be used as a reference image in the training process. The initial diffusion model performs prediction based on the source domain training sample image to obtain an enhanced target domain prediction image corresponding to the source domain training sample image. The prediction image is compared with the target domain training sample image to adjust the model parameters of the initial diffusion model until the image enhancement diffusion model is obtained when the training completion condition is met.

[0066] In practical applications, it is usually difficult to directly obtain real paired source domain training sample images and target domain training sample images, and the requirements for related technologies are relatively high. Therefore, in a second optional implementation, the training sample image pair can be a generated pseudo image pair. Specifically, the corresponding pseudo target domain image can be generated based on the real sample image of the source domain, or the corresponding pseudo source domain image can be generated based on the real sample image of the target domain. This can solve the problem that it is difficult to directly obtain real matching image pairs in practical applications.

[0067] Therefore, the initial diffusion model in the embodiment of the present application may include a non-diffusion network. In this case, the step of obtaining a training sample data set in step S21 includes:

[0068] Acquire a real sample image in the source domain; input the real sample image in the source domain into the non-diffusion network to generate a pseudo target domain image corresponding to the real sample image in the source domain; use the real sample image in the source domain and the pseudo target domain image as the training sample image pair; at this time, the real sample image in the source domain and the corresponding pseudo target domain image are also the source domain training sample image and the corresponding target domain training sample image in the training sample image pair mentioned above;

[0069] and / or,

[0070] Acquire a real sample image in the target domain; input the real sample image in the target domain into the non-diffusion network to generate a pseudo source domain image corresponding to the real sample image in the target domain; use the real sample image in the target domain and the pseudo source domain image as the training sample image pair; at this time, the real sample image in the target domain and the corresponding pseudo source domain image are also the target domain training sample image and the corresponding source domain training sample image in the training sample image pair mentioned above.

[0071] Taking the source domain real sample image as a two-dimensional X-ray image and the target domain real sample image as a target two-dimensional image obtained by projection processing of a three-dimensional CT image as an example, the process of obtaining the above training sample image pair is explained.

[0072] Specifically, in this embodiment, a two-dimensional X-ray image of the lumbar spine can be obtained by an X-ray machine, for example, see Figure 3 As shown in , the frontal and lateral X-ray images of the lumbar spine can be obtained and used as the real sample images in the source domain. The three-dimensional CT image of the lumbar spine can be obtained by tomography, which contains voxel information, spatial resolution and image position information in physical space. The high-quality lumbar image obtained by multi-angle projection of the three-dimensional CT image is used as the real sample image in the target domain, as shown in Figure 4 As shown, both the source domain true sample images and the target domain true sample images are unpaired images.

[0073] The real sample image of the source domain is input into the non-diffusion network to obtain a pseudo target domain image corresponding to the real sample image of the source domain. The real sample image of the target domain is input into the non-diffusion network to obtain a pseudo source domain image corresponding to the real sample image of the target domain.

[0074] It should be noted that the non-diffusion network in the embodiment of the present application may be a non-diffusion generative adversarial network, and the step of obtaining a training sample data set includes:

[0075] Input the source domain real sample image into the first non-diffusion generator of the non-diffusion generative adversarial network to obtain the pseudo target domain image; input the target domain training sample image into the second non-diffusion generator of the non-diffusion generative adversarial network to obtain the pseudo source domain image;

[0076] The training based on the training sample data set and the initial diffusion model includes: inputting the pseudo target domain image and the target domain real sample image into a first non-diffusion discriminator of the non-diffusion generative adversarial network; and inputting the pseudo source domain image and the source domain real sample image into a second non-diffusion discriminator of the non-diffusion generative adversarial network.

[0077] In this embodiment, the training can be performed with reference to the first embodiment described above, that is, the target domain training sample image is used as a reference image for model training. Exemplarily, the initial diffusion model may include a reverse diffusion generative adversarial network, and the training based on the training sample data set and the initial diffusion model includes:

[0078] Noise is added to the real sample image of the target domain to obtain a target domain noise image; source edge information of the pseudo source domain image is extracted; the target domain noise image and the source edge information are input into the first diffusion generator of the back-diffusion generative adversarial network to obtain an enhanced reconstructed target domain image; the reconstructed target domain image is input into the first diffusion discriminator of the back-diffusion generative adversarial network.

[0079] Since the paired source domain training sample images and target domain training sample images are actually pseudo-paired, in order to improve the accuracy of model training and obtain a more accurate image enhancement diffusion model, a cycle-consistent architecture can be designed based on the above model training, that is, the reverse diffusion generative adversarial network can also include a second diffusion generator for converting the target domain data into source domain data during the reverse diffusion process. Specifically, the training based on the training sample data set and the initial diffusion model also includes:

[0080] Noise is added to the real sample image in the source domain to obtain a source domain noise image; target edge information of the training sample image in the target domain is extracted; the source domain noise image and the target edge information are input into a second diffusion generator of the inverse diffusion generative adversarial network to obtain an enhanced reconstructed source domain image; the reconstructed source domain image is input into a second diffusion discriminator of the inverse diffusion generative adversarial network.

[0081] In step S12 of the embodiment of the present application, the edge information of the source domain image can be extracted by frequency domain filtering technology. Similarly, the source edge information of the source domain training sample image can be extracted by frequency domain filtering technology, and the target edge information of the target domain training sample image can be extracted by frequency domain filtering technology.

[0082] See also Figure 5 As shown, the contour information of the image is mainly obtained through low-pass filtering and high-pass filtering. Specifically, given an image in the source domain First, we can use a bilateral filter ω(i,j,k,l) ​​to smooth the low-frequency part while retaining the high-frequency information:

[0083]

[0084] Among them, (k, l) represents the center position of the filter, f(i, j) represents the gray value of the image at (i, j), σ s , σ r Randomly generated during training within the specified range.

[0085] Suppose the filtered image is Then the Sobel operator can be used to calculate the gradient value g of the image in the horizontal direction. x And the gradient value g of the image in the vertical direction y :

[0086]

[0087] The gradient amplitude is The final edge information is obtained by threshold segmentation:

[0088]

[0089] The edge information is used to guide the reverse diffusion process during the diffusion process, achieving high-fidelity image enhancement.

[0090] For better understanding, the training process of the image enhancement diffusion model is explained below with reference to a specific example.

[0091] The initial diffusion model in this example includes a non-diffusion generative adversarial network and a reverse diffusion generative adversarial network.

[0092] Among them, the non-diffusion generation adversarial network is used to solve the problem that the source domain data and the target domain data are not paired. The non-diffusion generation adversarial network includes a first non-diffusion generator With the first non-diffusing generator The corresponding first non-diffusion discriminator Second non-diffusing generator and the second non-diffusing generator The corresponding second non-diffusion discriminator

[0093] See also Figure 6 As shown, the embodiment of the present application utilizes a first non-diffusion generator To predict the pseudo target domain image corresponding to the real sample image in the source domain, see Figure 7 As shown, using the second non-diffusion generator To predict the pseudo source domain image corresponding to the real sample image in the target domain, there is the following process:

[0094]

[0095] Among them, x 0 represents the real sample image in the target domain, y 0 represents the real sample image in the source domain, represents the target domain prediction image generated by the first non-diffusion generator, that is, the pseudo target domain image, represents the source domain prediction image generated by the second non-diffusion generator, that is, the pseudo source domain image, y 0 and Constitute a training sample image pair, and x 0 It also constitutes a training sample image pair.

[0096] The first non-diffusion discriminator x 0 and As input, the second non-diffusion discriminator With y 0 and as input.

[0097] The structure of the initial diffusion model can be found in Figure 8-1 and Figure 8-2 The specific training process can be found in Fig. 9 As shown, the initial diffusion model in the embodiment of the present application has a forward diffusion process and a reverse diffusion process, wherein the forward diffusion process corresponds to the noise addition process, and the reverse diffusion process is implemented by the reverse diffusion generative adversarial network, corresponding to the denoising process of the initial diffusion model. The reverse diffusion generative adversarial network includes a first diffusion generator and the corresponding first diffusion discriminator Second diffusion generator and the corresponding second diffusion discriminator

[0098] The noise adding process is described as a Markov chain, that is, the image at the current moment depends only on the image at the previous moment. 0 For example, by adding the original data x 0 Continuously add noise to get x 1 ,x 2 ....x T , a total of T noises are added, and the mean and variance of each added noise are determined by β t Control, βt is a linearly increasing value. Therefore x t It can be expressed as a conditional probability distribution:

[0099]

[0100] in, Indicates the mean The variance is β t I is a Gaussian distribution, let α t =1-β t , Then the image after adding noise at time t can be expressed as:

[0101]

[0102] Where ε represents the standard Gaussian distribution. t It is equivalent to the image at time t obtained by adding noise to the real sample image in the target domain, which is equivalent to the pure noise image obtained based on the real sample image in the target domain, that is, the target domain noise image, which is used as the input of the network model in the reverse diffusion process, and the image x at time t-1 is calculated through the output of the network model. t-1 .

[0103] That is, according to x t Get x t-1 Although it is not possible to directly obtain q(x t-1 |x t ), but if x is known 0 , then q(x t-1 |x t ,x 0 ) can be expressed as the mean The variance is Gaussian distribution of:

[0104]

[0105]

[0106] therefore,

[0107] Where ε represents the standard Gaussian distribution, the above formula shows that according to x t and the predicted x 0 Calculate the denoised image x at the previous moment t-1 .

[0108] Therefore, in the embodiment of the present application, the first diffusion generator Take the target domain noisy image x t Source edge information of pseudo source domain images As input, it is trained to generate

[0109] Right now

[0110] represents the enhanced reconstructed target domain image. According to the above formula:

[0111]

[0112] Reverse Second Diffusion Generator With the first diffusion generator A cycle-consistent architecture is constructed to generate source domain data from frequency domain information in the target domain during the back-diffusion process, i.e., the second diffusion generator Take the source domain noise image y t Target edge information with pseudo target domain image As input, it is trained to generate

[0113]

[0114] Represents the enhanced reconstructed source domain image, two diffusion discriminators Input real data and synthetic data respectively for discrimination.

[0115] by For example, the corresponding real data input is The synthetic data input is

[0116] In this example, in order to cope with the difficulty of obtaining paired images, unsupervised training is performed through a cycle-consistent architecture. Specifically, the target real sample image is converted into a pseudo source domain image through a non-diffusion generative adversarial network, and the source domain real sample image is converted into a pseudo target domain image. Then, the edge information of the pseudo source domain image and the edge information of the pseudo target domain image are obtained through frequency domain filtering, and the edge information is used to guide the denoising process of the back-diffusion generative adversarial network. At the same time, in the denoising process, the discriminator is used to discriminate the denoised image at each step and back-propagate the gradient to obtain the final image enhancement diffusion model.

[0117] The loss in the model training process of this example includes at least one of a diffusion process loss, a non-diffusion process loss, and a cycle consistency loss.

[0118] The diffusion process losses include:

[0119] The input of the first diffusion discriminator Abbreviated as The input of the second diffusion discriminator Abbreviated as The generator of the diffusion process has the following loss term:

[0120] First diffusion generator loss:

[0121] Second diffusion generator loss:

[0122] The discriminator of the diffusion process has the following loss terms:

[0123] First diffusion discriminator loss:

[0124] Second diffusion discriminator loss:

[0125] Non-diffusion process losses include:

[0126] In the non-diffusion process, there is an adversarial loss for the non-diffusion generator:

[0127] First non-diffusing generator loss:

[0128] Second non-diffusing generator loss:

[0129] in represents the target domain conditional probability distribution for a given image predicted by the network parameters.

[0130] The discriminator of the non-diffusion process has the following loss:

[0131] The first non-diffusion discriminator loss:

[0132]

[0133] Second non-diffusion discriminator loss:

[0134]

[0135] in Represents the true distribution of target domain images.

[0136] Cycle consistency losses include:

[0137] For a non-diffusing generator the following process is used:

[0138]

[0139] in is a non-diffusive generator according to The reconstructed image, in theory In addition, the above content mentioned that in the reverse diffusion process Therefore, we take the 1-norm of the difference between the two images as the consistency measure, and the cycle consistency loss L cycle Defined as:

[0140]

[0141] where λ φ1 ,λ θ1 are the weights of the cycle consistency loss term for the non-diffusion process and the diffusion process, respectively.

[0142] For example, the global generator loss in this example can be defined as:

[0143]

[0144] The global discriminator loss in this example can be defined as:

[0145]

[0146] Among them, λ φ2 is the global loss term weight of the non-diffusion process, λ θ2 is the global loss term weight of the diffusion process, and the global loss function is

[0147] In practical applications, in order to train the model, it is necessary to obtain a training sample data set. The data set in the experiment is made from 100 real training sample image pairs. In order to obtain paired source domain-target domain images, a 2D-3D registration method can be used to regress the posture of the three-dimensional CT image in the X-ray imaging perspective space, and project the three-dimensional CT image according to the regressed posture information, thereby obtaining 100 pairs of paired source domain-target domain images.

[0148] On the basis of the above, 100 real training sample image pairs were augmented and expanded to obtain 1,000 samples, which were divided into training set and test set in a ratio of 9: 1. Although paired images are not required in the unsupervised training process, a small number of paired source domain-target domain images are still required in the test process to evaluate the performance of the model.

[0149] All the overall network frameworks can be implemented in Python using the PyTorch framework. 1 =0.5,β 2 The model is trained with the Adam optimizer with 0.9. The model is trained on a workstation equipped with an Nvidia RTX 3090 GPU. The model performance is evaluated on the test set.

[0150] See also Fig.10 As shown in Figure 2, during the test, only the two-dimensional X-ray image can be used as the source domain image y, and the source edge information y can be extracted. edge , using the diffusion generator After a specified number of iterations, the target domain image can be predicted The corresponding true label is x 0 The performance of the model is evaluated by the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) indicators.

[0151] PSNR can be calculated using the following formula:

[0152]

[0153] Max=255;

[0154]

[0155] Among them, x 0 (i, j) represents the grayscale value of the image at (i, j), m represents the number of pixels of the image in the horizontal direction, and n represents the number of pixels of the image in the vertical direction.

[0156] PSNR mainly focuses on pixel-level errors and cannot accurately reflect the perceptual differences of the human visual system. Therefore, the structural similarity index SSIM is used, which takes into account the similarities in brightness, contrast and structure and can better simulate the visual perception of the human eye. It can be expressed as follows:

[0157]

[0158] C 1 , C 2 Used to avoid the situation where the denominator is 0, where:

[0159]

[0160] The noise image in the embodiment of the present application is a full noise image that obeys Gaussian distribution. In specific applications, edge information can be extracted from the source domain image to be processed, and then a noise image can be sampled from the standard Gaussian noise, and the noise image and edge information are sent to the image enhancement diffusion model together, and a high-quality target domain image is output after iterating T times through the first diffusion generator of the image enhancement diffusion model.

[0161] This example provides an unsupervised medical image enhancement method based on a frequency-domain guided adversarial diffusion model. Training sample image pairs can be constructed based on projection images of two-dimensional X-ray imaging and three-dimensional CT images to form a training sample data set. Unsupervised training is achieved in an unpaired data set through a non-diffusion network, a diffusion network, and a cycle-consistent architecture, thereby ensuring the accuracy of the training results. In addition, in order to ensure the structural integrity of the image during the image enhancement process, the edge information of the source domain image is used to guide the back diffusion process, ultimately achieving high-fidelity, geometrically complete X-ray image enhancement.

[0162] It should be understood that, although the various steps in the above-mentioned flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above-mentioned flow chart may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0163] Embodiment 2:

[0164] Based on the same inventive concept, see Fig.11 As shown, this embodiment provides an image enhancement device, including:

[0165] An acquisition module 1101 is used to acquire a source domain image and a noise image to be processed;

[0166] An extraction module 1102, configured to extract edge information of the source domain image;

[0167] The processing module 1103 is used to input the edge information and the noise image into a preset image enhancement diffusion model to obtain an enhanced target domain image.

[0168] It should be understood that for the sake of brevity, the contents described in some embodiments will not be repeated in this embodiment.

[0169] Embodiment three:

[0170] See also Fig.12 As shown, an embodiment of the present application provides an electronic device, including a processor 1201 and a memory 1202, wherein the memory 1202 stores a computer program, and the processor 1201 executes the computer program to implement the steps of the method introduced above, which will not be repeated here.

[0171] The processor 1201 may be an integrated circuit chip with signal processing capabilities. The processor 1201 may be a general-purpose processor, including a CPU (central processing unit), an NP (network processor), etc.; it may also be a DSP (digital signal processor), an ASIC (application-specific integrated circuit), an FPGA (off-the-shelf programmable gate array), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. It may implement or execute various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0172] The memory 1202 may include, but is not limited to, RAM (Random Access Memory), ROM (Read Only Memory), PROM (Programmable Read Only Memory), EPROM (Erasable Read Only Memory), EEPROM (Electrically Erasable Read Only Memory), and the like.

[0173] This embodiment also provides a computer-readable storage medium, such as a floppy disk, a CD, a hard disk, a flash memory, a USB flash disk, an SD card, an MMC card, etc., in which one or more programs for implementing the above steps are stored. These one or more programs can be executed by one or more processors to implement the steps of the method in the above embodiment 1, which will not be repeated here.

[0174] It should be noted that the diagram provided in the present embodiment only illustrates the basic concept of the present invention in a schematic manner, so the diagram only shows the components related to the present invention rather than drawing according to the number, shape and size of the components during actual implementation. The type, quantity and ratio of each component during actual implementation can be a random change, and the component layout type may also be more complicated. The structure, ratio, size, etc. illustrated in the drawings of the present specification are only used to match the content disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the limiting conditions that the present invention can implement, so they have no technical substantive significance. Any modification of the structure, change of the proportional relationship or adjustment of the size should still fall within the scope of the technical content disclosed by the present invention without affecting the effect that the present invention can produce and the purpose that can be achieved. At the same time, the terms such as "upper", "lower", "left", "right", "middle" and "one" quoted in this specification are only for the convenience of narration, and are not used to limit the scope of the present invention. The change or adjustment of its relative relationship should also be regarded as the scope of the present invention without substantially changing the technical content.

[0175] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0176] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. An image enhancement method, characterized in that: include: Obtaining a source domain image and a noise image to be processed; Extracting edge information of the source domain image; The edge information and the noise image are input into a preset image enhancement diffusion model to obtain an enhanced target domain image.

2. The image enhancement method according to claim 1, characterized in that: The image enhancement diffusion model is a model obtained by training in the following manner: Acquire a training sample data set and an initial diffusion model; the training sample data set includes a plurality of training sample image pairs, wherein the training sample image pairs include a source domain training sample image and a corresponding target domain training sample image; Training is performed based on the training sample data set and the initial diffusion model, and when a training completion condition is met, the image enhancement diffusion model is obtained.

3. The image enhancement method according to claim 2, characterized in that: The initial diffusion model includes a non-diffusion network; The step of obtaining a training sample data set includes: Acquire a real sample image in the source domain; input the real sample image in the source domain into the non-diffusion network to generate a pseudo target domain image corresponding to the real sample image in the source domain; use the real sample image in the source domain and the pseudo target domain image as the training sample image pair; and / or, Acquire a real sample image in the target domain; input the real sample image in the target domain into the non-diffusion network to generate a pseudo source domain image corresponding to the real sample image in the target domain; and use the real sample image in the target domain and the pseudo source domain image as the training sample image pair.

4. The image enhancement method according to claim 3, characterized in that: The source domain real sample image is a two-dimensional X-ray image, and the target domain real sample image is a target two-dimensional image obtained by projecting a three-dimensional CT image.

5. The image enhancement method according to claim 3, characterized in that: The non-diffusion network is a non-diffusion generative adversarial network, and the acquiring of a training sample data set includes: Input the source domain real sample image into the first non-diffusion generator of the non-diffusion generative adversarial network to obtain the pseudo target domain image; input the target domain real sample image into the second non-diffusion generator of the non-diffusion generative adversarial network to obtain the pseudo source domain image; The training based on the training sample data set and the initial diffusion model includes: The pseudo target domain image and the target domain real sample image are input into a first non-diffusion discriminator of the non-diffusion generative adversarial network; and the pseudo source domain image and the source domain real sample image are input into a second non-diffusion discriminator of the non-diffusion generative adversarial network.

6. The image enhancement method according to claim 3, characterized in that: The initial diffusion model includes a reverse diffusion generative adversarial network, and the training based on the training sample data set and the initial diffusion model includes: Adding noise to the target domain real sample image to obtain a target domain noise image; Extracting source edge information of the pseudo source domain image; Inputting the target domain noise image and the source edge information into a first diffusion generator of the back-diffusion generative adversarial network to obtain an enhanced reconstructed target domain image; The reconstructed target domain image is input into the first diffusion discriminator of the back-diffusion generative adversarial network.

7. The image enhancement method according to claim 6, characterized in that: The training based on the training sample data set and the initial diffusion model includes: Adding noise to the real sample image in the source domain to obtain a source domain noise image; Extracting target edge information of the target domain training sample image; Inputting the source domain noise image and the target edge information into a second diffusion generator of the back-diffusion generative adversarial network to obtain an enhanced reconstructed source domain image; The reconstructed source domain image is input into the second diffusion discriminator of the back-diffusion generative adversarial network.

8. An image enhancement device, characterized in that: include: An acquisition module, used for acquiring a source domain image and a noise image to be processed; An extraction module, used for extracting edge information of the source domain image; The processing module is used to input the edge information and the noise image into a preset image enhancement diffusion model to obtain an enhanced target domain image.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and the processor executes the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by at least one processor, the method according to any one of claims 1 to 7 is implemented.

Images