Medical image enhancement method, electronic equipment and readable storage medium

Abstract

The invention provides a medical image enhancement method, electronic equipment and a readable storage medium, and the method comprises the steps: carrying out the forward diffusion of an obtained to-be-enhanced medical image through employing a pre-trained low-quality domain model, so as to gradually add noise to the to-be-enhanced medical image until a preset forward diffusion end condition is satisfied, therefore, a corresponding hidden space image is obtained; and carrying out reverse diffusion on the hidden space image by adopting a pre-trained high-quality domain model so as to de-noise the hidden space image step by step until a preset reverse diffusion ending condition is met, thereby obtaining a corresponding high-quality domain image. According to the method, an unsupervised training method based on a diffusion model can be used, so that a corresponding medical image (such as an ultrasonic image) in a high-quality domain can be generated from a medical image (such as an ultrasonic image) in a low-quality domain without matching high-quality and low-quality image pair data; and the collection and manufacturing cost of a data set required for model training is greatly reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to a medical image enhancement method, an electronic device, and a readable storage medium. Background Art

[0002] Ultrasound imaging is a commonly used diagnosis and treatment method, and has been more and more widely used due to its advantages such as non-invasiveness, no radiation, and low cost. Among them, handheld ultrasound devices have a wide application prospect due to their portability and low cost compared with large desktop ultrasound machines. However, due to hardware limitations, the imaging quality of handheld ultrasound devices is inferior to that of large desktop ultrasound devices. Therefore, enhancing low-quality ultrasound images to achieve better image quality under the same hardware level is one of the bases for the wider application of handheld ultrasound devices.

[0003] In recent years, with the development of deep learning, especially convolutional neural network (CNN), there have been many studies on image enhancement methods such as denoising, super-resolution, and style change using deep learning. However, most of them use supervised learning methods, that is, a "gold standard" high-quality image is required to form a paired matching data pair with the low-quality image to be enhanced. And on this basis, the neural network is used to learn the mapping relationship from the low-quality image to the high-quality image. However, this kind of paired matching data is difficult to obtain and has a small data volume, especially in the medical scenario, which is likely to affect the effect and robustness of the network.

[0004] It should be noted that the information disclosed in the background art of the present invention is only intended to deepen the understanding of the general background technology of the present invention, and should not be regarded as an admission or any form of implication that this information constitutes the prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of the present invention is to provide a medical image enhancement method, an electronic device, and a readable storage medium, which can use an unsupervised training method based on a diffusion model, so that it is not necessary to match high- and low-quality image pairs of data, and a medical image (such as an ultrasound image) in a low-quality domain can be used to generate a corresponding medical image (such as an ultrasound image) in a high-quality domain, greatly reducing the collection and production costs of the data set required for model training.

[0006] To achieve the above object, the present invention provides a medical image enhancement method, including:

[0007] Perform forward diffusion on the acquired medical image to be enhanced using a pre-trained low-quality domain model, gradually adding noise to the medical image to be enhanced until a preset forward diffusion end condition is met, thereby obtaining a corresponding latent space image;

[0008] Perform reverse diffusion on the latent space image using a pre-trained high-quality domain model, gradually denoising the latent space image until a preset reverse diffusion end condition is met, thereby obtaining a corresponding high-quality domain image.

[0009] Optionally, the step of performing forward diffusion on the acquired medical image to be enhanced using a pre-trained low-quality domain model, gradually adding noise to the medical image to be enhanced until a preset forward diffusion end condition is met, thereby obtaining a corresponding latent space image, includes:

[0010] Step A: Use the medical image to be enhanced as the current pre-forward diffusion image, and set the current forward diffusion step number to 0;

[0011] Step B: Input the current pre-forward diffusion image and the current forward diffusion step number into the low-quality domain model to obtain a corresponding first estimated noise;

[0012] Step C: Add noise to the current pre-forward diffusion image according to the first estimated noise to obtain a current post-forward diffusion image;

[0013] Step D: Determine whether the current forward diffusion step number is equal to the preset forward diffusion step number. If not, execute Step E; if so, execute Step F;

[0014] Step E: Use the current post-forward diffusion image as the new current pre-forward diffusion image, increment the current forward diffusion step number by 1 as the new current forward diffusion step number, and return to execute Steps B to D;

[0015] Step F: Use the current post-forward diffusion image as the latent space image.

[0016] Optionally, the step of adding noise to the current pre-forward diffusion image according to the first estimated noise to obtain a current post-forward diffusion image includes:

[0017] Obtain a current first estimated noise-free image according to the first estimated noise, the current pre-forward diffusion image, and the current hyperparameter corresponding to the current forward diffusion step number;

[0018] Obtain a current first estimated Gaussian noise according to the current first estimated noise-free image, the current pre-forward diffusion image, and the current hyperparameter;

[0019] Obtain the current image after forward diffusion based on the current first estimated Gaussian noise and the first estimated noise-free image.

[0020] Optionally, the obtaining of the current first estimated noise-free image according to the first estimated noise, the image before the current forward diffusion, and the current hyperparameter corresponding to the current number of forward diffusion steps includes:

[0021] Obtain the current first estimated noise-free image according to the following formula:

[0022]

[0023] In the formula, represents the current first estimated noise-free image, represents the image before the current forward diffusion, represents the current hyperparameter, ε LQ represents the first estimated noise.

[0024] Optionally, the obtaining of the current first estimated Gaussian noise according to the current first estimated noise-free image, the image before the current forward diffusion, and the current hyperparameter includes:

[0025] Obtain the current first estimated Gaussian noise according to the following formula:

[0026]

[0027] In the formula, represents the current first estimated Gaussian noise, represents the image before the current forward diffusion, represents the current hyperparameter, represents the current first estimated noise-free image.

[0028] Optionally, the obtaining of the current image after forward diffusion according to the current first estimated Gaussian noise and the first estimated noise-free image includes:

[0029] Obtain the current image after forward diffusion according to the following formula:

[0030]

[0031] In the formula, represents the current image after forward diffusion, represents the current first estimated noise-free image, represents the current first estimated Gaussian noise, represents the hyperparameter corresponding to the next number of forward diffusion steps.

[0032] Optionally, the method of performing reverse diffusion on the latent space image using the pre-trained high-quality domain model to gradually denoise the latent space image until a preset reverse diffusion end condition is met, so as to obtain the corresponding high-quality domain image, includes:

[0033] Step a: Use the latent space image as the image before the current reverse diffusion, and set the current reverse diffusion step number to the preset reverse diffusion step number;

[0034] Step b: Input the image before the current reverse diffusion and the current reverse diffusion step number into the low-quality domain model together to obtain the corresponding second estimated noise;

[0035] Step c: Denoise the image before the current reverse diffusion according to the second estimated noise to obtain the image after the current reverse diffusion;

[0036] Step d: Determine whether the current reverse diffusion step number is equal to 1. If not, execute step e; if so, execute step f;

[0037] Step e: Use the image after the current reverse diffusion as the new image before the current reverse diffusion, subtract 1 from the current reverse diffusion step number as the new current reverse diffusion step number, and return to execute steps b to d;

[0038] Step f: Use the image after the current reverse diffusion as the high-quality domain image.

[0039] Optionally, the method of denoising the image before the current reverse diffusion according to the second estimated noise to obtain the image after the current reverse diffusion, includes:

[0040] Obtain the current second estimated noiseless image according to the second estimated noise, the image before the current reverse diffusion, and the current hyperparameter corresponding to the current reverse diffusion step number;

[0041] Obtain the current second estimated Gaussian noise according to the current added noise image, the image before the current reverse diffusion, and the current hyperparameter;

[0042] Obtain the image after the current reverse diffusion according to the current second estimated noiseless image and the current second estimated Gaussian noise.

[0043] Optionally, the method of obtaining the current second estimated noiseless image according to the second estimated noise, the image before the current reverse diffusion, and the current hyperparameter corresponding to the current reverse diffusion step number, includes:

[0044] Obtain the current second estimated noiseless image according to the following formula:

[0045]

[0046]

[0047] wherein, represents the current second estimated noise-free image, t represents the current reverse diffusion step number, represents the current hyperparameter, represents the image before the current reverse diffusion, ε HQ represents the second estimated noise, x represents the medical image to be enhanced, and λ is a constant, represents the gradient operator of the image before the current reverse diffusion, and z represents random noise having the same size as the medical image to be enhanced and satisfying a Gaussian distribution with a mean of 0 and a variance of 1.

[0048] Optionally, obtaining the current second estimated Gaussian noise according to the current noise-added image, the image before the current reverse diffusion, and the current hyperparameter includes:

[0049] Obtaining the current second estimated Gaussian noise according to the following formula:

[0050]

[0051] wherein, represents the current second estimated Gaussian noise, represents the image before the current reverse diffusion, represents the current hyperparameter, represents the current second estimated noise-free image.

[0052] Optionally, obtaining the image after the current reverse diffusion according to the current second estimated noise-free image and the current second estimated Gaussian noise includes:

[0053] Obtaining the image after the current reverse diffusion according to the following formula:

[0054]

[0055] wherein, represents the image after the current reverse diffusion, represents the current second estimated noise-free image, represents the current hyperparameter, represents the current second estimated Gaussian noise.

[0056] Optionally, the calculation formula of the hyperparameter is as follows:

[0057]

[0058] α i = 1 - βi

[0059] In the formula, t is the number of forward diffusion steps or reverse diffusion steps, and β i is to satisfy β 0 equal to a constant that linearly increases between the first preset value and β T-1 equal to the second preset value, and T is the total number of preset forward diffusion steps or the total number of preset reverse diffusion steps.

[0060] To achieve the above object, the present invention further provides an electronic device, including a processor and a memory. A computer program is stored on the memory. When the computer program is executed by the processor, the medical image enhancement method described above is implemented.

[0061] To achieve the above object, the present invention further provides a readable storage medium. A computer program is stored in the readable storage medium. When the computer program is executed by a processor, the medical image enhancement method described above is implemented.

[0062] Compared with the prior art, the medical image enhancement method, electronic device, and readable storage medium provided by the present invention have the following beneficial effects:

[0063] The present invention first performs forward diffusion on the acquired medical image to be enhanced by using a pre-trained low-quality domain model, so as to gradually add noise to the medical image to be enhanced until the preset forward diffusion end condition is satisfied, thereby obtaining a corresponding latent space image; then uses a pre-trained high-quality domain model to perform reverse diffusion on the latent space image, so as to gradually denoise the latent space image until the preset reverse diffusion end condition is satisfied, thereby obtaining a corresponding high-quality domain image. It can be seen that the medical image enhancement method provided by the present invention can use the method of unsupervised training based on a diffusion model to obtain a trained low-quality domain model and high-quality domain model, so that it is not necessary to match high- and low-quality image pairs of data, and a medical image in the low-quality domain (such as an ultrasound image) can be used to generate a corresponding medical image in the high-quality domain (such as an ultrasound image), greatly reducing the collection and production cost of the data set required for model training. In addition, due to the nature of gradual fitting of the medical image enhancement method provided by the present invention, it can ensure that the quality of the generated high-quality domain image is relatively high and can maintain the structural features of the original domain image (i.e., the medical image to be enhanced). In addition, compared with other image enhancement methods based on a generative network, the low-quality domain model and high-quality domain model in the medical image enhancement method provided by the present invention can be trained separately, so that there can be easy expandability and data privacy between different domains, and thus it can be extended to image conversion between multiple domains.

[0064] Since the electronic device and the readable storage medium provided by the present invention belong to the same inventive concept as the medical image enhancement method provided by the present invention, the electronic device and the readable storage medium provided by the present invention at least have all the beneficial effects of the medical image enhancement method provided by the present invention. For specific details, reference may be made to the relevant descriptions of the beneficial effects of the medical image enhancement method provided by the present invention in the above text. Therefore, the beneficial effects of the electronic device and the readable storage medium provided by the present invention will not be elaborated herein one by one. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 FIG. is a schematic flowchart of a medical image enhancement method provided by an embodiment of the present invention;

[0066] Figure 2 FIG. is a schematic diagram of image enhancement using the medical image enhancement method provided by the present invention;

[0067] Figure 3 FIG. is a schematic flowchart of a forward diffusion process provided by an embodiment of the present invention;

[0068] Figure 4 FIG. is a schematic flowchart of a reverse diffusion process provided by an embodiment of the present invention;

[0069] Figure 5 FIG. is a schematic block diagram of an electronic device provided by an embodiment of the present invention.

[0070] Among them, the reference numerals are as follows:

[0071] Processor - 101; Communication interface - 102; Memory - 103; Communication bus - 104. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0072] The following further elaborates on the medical image enhancement method, electronic device, and readable storage medium proposed by the present invention in conjunction with the accompanying drawings and specific embodiments. According to the following description, the advantages and features of the present invention will be clearer. It should be noted that the accompanying drawings are in a very simplified form and use non - precise scales, only for the convenience of clearly assisting in explaining the purpose provided by the present invention. In order to make the purpose, features, and advantages of the present invention more obvious and understandable, please refer to the accompanying drawings. It should be noted that the structures, ratios, sizes, etc. shown in the drawings of this specification are only used to cooperate with the content disclosed in the specification for those familiar with this technology to understand and read, and are not used to limit the limiting conditions for the implementation of the present invention. Any modification of the structure, change in the proportional relationship, or adjustment of the size, in the case of being the same or similar to the effects that the present invention can produce and the purposes that can be achieved, should still fall within the scope covered by the technical content disclosed by the present invention.

[0073] It should be noted that in this text, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising a..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element. The singular forms "a", "an" and "the" include plural objects, the term "or" is generally used in the sense of "and / or", the term "several" is generally used in the sense of "at least one", the term "at least two" is generally used in the sense of "two or more", and in addition, the terms "first", "second", "third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features.

[0074] In addition, in the description of this specification, the description with reference to terms such as "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0075] The core idea of the present invention is to provide a medical image enhancement method, an electronic device and a readable storage medium, which can use an unsupervised training method based on a diffusion model, so that it is not necessary to match high-quality and low-quality image pairs of data, and a medical image in a low-quality domain (such as an ultrasound image) can be used to generate a corresponding medical image in a high-quality domain (such as an ultrasound image), greatly reducing the collection and production cost of the data set required for model training.

[0076] It should be noted that the medical image enhancement method provided by the present invention can be applied to the electronic device provided by the present invention. Among them, the electronic device can be a personal computer, a mobile terminal, etc. The mobile terminal can be a hardware device such as a mobile phone, a tablet computer, etc. with various operating systems. It should also be noted that, as can be understood by those skilled in the art, the medical image enhancement method provided by the present invention can not only enhance ultrasonic images (i.e., convert low-quality ultrasonic images into high-quality ultrasonic images), but also enhance medical images in other fields except ultrasonic images.

[0077] To achieve the above idea, the present invention provides a medical image enhancement method. Please refer to Figure 1 which is a schematic flowchart of the medical image enhancement method provided by an embodiment of the present invention. As Figure 1 shown, the medical image enhancement method provided by the present invention includes the following steps:

[0078] Step S100: Use the pre-trained low-quality domain model to perform forward diffusion on the acquired medical image to be enhanced, so as to gradually add noise to the medical image to be enhanced until the preset forward diffusion end condition is met, thereby obtaining the corresponding latent space image.

[0079] Step S200: Use the pre-trained high-quality domain model to perform reverse diffusion on the latent space image, so as to gradually denoise the latent space image until the preset reverse diffusion end condition is met, thereby obtaining the corresponding high-quality domain image.

[0080] It can be seen that the medical image enhancement method provided by the present invention can use the unsupervised training method based on the diffusion model to obtain the trained low-quality domain model and high-quality domain model, so that it is not necessary to match high- and low-quality image pairs of data, and the corresponding high-quality domain medical image (such as an ultrasonic image) can be generated from the low-quality domain medical image (such as an ultrasonic image), greatly reducing the collection and production cost of the data set required for model training. In addition, due to its gradually fitting nature, the medical image enhancement method provided by the present invention can ensure that the generated high-quality domain image has high quality and can maintain the structural characteristics of the original domain image (i.e., the medical image to be enhanced). In addition, compared with other image enhancement methods based on generative networks, the low-quality domain model and high-quality domain model in the medical image enhancement method provided by the present invention can be trained separately, so that there can be easy expandability and data privacy between different domains, and thus it can be extended to image conversion between multiple domains.

[0081] Specifically, the low-quality domain model is a diffusion model trained using a low-quality domain image dataset, and the high-quality domain model is a diffusion model trained using a high-quality domain image dataset. Specifically, taking the training of the low-quality domain model as an example, the input of the low-quality domain model is the forward diffusion step t and the noisy image x at the forward diffusion step t t (obtained by adding noise to the low-quality domain image), and the output is the estimated noise σ t , denoted as σ t =Net θ (x t , t). Among them, the forward diffusion step t is an integer between 1 and T (the preset total number of forward diffusion steps). It should be noted that, as can be understood by those skilled in the art, the specific value of the preset total number of forward diffusion steps T can be set according to the actual situation, and the present invention does not limit this. For example, the specific value of the preset total number of forward diffusion steps T can be 1000. It should also be noted that, as can be understood by those skilled in the art, when t = T, the noisy image x t is a pure noise image.

[0082] Further, the analytical formula of the noisy image x t corresponding to the forward diffusion step t is: Among them, x is the original image, z is a random noise with the same size as the original image and satisfying a Gaussian distribution with a mean of 0 and a variance of 1, represents the hyperparameter corresponding to the forward diffusion step t.

[0083] Specifically, where α i =1-β i , β i is a constant that linearly increases between β 0 equal to the first preset value (for example, 0.0001) and β T-1 equal to the second preset value (for example, 0.002). It should be noted that, as can be understood by those skilled in the art, the specific values of the first preset value and the second preset value can be set according to the actual situation, and the present invention does not limit this.

[0084] Further, the loss function during the training of the low-quality domain model can be, but is not limited to, the mean square error. The specific calculation formula for the mean square error can refer to the related technologies well-known to those skilled in the art, and will not be elaborated here.

[0085] It should be noted that, as can be understood by those skilled in the art, the training process of the high-quality domain model is similar to that of the low-quality domain model. For the specific training process of the high-quality domain model, reference can be made to the relevant content of the specific training process of the low-quality domain model in the above text for adaptive understanding, and details will not be elaborated here. It should be noted that for the high-quality domain model, the input of the high-quality domain model is the reverse diffusion step number t and the noisy image x at the reverse diffusion step number t t (obtained by adding noise to the high-quality domain image), and the specific values of the preset total number of reverse diffusion steps and the preset total number of forward diffusion steps are the same, and can both be T. It should also be noted that, as can be understood by those skilled in the art, more content on how to train the low-quality domain model and the high-quality domain model can be referred to the relevant content of the training process of the diffusion model known to those skilled in the art, and details will not be elaborated here. In addition, it should be noted that, as can be understood by those skilled in the art, the present invention does not limit the specific network structures of the low-quality domain model and the high-quality domain model, and the network structures of the low-quality domain model and the high-quality domain model can be, but are not limited to, the U-net structure. Additionally, it should be noted that, as can be understood by those skilled in the art, the network structures of the low-quality domain model and the high-quality domain model can be the same or different, and the present invention does not limit this

[0086] Please continue to refer to Figure 2 , which is a schematic diagram of image enhancement using the medical image enhancement method provided by the present invention. As Figure 2 shown, by first performing forward diffusion on the low-quality domain image using a pre-trained low-quality domain model, the corresponding latent space image can be obtained, and then performing reverse diffusion on the latent space image using a pre-trained high-quality domain model, a high-quality domain image can be generated, thereby completing the enhancement of the image. Further, as Figure 2 shown, the medical image enhancement method provided by the present invention can not only ensure that the generated high-quality domain image has high quality, but also maintain the structural features of the original domain image

[0087] Please continue to refer to Figure 3 , which is a schematic diagram of the forward diffusion process provided by an embodiment of the present invention. As Figure 3 shown, in some exemplary embodiments, performing forward diffusion on the to-be-enhanced medical image obtained using a pre-trained low-quality domain model to gradually add noise to the to-be-enhanced medical image until a preset forward diffusion end condition is met, thereby obtaining the corresponding latent space image, includes:

[0088] Step A: Using the to-be-enhanced medical image as the current image before forward diffusion, and setting the current forward diffusion step number to 1

[0089] Step B: Input the current pre-forward-diffusion image and the current forward diffusion step number into the low-quality domain model to obtain the corresponding first estimated noise;

[0090] Step C: Add noise to the current pre-forward-diffusion image according to the first estimated noise to obtain the current post-forward-diffusion image;

[0091] Step D: Determine whether the current forward diffusion step number is equal to the preset forward diffusion step number. If not, execute Step E; if so, execute Step F;

[0092] Step E: Use the current post-forward-diffusion image as the new current pre-forward-diffusion image, increment the current forward diffusion step number by 1 as the new current forward diffusion step number, and return to execute Steps B to D;

[0093] Step F: Use the current post-forward-diffusion image as the latent space image.

[0094] Thus, the medical image enhancement method provided by the present invention estimates the first estimated noise according to the pre-forward-diffusion image and the corresponding forward diffusion step number by using a pre-trained low-quality domain model, so as to obtain the post-forward-diffusion image according to the first estimated noise, thereby ensuring that the noise added in each step of forward diffusion is directionally reversible noise, and further maintaining the structural features of the original domain image (i.e., the medical image to be enhanced), which helps to ensure that the finally generated high-quality domain image has both high quality and can maintain the structural features of the original domain image (i.e., the medical image to be enhanced).

[0095] It should be noted that the specific value of the preset forward diffusion step number can be set according to the actual situation, and the present invention does not limit this. However, as can be understood by those skilled in the art, the preset forward diffusion step number should be less than the preset total forward diffusion step number when training the low-quality domain model.

[0096] Preferably, the preset number of forward diffusion steps is equal to half of the preset total number of forward diffusion steps minus 1 (for example, when the preset total number of forward diffusion steps is equal to 1000, the preset number of forward diffusion steps is equal to 499). Thus, such a setting can not only ensure that the latent space image obtained through multiple steps of forward diffusion (when the preset number of forward diffusion steps is equal to 499, the latent space image is obtained by gradually (a total of 500 steps) performing forward diffusion on the medical image to be enhanced with the forward diffusion steps from 0 to 499) can maintain the structural features of the original domain image (i.e., the medical image to be enhanced), that is, the structural features of the latent space image are not much different from those of the medical image to be enhanced, and further ensure that the finally generated high-quality domain image has both high quality and can maintain the structural features of the original domain image (i.e., the medical image to be enhanced); moreover, such a setting can also ensure the generation speed of the latent space image, and thus contribute to improving the speed of generating the high-quality domain image by the medical image enhancement method provided by the present invention.

[0097] In some exemplary embodiments, adding noise to the current image before forward diffusion according to the first estimated noise to obtain the current image after forward diffusion includes:

[0098] Obtaining a current first estimated noise-free image according to the first estimated noise, the current image before forward diffusion, and a current hyperparameter corresponding to the current number of forward diffusion steps;

[0099] Obtaining a current first estimated Gaussian noise according to the current first estimated noise-free image, the current image before forward diffusion, and the current hyperparameter;

[0100] Obtaining the current image after forward diffusion according to the current first estimated Gaussian noise and the first estimated noise-free image.

[0101] Thus, by first obtaining a current first estimated noise-free image according to the first estimated noise, the current image before forward diffusion, and a current hyperparameter corresponding to the current number of forward diffusion steps; then obtaining a current first estimated Gaussian noise according to the current first estimated noise-free image, the current image before forward diffusion, and the current hyperparameter; and finally obtaining the current image after forward diffusion according to the current first estimated Gaussian noise and the first estimated noise-free image, it can effectively ensure that the latent space image obtained through multiple steps of forward diffusion can maintain the structural features of the original domain image (i.e., the medical image to be enhanced), that is, the structural features of the latent space image are not much different from those of the medical image to be enhanced, and further ensure that the finally generated high-quality domain image has both high quality and can maintain the structural features of the original domain image (i.e., the medical image to be enhanced).

[0102] In some exemplary embodiments, obtaining the current first estimated noise-free image according to the first estimated noise, the current pre-forward-diffusion image, and the current hyperparameter corresponding to the current number of forward diffusion steps includes:

[0103] Obtaining the current first estimated noise-free image according to the following formula:

[0104]

[0105] In the formula, represents the current first estimated noise-free image, t represents the current number of forward diffusion steps, represents the current pre-forward-diffusion image, represents the current hyperparameter, and ε LQ represents the first estimated noise.

[0106] Specifically, for the specific content on how to obtain the current hyperparameter corresponding to the current number of forward diffusion steps t, reference can be made to the relevant descriptions in the foregoing, and details will not be elaborated herein. It should be noted that, as can be understood by those skilled in the art, wherein, Net LQ represents a low-quality domain model. It should also be noted that, as can be understood by those skilled in the art, (x represents the medical image to be enhanced).

[0107] In some exemplary embodiments, obtaining the current first estimated Gaussian noise according to the current first estimated noise-free image, the current pre-forward-diffusion image, and the current hyperparameter includes:

[0108] Obtaining the current first estimated Gaussian noise according to the following formula:

[0109]

[0110] In the formula, represents the current first estimated Gaussian noise, represents the current pre-forward-diffusion image, represents the current hyperparameter, represents the current first estimated noise-free image.

[0111] In some exemplary embodiments, obtaining the current post-forward-diffusion image according to the current first estimated Gaussian noise and the first estimated noise-free image includes:

[0112] Obtaining the current post-forward-diffusion image according to the following formula:

[0113]

[0114] In the formula, represents the image after the current forward diffusion, represents the current first estimated noise-free image, represents the current first estimated Gaussian noise, represents the hyperparameter corresponding to the next forward diffusion step number.

[0115] Please continue to refer to Figure 4 , which is a schematic diagram of the reverse diffusion process provided by an embodiment of the present invention. In some exemplary embodiments, the reverse diffusion is performed on the latent space image by using a pre-trained high-quality domain model to gradually denoise the latent space image until a preset reverse diffusion end condition is satisfied, so as to obtain a corresponding high-quality domain image, including:

[0116] Step a: Take the latent space image as the image before the current reverse diffusion, and set the current reverse diffusion step number to the preset reverse diffusion step number;

[0117] Step b: Input the image before the current reverse diffusion and the current reverse diffusion step number into the low-quality domain model together to obtain a corresponding second estimated noise;

[0118] Step c: Denoise the image before the current reverse diffusion according to the second estimated noise to obtain the image after the current reverse diffusion;

[0119] Step d: Determine whether the current reverse diffusion step number is equal to 1. If not, execute step e. If so, execute step f;

[0120] Step e: Take the image after the current reverse diffusion as the new image before the current reverse diffusion, subtract 1 from the current reverse diffusion step number as the new current reverse diffusion step number, and return to execute steps b to d;

[0121] Step f: Take the image after the current reverse diffusion as the high-quality domain image.

[0122] Thus, the medical image enhancement method provided by the present invention can effectively ensure that the finally generated high-quality domain image has both high quality and can maintain the structural features of the original domain image (i.e., the medical image to be enhanced) by using a pre-trained high-quality domain model to estimate the second estimated noise according to the image before the reverse diffusion and the corresponding reverse diffusion step number, so as to obtain the image after the reverse diffusion according to the second estimated noise.

[0123] It should be noted that the specific value of the preset reverse diffusion step number can be set according to actual situations, and the present invention does not limit this. However, as can be understood by those skilled in the art, the preset reverse diffusion step number should be less than the total preset reverse diffusion step number when training the high-quality domain model.

[0124] Preferably, the preset number of reverse diffusion steps is equal to the preset number of forward diffusion steps plus 1 (for example, when the preset number of forward diffusion steps is equal to 499, the preset number of reverse diffusion steps is equal to 500). Thus, such a setting can ensure that the total number of reverse diffusion steps is equal to the total number of forward diffusion steps (for example, both are equal to 500 steps). Furthermore, it can not only ensure that the finally generated high-quality domain image (when the preset number of reverse diffusion steps is equal to 500, the high-quality domain image is obtained by reverse-diffusing the latent space image step by step (a total of 500 steps) from 500 to 1) has high quality, but also ensure that the finally generated high-quality domain image can maintain the structural features of the original domain image (i.e., the medical image to be enhanced).

[0125] In some exemplary embodiments, the denoising the current image before reverse diffusion according to the second estimated noise to obtain the current image after reverse diffusion includes:

[0126] Obtaining a current second estimated noise-free image according to the second estimated noise, the current image before reverse diffusion, and a current hyperparameter corresponding to the current number of reverse diffusion steps;

[0127] Obtaining a current second estimated Gaussian noise according to the current added-noise image, the current image before reverse diffusion, and the current hyperparameter;

[0128] Obtaining the current image after reverse diffusion according to the current second estimated noise-free image and the current second estimated Gaussian noise.

[0129] Thus, by first obtaining a current second estimated noise-free image according to the second estimated noise, the current image before reverse diffusion, and a current hyperparameter corresponding to the current number of reverse diffusion steps; then obtaining a current second estimated Gaussian noise according to the current added-noise image, the current image before reverse diffusion, and the current hyperparameter; and finally obtaining the current image after reverse diffusion according to the current second estimated noise-free image and the current second estimated Gaussian noise, it can effectively ensure that the high-quality domain image generated by multi-step reverse diffusion has both high quality and can maintain the structural features of the original domain image (i.e., the medical image to be enhanced).

[0130] In some exemplary embodiments, the obtaining a current second estimated noise-free image according to the second estimated noise, the current image before reverse diffusion, and a current hyperparameter corresponding to the current number of reverse diffusion steps includes:

[0131] Obtaining the current second estimated noise-free image according to the following formula:

[0132]

[0133]

[0134] wherein represents the current second estimated noiseless image, t represents the current reverse diffusion step number, represents the current hyperparameter, represents the image before the current reverse diffusion, ε HQ represents the second estimated noise, x represents the medical image to be enhanced, and λ is a constant, represents the gradient operator of the image before the current reverse diffusion, and z represents random noise having the same size as the medical image to be enhanced and satisfying a Gaussian distribution with a mean of 0 and a variance of 1.

[0135] Specifically, regarding how to obtain the current hyperparameter corresponding to the current forward diffusion step number t the specific content can refer to the relevant descriptions in the above text and will not be elaborated here. It should be noted that, as can be understood by those skilled in the art, wherein, Net HQ represents the high-quality domain model. It should also be noted that, as can be understood by those skilled in the art, when the preset reverse diffusion step number is τ, represents the latent space image. In addition, it should be noted that the present invention does not limit the specific value of λ. For example, λ can be set to 0.01 according to empirical values.

[0136] In some exemplary embodiments, obtaining the current second estimated Gaussian noise according to the current noise-added image, the image before the current reverse diffusion, and the current hyperparameter includes:

[0137] Obtaining the current second estimated Gaussian noise according to the following formula:

[0138]

[0139] wherein represents the current second estimated Gaussian noise, represents the image before the current reverse diffusion, represents the current hyperparameter, represents the current second estimated noiseless image.

[0140] In some exemplary embodiments, obtaining the image after the current reverse diffusion according to the current second estimated noiseless image and the current second estimated Gaussian noise includes:

[0141] Obtaining the image after the current reverse diffusion according to the following formula:

[0142]

[0143] In the formula, represents the image after the current backward diffusion, represents the current second estimated noise-free image, represents the current hyperparameter, represents the current second estimated Gaussian noise.

[0144] Based on the same inventive concept, the present invention further provides an electronic device. Please refer to Figure 5 , which is a schematic block diagram of the electronic device provided by an embodiment of the present invention. As Figure 5 shown, the electronic device includes a processor 101 and a memory 103. A computer program is stored on the memory 103. When the computer program is executed by the processor 101, the medical image enhancement method described above is implemented. Since the electronic device provided by the present invention and the medical image enhancement method provided by the present invention belong to the same inventive concept, the electronic device provided by the present invention has at least all the beneficial effects of the medical image enhancement method provided by the present invention. For specific details, reference can be made to the relevant descriptions of the beneficial effects of the medical image enhancement method provided by the present invention above, and details will not be repeated here.

[0145] As Figure 5 shown, the electronic device further includes a communication interface 102 and a communication bus 104. The processor 101, the communication interface 102, and the memory 103 complete mutual communication through the communication bus 104. The communication bus 104 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus 104 can be divided into an address bus, a data bus, a control bus, etc. For the sake of simplicity, only a thick line is used to represent it in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface 102 is used for communication between the above-mentioned electronic device and other devices.

[0146] The processor 101 referred to in the present invention may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The processor 101 is the control center of the electronic device and connects all parts of the electronic device through various interfaces and lines.

[0147] The memory 103 can be used to store the computer program. The processor 101 realizes various functions of the electronic device by running or executing the computer program stored in the memory 103 and calling the data stored in the memory 103. The memory 103 may include non-volatile and / or volatile memory. The non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. The volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0148] The present invention also provides a readable storage medium with a computer program stored therein. When the computer program is executed by a processor, it can implement the medical image enhancement method described above. Since the readable storage medium provided by the present invention and the medical image enhancement method provided by the present invention belong to the same inventive concept, the readable storage medium provided by the present invention has at least all the beneficial effects of the medical image enhancement method provided by the present invention. For specific details, reference may be made to the relevant description of the beneficial effects of the medical image enhancement method provided by the present invention above, and no further elaboration will be provided here.

[0149] The readable storage medium provided by the present invention may adopt any combination of one or more computer-readable media. The readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the computer-readable storage medium include: an electrical connection having one or more wires, a portable computer hard disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this article, the computer-readable storage medium may be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, apparatus, or device.

[0150] The computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries the computer-readable program code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium, and this computer-readable medium can send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted by any appropriate medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0151] In summary, compared with the prior art, the medical image enhancement method, electronic device, and readable storage medium provided by the present invention have the following beneficial effects:

[0152] In the present invention, first, a pre-trained low-quality domain model is used to perform forward diffusion on the acquired medical image to be enhanced, so as to gradually add noise to the medical image to be enhanced until a preset forward diffusion end condition is met, thereby obtaining a corresponding latent space image; then, a pre-trained high-quality domain model is used to perform reverse diffusion on the latent space image, so as to gradually denoise the latent space image until a preset reverse diffusion end condition is met, thereby obtaining a corresponding high-quality domain image. It can be seen that the present invention can obtain a trained low-quality domain model and a high-quality domain model by using an unsupervised training method based on a diffusion model, so that it is possible to generate a corresponding high-quality domain medical image (such as an ultrasound image) from a low-quality domain medical image (such as an ultrasound image) without a matching pair of high- and low-quality images, greatly reducing the collection and production costs of the data set required for model training. In addition, due to its gradually fitting nature, the present invention can ensure that the generated high-quality domain image has high quality and can maintain the structural features of the original domain image (i.e., the medical image to be enhanced). Furthermore, compared with other image enhancement methods based on a generative network, the low-quality domain model and the high-quality domain model in the present invention can be trained separately, so that there can be easy expandability and data privacy between different domains, and thus it can be extended to image conversion between multiple domains.

[0153] It should be noted that, as can be understood by those skilled in the art, computer program code for performing the operations of the present invention can be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages - such as Java, Smalltalk, C++, and also include conventional procedural programming languages - such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network - including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, by using an Internet service provider to connect through the Internet).

[0154] It should be noted that the devices and methods disclosed in the embodiments of this article can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of devices, methods, and computer program products according to multiple embodiments of this article. In this regard, each block in the flowchart or block diagram may represent a module, program, or part of code that contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, as well as the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions. Additionally, in each embodiment of this article, the functional modules can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part.

[0155] It should also be noted that the above description is only a description of the preferred embodiments of the present invention and does not limit the scope of the present invention in any way. Any changes and modifications made by those of ordinary skill in the art of the present invention based on the above disclosure belong to the protection scope of the present invention. Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations fall within the scope of the present invention and its equivalent technologies, the present invention is also intended to include these modifications and variations.

Claims

1. A medical image enhancement method, characterized in that, comprising: Performing forward diffusion on the acquired medical image to be enhanced using a pre-trained low-quality domain model to gradually add noise to the medical image to be enhanced until a preset forward diffusion end condition is met, thereby obtaining a corresponding latent space image; Performing reverse diffusion on the latent space image using a pre-trained high-quality domain model to gradually denoise the latent space image until a preset reverse diffusion end condition is met, thereby obtaining a corresponding high-quality domain image.

2. The medical image enhancement method according to claim 1, characterized in that, The step of performing forward diffusion on the acquired medical image to be enhanced using a pre-trained low-quality domain model to gradually add noise to the medical image to be enhanced until a preset forward diffusion end condition is met, thereby obtaining a corresponding latent space image, includes: Step A: Taking the medical image to be enhanced as the current pre-forward diffusion image and setting the current forward diffusion step number to 0; Step B: Inputting the current pre-forward diffusion image and the current forward diffusion step number into the low-quality domain model together to obtain a corresponding first estimated noise; Step C: Adding noise to the current pre-forward diffusion image according to the first estimated noise to obtain a current post-forward diffusion image; Step D: Judging whether the current forward diffusion step number is equal to the preset forward diffusion step number. If not, execute step E; if so, execute step F; Step E: Taking the current post-forward diffusion image as the new current pre-forward diffusion image, adding 1 to the current forward diffusion step number as the new current forward diffusion step number, and returning to execute steps B to D; Step F: Taking the current post-forward diffusion image as the latent space image.

3. The medical image enhancement method according to claim 2, characterized in that, The step of adding noise to the current pre-forward diffusion image according to the first estimated noise to obtain a current post-forward diffusion image, includes: Obtaining a current first estimated noise-free image according to the first estimated noise, the current pre-forward diffusion image, and a current hyperparameter corresponding to the current forward diffusion step number; Obtaining a current first estimated Gaussian noise according to the current first estimated noise-free image, the current pre-forward diffusion image, and the current hyperparameter; Obtaining a current post-forward diffusion image according to the current first estimated Gaussian noise and the first estimated noise-free image.

4. The medical image enhancement method according to claim 3, characterized in that, The step of obtaining a current first estimated noise-free image according to the first estimated noise, the current pre-forward diffusion image, and a current hyperparameter corresponding to the current forward diffusion step number, includes: Obtaining the current first estimated noise-free image according to the following formula: wherein, represents the current first estimated noise-free image, represents the current image before forward diffusion, represents the current hyperparameter, ε LQ represents the first estimated noise; and / or The step of obtaining a current first estimated Gaussian noise according to the current first estimated noise-free image, the current pre-forward diffusion image, and the current hyperparameter, includes: Obtaining the current first estimated Gaussian noise according to the following formula: Wherein, represents the current first estimated Gaussian noise, represents the current image before forward diffusion, represents the current hyperparameter, represents the current first estimated noise-free image; and / or Obtaining the current forward-diffused image according to the current first estimated Gaussian noise and the first estimated noise-free image includes: Obtaining the current forward-diffused image according to the following formula: Wherein, represents the image after the current forward diffusion, represents the current first estimated noise-free image, represents the current first estimated Gaussian noise, represents the hyperparameter corresponding to the next forward diffusion step.

5. The medical image enhancement method according to claim 1, wherein, Performing reverse diffusion on the latent space image by using a pre-trained high-quality domain model to gradually denoise the latent space image until a preset reverse diffusion end condition is satisfied, thereby obtaining a corresponding high-quality domain image, includes: Step a: Taking the latent space image as the current pre-reverse-diffusion image and setting the current reverse diffusion step number to the preset reverse diffusion step number; Step b: Inputting the current pre-reverse-diffusion image and the current reverse diffusion step number into the low-quality domain model together to obtain a corresponding second estimated noise; Step c: Denoising the current pre-reverse-diffusion image according to the second estimated noise to obtain the current post-reverse-diffusion image; Step d: Judging whether the current reverse diffusion step number is equal to 1. If not, execute step e; if so, execute step f; Step e: Taking the current post-reverse-diffusion image as the new current pre-reverse-diffusion image, subtracting 1 from the current reverse diffusion step number as the new current reverse diffusion step number, and returning to execute steps b to d; Step f: Taking the current post-reverse-diffusion image as the high-quality domain image.

6. The medical image enhancement method according to claim 5, wherein, Denoising the current pre-reverse-diffusion image according to the second estimated noise to obtain the current post-reverse-diffusion image, includes: Obtaining the current second estimated noise-free image according to the second estimated noise, the current pre-reverse-diffusion image, and the current hyperparameter corresponding to the current reverse diffusion step number; Obtaining the current second estimated Gaussian noise according to the current added-noise image, the current pre-reverse-diffusion image, and the current hyperparameter; Obtaining the current post-reverse-diffusion image according to the current second estimated noise-free image and the current second estimated Gaussian noise.

7. The medical image enhancement method according to claim 6, wherein, Obtaining the current second estimated noise-free image according to the second estimated noise, the current pre-reverse-diffusion image, and the current hyperparameter corresponding to the current reverse diffusion step number, includes: Obtaining the current second estimated noise-free image according to the following formula: In the formula, represents the current second estimated noise-free image, t represents the current reverse diffusion step number, represents the current hyperparameter, represents the image before the current reverse diffusion, and ε HQ represents the second estimated noise, x represents the medical image to be enhanced, λ is a constant, represents the gradient operator of the image before the current reverse diffusion, z represents a random noise having the same size as the medical image to be enhanced and satisfying a Gaussian distribution with a mean of 0 and a variance of 1; and / or Obtaining the current second estimated Gaussian noise according to the current added-noise image, the current pre-reverse-diffusion image, and the current hyperparameter, includes: Obtaining the current second estimated Gaussian noise according to the following formula: In the formula, represents the current second estimated Gaussian noise, represents the current image before reverse diffusion, represents the current hyperparameter, represents the current second estimated noise-free image; and / or Obtaining the current post-reverse-diffusion image according to the current second estimated noise-free image and the current second estimated Gaussian noise, includes: Obtaining the current post-reverse-diffusion image according to the following formula: In the formula, represents the image after the current reverse diffusion, represents the current second estimated noise-free image, represents the current hyperparameter, represents the current second estimated Gaussian noise.

8. The medical image enhancement method according to any one of claims 3 to 7, wherein, The calculation formula of the hyperparameter is as follows: α i ＝1-β i where t is the number of forward diffusion steps or reverse diffusion steps, and β i is a constant that linearly increases between β 0 equal to the first preset value and β T-1 equal to the second preset value, and T is the total number of preset forward diffusion steps or the total number of preset reverse diffusion steps.

9. An electronic device, wherein, It includes a processor and a memory, and a computer program is stored on the memory. When the computer program is executed by the processor, it implements the medical image enhancement method according to any one of claims 1 to 8.

10. A readable storage medium, characterized in that, a computer program is stored in the readable storage medium, and when the computer program is executed by a processor, it implements the medical image enhancement method according to any one of claims 1 to 8.

Images