Three-dimensional ultrasonic image generation method and device based on DALLE and Pix2Pix GAN

Abstract

The invention discloses a three-dimensional ultrasonic image generation method based on DALL.E and Pix2Pix GAN, and the method comprises the following steps: obtaining a medical text description, inputting the medical text description into a generator model, and obtaining an image representation corresponding to the medical text description; and inputting the image representation into a discriminator model, wherein the discriminator model outputs the image representation meeting the clinically available image quality standard as a high-quality three-dimensional ultrasound image. Compared with the prior art, the three-dimensional ultrasonic image generation method provided by the invention has the advantages that the generation accuracy, quality and quantity of three-dimensional medical images are remarkably improved, richer and more accurate marking data are provided for training of other medical image models, so that the over-fitting phenomenon of the other medical image models is reduced, and the training efficiency is improved. And the diagnostic capability of the medical image model is enhanced, so that the overall level of medical image analysis is improved.

Description

Technical Field

[0001] The present invention relates to the field of medical image technology, and in particular to a method and device for generating three-dimensional ultrasound images based on DALL·E and Pix2Pix GAN. Background Art

[0002] Image generation has gradually become a key link in medical image analysis, especially in generating realistic organ or lesion images in medical diagnostic images such as CT and MRI. Its purpose is to generate images that can accurately reflect the shape and volume of organs or tissues, but this task is extremely challenging in the field of medical image analysis. With the continuous advancement of computer hardware equipment, deep learning methods have begun to emerge and demonstrated their powerful capabilities in image generation tasks. Today, deep learning methods have become an important part of the field of image generation, especially in medical image generation. The main function of medical image generation is to more clearly show the changes in anatomical or pathological structures through the generated images, thereby playing a key role in computer-aided diagnosis and intelligent medical care, greatly improving the efficiency and accuracy of diagnosis.

[0003] The main tasks of medical image generation include generating images of the liver and its tumors, the brain and its tumors, cell images, lung images, etc. These generated medical images are mainly used to train other medical imaging models and / or help clinicians make more accurate diagnoses. Therefore, the generated medical images must be clear and realistic enough to extract the required precise information in order to provide large amounts of high-quality data for the training of other medical imaging models, thereby further improving the accuracy of auxiliary predictive diagnosis.

[0004] Although neural networks have achieved remarkable success in medical-assisted diagnosis, the training of medical imaging models still faces the challenge of insufficient datasets. The main reason is the lack of a large number of labeled datasets in medical images. First, the dataset itself has a limited number of defects. Second, defining the labels in the dataset is also very challenging. Training deep neural networks with a small dataset is prone to overfitting, which in turn affects the quality of medical imaging models. Therefore, developing a medical image generation model with excellent performance is crucial for training medical imaging models with high-quality data generated by medical image generation models to assist diagnosis. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and to provide a method and device for generating three-dimensional ultrasound images based on DALL·E and Pix2PixGAN, aiming to solve the problem of insufficient labeled data in current medical images.

[0006] The present invention is achieved through the following technical solutions:

[0007] A method for generating three-dimensional ultrasound images based on DALL·E and Pix2Pix GAN includes the following steps:

[0008] Obtaining a medical text description, and inputting the medical text description into a generator model to obtain an image representation;

[0009] The image representation is input into a discriminator model, which outputs the image representation that meets clinically usable image quality standards as a high-quality three-dimensional ultrasound image.

[0010] Compared with the prior art, the 3D ultrasound image generation method provided by the present invention significantly improves the generation accuracy, quality and quantity of 3D medical images by combining the technical advantages of DALL·E and Pix2Pix, and provides richer and more accurate labeled data for the training of other medical imaging models, thereby reducing the overfitting phenomenon of other medical imaging models, enhancing the diagnostic ability of medical imaging models, and further improving the overall level of medical image analysis.

[0011] Further, the method includes: performing image enhancement processing on the image representation obtained by the generator model, and inputting the image representation after image enhancement processing into the discriminator model;

[0012] and / or, performing image enhancement processing on the high-quality three-dimensional ultrasound image obtained by the discriminator model, and outputting the high-quality three-dimensional ultrasound image after the image enhancement processing;

[0013] The image enhancement process includes denoising, contrast adjustment and smoothing.

[0014] Further, the method includes converting the medical text description into a latent vector using a text encoder, and generating a preliminary image representation based on the latent vector using a DALL·E generator, wherein the DALL·E generator includes:

[0015] y = ReLU(BN(Conv(x; W conv , b conv )))

[0016] z=ReLU(BN(Deconv(y;W deconv , b deconv )))

[0017] Conv k (x)→ReLU→Conv p (y)→ReLU→Deconv q (z)

[0018] Among them, x is the potential vector, Wconv is the weight matrix of the convolutional layer, b conv is the bias vector of the convolutional layer, W deconv Transpose the weight matrix of the convolutional layer, b deconv The bias vector of the transposed convolutional layer, Conv k and Conv p Respectively represent the convolutional layers with different convolutional kernels, Deconv q represents the transposed convolutional layer, and ReLU is the activation function.

[0019] Further, the method comprises optimizing the preliminary image representation using a Pix2Pix generator to obtain the image representation, wherein the Pix2Pix generator comprises:

[0020] G(x)=ReLU(BN(Deconv(ReLU(BN(Conv(x))))))

[0021] Among them, BN stands for batch normalization, Conv and Deconv represent convolutional layer and transposed convolutional layer respectively, and ReLU is the activation function.

[0022] Furthermore, the method includes using a convolution layer to extract features from the image representation input into the discriminator model to obtain a feature map; using a pooling layer to downsample the feature map, and using a fully connected layer to map the feature map output by the pooling layer to obtain a probability value, and when the probability value is greater than a set threshold, the image representation corresponding to the probability value is output as a high-quality three-dimensional ultrasound image; wherein the formula of the convolution layer is as follows:

[0023] y = LeakyReLU(BN(Conv(x; W conv , b conv )))Among them, W conv 、b conv are the weights and bias parameters of the discriminator convolution layer, Conv(x) represents the convolution operation, BN represents batch normalization, and LeakyReLU is the activation function.

[0024] Furthermore, the method also includes: calculating the loss of the generator model, using the loss of the generator model as feedback information of the back-propagation algorithm to update the trainable parameters until the loss of the generator model tends to converge; calculating the loss of the discriminator model, using the loss of the discriminator model as feedback information of the back-propagation algorithm to update the trainable parameters until the loss of the discriminator model tends to converge.

[0025] Furthermore, the method also includes performing adversarial training on the generator model and the discriminator model based on the loss of the generator model and the loss of the discriminator model, and the adversarial training includes performing multiple rounds of alternating training on the discriminator model and the generator model, and each round of the alternating training includes: the discriminator model receives real images and generated images, and updates trainable parameters by calculating the loss of the discriminator model, and then the generator model receives medical text descriptions, and updates and adjusts the trainable parameters by calculating the loss of the generator model.

[0026] The present invention also provides a three-dimensional ultrasound image generation device based on DALL·E and Pix2Pix GAN, comprising:

[0027] A generator unit, which obtains a medical text description and inputs the medical text description into a generator model to obtain an image representation corresponding to the medical text description;

[0028] An image enhancement unit, performing image enhancement processing on the image representation;

[0029] The discriminator unit inputs the image representation after image enhancement processing into the discriminator model, and the discriminator model outputs the image representation that meets the clinically applicable image quality standard as a high-quality three-dimensional ultrasound image.

[0030] The present invention also provides a computer device, the computer device comprising a processor and a memory:

[0031] The memory is used to store a computer program and transmit the computer program to the processor;

[0032] The processor is used to execute the three-dimensional ultrasound image generation method based on DALL·E and Pix2PixGAN according to the instructions in the computer program.

[0033] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the processor executes the three-dimensional ultrasound image generation method based on DALL·E and Pix2PixGAN.

[0034] For better understanding and implementation, the present invention is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Schematic diagram of a process of generating a three-dimensional ultrasound image based on DALL·E and Pix2Pix GAN according to an embodiment of the present invention;

[0036] Figure 2Schematic diagram of the structure of a three-dimensional ultrasound image generation device based on DALL·E and Pix2Pix GAN according to an embodiment of the present invention;

[0037] Figure 3 A flowchart of a training method for a three-dimensional ultrasound image generation method based on DALL·E and Pix2Pix GAN;

[0038] Figure 4 Schematic diagram of the construction process of the model for 3D ultrasound image generation based on DALL·E and Pix2Pix GAN, namely the MinDALL-Ev2 model.

[0039] Figure 5 for Figure 3 A schematic diagram of the model training and validation process for data validation and iterative training in the training method shown;

[0040] Figure 6 for Figure 4 A schematic diagram of the structure of the MinDALL-Ev2 model;

[0041] Figure 7 The performance of different models and techniques in various organs and pathological conditions in the embodiment of the present invention is shown, wherein each column represents a specific organ or pathology (such as brain tumor, heart, liver, etc.), the numbers 1 and 2 represent different subclasses or labeling standards of the organ or pathology, and each row represents a model or technology. DETAILED DESCRIPTION

[0042] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0043] In view of the problems of limited number of data sets and inaccurate definition of labels in the data sets in the training of existing medical imaging models, the present invention intends to provide a method for generating medical images to solve the problem of insufficient labeled data in current medical images, increase the amount of data through advanced image generation technology, provide richer and more accurate labeled data for the training of other medical imaging models, thereby reducing the overfitting phenomenon of medical imaging models, and enhance the diagnostic ability of medical imaging models by improving the quality of generated images.

[0044] Based on this, the present invention develops a MinDALL-Ev2 model based on the DALL·E and Pix2Pix GAN models, which is specifically used for deep learning image generation, especially in the field of medical image generation. Figure 1 The three-dimensional ultrasound image generation method based on DALL·E and Pix2Pix GAN provided by the present invention comprises the following steps:

[0045] Obtaining a medical text description, and inputting the medical text description into a generator model to obtain an image representation;

[0046] Inputting the image representation into a discriminator model, the discriminator model outputting the image representation that meets clinically applicable image quality standards as a high-quality three-dimensional ultrasound image;

[0047] Among them, image enhancement processing is performed on the image representation obtained by the generator model, and the image representation after image enhancement processing is input into the discriminator model; and / or, image enhancement processing is performed on the high-quality three-dimensional ultrasound image obtained by the discriminator model, and the high-quality three-dimensional ultrasound image after image enhancement processing is output.

[0048] The image enhancement processing is set before the discriminator model, which can ensure that the images seen by the discriminator model are processed images, so that the feedback of the discriminator model is more in line with clinically available image quality standards; while the image enhancement processing is set after the discriminator model, the discriminator model is trained with images that have not been enhanced, which can ensure that the generator model generates high-quality images as directly as possible during generation, and then necessary fine-tuning is performed later.

[0049] At the same time, based on the above three-dimensional ultrasound image generation method, the present invention also proposes a method for optimizing the training parameters involved in the three-dimensional ultrasound image generation step.

[0050] In addition, in order to more clearly understand the technical solution provided by the embodiment of the present invention, some key terms involved in the embodiment of the present invention are first introduced here:

[0051] DALL·E: DALL·E is a model for image generation that can convert text descriptions into corresponding images. Although it is not directly used for report generation, when image assistance related to the report is needed, DALL·E can be used to generate corresponding image materials to make the report more diverse and easier for managers to understand. For example, various statistical charts are generated based on data, and pictures that can reflect on-site problems are automatically generated according to the description in the report. At the same time, the model also supports editing through descriptions on existing pictures, which greatly liberates productivity. For example, when the user needs materials, he can enter the corresponding text description such as "ultrasound image of the liver, showing mild fatty degeneration", and the model will generate some optional pictures based on the text description, and even generate multiple pictures at the same time for users to choose.

[0052] The Pix2Pix GAN model is an image-to-image conversion model based on the conditional generative adversarial network (CGAN). It can be used for various tasks, such as converting an input image into a corresponding output image. Compared with other neural network models (such as U-Net, Autoencoder, CVAE), the Pix2Pix GAN model has obvious advantages in colorization. It can generate high-quality color images, capture more details, and create more realistic images, and better restore the characteristics of specific organs or pathologies. At the same time, it can also handle nonlinear relationships and learn complex color mappings. Pix2Pix GAN is a type of GAN model that is mainly used for conversion between images, also known as image translation, to convert an input image into a corresponding output image. The network structure of the Pix2Pix GAN model includes a generator and a discriminator: the generator is a U-Net generator, which is used to convert the original image into a target image; the discriminator is a Patch GAN discriminator, which is used to distinguish between the image generated by the generator and the real target image.

[0053] The U-Net generator is a special type of convolutional neural network that can effectively handle image segmentation tasks. The encoder of the U-Net generator can convert the image into a smaller feature map while retaining important information about the image content. Its decoder can convert the feature map back to the same size as the original image and use skip connections to improve the accuracy of segmentation. This structure is very suitable for image-to-image conversion tasks, such as colorization.

[0054] The Patch GAN discriminator is a discriminator-based adversarial loss function that divides the input image into multiple small patches and discriminates each patch. This method has an advantage over discriminating the entire image because it can evaluate the difference between the generated image and the target image in a more granular way, thereby improving the quality and authenticity of the generated image.

[0055] Based on the relevant introduction of the above key terms, the three-dimensional ultrasound image generation method based on DALL·E and Pix2Pix GAN in an embodiment of the present invention is specifically described through the following two parts.

[0056] (I) Three-dimensional ultrasound image generation part

[0057] This embodiment provides a method for generating a three-dimensional ultrasound image based on DALL·E and Pix2Pix GAN. Figure 1 and Figure 2 , Figure 1 is a flow chart of the steps of generating a three-dimensional ultrasound image in this embodiment, Figure 2Schematic diagram of the structure of the three-dimensional ultrasound image generation device of this embodiment. The three-dimensional ultrasound image generation device of the embodiment of the present invention comprises a generator unit 10, an image enhancement unit 20 and a discriminator unit 30.

[0058] The generator unit 10 of the three-dimensional ultrasound image generation device is used to perform step S10: obtain a medical text description, input the medical text description into a generator model, and obtain an image representation corresponding to the medical text description. Specifically, the generator unit 10 includes a text encoding module 11, a first generator module 12, and a second generator module 13.

[0059] The text encoding module 11 is used to execute step S11: using the text encoder of the DALL·E model to convert the input medical text description into a high-dimensional latent vector. Specifically, step S11 generates a latent representation vector of the text through the Transformer architecture.

[0060] The first generator module 12 is used to perform step S12: input the potential vector as a condition into the DALL·E generator, and the DALL·E generator generates a preliminary image representation according to the potential vector; the DALL·E generator includes multiple convolution layers, deconvolution layers and ReLU activation function, and the specific formula is as follows:

[0061] y = ReLU(BN(Conv(x; W conv , b conv )))

[0062] z=ReLU(BN(Deconv(y;W deconv , b deconv )))

[0063] Conv k (x)→ReLU→COnv p (y)→ReLU→Deconv q (z)

[0064] Where x is the input potential vector, W conv is the weight matrix of the convolutional layer, b conv is the bias vector of the convolutional layer, W deconv Transpose the weight matrix of the convolutional layer, b deconv The bias vector of the transposed convolutional layer, Conv k and Conv p Respectively represent the convolutional layers with different convolutional kernels, Deconv q Represents a transposed convolutional layer. Each convolutional layer is followed by a ReLU activation function to introduce non-linearity.

[0065] The second generator module 13 is used to perform step S13: input the preliminary image representation into the Pix2Pix generator to obtain the image representation. The Pix2Pix generator uses a convolution layer and a transposed convolution layer structure similar to DALL·E to optimize the details and style of the preliminary image representation to achieve higher visual quality and more accurate conformance to the target features. The specific formula of the Pix2Pix generator is as follows:

[0066] G(x)=ReLU(BN(Deconv(ReLU(BN(Conv(x))))))

[0067] Among them, BN stands for batch normalization, Conv and Deconv represent the convolution layer and transposed convolution layer respectively, and ReLU is the activation function.

[0068] By combining the DALL·E generator and the Pix2Pix generator to form the generator unit 10, the generator unit 10 can not only generate a rough three-dimensional image according to the input text, but also use the image conversion capability of Pix2Pix to further enhance the details and style of the image, ensuring that the generated result is more in line with the text description and visual requirements.

[0069] Furthermore, the image enhancement unit 20 is used to execute step S20: perform image enhancement processing on the image representation obtained in step S10 to obtain a generated image. Specifically, the image enhancement processing includes denoising processing, contrast adjustment and smoothing processing. The denoising processing includes removing noise and artifacts in the image representation, which can improve the image quality and make the image clearer; the contrast adjustment includes optimizing the contrast of the image representation, making the structure clearer and enhancing the accuracy of medical diagnosis; the smoothing processing includes smoothing the image edges and reducing the impact of noise on image interpretation.

[0070] In some embodiments, Gaussian filtering is used to perform the denoising process. The standard deviation of the filter of the Gaussian filtering is set according to the image noise level, generally set to 1.0. The specific formula is:

[0071]

[0072] Among them, I'(x,y) is the image pixel and G(i,j) is the Gaussian kernel function.

[0073] Furthermore, the discriminator unit 30 is used to perform step S30: input the generated image of step S20 as an input image into the discriminator model, the discriminator model determines whether the input image meets the clinically applicable image quality standard, and outputs the input image that meets the clinically applicable image quality standard as a high-quality three-dimensional ultrasound image. In this embodiment, the discriminator unit 30 is arranged after the image enhancement unit 20 to ensure that the image seen by the discriminator model is a processed image, so that the feedback of the discriminator model is more in line with the clinically applicable image quality standard.

[0074] Specifically, the discriminator model is based on the discriminator architecture of Pix2Pix GAN, and is composed of multiple convolutional layers, pooling layers and fully connected layers. The discriminator unit 30 includes a convolution module 31, a pooling module 32 and a fully connected module 33.

[0075] The convolution module 31 is used to perform step S31: extract features from the input image using the convolution layer of the discriminator model to obtain a feature map, wherein the input image includes a real image directly input to the discriminator model from the outside, an image representation obtained by the generator unit 10, or an image representation processed by the image enhancement processing unit 20. In this embodiment, each convolution layer is followed by a batch normalization (BN) and a Leaky ReLU activation function to enhance the feature expression capability, and the formula of the convolution layer is as follows:

[0076] y = LeakyReLU(BN(Conv(x; W conv , b conv )))

[0077] Among them, W conv 、b conv are the weights and bias parameters of the convolutional layer of the discriminator model, Conv(x) represents the convolution operation, BN represents batch normalization, and LeakyReLU is the activation function used to introduce nonlinearity.

[0078] The pooling module 32 is used to execute step S32: down-sample the feature map output by the convolution layer through the pooling layers to reduce the size of the feature map and retain the most significant features in the feature map. In this embodiment, the feature map is processed by using the maximum pooling method, and the specific formula is as follows:

[0079] z=MaxPooling(y)

[0080] The fully connected module 33 is used to execute step S33: integrate the features retained after the feature map is processed by the pooling layer through the fully connected layers to obtain a probability value, and judge whether the input image corresponding to the feature map (i.e., the image representation after image enhancement processing) meets the clinically applicable image quality standards according to the probability value, specifically including: the fully connected layer flattens the feature map output by the pooling layer and maps it to a probability value, the probability value represents the possibility of the input image representation being real or generated, and is used to determine the authenticity of the input image, classifying the input image with a probability value greater than a set threshold as real and retaining it, and outputting the retained input image as a high-quality three-dimensional ultrasound image, wherein the formula of the fully connected layer is:

[0081] p=σ(W fc Flatten(z)+b fc )

[0082] W fc 、b fc are the weight and bias parameters of the fully connected layer, and σ is the Sigmoid activation function, which maps the output to a probability value between [0,1].

[0083] The discriminator model performs binary classification through the probability value output by the fully connected layer: when the probability value p is close to 1, that is, greater than the set threshold, the discriminator model believes that the input image is real; when the probability value p is close to 0, that is, less than the set threshold, the discriminator believes that the image is generated. In this embodiment, the threshold is set to 0.5, that is, when p>0.5, the image is classified as real; when p≤0.5, the image is classified as generated. The basis for the judgment of the discriminator model is the features extracted from the input image: real medical images usually have more delicate textures, clear boundaries and consistent structures, while the generated fake images may show deficiencies in these aspects. The discriminator model extracts these differences through the convolution layer and distinguishes them based on the output of the fully connected layer; through this distinction method, the discriminator model can effectively distinguish between the generated fake images and the real medical images, ensuring that the generation model can be gradually optimized to generate medical images that are closer to the real ones.

[0084] In this embodiment, the initial weights of the DALL·E generator and the Pix2Pix generator in the generator unit 10 are usually randomly initialized from a standard normal distribution, and may also adopt a uniform distribution or other weight initialization method, such as Xavier initialization, according to specific needs to ensure that the initial state of the generator network is stable; the initial weights of the discriminator model in the discriminator unit 30 may also be randomly initialized through a standard normal distribution, or adopt He initialization and other methods to optimize the convergence performance of the deep network.

[0085] In addition, in this embodiment, the image enhancement processing is arranged before the discriminator model, and in other embodiments, the image enhancement processing may be arranged after the discriminator model, that is, the discriminator model is trained with unprocessed images to ensure that the generator generates high-quality images as directly as possible during generation, and then performs necessary fine-tuning afterwards.

[0086] Compared with the prior art, the present invention combines the advantages of DALL·E and Pix2Pix GAN to develop an efficient image generation method. The DALL·E model is first used to pass the potential vector through the generator network to generate a preliminary two-dimensional or three-dimensional medical image, and then the generated preliminary image is input into the Pix2Pix model for further transformation and optimization, so that the image is more realistic and has medical diagnostic significance, and then the image is enhanced to ensure that the image can meet the standards of medical images for doctors' diagnosis or for further model training. Among them, the image generated by the DALL·E model already has certain medical features, such as the basic shape and structure of the organ, but lacks in details and texture, while the Pix2Pix model can enhance the edge details of the image, improve the texture details, etc., and the subsequent image enhancement processing ensures the clinical availability of the generated image through denoising, contrast adjustment and other processing. Compared with the prior art, the innovation of the present invention lies in its unique model architecture design and effective data processing process, which can generate more realistic and accurate medical images, assist clinical diagnosis, and significantly improve the generation quality and efficiency of three-dimensional ultrasound images.

[0087] The high-quality medical images generated by the three-dimensional ultrasound image generation method of the present invention can be used to train other medical models, such as deep learning models for pathology classification, segmentation and other tasks. These high-quality generated images can expand the scale of the training data set, improve the generalization ability of the model, and ultimately improve the accuracy of the model in actual diagnostic tasks.

[0088] The image enhancement processing of the present invention can ensure the clinical usability of the image: the generated three-dimensional ultrasound image is used for medical diagnosis or training of other medical models, and it must be ensured that the image has sufficient clarity, contrast and low noise so that doctors or other models can accurately interpret the image. Failure to perform image enhancement processing may result in: the unprocessed image may contain more noise, which will interfere with the effect of the medical diagnosis model when used for training other medical models; the unprocessed image may have insufficient contrast, resulting in unclear and difficult to distinguish important structures, especially when subtle abnormalities need to be identified; the unprocessed image may have artifacts or geometric distortions, which will affect the authenticity and reliability of the image. The present invention is arranged to perform image enhancement processing immediately after the generator output, so as to avoid the possibility that the lack of appropriate image post-processing may affect the clinical usability of the image, thereby affecting its effect in actual medical diagnosis and subsequent model training.

[0089] (II) Training

[0090] The method for generating a three-dimensional ultrasound image based on DALL·E and Pix2Pix GAN described in this embodiment also includes a training method, see Figure 3-5 Before training, the original three-dimensional ultrasound images are collected from a hospital or a medical image database, and the image format of the original three-dimensional ultrasound images is a standard DICOM format. Then, the collected original three-dimensional ultrasound images are preprocessed to obtain training set image data and verification set image data, which specifically includes the following steps:

[0091] The original three-dimensional ultrasound image is normalized, and the pixel values ​​of the image data are scaled to the range of [0,1]; then the normalized image data is denoised using Gaussian filtering, and the standard deviation of the Gaussian filter is set according to the image noise level, generally set to 1.0; the denoised image data is enhanced using rotation, translation, scaling, mirroring and other techniques to generate enhanced image data, the amount of which is 5 times the original data, which can increase the generalization ability of the model; the data-enhanced image data is divided into a training set and a validation set in a ratio of 8:2 to obtain training set image data and validation set image data. The specific formula for the normalization process is:

[0092]

[0093] Among them, x is the original pixel value, X is the image dataset, and x′ is the normalized pixel value.

[0094] The specific formula for the denoising process is:

[0095]

[0096] The training set image data is input as a real image into the discriminator unit 30 or into the discriminator model in step S30 after image preprocessing, and is used to train the discriminator model and the generator model. The image preprocessing includes the following steps: performing three-dimensional convolution (3D Convolution) processing on the training set image data: extracting the spatial features of the image, and the formula of the three-dimensional convolution is:

[0097]

[0098] Then, the maximum pooling operation is used on the image data output by the three-dimensional convolution to reduce the image size while retaining the main features. The formula of the maximum pooling is:

[0099]

[0100] The ReLU activation function is used to introduce nonlinearity into the image data output by the maximum pooling to obtain the final training set image. The specific formula is:

[0101] f(x) = max(0, x)

[0102] During training, the final training set image is input as a real image into the judgment module 30 or the discriminator model in step S30, and the input image features are extracted together with the image representation processed by image enhancement in step S20.

[0103] See also Figure 3 , Figure 3 The present invention is a flow chart of a training method for a three-dimensional ultrasound image generation method based on DALL·E and Pix2Pix GAN. The training method mainly includes step S40: calculating the losses corresponding to the generator model and the discriminator model, and using the losses of the generator model and the discriminator model as feedback information of the back propagation algorithm to update the trainable parameters until the losses of the generator model and the discriminator model tend to converge.

[0104] This embodiment also provides a training device 40 for generating three-dimensional ultrasound images, which is used to perform step S40. Figure 2 and Figure 6 The training device 40 includes a first training module 41, a second training module 42, a third training module 43 and a fourth training module 44. The first training module 41 and the third training module 43 are used to optimize the trainable parameters in the generator unit 10 in the three-dimensional ultrasonic image generating device. The second training module 42 is used to optimize the trainable parameters in the discriminator unit 30 in the three-dimensional ultrasonic image generating device. The fourth training module 44 is used to perform adversarial training on the generator unit 10 and the discriminator unit 30 and update the corresponding trainable parameters.

[0105] The first training module 41 is used to execute step S41: according to the result obtained by the discriminator model in step S30, calculate the loss L of the generator model G , using the loss L of the generator model G As feedback information of the back-propagation algorithm, the trainable parameters are updated until the loss L of the generator model G The goal of the first training module 41 is to make the generator model generate realistic three-dimensional images according to the input text description, so that the discriminator model has difficulty in distinguishing these images from real images.

[0106] Specifically, the loss L of the generator model is G Based on the output of the discriminator model for the generated image, the specific formula is:

[0107]

[0108] Among them, D(G(z i ; W gen )) represents the predicted probability of the discriminator model for the generated image, W gen are all the weight parameters in the generator model.

[0109] The generator model adjusts its weights and biases through the back-propagation algorithm so that the generated image is considered to be real in the discriminator model. In this process, the trainable parameters are updated using gradient descent or other optimization algorithms. The formula used in this embodiment is as follows:

[0110]

[0111] Where η is the learning rate.

[0112] Through the above steps, the generator model continuously improves the quality of the images it generates in each iteration, making the final generated images indistinguishable from the real images.

[0113] The second training module 42 is used to execute step S42: based on the preliminary generator model obtained in step S41, the discriminator model calculates and generates the image G(z i ; W gen ) and the real image x i The loss between them gives the loss L of the discriminator model. D , using the loss L of the discriminator model D As feedback information of the back-propagation algorithm, the trainable parameters are updated until the loss of the discriminator model converges to obtain a preliminary discriminator model. The generated image is an image representation obtained by the generator unit 10 or an image representation obtained by the image enhancement processing unit 20 after image enhancement processing. The real image x i It is the final training set image obtained before training; the goal of the second training module 42 is to distinguish the generated false images from the real images.

[0114] In this embodiment, the binary cross entropy loss function is used to calculate the generated image G(z i ; W gen ) and the real image x i The loss between is used to measure the classification accuracy of the discriminator model for real images and generated images. The loss L of the discriminator model is D The formula is:

[0115]

[0116] Among them, yi is the label of the real image (1 represents the real image, 0 represents the generated image), D(xi ; W disc ) is the predicted probability of the discriminator model for the real image, D(G(z i ; W gen );W disc ) is the discriminator’s predicted probability for the generated image.

[0117] The discriminator model updates its weights and bias parameters through the back propagation algorithm so that it can better distinguish between real images and generated images. In this process, gradient descent or other optimization algorithms are used to update the trainable parameters. The formula used in this embodiment is as follows:

[0118]

[0119] Among them, η is the learning rate, which indicates the step size of each parameter update.

[0120] Through the above steps, the discriminator model gradually improves its classification ability and can more accurately distinguish between real images and generated images, thereby effectively guiding the generator model to generate more realistic images.

[0121] The third training module 43 is used to execute step S43: based on the preliminary discriminator model obtained in step S42, the loss L of the generator model is calculated again G ′, using the loss L of the generator model G ' is used as feedback information of the back-propagation algorithm to update the trainable parameters until the loss L of the generator model G ' tends to converge, and an optimized generator model is obtained. The goal of the third training module 43 is to make the images generated by the generator model gradually realistic by continuously adjusting the weight parameters of the generator model, and to maximize the probability of misjudgment of the image representation generated by the generator model by the discriminator model, that is, to make the probability output by the discriminator model as close to 1 (the probability of a real image) as possible.

[0122]

[0123] Among them, D(G(z i ; W gen );W disc ) is the predicted probability of the discriminator model for the image representation generated by the generator model, W gen is the weight parameter of the generator model.

[0124] In order to minimize the loss LG′ of the generator model, the generator model adjusts its weight parameter W through the back-propagation algorithm. gen , the specific formula is:

[0125]

[0126] Among them, η is the learning rate, which controls the step size of each update.

[0127] The fourth training module 44 is used to execute step S44: based on the loss of the generator model and the loss of the discriminator model, the generator model and the discriminator model are adversarially trained. The adversarial training is the core training strategy of the Generative Adversarial Network (GAN). The generator model and the discriminator model continuously improve their respective performances through alternating training. The adversarial training includes multiple rounds of alternating training of the discriminator model and the generator model. Each round of the alternating training includes: first, the discriminator model receives the real image and the generated image, and calculates the loss L of the discriminator model. D And update its trainable parameters W disc , improving its ability to classify true and false images. Subsequently, the generator model receives text information, generates images and calculates the loss L of the generator model G , adjust its trainable parameters W gen , making the generated image more realistic, where the loss L of the discriminator model D and the loss L of the generator model G The calculation formula of is the same as that of step S41 and step S42. In each round of alternating training, the generator model attempts to generate images that can deceive the discriminator model, while the discriminator model continuously improves its ability to distinguish until one of the following situations occurs, at which time the adversarial training is terminated to obtain the final generator model and discriminator model: (1) the loss of the generator model and the discriminator model tends to be stable and no longer changes significantly; (2) the image quality generated by the generator model reaches the expected level; (2) the preset number of training rounds is reached.

[0128] In this embodiment, the validation set image data is input into the generator model and the discriminator model in step S44 for model validation. According to the validation results, the best performing generator model and discriminator model are saved as the trained generator model and discriminator model, i.e., the MinDALL-Ev2 model. The structure of the MinDALL-Ev2 model is as follows: Figure 6 Specifically, structural similarity (SSIM) and peak signal-to-noise ratio (PSNR) are used as evaluation indicators to evaluate the performance of the generator model and the discriminator model in step S44 on unseen data, and the evaluation includes the following steps: for each pair of generated images and real images, SSIM and PSNR are calculated; by averaging the SSIM and PSNR of the entire verification set, the overall performance index of the model is obtained; if the verification result does not meet the performance requirements, return to step S44 to train the generator model and the discriminator model again, and if the verification result meets the performance requirements, the generator model and the discriminator model that meet the requirements are used as the trained generator model and the discriminator model, that is, the MinDALL-Ev2 model.

[0129] Compared with the prior art, the training method of the present invention trains the discriminator model and the generator model separately, and then performs adversarial training on the discriminator model and the generator model, and finally obtains the trained generator model and the discriminator model. The generator model can generate high-quality images that are difficult to distinguish from real images; the discriminator model has good recognition ability, can effectively distinguish between real images and generated images, and significantly improves the performance of the generator model and the discriminator model.

[0130] Based on the same inventive concept, the present invention also provides an electronic device, which may be a terminal device such as a server, a desktop computing device or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). The device includes one or more processors and a memory, wherein the processor is used to execute a program to implement the three-dimensional ultrasound image generation method based on DALL·E and Pix2Pix GAN as described in the method embodiment; and the memory is used to store a computer program executable by the processor.

[0131] Based on the same inventive concept, the present invention further provides a computer-readable storage medium, corresponding to the embodiment of the aforementioned image processing method, wherein the computer-readable storage medium stores a computer program, and when the program is executed by a processor, the steps of the three-dimensional ultrasound image generation method based on DALL·E and Pix2Pix GAN recorded in any of the aforementioned embodiments are implemented.

[0132] The present invention may take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-usable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0133] Example 1

[0134] Step S11: Input the medical text description, such as "ultrasound image of the liver, showing mild fatty degeneration", into the text encoder of the DALL·E model, and the text encoder converts the medical text description into a latent vector representation. These latent vectors contain the semantic content of the text information and can guide subsequent image generation.

[0135] Step S12: The DALLE generator generates a preliminary two-dimensional or three-dimensional medical image using the latent vector representation. The preliminary two-dimensional or three-dimensional medical image may have certain medical features, such as the basic shape and structure of an organ, but may lack details and textures.

[0136] Step S13: The preliminary two-dimensional or three-dimensional medical image is input into the Pix2Pix model, and the Pix2Pix model uses a conditional GAN ​​(generative adversarial network) to further transform and optimize the image to obtain an image representation. In this embodiment, the transformation and optimization include enhancing the edge details of the image, improving the texture details, etc., which can make the image more realistic and have medical diagnostic significance.

[0137] Step S20: In order to ensure the clinical usability of the generated image, the image representation is subjected to image enhancement processing such as denoising and contrast adjustment. The image representation after image enhancement processing contains the details and features required for diagnosis, and can be used for doctors' diagnosis or for further model training.

[0138] Step S30: judging in a discriminator whether the image representation after image enhancement processing meets clinically applicable image quality standards, and outputting the image representation that meets the standards as a high-quality three-dimensional ultrasound image.

[0139] The high-quality three-dimensional ultrasound images finally output by this embodiment can be used to train other medical models, such as deep learning models for pathology classification, segmentation and other tasks. Furthermore, the high-quality three-dimensional ultrasound images finally output by this embodiment can expand the scale of the training data set and improve the generalization ability of the model, thereby improving the accuracy of other medical models in actual diagnostic tasks.

[0140] In summary, the present invention combines the advantages of DALL·E and Pix2Pix GAN to develop an efficient image generation method, which significantly improves the generation quality and efficiency of three-dimensional ultrasound images. Compared with the prior art, the innovation of the present invention lies in its unique model architecture design and effective data processing flow, which can generate more realistic and accurate medical images and assist clinical diagnosis.

[0141] Effect Example

[0142] In order to verify the effectiveness of the MinDALL-Ev2 model of the present invention, this example uses the following baseline models to conduct multiple comparative experiments:

[0143] DALL·E: Three-dimensional ultrasound image generation based on the DALL·E model.

[0144] Pix2Pix GAN: Image Generation Using Pix2Pix GAN.

[0145] DataAugment: Adds data augmentation technology based on Pix2Pix GAN.

[0146] DALL·E+Pix2Pix: Combines DALL·E and Pix2Pix GAN for image generation.

[0147] Test Set: The data of the test set is used to evaluate the final performance of the model.

[0148] Based on the test set data, this embodiment uses the MinDALL-Ev2 model of the present invention and other baseline models to generate three-dimensional ultrasound images of different organs. In each experiment, 80% of the data is randomly selected for training, and the remaining data is used for testing. The performance scores of each model under different organs and annotation standards are calculated to compare and evaluate the advantages and disadvantages of different models or technologies, that is, to reflect the advantages and disadvantages of the medical images generated by the model in terms of visual quality, authenticity, and effectiveness of auxiliary diagnosis model training. The performance scores are mainly related to the quality of the generated images and the accuracy of auxiliary diagnosis: (1) The quality of the generated images: These scores usually represent the visual quality and authenticity of the medical images generated by the model in various organs and pathological conditions; for example, if a generated liver image can accurately reflect the anatomical structure and pathological characteristics of the real liver, then the model will score higher in the "liver" category; (2) The accuracy of auxiliary diagnosis: If these generated images are subsequently used for the training and verification of medical imaging diagnosis models, then the quality of the generated images will also directly affect the accuracy of auxiliary diagnosis. High-quality generated images can make the trained model more accurate, thereby improving the reliability and effectiveness of auxiliary diagnosis.

[0149] See also Figure 7 , Figure 7 The data in the figure represent the performance scores of each model under different organs and annotation standards. Each column represents a specific organ or pathology (such as brain tumor, heart, liver, etc.), and the numbers 1 and 2 represent different subclasses or annotation standards of the organ or pathology; each row represents a model or technology. Figure 7This shows that the MinDALL-Ev2 model (based on DALL·E and Pix2Pix GAN three-dimensional ultrasound image generation model) provided by the present invention is significantly superior to the existing technology in terms of accuracy and realism in generating medical images, especially in terms of generating high-quality three-dimensional medical image data.

[0150] Compared with the prior art, the three-dimensional ultrasound image generation method provided by the present invention has the following advantages:

[0151] (1) Improved image generation accuracy: Through the text-to-image generation capability of the DALL·E model, the present invention can generate three-dimensional medical images with complex structures and fine features based on detailed text descriptions. These images more accurately reflect the actual anatomical structures and pathological features in medical images. Compared with single image generation technology, combined with the image conversion capability of the Pix2Pix model, the present invention can further optimize the initially generated images, enhance their details and contrast, and thus generate three-dimensional images that are closer to real images.

[0152] (2) Improvement in the accuracy of auxiliary diagnosis: The high-quality three-dimensional medical images generated by the present invention can significantly improve the model's feature extraction capability and diagnostic accuracy when used to train other medical imaging diagnostic models. Since the generated images perform well in multiple medical categories, this technology has significant clinical value for auxiliary diagnosis of complex lesions. By improving the quality of generated images, the present invention can effectively increase the diversity and quality of training data, allowing medical imaging diagnostic models to be better generalized and provide more reliable diagnostic support in clinical practice.

[0153] (3) Optimized image post-processing process: The present invention can not only generate high-quality three-dimensional medical images, but also provide necessary image post-processing steps, including denoising, contrast adjustment and smoothing. These post-processing measures ensure the usability of the generated images in actual clinical applications, so that they meet the standard requirements of medical imaging. The addition of image post-processing steps makes the final generated images not only have high visual quality, but also provide more reliable data support for subsequent medical diagnosis and analysis, reducing the possible risk of misdiagnosis.

[0154] (4) Broad application prospects: The generative model of the present invention has broad application prospects. It can not only be used to generate three-dimensional images of various organs and pathological conditions, but also adapt to different medical imaging standards and annotation methods, providing support for different medical research and clinical practices. In addition, the image data generated by the model can be directly applied to the training of various medical imaging models, which is helpful for the development of new diagnostic tools and methods and promotes the further development of medical imaging technology.

[0155] In summary, compared with the prior art, the three-dimensional ultrasound image generation method provided by the present invention significantly improves the generation accuracy, quality and quantity of three-dimensional medical images by combining the technical advantages of DALL·E and Pix2Pix, and provides richer and more accurate labeled data for the training of other medical imaging models, thereby reducing the overfitting phenomenon of other medical imaging models, enhancing the diagnostic ability of medical imaging models, and further improving the overall level of medical image analysis.

[0156] The above-described embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or changes made by those skilled in the art based on the present invention are within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A three-dimensional ultrasound image generation method based on DALL·E and Pix2Pix GAN, characterized in that: The steps include: Obtaining a medical text description, and inputting the medical text description into a generator model to obtain an image representation; The image representation is input into a discriminator model, which outputs the image representation that meets clinically usable image quality standards as a high-quality three-dimensional ultrasound image.

2. The method according to claim 1, characterized in that: include: Performing image enhancement processing on the image representation obtained by the generator model, and inputting the image representation after the image enhancement processing into the discriminator model; and / or, performing image enhancement processing on the high-quality three-dimensional ultrasound image obtained by the discriminator model, and outputting the high-quality three-dimensional ultrasound image after the image enhancement processing; The image enhancement process includes denoising, contrast adjustment and smoothing.

3. The method according to claim 1, characterized in that: The method comprises converting the medical text description into a latent vector using a text encoder, and generating a preliminary image representation according to the latent vector using a DALL·E generator, wherein the DALL·E generator comprises: y=ReLU(BN(Conv(x;W conv ,b conv ))) z=ReLU(BN(Deconv(y;W deconv ,b deconv ))) Conv k (x)→ReLU→Conv p (y)→ReLU→Deconv q (z) Among them, x is the potential vector, W conv is the weight matrix of the convolutional layer, b conv is the bias vector of the convolutional layer, W deconv Transpose the weight matrix of the convolutional layer, b deconv Transpose the bias vector of the convolutional layer, Conv k and Conv p Respectively represent the convolutional layers with different convolutional kernels, Deconv q represents the transposed convolutional layer, and ReLU is the activation function.

4. The method according to claim 2, characterized in that: The method comprises optimizing the preliminary image representation using a Pix2Pix generator to obtain the image representation, wherein the Pix2Pix generator comprises: G(X)=ReLU(BN(Deconv(ReLU(BN(Conv(x)))))) Among them, BN stands for batch normalization, Conv and Deconv represent the convolution layer and transposed convolution layer respectively, and ReLU is the activation function.

5. The method according to claim 1, characterized in that The method comprises using a convolution layer to extract features of the image representation input into the discriminator model to obtain a feature map; using a pooling layer to downsample the feature map, using a fully connected layer to map the feature map output by the pooling layer to obtain a probability value, and when the probability value is greater than a set threshold, the image representation corresponding to the probability value is output as a high-quality three-dimensional ultrasound image; wherein the formula of the convolution layer is as follows: y=LeakyReLU(BN(Conv(x;W conv ,b conv ))) Among them, W conv , b conv are the weights and bias parameters of the convolutional layer of the discriminator model, Conv(x) represents the convolution operation, BN represents batch normalization, and LeakyReLU is the activation function.

6. The method according to any one of claims 1 to 5, characterized in that: Also includes: Calculate the loss of the generator model, and use the loss of the generator model as feedback information of the back-propagation algorithm to update the trainable parameters until the loss of the generator model converges; The loss of the discriminator model is calculated, and the loss of the discriminator model is used as feedback information of the back-propagation algorithm to update the trainable parameters until the loss of the discriminator model converges.

7. The method according to claim 6, characterized in that It also includes performing adversarial training on the generator model and the discriminator model based on the loss of the generator model and the loss of the discriminator model, wherein the adversarial training includes performing multiple rounds of alternating training on the discriminator model and the generator model, and each round of the alternating training includes: the discriminator model receives real images and generated images, and updates trainable parameters by calculating the loss of the discriminator model, and then the generator model receives medical text descriptions, and updates and adjusts the trainable parameters by calculating the loss of the generator model.

8. A three-dimensional ultrasound image generation device based on DALL·E and Pix2Pix GAN, characterized in that: The device comprises: A generator unit, which obtains a medical text description and inputs the medical text description into a generator model to obtain an image representation corresponding to the medical text description; An image enhancement unit, performing image enhancement processing on the image representation; The discriminator unit inputs the image representation after image enhancement processing into the discriminator model, and the discriminator model outputs the image representation that meets the clinically applicable image quality standard as a high-quality three-dimensional ultrasound image.

9. A computer device, characterized in that: The computer device comprises a processor and a memory: The memory is used to store a computer program and transmit the computer program to the processor; The processor is configured to execute the method according to any one of claims 1 to 7 according to instructions in the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the processor is enabled to perform the method according to any one of claims 1 to 7.

Images