Medical image generation method, device, equipment and computer medium
By processing multimodal medical data and diffusion models, images at specific time points are generated, solving the problem of low accuracy in single-modal generation and achieving high-accuracy image prediction to support personalized diagnosis and treatment.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CAPITAL NORMAL UNIVERSITY
- Filing Date
- 2026-03-04
- Publication Date
- 2026-06-05
AI Technical Summary
Current medical image generation technologies are mainly based on a single data modality, which cannot generate images at a specific point in time, resulting in low accuracy in predicting the direction of changes in patient images.
Using multimodal medical data and pre-set noise data, the target prediction image at a specific time point is generated through the denoising unit in the diffusion model. Feature extraction and stitching are performed using a multi-layer U-Net neural network, and feature fusion is performed by combining a cross-attention module.
It enables accurate prediction of changes in patient images, improves the accuracy of image generation, and supports personalized diagnosis and treatment decisions and planning.
Smart Images
Figure CN122156397A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of image generation technology, and more specifically, relates to a method, apparatus, device and computer medium for generating medical images. Background Technology
[0002] In related technologies, most diagnostic and treatment techniques based on intelligent image analysis rely on current image scanning and lack longitudinal, periodic imaging examinations to support more precise diagnosis and treatment. In recent years, with the rapid development of generative artificial intelligence (AI) technology, medical image synthesis has become a hot research area. Related research focuses on the generation of specific tissues, organs, and lesions from images. These studies are mostly based on diffusion models, using anatomical constraints to generate corresponding medical images. Other research focuses on image generation across different modalities. In predicting disease progression, related research focuses on the quantitative assessment of tissue, organ, or lesion development changes using AI technology, as well as image generation based on spatiotemporal diffusion models. The application of generative AI technology in medical imaging is a hot topic, but related research is mostly in its early stages, with scattered research directions and a lack of a systematic framework. Furthermore, existing research is mostly based on a single data modality, which does not fully match the multimodal data encountered in actual clinical diagnosis and treatment scenarios, and lacks technical methods for generating medical images at specific time points.
[0003] In summary, the related technologies all generate medical images based on a single data modality, which cannot generate medical images at a specific point in time. When predicting the direction of change in a patient's medical images, they can only be predicted manually, resulting in low accuracy. Summary of the Invention
[0004] In view of the above problems, this application provides a medical image generation method, apparatus, device and computer medium, thereby solving or at least alleviating one or more of the above-mentioned problems and other problems existing in the prior art.
[0005] A first aspect of this application provides a method for generating medical images, including: Acquire multimodal medical data of the patient at a reference time. The multimodal medical data includes medical images and text data. The text data includes basic patient information and the time interval between the predicted time and the reference time. The basic patient information includes the patient's age information and the patient's diagnosis report. Obtain preset noise data; The multimodal medical data and the preset noise data are processed based on a preset target denoising unit to obtain the target prediction image at the prediction time. The target denoising unit is a denoising unit in the diffusion model.
[0006] A second aspect of this application provides a medical image generation apparatus, comprising: The acquisition unit is used to acquire the patient's multimodal medical data at a reference time. The multimodal medical data includes medical images and text data. The text data includes the patient's basic information and the time interval between the predicted time and the reference time. The patient's basic information includes the patient's age information and the patient's diagnosis report. The acquisition unit is also used to acquire preset noise data; The generation unit is used to process the multimodal medical data and the preset noise data based on the preset target denoising unit to obtain the target prediction image at the prediction time. The target denoising unit is the denoising unit in the diffusion model.
[0007] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the medical image generation method described above.
[0008] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.
[0009] The beneficial effects of this application's embodiments are as follows: In this application's solution, multimodal medical data of the patient at a reference time is acquired. This multimodal medical data includes medical images and text data. The text data includes basic patient information and the time interval between the predicted time and the reference time. The basic patient information includes the patient's age and diagnostic report. Preset noise data is acquired. Based on a preset target denoising unit, the multimodal medical data and the preset noise data are processed to obtain the target predicted image at the predicted time. The target denoising unit is a denoising unit in a diffusion model. This introduces the analysis of multimodal medical data and enables the generation of medical images at specific time points. When predicting the direction of change in the patient's medical images, the data used is multimodal, considering more comprehensive data and improving the accuracy of predicting the direction of change in the patient's medical images. Attached Figure Description
[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0011] Figure 1 A schematic flowchart of a medical image generation method provided in an embodiment of this application; Figure 2 A schematic flowchart of a medical image generation method provided in an embodiment of this application; Figure 3 A flowchart illustrating the training process of a target diffusion model provided in an embodiment of this application; Figure 4 This is a structural block diagram of a medical image generation device provided in an embodiment of this application; Figure 5 This is a schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation
[0012] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments in the specific implementation of this application should fall within the protection scope of the embodiments of this application.
[0013] To keep the drawings concise, each drawing only schematically shows the parts relevant to the disclosure; these do not represent the actual structure of the product. Furthermore, for ease of understanding, in some drawings, only one of components with the same structure or function is schematically shown, or only one is labeled. In this document, "one" not only means "only one," but can also mean "more than one," and "several" includes "two" and "more than two."
[0014] Furthermore, in the description of this application, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0015] Multiple imaging follow-ups play a crucial role in personalized precision medicine. Firstly, they allow for precise assessment of disease progression. Through regular imaging examinations (such as CT and MRI scans), physicians can accurately detect dynamic changes in the disease, which is essential for understanding its development patterns. For example, in suspected lung cancer patients, imaging follow-ups help physicians observe changes in tumor size, the appearance of new lesions, and the presence of metastases. For patients with fibrotic interstitial lung disease, imaging follow-ups can quantitatively assess the impact of fibrosis on the morphology and function of lung tissue. This continuous monitoring enables physicians to more accurately assess the condition, thus having significant clinical implications for developing more personalized treatment plans and evaluating post-treatment outcomes.
[0016] In addition, multiple imaging follow-ups can support more personalized treatment decisions, avoiding undertreatment or overtreatment. Personalized diagnosis and treatment emphasizes developing the most appropriate treatment strategy based on the patient's specific situation. By comparing the images and corresponding measurement results from follow-up over a period of time, doctors can develop corresponding personalized treatment plans for patients and gain a deeper understanding of the response of tissues, organs, or lesions to treatment.
[0017] Current diagnostic and treatment technologies based on intelligent image analysis are mostly based on current image scans, lacking longitudinal, periodic imaging examinations to support more precise diagnosis and treatment. In recent years, with the rapid development of generative artificial intelligence (AI) technology, medical image synthesis has become a hot research area. Related research focuses on the generation of specific tissues, organs, and lesions from images. These studies are mostly based on diffusion models, using anatomical constraints to generate corresponding medical images. Other research focuses on image generation across different modalities. In predicting disease progression, related research focuses on the quantitative assessment of tissue, organ, or lesion development changes using AI technology, and image generation based on spatiotemporal diffusion models. The application of generative AI technology in medical imaging is a hot topic, but related research is mostly in its early stages, with scattered research directions and a lack of a systematic framework. Furthermore, existing research is mostly based on single data modalities, which cannot fully match the multimodal data encountered in actual clinical diagnosis and treatment scenarios, and there is a lack of technical methods for generating medical images at specific time points.
[0018] In summary, the related technologies all generate medical images based on a single data modality, which cannot generate medical images at a specific point in time. When determining the direction of change in a patient's medical images, they can only rely on manual prediction, resulting in low accuracy.
[0019] This application focuses on image generation by incorporating existing images and diagnostic reports, as well as multimodal clinical information such as the patient's current individual status (e.g., age, laboratory tests), to generate images at specific future moments. This enables a visual and intuitive prediction of disease progression, thereby helping doctors set personalized follow-up intervals and subsequent precise treatment plans.
[0020] To make the objectives, technical solutions, and advantages of this application clearer, the following description will be provided in conjunction with the accompanying drawings and specific embodiments.
[0021] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a medical image generation method provided in an embodiment of this application. The method can be executed by any electronic device and may include the following steps S101-S103: S101. Obtain multimodal medical data of the patient at a reference time. The multimodal medical data includes medical images and text data. The text data includes basic patient information and the time interval between the predicted time and the reference time. The basic patient information includes the patient's age information and the patient's diagnosis report. In some optional embodiments of this application, the reference time is the current time or a historical time; When the reference time is a historical time, the method further includes the following steps S1-S2: S1. Obtain the detection results of the detection device for the patient after the reference time; S2. Correct the target prediction image based on the detection results to obtain the corrected prediction image.
[0022] Alternatively, the detection device can be a portable detection device for the patient, used to detect medical parameters associated with the target predicted image, such as a blood pressure monitor, holter, etc.
[0023] Optionally, the aforementioned S2 can be implemented by a corresponding neural network model or by human operation of the electronic device; this application does not limit this.
[0024] The corrected predicted image can be used as the final target predicted image.
[0025] S102. Obtain preset noise data; Optionally, the distribution of the noise data itself can be preset to a normal distribution.
[0026] S103. Based on the preset target denoising unit, the multimodal medical data and the preset noise data are processed to obtain the target prediction image at the prediction time. The target denoising unit is the denoising unit in the diffusion model.
[0027] Optionally, the aforementioned target denoising unit is obtained by training the initial denoising unit.
[0028] In this application, the use of generative artificial intelligence technology to predict and generate medical images of changes in tissues, organs, or lesions can provide more reference for clinical diagnosis and treatment.
[0029] Figure 2This is a flowchart illustrating a medical image generation method provided in this application. In some optional embodiments of this application, the aforementioned target denoising unit includes a multi-layer U-Net neural network. The process of training the target denoising unit includes training each layer of the multi-layer U-Net neural network, where the U-Net neural network is a 3D U-Net network used for the denoising process. The training of the multi-layer U-Net neural network in the diffusion model can refer to related techniques, which will not be elaborated here.
[0030] In the aforementioned S103, the multimodal medical data and the preset noise data are processed based on a preset target denoising unit to obtain the target prediction image at the prediction time, including the following S1031-S1034: S1031. The medical image is encoded using a target image encoder to obtain encoded image features; Medical imaging can be understood as baseline imaging, specifically the medical examination images of a patient.
[0031] S1032. The text data is encoded using a target text encoder to obtain encoded text features; S1033. The target image and text feature processing unit performs feature extraction and feature concatenation on the encoded image features and the encoded text features to obtain the target concatenation result. S1034. The target stitching result and the preset noise data are processed by a preset target denoising unit to obtain the target prediction image at the prediction time. The above-mentioned target splicing results are fused with the features of each intermediate feature layer of each U-Net neural network through a cross-attention module to realize the denoising process of each U-Net neural network.
[0032] The aforementioned preset noise data can be obtained during the training of the target diffusion model, or it can be noise data with a normal distribution that is directly acquired. The size of the preset noise data can be consistent with the size of the encoded image features.
[0033] Further, see Figure 3 As shown, correspondingly, in some optional embodiments of this application, the method further includes the following steps S01-S09: S01. Obtain multimodal medical data of sample patients and corresponding sample prediction images. The multimodal medical data of sample patients includes: sample medical images and sample text data. The sample prediction image is a medical image at a second sample time that is a preset time interval away from the first sample time. The first sample time is the time corresponding to the sample medical image. Optionally, the first sample time is the time when the medical image of the sample was acquired.
[0034] Optionally, the collected follow-up images and report data can be organized. For data from only two follow-ups, it can be compiled into (I) t1 I t2 ) is used as a training sample for the generative model; where I t1 Medical data at the first sample time, I t2 This refers to sample data at the follow-up time. For data with three follow-up periods, it can be compiled into (I... t1 I t2 ), (I t1 I t3 ), (I t2 I t3 Three training samples are used; for data from more than three follow-ups, this process is repeated to complete the composition and organization of the training dataset. The latter data point in each sample pair corresponds to the gold standard for prediction based on the former data point.
[0035] S02. The sample prediction image is encoded using an image encoder to obtain encoded data; S03. The encoded data is processed by the noise introduction unit in the diffusion model to obtain the 3D noise data corresponding to the sample prediction image; Optionally, the noise introduction unit includes a multi-layer noise introduction unit, and the noise addition process of the diffusion model in the relevant art can be referred to in detail here.
[0036] S04. The sample medical image is encoded using the target image encoder to obtain the encoded sample image; Regarding conditional input, existing sample medical images have features extracted through a target image encoder, which can be Med CLIP or a pre-trained Transformer, both of which can extract visual features.
[0037] S05. The sample text data is encoded using the target text encoder to obtain the encoded sample text; The sample text data includes: basic patient information and preset time intervals. The basic patient information includes: the patient's age information and the patient's diagnosis report.
[0038] The text data in this application affects the morphology and growth rate of lesions.
[0039] The target text encoder can use parameters pre-trained with CLIP or MedCLIP.
[0040] S06. The target image and text feature processing unit performs feature extraction and feature concatenation on the encoded image and the encoded text to obtain the sample concatenation result. S07. The sample splicing result and the 3D noise data are processed by the initial denoising unit in the diffusion model to obtain the denoised data; S08. The denoised data is decoded by an image decoder to obtain the patient prediction image at the second sample time. S09. The initial denoising unit is trained based on the patient prediction image and the sample prediction image to obtain the target denoising unit.
[0041] Specifically, training the initial denoising unit based on the patient predicted image and the sample predicted image means training the initial denoising unit according to the principle of minimizing the difference between the patient predicted image and the sample predicted image.
[0042] Optionally, at least one of the image encoder, the image decoder, the target image encoder, the target text encoder, and the target image and text feature processing unit is obtained through training.
[0043] exist Figure 3 In this model, the main data processing component is a diffusion model based on a latent space combined with multiple input conditions.
[0044] First, the latent space can mitigate the computational overhead and slow generation speed of diffusion models, offering a computationally efficient solution for generating 3D medical images. The inverse diffusion process employs a 3D denoising U-Net structure. By processing volumetric data through 3D convolutional layers, it can capture the spatial context within the data, thus maintaining the consistency of the generated structure. Second, through 3D downsampling and upsampling layers, multi-scale feature fusion is achieved, enhancing the modeling ability of the overall morphology and local details of tissues, organs, or lesions. Finally, through cross-attention modules in the intermediate layers of U-Net, it supports various input conditions such as text and images, thereby generating controllable medical images.
[0045] This architecture enables the model to efficiently generate high-fidelity 3D medical images with reasonable anatomical structures. During the inference phase, only existing images, corresponding diagnostic reports, patient age, and generation time interval Δt need to be input. The trained model can then generate the medical images for the required time interval.
[0046] Further, the model training steps may include: (1) For 3D medical images or image blocks containing organs, tissues or lesions in the training set, the Z0 data of the 3D latent space is obtained after feature extraction processing by the 3D encoder; (2) The 3D noise data ZT of the latent space is obtained through the forward process of the 3D diffusion model; (3) Visual features are extracted from the reference image, and textual features are extracted from the information such as age, diagnosis report, and data interval of the generated image corresponding to the reference image, and the two features are spliced together; (4) The spliced features are fused with the intermediate feature layers of U-Net in the reverse denoising process of the 3D diffusion model through the cross-attention module, thereby generating the Z0 data in the 3D latent space; (5) Finally, the Z0 data of the latent space is decoded by the 3D decoder to obtain the generated image at a specific time point.
[0047] In some optional embodiments of this application, the reference time is a historical time, and the method further includes: acquiring the patient's symptom information; correcting the target prediction image based on the symptom information to obtain a corrected prediction image.
[0048] The patient's symptom information refers to the patient's symptoms within a preset time period after the target prediction effect has been determined. Correcting the target prediction image based on the patient's symptom information can improve the accuracy of the final target prediction image.
[0049] In some optional embodiments of this application, the step of correcting the target predicted image based on the symptom information to obtain a corrected predicted image includes: querying a reference patient matching the patient from a patient database based on the symptom information; obtaining medical image change information of the reference patient; and correcting the target predicted image based on the medical image change information to obtain a corrected predicted image.
[0050] Specifically, the target prediction image can be corrected by referring to the medical image change information of the reference patients obtained from existing actual tests, which can make the final target prediction image more accurate and realistic.
[0051] Optionally, the encoder in this application can be a 3D encoder, which can be a pre-trained visual model such as CNN or Transformer, such as ResNet, Swing Transformer, or a pre-trained image-text multimodal model such as CLIP; the decoder in this application can be a 3D decoder, which can be a transposed convolutional CNN or a Transformer decoder, etc.; the text encoder is a Transformer encoder structure such as LSTM or BERT.
[0052] In this application, the controllable generation technology based on the diffusion model learns from the multimodal information of multiple sets of images and reports from two time points. It can generate images at specific future time points based on existing images, diagnostic reports, patient age, and other information, thereby assisting doctors in making predictable assessments of changes in tissues and organs as well as changes in lesions themselves.
[0053] Corresponding to the medical image generation method in the above embodiments, Figure 4 A structural block diagram of a medical image generation apparatus provided in an embodiment of this application. The apparatus includes: The acquisition unit 41 is used to acquire the patient's multimodal medical data at a reference time. The multimodal medical data includes medical images and text data. The text data includes the patient's basic information and the time interval between the predicted time and the reference time. The patient's basic information includes the patient's age information and the patient's diagnosis report. The acquisition unit 41 is also used to acquire preset noise data; The generation unit 42 is used to process the multimodal medical data and the preset noise data based on the preset target denoising unit to obtain the target prediction image at the prediction time. The target denoising unit is the denoising unit in the diffusion model.
[0054] Optionally, when the generation unit processes the multimodal medical data and the preset noise data based on the preset target denoising unit to obtain the target prediction image at the prediction time, it is specifically used for: The medical image is encoded using a target image encoder to obtain encoded image features; The text data is encoded using a target text encoder to obtain encoded text features; The target image and text feature processing unit performs feature extraction and feature concatenation on the encoded image features and the encoded text features to obtain the target concatenation result. The target stitching result and the preset noise data are processed by a preset target denoising unit to obtain the target prediction image at the predicted time.
[0055] Optionally, the aforementioned device is also used for: Acquire multimodal medical data of sample patients and corresponding sample prediction images. The multimodal medical data of sample patients includes: sample medical images and sample text data. The sample prediction image is a medical image at a second sample time that is a preset time interval away from the first sample time. The first sample time is the time corresponding to the sample medical image. The sample prediction image is encoded using an image encoder to obtain encoded data; The encoded data is processed by the noise introduction unit in the diffusion model to obtain the 3D noise data corresponding to the sample prediction image; The sample medical image is encoded using the target image encoder to obtain the encoded sample image; The sample text data is encoded using the target text encoder to obtain the encoded sample text. The target image and text feature processing unit performs feature extraction and feature concatenation on the encoded image and the encoded text to obtain the sample concatenation result. The sample stitching result and the 3D noise data are processed by the initial denoising unit in the diffusion model to obtain denoised data; The denoised data is decoded by an image decoder to obtain the patient prediction image at the second sample time. The initial denoising unit is trained based on the patient's predicted image and the sample's predicted image to obtain the target denoising unit.
[0056] Optionally, at least one of the image encoder, the image decoder, the target image encoder, the target text encoder, and the target image and text feature processing unit is obtained through training.
[0057] Optionally, the reference time is the current time or a historical time; When the reference time is a historical time, the device is further used to: The detection results of the detection device for the patient were obtained after the reference time was acquired; The target prediction image is corrected based on the detection results to obtain the corrected prediction image.
[0058] Optionally, the reference time is a historical time, and the device is further used to: Obtain the patient's symptom information; The target prediction image is corrected based on the symptom information to obtain the corrected prediction image.
[0059] Optionally, when the device is used to correct the target prediction image based on the symptom information to obtain a corrected prediction image, it is specifically used for: Based on the symptom information, a reference patient matching the patient is retrieved from the patient database; Obtain the medical imaging changes of the reference patient; The target prediction image is corrected based on the medical image change information to obtain the corrected prediction image.
[0060] See Figure 5 , Figure 5 This is a schematic block diagram of an electronic device provided according to an embodiment of this application. Figure 5 The electronic device 400 in this embodiment may include one or more processors 401, one or more input devices 402, one or more output devices 403, and one or more memories 404. The processors 401, input devices 402, output devices 403, and memories 404 communicate with each other via a communication bus 405. The memories 404 store computer programs, including program instructions. The processors 401 execute the program instructions stored in the memories 404. The processors 401 are configured to invoke the program instructions to perform the functions of the units in the above-described device embodiments.
[0061] It should be understood that, in the embodiments of this application, the processor 401 may be a central processing unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.
[0062] Input device 402 may include a touchpad, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint orientation information), a microphone, etc., and output device 403 may include a display (LCD, etc.), a speaker, etc.
[0063] The memory 404 may include read-only memory and random access memory, and provides instructions and data to the processor 401. A portion of the memory 404 may also include non-volatile random access memory.
[0064] In specific implementations, the processor 401, input device 402, and output device 403 described in the embodiments of this application can execute the implementation methods described in the embodiments of this application, or they can execute the implementation methods of the electronic devices described in the embodiments of this application, which will not be repeated here.
[0065] In another embodiment of this application, a computer-readable storage medium is provided. This computer-readable storage medium stores a computer program, which includes program instructions. When executed by a processor, the program instructions implement all or part of the processes in the methods described above. Alternatively, the computer program can instruct related hardware to complete the process. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include any entity or device capable of carrying computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.
[0066] The computer-readable storage medium can be an internal storage unit of the electronic device in any of the foregoing embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device. Furthermore, the computer-readable storage medium can include both internal and external storage units of the electronic device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.
[0067] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.
[0068] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the electronic devices and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0069] In the several embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces or units, or it may be an electrical, mechanical, or other form of connection.
[0070] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.
[0071] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0072] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A method for generating medical images, characterized in that, include: Acquire multimodal medical data of the patient at a reference time. The multimodal medical data includes medical images and text data. The text data includes basic patient information and the time interval between the predicted time and the reference time. The basic patient information includes the patient's age information and the patient's diagnosis report. Obtain preset noise data; The multimodal medical data and the preset noise data are processed based on a preset target denoising unit to obtain the target prediction image at the prediction time. The target denoising unit is a denoising unit in the diffusion model.
2. The method as described in claim 1, characterized in that, The preset target denoising unit processes the multimodal medical data and the preset noise data to obtain the target prediction image at the predicted time, including: The medical image is encoded using a target image encoder to obtain encoded image features; The text data is encoded using a target text encoder to obtain encoded text features; The target image and text feature processing unit performs feature extraction and feature concatenation on the encoded image features and the encoded text features to obtain the target concatenation result. The target stitching result and the preset noise data are processed by a preset target denoising unit to obtain the target prediction image at the predicted time.
3. The method as described in claim 2, characterized in that, The method further includes: Acquire multimodal medical data of sample patients and corresponding sample prediction images. The multimodal medical data of sample patients includes: sample medical images and sample text data. The sample prediction image is a medical image at a second sample time that is a preset time interval away from the first sample time. The first sample time is the time corresponding to the sample medical image. The sample prediction image is encoded using an image encoder to obtain encoded data; The encoded data is processed by the noise introduction unit in the diffusion model to obtain the 3D noise data corresponding to the sample prediction image; The sample medical image is encoded using the target image encoder to obtain the encoded sample image; The sample text data is encoded using the target text encoder to obtain the encoded sample text. The target image and text feature processing unit performs feature extraction and feature concatenation on the encoded image and the encoded text to obtain the sample concatenation result. The sample stitching result and the 3D noise data are processed by the initial denoising unit in the diffusion model to obtain denoised data; The denoised data is decoded by an image decoder to obtain the patient prediction image at the second sample time. The initial denoising unit is trained based on the patient's predicted image and the sample's predicted image to obtain the target denoising unit.
4. The method as described in claim 3, characterized in that, At least one of the image encoder, the image decoder, the target image encoder, the target text encoder, and the target image and text feature processing unit is obtained through training.
5. The method as described in claim 1, characterized in that, The reference time can be the current time or a historical time. When the reference time is a historical time, the method further includes: The detection results of the detection device for the patient were obtained after the reference time was acquired; The target prediction image is corrected based on the detection results to obtain the corrected prediction image.
6. The method as described in claim 1, characterized in that, The reference time is a historical time, and the method further includes: Obtain the patient's symptom information; The target prediction image is corrected based on the symptom information to obtain the corrected prediction image.
7. The method as described in claim 6, characterized in that, The step of correcting the target prediction image based on the symptom information to obtain the corrected prediction image includes: Based on the symptom information, a reference patient matching the patient is retrieved from the patient database; Obtain the medical imaging changes of the reference patient; The target prediction image is corrected based on the medical image change information to obtain the corrected prediction image.
8. A medical image generation device, characterized in that, include: The acquisition unit is used to acquire the patient's multimodal medical data at a reference time. The multimodal medical data includes medical images and text data. The text data includes the patient's basic information and the time interval between the predicted time and the reference time. The patient's basic information includes the patient's age information and the patient's diagnosis report. The acquisition unit is also used to acquire preset noise data; The generation unit is used to process the multimodal medical data and the preset noise data based on the preset target denoising unit to obtain the target prediction image at the prediction time. The target denoising unit is the denoising unit in the diffusion model.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.