Text and image multi-modal based brain lesion image generation method and system

Abstract

The application discloses a kind of brain lesion image generation method and system based on text and image multimodal, comprising: data collection module: collect relevant brain lesion public data set, with text template for each image, form text-image template data set;Fine-tuning module: by image and text are encoded respectively, convert into a common embedding space, then calculate similarity and match.Data augmentation module: using fine-tuned DALLE2 model according to the text requirement of need carries out image generation and variant generation.Label image generation module: using the model of cyclic fine-tuning to target image editing, provide mask specified editing area, text description is generated target lesion. To generate lesion image with label.The application has the advantages that: it can process image and text information simultaneously, generate more realistic, accurate and diverse brain lesion images, realize unsupervised brain lesion segmentation task, and be more efficient, accurate and diversified.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence-generated medical image technology, and in particular to a method and system for generating multimodal brain lesion images based on deep learning, using text and images. Background Technology

[0002] Brain lesion images refer to the images of abnormal lesions discovered through imaging techniques (such as magnetic resonance imaging and computed tomography) on the human brain. These images can be used by doctors to diagnose and treat brain diseases. Common brain lesions include cerebral hemorrhage, cerebral infarction, and tumors. The application of computer-aided diagnostics (CAD) and artificial intelligence technologies can more accurately identify and locate brain lesions, providing important references for doctors to formulate treatment plans.

[0003] Existing technologies have proposed deep learning-based models for generating images of brain diseases, such as GANs. [1][2]

[0004] (Generative Adversarial Networks, etc.) These models typically generate brain lesion images based on a single data source, such as brain MRI images. However, these methods have several drawbacks. First, they cannot utilize other available data sources, such as lesion description text, to improve the accuracy and realism of the generated images, nor can they add prior information through text descriptions. Second, these methods usually require large amounts of training data, which are often difficult to obtain, especially for rare brain diseases where data acquisition is even more challenging. Furthermore, the hyperparameters of GAN models are highly sensitive, training is unstable, and their versatility in generating images needs improvement.

[0005] In addition, existing technologies have proposed multimodal image generation models based on GANs. [3] Examples of such models include text-to-image generation (CTG) models. These models can combine text and image information to generate images. However, these models are generally not well-suited for generating images of brain diseases because the pathological features of brain diseases differ from those of other objects, requiring specialized models to handle more detailed textures. Most importantly, these generative models cannot simultaneously provide labels for the generated image and the edited lesion regions, thus hindering the creation of labeled image pairs for training in segmentation tasks.

[0006] As can be seen from the above, existing models are typically based on a single data source, making it difficult to simultaneously utilize multimodal information from text and images to generate realistic and accurate images of brain lesions. Furthermore, existing technologies require substantial training data and computational resources to train the models, and are not suitable for the specific needs of the medical imaging field. Simply generating images without creating labeled regions for edited areas (e.g., added lesions) makes it impossible to pair images with labels for training the segmentation model.

[0007] [1]. Han, Changhee, et al. "GAN-based synthetic brain MR image generation." 2018IEEE 15th international symposium on biomedical imaging (ISBI2018). IEEE, 2018.

[0008] [2]. Huang, Yawen, et al. "MCMT-GAN: Multi-task coherent modalitytransferable GAN for 3D brain image synthesis." IEEE Transactions on ImageProcessing 29(2020):8187-8198.

[0009] [3]. Zhu, Minfeng, et al. "Dm-gan: Dynamic memory generative adversarial networks for text-to-image synthesis." Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2019. Summary of the Invention

[0010] This invention addresses the shortcomings of existing technologies by providing a method and system for generating brain lesion images based on text and image multimodal processing. It can simultaneously process image and text information. It can utilize both text and image multimodal information, along with specified editing regions, to generate more realistic and accurate labeled brain lesion images, while reducing the need for training data and computational resources.

[0011] To achieve the above-mentioned objectives, the technical solution adopted by the present invention is as follows:

[0012] A method for generating brain lesion images based on text and image multimodal methods includes the following steps:

[0013] Step 1: Collect publicly available datasets of relevant brain lesions and assign a text template to each image. Fill in the reserved positions in the text template according to each image sample to form a text-image template dataset.

[0014] Step 2: Use the paired text-image template dataset to fine-tune the CLIP module of DALLE2 to match the text feature space of the image.

[0015] Step 3: Using the fine-tuned DALLE2 model, image and variant generation are performed based on the required text. The CLIP module encodes the required text and maps it to a text feature space. The text feature space is then transformed into an image feature space through a prior model. The image features are then decoded to generate images with the required semantics. The generated images serve as supplementary data, further satisfying the DALLE2 model's data volume requirements on top of a limited public dataset.

[0016] Step 4: Combine the paired image text generated in S3 with the original text-image template dataset from S1 and fine-tune the model again in a loop.

[0017] Step 5: Use the finely tuned model to edit the target image. Provide a mask to specify the area of ​​the image to be edited, and input text descriptions to generate target lesions, thereby generating labeled lesion images for subsequent segmentation tasks.

[0018] Furthermore, the fine-tuning described in S2 specifically involves: loading a weighted model pre-trained from large text-image pairs as an initialization model, and continuing to train this initialization model using locally generated text-image data based on brain lesion images, thereby completing the fine-tuning of the model weights.

[0019] Furthermore, the cyclic fine-tuning in S4 specifically involves using the preliminary fine-tuned model from S2 as the initial model, and then using the augmented data generated in S3 together with the original data to train the model again, thus completing the secondary fine-tuning of the model parameters.

[0020] This invention discloses a brain lesion image generation system, which can be used to implement the above-mentioned brain lesion image generation method. Specifically, it includes: a data collection module, a fine-tuning module, a data expansion module, and an annotated image generation module.

[0021] Data collection module: used to collect publicly available datasets of relevant brain lesions and to assign a text template to each image, forming a text-image template dataset;

[0022] Fine-tuning module: It encodes images and text separately, that is, converts them into a common embedding space, and then performs matching by calculating the similarity between them.

[0023] Data augmentation module: Uses the fine-tuned DALLE2 model to generate images and variants according to the text requirements.

[0024] The labeled image generation module edits the target image using a finely tuned model, providing a mask to specify the editing area and text descriptions to generate labeled lesion images.

[0025] The present invention also discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method for generating brain lesion images.

[0026] Compared with the prior art, the advantages of the present invention are as follows:

[0027] 1. It leverages the advantages of large-scale multimodal models (such as CLIP, DALLE2, etc.) to process image and text information simultaneously, and can use prior descriptions of language to guide image generation as needed.

[0028] 2. It can generate more realistic, accurate and diverse brain lesion images, and realize the generation of unlabeled lesion segmentation image pairs, providing better tools and support for medical diagnosis and research.

[0029] 3. Better adapted to brain lesion image generation tasks.

[0030] 4. Employ the state-of-the-art diffusion generative model (DALLE2) to generate highly realistic and diverse brain lesion images and corresponding lesion labels. Use the generated image label pairs to train the individual models, achieving unsupervised brain lesion segmentation tasks.

[0031] 5. By using the large-scale multimodal model DALLE2 and the state-of-the-art diffusion generation model, this invention can generate brain lesion image annotation pairs more efficiently, accurately and diversely, thereby providing better medical lesion segmentation diagnosis and research support. Attached Figure Description

[0032] Figure 1 This is a flowchart of the brain lesion image generation method according to an embodiment of the present invention.

[0033] Figure 2 This is a flowchart of the fine-tuning process of an embodiment of the present invention.

[0034] Figure 3 This is a flowchart illustrating the generation of labeled lesion images according to an embodiment of the present invention. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and examples.

[0036] like Figure 1 As shown, a method for generating brain lesion images based on text and image multimodal methods includes the following steps:

[0037] Step 1: Collect relevant publicly available datasets of brain lesions, such as the BraTS brain tumor segmentation dataset, the ATLAS old stroke segmentation dataset, the ISLES ischemic stroke dataset, and the MSSEG multiple sclerosis lesion segmentation dataset. Assign a text template to each image, for example: [Stroke]with[hyper]intensity on[DWI]modality. Fill in the [] positions in the text template according to each image sample to form a text-image template dataset.

[0038] Step 2: The text-image template dataset generated by the pairing is used to fine-tune the CLIP module of DALLE2 to match the text-image feature space. Specifically, a weighted model pre-trained with large text-image pairs is loaded as the initial model, and locally generated text-image data based on brain lesion images is used to continue training on this initial model, thereby completing the fine-tuning of the model weights.

[0039] Step 3: The fine-tuned DALLE2 model is used to generate images and variants based on the required text. Specifically, the CLIP module encodes the required text and maps it to a text feature space. The text feature space is then transformed into an image feature space through a prior model. The image features are then decoded to generate images with the required semantics. DALLE2's generation mode is based on a diffusion model; therefore, for the same model with fixed weight parameters, different initial noise levels will result in different generated data. Thus, the generated images serve as a form of data augmentation, further satisfying the DALLE2 model's data volume requirements on top of a limited public dataset.

[0040] Step 4: Use the paired image-text generated in S3 together with the original text-image template dataset from S1 to fine-tune the model again. Specifically, use the preliminary fine-tuned model from S2 as the initial model in this step. That is, based on the parameter weights of this model, use the augmented data generated in S3 together with the original data to train the model again, completing the second fine-tuning of the model parameters.

[0041] Step 5: Use the finely tuned model to edit the target image. A mask can be provided to specify the editing area, along with text descriptions to generate the target lesions. This produces an labeled lesion image that can be used for subsequent segmentation tasks.

[0042] Specifically, in step 1, the fine-tuning details in the CLIP module and the diffusion model are as follows: Figure 2 As shown, by encoding both images and text separately, that is, converting them into a common embedding space, matching is performed by calculating the similarity between them.

[0043] In step 5, as Figure 3 As shown, input text ( Figure 3 Mark ②) as "A tumor at Axial view with hyper-intensity" and specify the input image for editing ( Figure 3 The region near the ventricles (marked ①) is a binary mask. Figure 3 The annotation pointed to by the arrow in marker ③ (generated by 3D graphics tools) can then generate the final image with tumor markers (the image with lesions is marked ③, and the lesion is labeled as...). Figure 3 The label pointed to by the arrow in mark ③.

[0044] Figure 3 As can be seen, for the same input image, different language descriptions and specified editing regions can generate lesion images that meet the requirements. All features and details of the original image are preserved outside the editing regions. The generated sample data and labeled regions can be used for subsequent segmentation tasks, greatly reducing and avoiding dependence on labeled data.

[0045] In another embodiment of the present invention, a brain lesion image generation system is provided. This system can be used to implement the above-described brain lesion image generation method. Specifically, it includes: a data collection module, a fine-tuning module, a data expansion module, and an annotated image generation module.

[0046] Data collection module: used to collect publicly available datasets of relevant brain lesions and to assign a text template to each image, forming a text-image template dataset;

[0047] Fine-tuning module: It encodes images and text separately, that is, converts them into a common embedding space, and then performs matching by calculating the similarity between them.

[0048] Data augmentation module: Uses the fine-tuned DALLE2 model to generate images and variants according to the text requirements.

[0049] The labeled image generation module edits the target image using a finely tuned model, providing a mask to specify the editing area and text descriptions to generate labeled lesion images.

[0050] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory). This computer-readable storage medium is a memory device in a terminal device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and extended storage media supported by the terminal device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, this storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device.

[0051] One or more instructions stored in a computer-readable storage medium can be loaded and executed by a processor to implement the corresponding steps of the brain lesion image generation method in the above embodiments; one or more instructions in the computer-readable storage medium are loaded and executed by the processor in the following steps:

[0052] Step 1: Collect publicly available datasets of relevant brain lesions and assign a text template to each image. Fill in the reserved positions in the text template according to each image sample to form a text-image template dataset.

[0053] Step 2: Use the paired text-image template dataset to fine-tune the CLIP module of DALLE2 to match the text feature space of the image.

[0054] Step 3: Using the fine-tuned DALLE2 model, image and variant generation are performed based on the required text. The CLIP module encodes the required text and maps it to a text feature space. The text feature space is then transformed into an image feature space through a prior model. The image features are then decoded to generate images with the required semantics. The generated images serve as supplementary data, further satisfying the DALLE2 model's data volume requirements on top of a limited public dataset.

[0055] Step 4: Combine the paired image text generated in S3 with the original text-image template dataset from S1 and fine-tune the model again in a loop.

[0056] Step 5: Use the finely tuned model to edit the target image. Provide a mask to specify the area of ​​the image to be edited, and input text descriptions to generate target lesions, thereby generating labeled lesion images for subsequent segmentation tasks.

[0057] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0058] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0059] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0060] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0061] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the implementation methods of the present invention, and should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of the present invention.

Claims

1. A method for generating brain lesion images based on text and image multimodal methods, characterized in that: Includes the following steps: Step 1: Collect publicly available datasets of relevant brain lesions and assign a text template to each image. Fill in the reserved positions in the text template according to each image sample to form a text-image template dataset. Step 2: Use the paired text-image template dataset to fine-tune the CLIP module of DALLE2 to match the text feature space of the image; The fine-tuning specifically involves: loading a weighted model pre-trained from large text-image pairs as an initialization model, and continuing to train this initialization model using locally generated text-image data based on brain lesion images, thereby completing the fine-tuning of the model weights; Step 3: Use the fine-tuned DALLE2 model to generate images and variants according to the text requirements; the CLIP module encodes the text requirements and maps them to the text feature space, which is then transformed into the image feature space through the prior model. The image features are then decoded to generate images with the semantics of the requirements; the generated images serve as supplementary data to further meet the data volume requirements of the DALLE2 model on the basis of the limited public dataset. Step 4: Combine the paired image text generated in S3 with the original text-image template dataset from S1 and fine-tune the model again in a loop; Step 5: Use the finely tuned model to edit the target image. Provide a mask to specify the area of ​​the image to be edited, and input text descriptions to generate target lesions, thereby generating labeled lesion images for subsequent segmentation tasks.

2. The method for generating brain lesion images based on text and image multimodal processing according to claim 1, characterized in that: The specific process of fine-tuning in S4 is as follows: the preliminary fine-tuned model in S2 is used as the initial model, and the model is trained again using the data generated in S3 and the original data to complete the second fine-tuning of the model parameters.

3. A brain lesion image generation system, characterized in that: This system can be used to implement the brain lesion image generation method according to claim 1 or 2; The brain lesion image generation system includes: a data collection module, a fine-tuning module, a data expansion module, and an annotated image generation module; Data collection module: used to collect publicly available datasets of relevant brain lesions and to assign a text template to each image, forming a text-image template dataset; Fine-tuning module: It encodes images and text separately, that is, converts them into a common embedding space, and then performs matching by calculating the similarity between them; Data augmentation module: Uses the fine-tuned DALLE2 model to generate images and variants according to the required text; The labeled image generation module edits the target image using a finely tuned model, provides a mask to specify the editing area, and text descriptions to generate the target lesion; thus generating a labeled lesion image.

4. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the brain lesion image generation method of claim 1 or 2.

Images