A tumor segmentation method and system based on multimodal generative adversarial network
Through the method of generating adversarial networks based on multimodality, the limitations of traditional tumor segmentation methods are solved when processing multimodal data, and the precise segmentation of multimodal tumor image data is realized, which improves the accuracy and medicine of tumor segmentation. Efficiency of image analysis.
Patent Information
- Application Number
- CN202311148615.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-07
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2043-09-07
AI Technical Summary
Traditional tumor segmentation methods have limitations in processing multimodal medical imaging data, and cannot fully mine and integrate information in different modal data, resulting in inconsistent and inaccurate segmentation results, which limits the reliability of radiotherapy physicians to formulate treatment plans and Accuracy.
Using the tumor segmentation method based on multimodal generative adversarial network, the tumor segmentation generation adversarial network is generated through adversarial training, and the tumor segmentation generation adversarial network is built using standard multimodal tumor image data sets to obtain tumor image segmentation model to realize multimodal tumor image Accurate segmentation of data.
It improves the accuracy and credibility of tumor segmentation, provides more accurate tumor segmentation results, helps doctors formulate personalized radiotherapy plans, and improves the efficiency of medical imaging analysis.
Smart Images

Figure CN117115185B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of medical image analysis, and in particular to a tumor segmentation method and system based on a multimodal generative adversarial network. Background Art
[0002] In today's medical field, tumor segmentation is not only a critical task, but also an indispensable part of medical practice. Especially in the radiotherapy department, accurate tumor segmentation is of decisive significance for the formulation of personalized radiotherapy plans. At present, tumor imaging technology has achieved the acquisition of multimodal data, and each modality provides unique anatomical and physiological information in specific aspects. For example, CT images can clearly depict the density distribution of tissues, MRI images highlight the contrast of soft tissues, and PET images reveal cell metabolic activity. However, traditional tumor segmentation methods have limitations in processing multimodal data. Due to the complex relationship and differences between different modal data, traditional methods are often unable to fully mine and integrate this valuable information, which leads to inconsistency and inaccuracy in segmentation results, limiting the reliability and accuracy of radiotherapy doctors in formulating treatment plans. Summary of the invention
[0003] In view of the shortcomings of current technology, in the first aspect, the present invention provides a tumor segmentation method based on a multimodal generative adversarial network, aiming to break through the limitations of traditional tumor segmentation methods in processing multimodal medical imaging data. The tumor segmentation method based on a multimodal generative adversarial network includes the following steps: obtaining a standard multimodal tumor image dataset and building a tumor segmentation generative adversarial network; using the standard multimodal tumor image dataset, adversarially training the tumor segmentation generative adversarial network; obtaining a tumor image segmentation model through the trained tumor segmentation generative adversarial network; obtaining multimodal tumor image data of the target patient at the same site; using the tumor image segmentation model, segmenting a multimodal tumor segmentation mask image from the multimodal tumor image data; fusing the multimodal tumor segmentation mask image to obtain the tumor segmentation result image of the target patient. The present invention uses a standard multimodal tumor image dataset to train the tumor segmentation generative adversarial network with an adversarial training method, thereby obtaining a tumor image segmentation model through the trained tumor segmentation generative adversarial network, and then using the tumor image segmentation model to achieve accurate tumor segmentation in the field of medical imaging, so as to improve the efficiency of medical image analysis and the level of personalized treatment.
[0004] Optionally, the standard multimodal tumor image dataset includes a CT image dataset, an MRI image dataset and a PET image dataset.
[0005] Optionally, the tumor segmentation generative adversarial network includes a tumor image generator and a tumor image discriminator; the tumor image generator includes a generator input layer, a generator encoding network, a generator decoding network and a generator output layer in sequence; the tumor image discriminator includes a discriminator input layer, a discriminator feature extraction layer, a discriminator Flatten layer, a discriminator fully connected layer, and a discriminator support vector machine layer in sequence. This option proposes a specific structure of a tumor segmentation generative adversarial network, which enables the tumor segmentation generative adversarial network to more effectively learn the features of multimodal tumor images through the multi-layer synergy between the tumor image generator and the tumor image discriminator.
[0006] Optionally, the generator encoding network includes a first convolution layer, a second convolution layer, a third convolution layer, a maximum pooling layer, a fifth convolution layer and a sixth convolution layer in sequence; wherein the first convolution layer includes 128 convolution kernels of size 3×3, a stride of 1, and an activation function of Leaky ReLU; the second convolution layer includes 256 convolution kernels of size 3×3, a stride of 1, and an activation function of Leaky ReLU; the third convolution layer includes 512 convolution kernels of size 3×3, a stride of 1, and an activation function of Leaky ReLU; the pooling size of the maximum pooling layer is 2x2; the fifth convolution layer includes 512 convolution kernels of size 3×3, a stride of 1, and an activation function of Leaky ReLU; the sixth convolution layer includes 512 convolution kernels of size 3×3, a stride of 1, and an activation function of Leaky ReLU. This option further optimizes the architecture of the generator encoding network by setting the number and size of different convolution kernels and the Leaky ReLU activation function, and further compressing the feature dimension through the maximum pooling layer, so that the generator encoding network can capture image information at different levels. Such a generator encoding network architecture design enhances the representation ability of the tumor image generator and improves the tumor segmentation accuracy of the corresponding tumor image segmentation model.
[0007] Optionally, the use of the standard multimodal tumor image dataset to adversarially train the tumor segmentation generative adversarial network includes the following steps: fixing the parameters of the tumor image discriminator, inputting the unlabeled standard multimodal tumor image dataset into the tumor image generator, and using the tumor image generator to generate a corresponding tumor segmentation mask image; inputting the tumor segmentation mask image and the labeled standard multimodal tumor image dataset into the tumor image discriminator, and adjusting the parameters of the tumor image discriminator and the tumor image generator respectively according to the input result of the tumor image discriminator, and the adjustment method includes the following steps: if the output result of the tumor image discriminator is false, fixing the tumor image discriminator; The parameters of the image discriminator are updated, and the parameters of the tumor image generator are updated; the output of the updated tumor image generator is used as the input of the tumor image discriminator; if the output result of the tumor image discriminator is still false, the parameters of the tumor image discriminator are fixed, and the parameters of the tumor image generator are updated until the output result is judged to be true by the tumor image discriminator; if the output result of the tumor image discriminator is true, the parameters of the tumor image generator are fixed, and the parameters of the tumor image discriminator are updated; the input of the tumor image generator whose output result is true before is judged again by the updated tumor image discriminator, and if the output result of the tumor image discriminator is still true, the training is stopped. This option fine-tunes the parameters of the tumor image generator and the tumor image discriminator through adversarial training, so that the tumor image discriminator can more effectively distinguish between real and generated images, and the tumor image generator can generate more accurate tumor segmentation mask images. Such an alternating training process optimizes the stability and performance of the model, and improves the accuracy and quality of tumor image segmentation by the tumor image segmentation model corresponding to the tumor image generator.
[0008] Optionally, the tumor image segmentation model is a tumor image generator in a trained tumor segmentation generative adversarial network. This option uses the tumor image generator obtained through adversarial training as the tumor image segmentation model to achieve subsequent accurate segmentation of multimodal tumor imaging data. The optimized parameters of the tumor image generator can reconstruct the tumor segmentation mask with high accuracy, providing a reliable basis for medical diagnosis and treatment, and helping doctors to formulate more accurate radiotherapy plans.
[0009] Optionally, obtaining multimodal tumor imaging data of the target patient at the same site includes the following steps: determining the modality type of the target patient's multimodal tumor imaging data; obtaining tumor imaging data of different modalities at the same site of the target patient according to the modality type; processing tumor imaging data of different modalities to obtain multimodal tumor imaging data of the target patient. This option helps to comprehensively utilize tumor imaging data of different modalities to obtain more comprehensive and accurate multimodal tumor information for the target patient; at the same time, this option improves the accuracy and credibility of tumor segmentation by processing and fusing data of different modalities, providing a more reliable basis for doctors to formulate personalized treatment plans.
[0010] Optionally, the use of the tumor image segmentation model to segment a multimodal tumor segmentation mask image from the multimodal tumor image data includes the following steps: inputting tumor images of different modalities in the multimodal tumor image data into the tumor image segmentation model; using the tumor image segmentation model to obtain tumor segmentation mask images of different modalities; and aggregating tumor segmentation mask images of different modalities to obtain a multimodal tumor segmentation mask image of the target patient. This option uses the tumor image segmentation model to perform modality-by-modality segmentation on the multimodal tumor image data, and then fuses the segmentation results to produce a more comprehensive and accurate multimodal tumor segmentation mask image. This helps doctors analyze tumor information of different modalities more carefully and supports more accurate diagnosis and treatment decisions.
[0011] Optionally, the fusion of the multimodal tumor segmentation mask image to obtain the tumor segmentation result image of the target patient includes the following steps: binarizing the multimodal tumor segmentation mask image at the same position; obtaining a two-dimensional matrix of multimodal tumor image pixels according to the binarization result; superimposing the two-dimensional matrix of multimodal tumor image pixels; and obtaining the tumor segmentation result image of the target patient according to the superposition result. This option obtains a comprehensive tumor segmentation result image by binarizing, superimposing and processing the multimodal tumor segmentation mask image. This comprehensive image helps to provide more comprehensive tumor segmentation information and provides a more reliable basis for doctors to formulate treatment plans and evaluate disease progression.
[0012] Optionally, the two-dimensional matrix of pixels of the tumor segmentation result image satisfies the following model: I t =[a ij ] t , a ij = 0 or 1, where K represents the pixel two-dimensional matrix of the tumor segmentation result image, N represents the number of modalities, and I t represents the two-dimensional matrix of tumor image pixels corresponding to the t-th modality, a ijRepresents the pixel value of the i-th row and j-th column in the two-dimensional matrix of tumor image pixels. This option defines the mathematical model of the tumor segmentation result image pixel matrix, so that tumor images of different modalities can be effectively fused and represented. The mathematical model provides a clear way to combine multimodal information, thereby producing more accurate and comprehensive tumor segmentation results, providing doctors with more valuable clinical decision support.
[0013] In the second aspect, in order to better implement the above-mentioned tumor segmentation method based on the multimodal generative adversarial network, the present invention also provides a tumor segmentation system based on the multimodal generative adversarial network. The tumor segmentation system based on the multimodal generative adversarial network includes an input device, a processor, a memory and an output device, wherein the input device, the processor, the memory and the output device are interconnected, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions to execute the tumor segmentation method based on the multimodal generative adversarial network described in the first aspect of the present invention. The tumor segmentation system based on the multimodal generative adversarial network provided by the present invention provides an efficient and integrated tool for medical professionals. Through the multimodal generative adversarial network technology, tumor segmentation can be automatically performed, thereby reducing the workload of doctors and improving the accuracy and reliability of tumor diagnosis. At the same time, the tumor segmentation system based on the multimodal generative adversarial network provided by the present invention has a clear structure and is easy to operate, which provides effective support for tumor segmentation tasks in medical practice. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Figure 1 A flow chart of a tumor segmentation method based on a multimodal generative adversarial network provided in an embodiment of the present invention;
[0015] Figure 2 A schematic diagram of the structure of a tumor segmentation system based on a multimodal generative adversarial network provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0016] The specific embodiments of the present invention will be described in detail below. It should be noted that the embodiments described herein are only for illustration and are not intended to limit the present invention. In the following description, a large number of specific details are set forth in order to provide a thorough understanding of the present invention. However, it is obvious to those of ordinary skill in the art that these specific details do not need to be adopted to implement the present invention. In other examples, in order to avoid confusing the present invention, known circuits, software or methods are not specifically described.
[0017] Throughout the specification, references to "one embodiment," "an embodiment," "an example," or "an example" mean that a particular feature, structure, or characteristic described in conjunction with the embodiment or example is included in at least one embodiment of the present invention. Therefore, the phrases "in one embodiment," "in an embodiment," "an example," or "an example" appearing in various places throughout the specification do not necessarily all refer to the same embodiment or example. In addition, particular features, structures, or characteristics may be combined in one or more embodiments or examples in any suitable combination and / or subcombination. In addition, it should be understood by those of ordinary skill in the art that the figures provided herein are for illustrative purposes and that the figures are not necessarily drawn to scale.
[0018] In an alternative embodiment, see Figure 1 , Figure 1 The flowchart of the tumor segmentation method based on the multimodal generative adversarial network provided by the embodiment of the present invention is as follows. Figure 1 As shown, the tumor segmentation method based on the multimodal generative adversarial network includes the following steps:
[0019] S01. Obtain a standard multimodal tumor imaging dataset and build a tumor segmentation generative adversarial network.
[0020] The standard multimodal tumor image dataset described in the present invention refers to a set of tumor image datasets containing multiple modalities, which are used to train and evaluate tumor segmentation generative adversarial networks.
[0021] Furthermore, multimodal tumor imaging data refers to data with different modal influences obtained using tumor imaging devices of different modalities, such as CT (Computed Tomography) images, MRI (Magnetic Resonance Imaging) images, and PET (Positron Emission Tomography) images. Each modality provides a specific type of information that can complement each other in the segmentation task to improve accuracy.
[0022] Furthermore, the acquisition of standard multimodal tumor imaging datasets can be collected from existing tumor imaging databases or obtained through image scanning of corresponding modalities in cooperation with medical institutions. It should be noted that in the process of obtaining standard multimodal tumor imaging datasets, it is necessary to protect patient privacy and comply with relevant laws and regulations to ensure the security and legality of the data.
[0023] In an optional embodiment, the tumor segmentation generative adversarial network constructed in step S01 includes a tumor image generator and a tumor image discriminator.
[0024] In this embodiment, a variational autoencoder (VAE) generation model is used to achieve more fine-grained tumor segmentation through encoding and decoding. Specifically, the constructed tumor image generator includes a generator input layer, a generator encoding network, a generator decoding network, and a generator output layer.
[0025] The generator input layer is used to input a multimodal tumor image dataset, specifically including a standard multimodal tumor image dataset input during training, and multimodal tumor image data of a target patient.
[0026] Furthermore, the generator encoding network includes the first convolution layer, the second convolution layer, the third convolution layer, the maximum pooling layer, the fifth convolution layer, and the sixth convolution layer in sequence. The input of the first convolution layer is connected to the output of the output layer, and the first convolution layer includes 128 convolution kernels of size 3×3, a stride of 1, and an activation function of Leaky ReLU; the first convolution layer performs preliminary feature extraction on the input multimodal tumor imaging data to capture basic image structure and texture information. The second convolution layer includes 256 convolution kernels of size 3×3, a stride of 1, and an activation function of Leaky ReLU; the second convolution layer is further convolutionally operated, which helps to extract higher-level features, such as more complex tissue structures and edges. The third convolution layer includes 512 convolution kernels of size 3×3, a stride of 1, and an activation function of LeakyReLU; the third convolution layer further deepens the abstract level of features through further convolution operations and captures more abstract image features, such as organ shape and distribution. The pooling size of the maximum pooling layer is 2x2. The addition of the maximum pooling layer helps to reduce the size of the feature map, extract the main features, and retain key information. The fifth convolutional layer includes 512 convolution kernels of size 3×3, a stride of 1, and an activation function of Leaky ReLU; the fifth convolutional layer further enhances the representation ability of the features through convolution operations and captures higher-level abstract features. The sixth convolutional layer includes 512 convolution kernels of size 3×3, a stride of 1, and an activation function of Leaky ReLU; the sixth convolutional layer continues to deepen the abstract level of the features through convolution operations and captures richer image information.
[0027] Furthermore, the generator decoding network includes a first deconvolution layer, a second deconvolution layer, a third deconvolution layer, a maximum depooling layer, a fifth deconvolution layer, and a sixth deconvolution layer in sequence; wherein the input of the first deconvolution layer is connected to the output of the sixth convolution layer, the first deconvolution layer includes 512 deconvolution kernels of size 3×3, a stride of 1, and an activation function of ReLU; the deconvolution operation of the first deconvolution layer is used to remap the abstract features to the image space and gradually restore the image details. The second deconvolution layer includes 256 deconvolution kernels of size 3×3, a stride of 1, and an activation function of ReLU; the deconvolution operation of the second deconvolution layer further restores the resolution of the feature map, which helps to restore more image details. The third deconvolution layer includes 256 deconvolution kernels of size 3×3, a stride of 1, and an activation function of ReLU; the third deconvolution layer further enhances the feature representation of the image based on the continuous deconvolution operations of the first deconvolution layer and the second deconvolution layer, and prepares for higher-level feature restoration. The pooling size of the maximum de-pooling layer is 2x2; maximum de-pooling is used to further expand the dimension of the feature map and restore image details. The fifth deconvolution layer includes 128 deconvolution kernels of size 3×3, stride 1, and activation function ReLU; further deconvolution operations in the fifth deconvolution layer help to restore more detail information and prepare for the generation of more refined segmentation masks. The sixth deconvolution layer includes 64 deconvolution kernels of size 3×3, stride 1, and activation function ReLU; the sixth deconvolution layer continues to restore the spatial resolution of the image and enhance the ability to restore features.
[0028] Furthermore, the input of the generator output layer is connected to the output of the sixth deconvolution layer, the generator output layer includes a convolution kernel of size 3x3, and the generator output layer uses a sigmoid activation function to generate a more fine-grained tumor segmentation mask image. In the generator output layer, the generated feature map is mapped to a range of 0 to 1 by the sigmoid function to obtain a fine-grained tumor segmentation result, where a value close to 1 indicates a tumor area.
[0029] In this embodiment, the constructed tumor image discriminator includes a discriminator input layer, a discriminator feature extraction layer, a discriminator Flatten layer, a discriminator fully connected layer, and a discriminator support vector machine layer in sequence.
[0030] Furthermore, the discriminator input layer is used to input the tumor segmentation mask from the tumor image generator, and the tumor image with tumor area pixels marked corresponding to the tumor segmentation mask image.
[0031] Furthermore, the discriminator feature extraction layer includes a first convolution layer, a second convolution layer, a third convolution layer, a maximum pooling layer, a fifth convolution layer, and a sixth convolution layer in sequence, that is, the discriminator feature extraction layer can adopt the structure of a generator encoding network to gradually extract features of two types of images, from low-level to high-level, and capture texture, shape and structural information of different levels of the image.
[0032] Furthermore, the input of the discriminator Flatten layer is connected to the output of the discriminator feature extraction layer. The Flatten layer is a network layer commonly used in deep learning, and its function is to convert multi-dimensional input data (such as feature maps) into a one-dimensional vector. In this embodiment, the extracted feature map is flattened into a one-dimensional vector to prepare for the subsequent fully connected layer.
[0033] Furthermore, the input of the discriminator fully connected layer is connected to the output of the discriminator Flatten layer, and the discriminator fully connected layer receives the flattened feature vector as input for further feature mapping and combination.
[0034] Furthermore, the input of the discriminator support vector machine layer is connected to the output of the discriminator fully connected layer, and the discriminator support vector machine layer uses the support vector machine as a classifier to classify the input feature vector. Furthermore, when the discriminator support vector machine layer outputs 0, the tumor image discriminator discriminates that the image input this time is a tumor segmentation mask image generated by the tumor image generator, and when the discriminator support vector machine layer outputs 1, the tumor image discriminator discriminates that the image input this time is a real tumor image not generated by the tumor image generator. Among them, the discriminator support vector machine layer described in this embodiment realizes the classification of images by learning the optimal hyperplane.
[0035] S02. Using the standard multimodal tumor image dataset, adversarially train the tumor segmentation generative adversarial network.
[0036] Generative Adversarial Network (GAN) is a deep learning model architecture. GAN is trained by two neural networks that compete with each other. One network generates data, and the other network evaluates the authenticity of the generated data, thereby prompting the generator to gradually generate more realistic data.
[0037] The present invention applies the generative adversarial network technology to build a tumor segmentation generative adversarial network. Therefore, the tumor segmentation generative adversarial network built by the present invention is more focused on tumor area segmentation. The task of the tumor image generator is to generate realistic tumor segmentation masks based on the input original tumor slice image, while the task of the tumor image discriminator is to evaluate the authenticity of these generated segmentation masks. Through repeated iterative training, the tumor segmentation generative adversarial network can gradually learn and generate more accurate tumor segmentation results.
[0038] In an optional embodiment, the adversarial training of the tumor segmentation generative adversarial network using the standard multimodal tumor image dataset comprises the following steps:
[0039] S021. Fix the parameters of the tumor image discriminator, input the unlabeled standard multimodal tumor image data set into the tumor image generator, and use the tumor image generator to generate a corresponding tumor segmentation mask image.
[0040] In this embodiment, the tumor segmentation mask image is the result of marking and segmenting the tumor area in the original tumor image using the tumor image generator. It distinguishes the tumor area from the non-tumor area in the image through pixel-level marking.
[0041] In the tumor segmentation task, each pixel is labeled as belonging to the tumor area (usually labeled as 1) or the non-tumor area (usually labeled as 0), thereby creating a mask image with the same size as the original image. This mask image provides information about the location and shape of the tumor in the image and is one of the key results generated by the tumor segmentation model.
[0042] S022. Input the tumor segmentation mask image and the annotated standard multimodal tumor image data set into the tumor image discriminator, and adjust the parameters of the tumor image discriminator and the tumor image generator respectively according to the input result of the tumor image discriminator.
[0043] It should be understood that the standard multimodal tumor image dataset includes tumor image datasets of different modality types, and any modality type of tumor image dataset includes multiple tumor images acquired under the modality, and the tumor images include tumor areas and non-tumor areas.
[0044] In this embodiment, the labeling of the tumor area pixels of any tumor image in the standard multimodal tumor imaging data set specifically refers to distinguishing the tumor area and non-tumor area in the tumor image by using binary pixel points, that is, marking the pixels in the tumor area as 1 and marking the pixels in the non-tumor area as 0.
[0045] Furthermore, in this embodiment, the adjustment method includes the following: if the output result of the tumor image discriminator is false, the parameters of the tumor image discriminator are fixed and the parameters of the tumor image generator are updated.
[0046] The output of the updated tumor image generator is used as the input of the tumor image discriminator. If the output result of the tumor image discriminator is still false, the parameters of the tumor image discriminator are continued to be fixed, and the parameters of the tumor image generator are continued to be updated until the output result is judged to be true by the tumor image discriminator.
[0047] If the output result of the tumor image discriminator is true, the parameters of the tumor image generator are fixed and the parameters of the tumor image discriminator are updated. The updated tumor image discriminator is used to judge the input of the tumor image generator whose output result is true before again. If the output result of the tumor image discriminator is still true, the training is stopped.
[0048] It should be understood that the parameters of the updated tumor image generator and the parameters of the tumor image discriminator refer to the weight parameters inside the model. Among them, the parameters of the tumor image discriminator include the weights and biases in the discriminator network, which are used to determine whether the input image is a real image (from a standard multimodal tumor imaging dataset) or a tumor segmentation mask image (from a tumor image generator). During the training process, the parameters of the tumor image discriminator are used to calculate the loss function of the discriminator to help the discriminator better distinguish between real and generated images. The parameters of the tumor image generator include the weights and biases in the generator network, which are used to convert random noise or other inputs into generated images. During the training process, the parameters of the tumor image generator are used to calculate the loss function of the generator to make the generated images more realistic to deceive the discriminator.
[0049] S03. Obtain a tumor image segmentation model by generating an adversarial network for trained tumor segmentation.
[0050] In an optional embodiment, step S03 uses the trained tumor segmentation generative adversarial network as a tumor image segmentation model, and uses it for tumor segmentation of subsequent multimodal tumor imaging data of the same part of the target patient.
[0051] In this embodiment, the tumor image generator obtained through adversarial training is used as a tumor image segmentation model to achieve subsequent accurate segmentation of multimodal tumor image data. The optimized parameters of the tumor image generator can reconstruct the tumor segmentation mask with high accuracy, providing a reliable basis for medical diagnosis and treatment, and helping doctors to formulate more accurate radiotherapy plans.
[0052] S04. Acquire multimodal tumor imaging data of the target patient at the same location.
[0053] It should be understood that the target patients described in the present invention are patients who need to undergo tumor segmentation and diagnosis. The target patients may need to determine the tumor boundary before radiotherapy, or need to monitor the growth of the tumor after treatment. These target patients usually need to undergo multimodal tumor imaging scans to facilitate doctors to accurately segment, diagnose and treat the tumor.
[0054] In an optional embodiment, the step S01 of acquiring multimodal tumor imaging data of the target patient at the same site includes the following steps:
[0055] S011. Determine the modality type of the multimodal tumor imaging data.
[0056] The modality type specifically refers to the modality type of tumor imaging data. Specifically, the specific modality type of tumor imaging data can be selected according to actual needs or the guidance of professional doctors. Tumor imaging data of different modality types assist each other, which helps to more accurately mark the tumor area in the future and help doctors develop personalized treatment plans.
[0057] In this embodiment, the selected multimodal tumor image data suitable for the target patient includes CT images, MRI images, and PET images.
[0058] The CT image is an image obtained by X-ray imaging technology, which can display tissue density information. CT images can provide edge and contour information in tumor segmentation to help locate the location and size of the tumor. However, the contrast of CT images for soft tissue is not as good as that of MRI images, so it may not be able to accurately distinguish between tumors and surrounding tissues in segmentation.
[0059] The MRI image is an image generated using a magnetic field and harmless radio waves, and has better contrast for soft tissue. Further, the MRI image includes a variety of sequence images, for example, T1-weighted image (T1-Weighted Image), T2-weighted image (T2-Weighted Image), enhanced T1-weighted image (Contrast-Enhanced T1-Weighted Image), DWI (Diffusion-Weighted Imaging, diffusion-weighted image), it can be understood that in MRI, different sequences can provide different types of information. Further, in this embodiment, the sequence selected by the MRI is enhanced T1-weighted image (Contrast-Enhanced T1-Weighted Image), and the enhanced T1-weighted image can more clearly display blood vessels, tumors and other abnormal tissues by injecting contrast agents. Tumors can usually show more obvious signal enhancement on such enhanced T1-weighted images, which helps to locate the position, boundary and size of the tumor.
[0060] The PET images use radioactive markers to measure the metabolic activity of tissue, which can provide information about the activity of a tumor, but have relatively low resolution and may not provide detailed anatomical information.
[0061] S012. Obtain multimodal tumor imaging data of the target patient at the same location according to the modality type.
[0062] According to the determined modality type, the tumor imaging device is used to scan and photograph to obtain the corresponding multimodal tumor imaging data. It should be noted that in the process of acquiring the corresponding modality tumor imaging data, it is necessary to ensure that the scanning parameters, scanning position and posture remain consistent to obtain high-quality tumor imaging data.
[0063] In this embodiment, corresponding multimodal tumor imaging data are formed based on the above selected CT images, MRI images and PET images. Further, for CT images, X-ray imaging technology is used for scanning; for MRI images, enhanced T1-weighted sequences are used for scanning; for PET images, we inject radioactive markers and scan them.
[0064] S013. Process tumor imaging data of different modalities to obtain multimodal tumor imaging data of the target patient.
[0065] Step S013 processes the acquired tumor image data of different modalities, for example, by noise removal, artifact elimination, edge enhancement, image registration, and grayscale value standardization, so that they can be spatially aligned and have a consistent scale and grayscale range for subsequent network processing.
[0066] In this embodiment, for the tumor image data of three modalities, CT images, MRI images and PET images, are acquired. First, each image data is preprocessed, such as removing artifacts and noise. Then, the MRI image is registered to the CT image to ensure that they are aligned in the same coordinate system, and then the PET image is registered to the CT image to ensure that they are aligned in the same coordinate system. Next, the grayscale values of the images of each modality are standardized, and the grayscale value range of all images is mapped to between 0 and 1. In this way, we obtain multi-modal tumor image data that have been preprocessed, registered and standardized. They are aligned in space and have a consistent scale and grayscale range, which provides preparation for subsequent tumor segmentation and analysis.
[0067] This embodiment acquires and processes multimodal tumor image data through steps S011 to S013, so that tumor image data of different modalities are spatially aligned and have a consistent scale and grayscale range, providing preparation for subsequent tumor segmentation and analysis.
[0068] S05. Using the tumor image segmentation model, segment a multimodal tumor segmentation mask image from the multimodal tumor image data.
[0069] Furthermore, for each modality (such as CT images, MRI images, and PET images), the tumor image segmentation model (tumor image generator) will generate tumor segmentation mask images that correspond to the tumor images in the multimodal tumor imaging data of the target patient. These tumor segmentation mask images will provide pixel-level labels for the tumor area for each modality, which will help to accurately locate and segment the tumor.
[0070] In an optional embodiment, the multimodal tumor imaging data of the target patient includes CT images, MRI images and PET images, and the corresponding selected tumor image segmentation model is a tumor image generator obtained by training with a standard multimodal tumor imaging data set containing the corresponding modalities. This means that the model learns during the training process how to effectively extract tumor information from imaging data of different modalities (including CT modality, MRI modality and PET modality), so that it can be applied to the multimodal data of the target patient to accurately segment the tumor segmentation mask image under each modality.
[0071] S06. Fusing the multimodal tumor segmentation mask image to obtain a tumor segmentation result image of the target patient.
[0072] In step S06, by fusing the multi-modal tumor segmentation mask images, a tumor segmentation result image of the target patient is finally generated. This process aims to integrate the tumor segmentation mask images from different modalities to produce a comprehensive and refined tumor segmentation result.
[0073] Specifically, for each modality (such as CT, MRI, and PET), corresponding tumor segmentation mask images have been obtained, which indicate the tumor area in each modality. In the fusion process, different methods can be used, such as performing a logical OR operation on each pixel position according to the mask images of different modalities, so as to merge the tumor information of the same pixel position in different modalities. In this way, a comprehensive tumor segmentation result image can be obtained, in which each pixel position reflects the fusion of tumor information under different modalities.
[0074] In an optional embodiment, the step S06 of fusing the multimodal tumor segmentation mask image to obtain the tumor segmentation result image of the target patient includes the following steps:
[0075] S061. Binarize the multimodal tumor segmentation mask image at the same position.
[0076] Binarization is the process of converting a grayscale image into a black and white image, so that the value of each pixel becomes 0 or 1, representing the tumor area and non-tumor area, respectively.
[0077] S062. Obtain a two-dimensional matrix of multimodal tumor image pixels according to the binarization result.
[0078] The tumor segmentation mask image of each modality after binarization is converted into a two-dimensional matrix with pixel values of 0 or 1. In this matrix, the pixel value corresponding to each pixel position indicates whether the corresponding position belongs to the tumor area.
[0079] S063. Overlay the two-dimensional matrix of multimodal tumor image pixels.
[0080] For each modality, the corresponding two-dimensional matrix of binarized tumor image pixels is superimposed. In this step, a logical OR operation is performed on pixels at the same position to merge tumor information under different modalities, thereby obtaining a comprehensive two-dimensional matrix of tumor image pixels.
[0081] S064. Obtain a tumor segmentation result image of the target patient according to the superposition result.
[0082] The target patient's tumor segmentation result image is generated based on the superimposed two-dimensional matrix of comprehensive tumor image pixels. In this image, each pixel position reflects the fusion information of multi-modal tumor images, that is, if a position is a tumor in any modality, then the position will also be marked as a tumor in the comprehensive result.
[0083] Specifically, in this embodiment, the pixel two-dimensional matrix of the tumor segmentation result image satisfies the following model: I t =[a ij ] t , a ij = 0 or 1, where K represents the pixel two-dimensional matrix of the tumor segmentation result image, N represents the number of modalities, and I t represents the two-dimensional matrix of tumor image pixels corresponding to the t-th modality, a ij Represents the pixel value of the i-th row and j-th column in the two-dimensional matrix of tumor image pixels.
[0084] In this embodiment, through step S061 to step S063, the multimodal tumor segmentation mask image is binarized, superimposed and processed in sequence, thereby obtaining a comprehensive tumor segmentation result image. This comprehensive image helps to provide more comprehensive tumor segmentation information, providing a more reliable basis for doctors to formulate treatment plans and evaluate disease progression.
[0085] In an optional embodiment, in order to better implement the above-mentioned tumor segmentation method based on multimodal generative adversarial network, the present invention also provides a tumor segmentation system based on multimodal generative adversarial network. Figure 2 , Figure 2 A schematic diagram of the structure of a tumor segmentation system based on a multimodal generative adversarial network provided in an embodiment of the present invention.
[0086] like Figure 2 As shown, the tumor segmentation system based on the multimodal generative adversarial network includes an input device, a processor, a memory and an output device, wherein the input device, the processor, the memory and the output device are interconnected, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions to execute the tumor segmentation method based on the multimodal generative adversarial network provided by the present invention.
[0087] Furthermore, the input device is an interface of the tumor segmentation system based on the multimodal generative adversarial network, and is used to receive multimodal tumor image data (or a standard multimodal tumor image data set), which may come from different imaging devices, such as CT scans, MRI scans, and PET scans, etc. The input device transmits these multimodal image data to the processor for further processing.
[0088] Furthermore, the processor is the core component of the tumor segmentation system based on the multimodal generative adversarial network, responsible for performing various tasks. In this system, the processor calls the computer program stored in the memory, that is, the tumor segmentation method based on the multimodal generative adversarial network is implemented. It processes the input multimodal image data, including generating a tumor segmentation mask image through the trained generative adversarial network, and generating the final tumor segmentation result image through the fusion algorithm.
[0089] Furthermore, the memory is used to store computer programs, model parameters, training data and intermediate results required by the system, including the model of the multimodal generative adversarial network, the training data set, and possible previously generated tumor segmentation mask images, etc.
[0090] Furthermore, the output device is used to display and present the final tumor segmentation result image. This can be a computer screen, a medical image browser, a printer, etc. Medical professionals can view the tumor segmentation results generated by the system through the output device for clinical analysis and diagnosis.
[0091] The tumor segmentation system based on the multimodal generative adversarial network provided by the present invention provides an efficient and integrated tool for medical professionals. Through the multimodal generative adversarial network technology, tumor segmentation can be automatically performed, thereby reducing the workload of doctors and improving the accuracy and reliability of tumor diagnosis. At the same time, the tumor segmentation system based on the multimodal generative adversarial network provided by the present invention has a clear structure and is easy to operate, providing effective support for tumor segmentation tasks in medical practice.
[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.
Claims
1. A tumor segmentation method based on multimodal generative adversarial network, It is characterized in that The tumor segmentation method based on a multimodal generative adversarial network comprises the following steps: Obtain a standard multimodal tumor imaging dataset and build a tumor segmentation generative adversarial network; Using the standard multimodal tumor image dataset, adversarially training the tumor segmentation generative adversarial network; The tumor image segmentation model is obtained by generating an adversarial network for trained tumor segmentation; Acquire multimodal tumor imaging data of the target patient at the same location; Using the tumor image segmentation model, segmenting a multimodal tumor segmentation mask image from the multimodal tumor image data; fusing the multimodal tumor segmentation mask image to obtain a tumor segmentation result image of the target patient; The tumor segmentation generative adversarial network includes a tumor image generator and a tumor image discriminator; The tumor image generator includes, in sequence, a generator input layer, a generator encoding network, a generator decoding network, and a generator output layer; The tumor image discriminator includes a discriminator input layer, a discriminator feature extraction layer, a discriminator Flatten layer, a discriminator fully connected layer, and a discriminator support vector machine layer in sequence; The method of using the standard multimodal tumor image dataset to adversarially train the tumor segmentation generative adversarial network comprises the following steps: Fixing the parameters of the tumor image discriminator, inputting the unlabeled standard multimodal tumor image data set into the tumor image generator, and using the tumor image generator to generate a corresponding tumor segmentation mask image; The tumor segmentation mask image and the annotated standard multimodal tumor image data set are input into the tumor image discriminator, and the parameters of the tumor image discriminator and the tumor image generator are adjusted according to the output result of the tumor image discriminator, and the adjustment method includes the following: If the output result of the tumor image discriminator is false, fixing the parameters of the tumor image discriminator and updating the parameters of the tumor image generator; Using the output of the updated tumor image generator as the input of the tumor image discriminator; If the output result of the tumor image discriminator is still false, the parameters of the tumor image discriminator are continued to be fixed, and the parameters of the tumor image generator are continued to be updated until the output result thereof is determined to be true by the tumor image discriminator; If the output result of the tumor image discriminator is true, fixing the parameters of the tumor image generator and updating the parameters of the tumor image discriminator; Use the updated tumor image discriminator to judge again the input of the tumor image generator whose output result was true before. If the output result of the tumor image discriminator is still true, stop training; The tumor image segmentation model is a tumor image generator in a trained tumor segmentation generative adversarial network.
2. The tumor segmentation method based on a multimodal generative adversarial network according to claim 1, It is characterized in that The standard multimodal tumor image dataset includes a CT image dataset, an MRI image dataset and a PET image dataset.
3. The tumor segmentation method based on a multimodal generative adversarial network according to claim 1, It is characterized in that Acquiring multimodal tumor imaging data of the target patient at the same location includes the following steps: Determine the modality type of the target patient's multimodal tumor imaging data; According to the modality type, respectively obtaining tumor image data of different modalities at the same part of the target patient; Process tumor imaging data of different modalities to obtain multimodal tumor imaging data of target patients.
4. The tumor segmentation method based on a multimodal generative adversarial network according to claim 3, It is characterized in that The step of segmenting a multimodal tumor segmentation mask image from the multimodal tumor image data using the tumor image segmentation model comprises the following steps: Inputting tumor images of different modalities in the multimodal tumor imaging data into the tumor image segmentation model; Using the tumor image segmentation model, obtaining tumor segmentation mask images of different modalities; Tumor segmentation mask images of different modalities are aggregated to obtain a multimodal tumor segmentation mask image of the target patient.
5. The tumor segmentation method based on a multimodal generative adversarial network according to claim 4, It is characterized in that The step of fusing the multimodal tumor segmentation mask image to obtain the tumor segmentation result image of the target patient comprises the following steps: Binarize the multimodal tumor segmentation mask image at the same position; According to the binarization results, a two-dimensional matrix of multimodal tumor image pixels is obtained; Overlay a two-dimensional matrix of multimodal tumor image pixels; According to the superposition result, a tumor segmentation result image of the target patient is obtained.
6. The tumor segmentation method based on a multimodal generative adversarial network according to claim 4, It is characterized in that The pixel two-dimensional matrix of the tumor segmentation result image satisfies the following model: , , ,in, Represents the pixel two-dimensional matrix of the tumor segmentation result image, represents the number of modes, Indicates The two-dimensional matrix of tumor image pixels corresponding to the middle modality, Represents the pixel value of the i-th row and j-th column in the two-dimensional matrix of tumor image pixels.
7. A tumor segmentation system based on multimodal generative adversarial network, It is characterized in that The tumor segmentation system based on a multimodal generative adversarial network comprises an input device, a processor, a memory and an output device, wherein the input device, the processor, the memory and the output device are interconnected, wherein the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions to execute the tumor segmentation method based on a multimodal generative adversarial network as described in any one of claims 1 to 6.
