Multimodal nuclear medicine image segmentation method, system, device and readable storage medium

Abstract

The present invention discloses a multimodal nuclear medicine image segmentation method, system, device and readable storage medium. The multimodal nuclear medicine image segmentation method comprises: acquiring PET and / or CT / MRI images of a patient; inputting the acquired PET and / or CT / MRI images into a trained multimodal nuclear medicine image segmentation model; processing and analyzing the input PET and / or CT / MRI images using the multimodal nuclear medicine image segmentation model; and generating medical images containing organ segmentation and tumor segmentation or only generating medical images containing tumor segmentation. By improving the U‑Net structure and training strategy, the present invention has the ability to process missing modalities, improves the ability to process multimodal data, improves prediction performance, improves segmentation accuracy, and realizes effective processing and analysis of multimodal medical images, especially in the aspects of prediction, segmentation and missing sequence combination, which has great advantages.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical imaging and image processing, and specifically relates to a multimodal nuclear medicine image segmentation method, system, device and readable storage medium for PET-CT and PET-MRI. Background Art

[0002] With the advancement of medical technology, medical imaging technology has become an important means of modern medical diagnosis. Among them, nuclear medicine imaging technology, especially positron emission tomography (PET), computed tomography (CT) and magnetic resonance imaging (MRI), has played an important role in clinical diagnosis with its unique advantages. PET-CT and PET-MRI multimodal imaging technologies combine their respective advantages and can provide richer and more accurate biological tissue information.

[0003] However, the processing and analysis of these high-resolution multimodal images is a huge challenge. Image segmentation is a key step in image processing, and its purpose is to distinguish the region of interest in the image from the background. However, due to the characteristics of nuclear medicine images, such as high noise, low contrast and blurred boundaries, image segmentation is very difficult.

[0004] In recent years, deep learning technology has been widely used in the field of medical image processing, especially in tumor detection and organ segmentation. In particular, convolutional neural networks (CNNs) have achieved remarkable results in medical image segmentation. Figure 3 The U-Net shown is a CNN structure specially designed for medical image segmentation. It uses a fully convolutional network consisting of an Encoder and a Decoder to form a U-shaped architecture. It can perform pixel-level prediction while maintaining the original image size, thereby achieving high-precision image segmentation.

[0005] However, the traditional U-Net is mainly designed for single-modality images, and the processing effect of multi-modality images (such as PET-CT and PET-MRI) is not ideal, especially in the automatic segmentation of organs and tumors. The specific shortcomings are as follows:

[0006] Insufficient ability to handle multimodal data:

[0007] First, images of different modalities have different characteristics and information, and traditional U-Net often cannot make full use of this information. Second, the traditional U-Net structure usually contains only one encoder and one decoder. This structure may be subject to some limitations when processing multimodal medical images (such as PET-CT and PET-MRI), making it impossible to effectively process multimodal data. For example, if you want to process PET and CT images at the same time, you need to merge them into one input or process them separately with two independent networks. Both methods have disadvantages. The former may cause confusion of information, and the latter requires more computing resources.

[0008] Insufficient prediction performance:

[0009] Since the encoder and decoder of the traditional U-Net structure are designed for the current input image, they cannot effectively predict the image of the missing modality.

[0010] The segmentation accuracy is not high:

[0011] When processing complex medical images, especially tumor and organ segmentation, the single encoder-decoder structure in the traditional U-Net may not provide sufficient segmentation accuracy.

[0012] Insufficient ability to handle missing modalities:

[0013] Multi-sequence nuclear medicine imaging data are valuable for reliable diagnosis and treatment prognosis in clinical settings. However, in practical applications, due to various reasons such as equipment limitations, patient conditions or economic factors, it may not be possible to obtain all imaging modalities, and some sequences may be unavailable or missing. Therefore, traditional methods may not be able to effectively deal with the problem of missing modalities. Summary of the invention

[0014] In view of the problems existing in the above-mentioned prior art, the present invention provides a multimodal nuclear medicine image segmentation method, system, device and readable storage medium to realize the processing of multimodal medical images of PET-CT and PET-MRI images, and perform prediction, segmentation and missing sequence synthesis.

[0015] In order to solve the above technical problems, the present invention is implemented through the following technical solutions:

[0016] A multimodal nuclear medicine image segmentation method, comprising:

[0017] Step 1) Acquire PET and / or CT / MRI images of the patient;

[0018] Step 2) inputting the acquired PET and / or CT / MRI images into the trained multimodal nuclear medicine image segmentation model;

[0019] Step 3) using the multimodal nuclear medicine image segmentation model to process and analyze the input PET and / or CT / MRI images;

[0020] Step 4) Generate a medical image containing both organ segmentation and tumor segmentation or only generate a medical image containing tumor segmentation.

[0021] Furthermore, the multimodal nuclear medical image segmentation model simultaneously generates a medical image containing organ segmentation and tumor segmentation in the following manner:

[0022] 1) When the PET image and the CT / MRI image are simultaneously used as input, the multimodal nuclear medicine image segmentation model directly performs segmentation processing on the input PET image and CT image, thereby simultaneously obtaining a medical image containing organ segmentation and tumor segmentation;

[0023] 2) When only the PET image is used as input, the multimodal nuclear medicine image segmentation model first uses the PET image to perform missing modality synthesis processing to generate a corresponding CT / MRI prediction image, and then uses the PET image and the CT / MRI prediction image as input again, and the multimodal nuclear medicine image segmentation model performs segmentation processing, thereby simultaneously obtaining a medical image containing organ segmentation and tumor segmentation;

[0024] 3) When only CT / MRI images are used as input, the multimodal nuclear medicine image segmentation model network first uses the CT / MRI images to perform missing modality synthesis processing to generate a corresponding PET prediction image, and then uses the CT / MRI images and the PET prediction image as input again, allowing the multimodal nuclear medicine image segmentation model to perform segmentation processing, thereby obtaining a medical image containing organ segmentation and tumor segmentation at the same time.

[0025] Furthermore, the missing modality synthesis processing method of the multimodal nuclear medicine image segmentation model is:

[0026] 1) Prediction of missing PET modalities:

[0027] First, multiple available sequences in the input CT / MRI image are encoded as corresponding features 1 respectively; then, by giving task-specific codes, multiple features 1 are fused to form fused feature B; at the same time, by giving task-specific codes, multiple features 1 are fused using the attention mechanism for a specific task to form fused feature A; finally, fused feature A is fused with fused feature B and then decoded to generate the missing PET image, i.e., PET predicted image;

[0028] 2) Prediction of missing CT / MRI modalities:

[0029] First, multiple available sequences in the input PET image are encoded as corresponding features 2 respectively; then, by giving task-specific codes, multiple features 2 are fused to form fused feature B; at the same time, by giving task-specific codes, multiple features 2 are fused using the attention mechanism for a specific task to form fused feature A; finally, the fused feature A is fused with the fused feature B and then decoded to generate the missing CT / MRI image, that is, the CT / MRI predicted image.

[0030] Furthermore, the multimodal nuclear medical image segmentation model generates only medical images containing tumor segmentation as follows:

[0031] 1) The obtained medical image containing organ segmentation, the PET image, and the CT / MRI image are again used as input, and the multimodal nuclear medical image segmentation model is used to perform deep segmentation processing, thereby obtaining a medical image containing only tumor segmentation;

[0032] 2) The obtained medical image containing organ segmentation, the PET image, and the CT / MRI predicted image are used as input again, and the multimodal nuclear medical image segmentation model is used to perform deep segmentation processing, thereby obtaining a medical image containing only tumor segmentation;

[0033] 3) The obtained medical image containing organ segmentation, the CT / MRI image, and the PET prediction image are used as input again, and the multimodal nuclear medicine image segmentation model is made to perform deep segmentation processing, thereby obtaining a medical image containing only tumor segmentation.

[0034] Furthermore, the multimodal nuclear medicine image segmentation model is composed of a segmentation model network structure and a missing modality processing structure;

[0035] 1) The segmentation model network structure is a convolutional neural network structure based on U-Net, which includes a segmentation encoder module and a segmentation decoder module;

[0036] The segmentation encoder module is a series of convolutional layers and pooling layers for extracting context information of an image; the segmentation encoder module includes two segmentation encoding units, one of which corresponds to a PET input and is responsible for performing convolution and pooling processing on an input PET image or a PET predicted image to generate a PET pooling feature map, and the other segmentation encoding unit corresponds to a CT / MRI input and is responsible for performing convolution and pooling processing on an input CT / MRI image or a CT / MRI predicted image to generate a PET pooling feature map;

[0037] The segmentation decoder module is a series of upsampling layers and convolution layers, which are used to restore the resolution and precise positioning of the image; the segmentation decoder module includes two segmentation decoding units, one of which corresponds to the organ segmentation output, is responsible for upsampling and refining the features of the input PET pooling feature map and the CT / MRI pooling feature map, and converting them into organ segmentation images, and the other segmentation decoding unit corresponds to the tumor segmentation output, is responsible for upsampling and refining the features of the input PET pooling feature map and the CT / MRI pooling feature map, and converting them into tumor segmentation images;

[0038] 2) The missing modality processing structure includes a synthesis encoder module, a multi-sequence fusion module and a synthesis decoder module;

[0039] The synthetic encoder module is an encoder based on a recurrent neural network (RNN), which is used to receive an input sequence and encode it into features; the synthetic encoder module includes two synthetic encoder units, one of which corresponds to CT / MRI input and is responsible for receiving multiple sequences available for CT / MRI images and converting the multiple sequences into corresponding features 1; the other synthetic encoder unit corresponds to PET input and is responsible for receiving multiple sequences available for PET images and converting the multiple sequences into corresponding features 2;

[0040] The multi-sequence fusion module is responsible for fusing feature 1 or feature 2 from multiple input sequences by giving task-specific codes to generate CT / MRI multi-sequence fusion features or PET multi-sequence fusion features;

[0041] The synthetic decoder module is a decoder based on a recurrent neural network (RNN), which is used to gradually generate an output sequence according to the features generated by the encoder; the synthetic decoder module includes two synthetic decoder units, one of which corresponds to the PET prediction output and is responsible for decoding the CT / MRI multi-sequence fusion features and converting them into the corresponding PET prediction image, and the other synthetic decoder unit corresponds to the CT / MRI prediction output and is responsible for decoding the PET multi-sequence fusion features and converting them into the corresponding CT / MRI prediction image.

[0042] Furthermore, the training strategy of the multimodal nuclear medicine image segmentation model is as follows:

[0043] 1) Dataset collection;

[0044] Collect a certain number of public PET-CT / MRI datasets, and classify and annotate them according to different modalities and whether they are tumors or organs;

[0045] 2) Dataset preprocessing;

[0046] Preprocessing of the collected datasets using techniques including intensity normalization, cropping and padding, or data augmentation;

[0047] Model training;

[0048] Taking paired PET and CT / MRI modality data as two inputs, the segmentation model network structure directly performs segmentation processing on the input PET and CT / MRI modality data, and outputs a medical image containing both organ segmentation and tumor segmentation;

[0049] Taking the PET modality data as an input and the input value set to -1 as another input to simulate the missing CT / MR modality data paired with the PET modality data, the missing modality processing structure first performs missing modality synthesis processing using the PET modality data to predict the missing CT / MRI modality data, and then reconstructs a new data set by combining the original input PET modality data and the predicted CT / MRI modality data, and inputs the reconstructed data set into the segmentation model network structure for segmentation processing, and outputs a medical image containing both organ segmentation and tumor segmentation;

[0050] Taking CT / MRI modality data as an input and an input value set to -1 as another input to simulate missing PET modality data paired with the CT / MRI modality data, the missing modality processing structure first performs missing modality synthesis processing using the CT / MRI modality data to predict the missing PET modality data, and then reconstructs a new data set by combining the original input CT / MRI modality data and the predicted PET modality data, and inputs the reconstructed data set into the segmentation model network structure for segmentation processing, and outputs a medical image containing both organ segmentation and tumor segmentation;

[0051] 4) Model enhancement;

[0052] Recombining the medical image data containing organ segmentation obtained during training with the original input PET modality data and the original input CT / MRI modality data into a new data set, and inputting the combined data set into the segmentation model network structure again for deep segmentation processing, only containing the medical image of tumor segmentation;

[0053] Recombining the medical image data containing organ segmentation obtained during training with the original input PET modality data and the predicted missing CT / MRI modality data into a new data set, and inputting the combined data set into the segmentation model network structure again for deep segmentation processing, only containing the medical image of tumor segmentation;

[0054] The medical image data containing organ segmentation obtained during training is reorganized into a new data set together with the original input CT / MRI modality data and the predicted missing PET modality data, and the combined data set is again input into the segmentation model network structure for deep segmentation processing, only containing medical images with tumor segmentation.

[0055] Furthermore, the training process of the missing modality processing structure is as follows:

[0056] Before inferring the missing modality, inputting the real existing PET-CT / MRI data into the missing modality processing structure;

[0057] The two synthetic encoder units in the missing modality processing structure receive CT / MRI and PET modality data respectively, convert them into their own features 1 and features 2, and concatenate the features 1 and features 2 of the two modalities;

[0058] Then, for different features 1 and 2, feature 1 and feature 2 are fused by giving specific task codes;

[0059] Finally, the new features 1 and 2 after fusion are respectively passed through two synthesis decoder units in the missing modality processing structure to output the predicted PET modality or CT / MRI modality.

[0060] A multimodal nuclear medicine image segmentation system comprises a data receiving unit, a data processing unit and a data output unit;

[0061] The data receiving unit is used to receive PET and / or CT images;

[0062] The data processing unit is used to process and analyze the PET and / or CT / MRI images according to the multimodal nuclear medicine image segmentation method to obtain corresponding organ segmentation images and / or tumor segmentation images;

[0063] The data output unit is used to display the obtained organ segmentation image and / or tumor segmentation image.

[0064] A computer device comprises: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus, and the memory is used to store at least one executable instruction, wherein the executable instruction enables the processor to perform operations corresponding to the above-mentioned multimodal nuclear medicine image segmentation method.

[0065] A computer-readable storage medium stores at least one executable instruction, wherein the executable instruction enables a processor to perform operations corresponding to the multimodal nuclear medicine image segmentation method.

[0066] The beneficial effects of the present invention are:

[0067] 1. Improve the ability to process multimodal data:

[0068] The present invention proposes a nuclear medicine image segmentation method for PET-CT and PET-MRI multimodalities, which involves an improvement in the U-Net network structure. The four encoders correspond to the PET and CT / MRI inputs of the segmentation task and the prediction task respectively, and the four decoders correspond to the PET prediction, CT / MRI prediction, organ segmentation, and tumor segmentation outputs respectively. This structure can process multimodal data more effectively, avoid information confusion, and does not require additional computing resources.

[0069] Improving PET and CT / MRI prediction performance:

[0070] The optimization training strategy proposed in the present invention is to force an input value in the encoder to become -1, so as to predict the image of the missing modality. This method can enable the network to learn to predict the image of the missing modality, thereby improving its prediction performance.

[0071] With missing mode handling capabilities:

[0072] The present invention can combine multiple available sequences based on a deep learning method to synthesize missing sequences, and then put them into the segmented neural network again for re-segmentation to obtain a more accurate result.

[0073] Improve segmentation accuracy:

[0074] The present invention can take PET, CT and segmented organs as input and output only one segmented tumor. This method can more effectively and centrally process tumor information, thereby improving the accuracy of tumor detection.

[0075] In general, the present invention realizes the effective processing and analysis of multimodal medical images by improving the U-Net structure and training strategy, especially in the aspects of prediction, segmentation and missing sequence combination, which has great advantages.

[0076] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. The specific implementation of the present invention is given in detail by the following embodiments and their accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0078] Figure 1 It is a schematic diagram of the process of the multimodal nuclear medicine image segmentation method of the present invention;

[0079] Figure 2 A schematic diagram of a sample of the AutoPET data set collected by the present invention;

[0080] Figure 3 This is the architecture diagram of the UNet network structure;

[0081] Figure 4 Schematic diagram of the overall architecture of the multimodal nuclear medicine image segmentation model of the present invention;

[0082] Figure 5 It is a schematic diagram of the architecture of the segmentation model network structure in the multimodal nuclear medicine image segmentation model of the present invention;

[0083] Figure 6 Schematic diagram of the architecture of the missing modality synthesis structure in the multimodal nuclear medicine image segmentation model of the present invention;

[0084] Figure 7 The present invention is a flowchart of an embodiment of a multimodal nuclear medicine image segmentation method. DETAILED DESCRIPTION

[0085] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings so that the purpose, features and advantages of the invention can be more clearly understood. It should be understood that the embodiments shown in the accompanying drawings are not intended to limit the scope of the present invention, but are only intended to illustrate the essential spirit of the technical solution of the present invention.

[0086] In the following description, certain specific details are set forth for the purpose of illustrating various disclosed embodiments to provide a thorough understanding of the various disclosed embodiments. However, those skilled in the relevant art will recognize that the embodiments may be practiced without one or more of these specific details. In other cases, well-known devices, structures, and techniques associated with the present application may not be shown or described in detail to avoid unnecessarily obscuring the description of the embodiments.

[0087] Unless the context requires otherwise, throughout the specification and claims, the word "comprise" and variations such as "include" and "have" should be construed in an open, inclusive sense, ie, should be interpreted as "including, but not limited to."

[0088] References throughout the specification to "one embodiment" or "an embodiment" indicate that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of "in one embodiment" or "in an embodiment" in various places throughout the specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any manner in one or more embodiments.

[0089] As used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. It should be noted that the term "or" is generally employed in its sense including "and / or" unless the context clearly dictates otherwise.

[0090] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0091] The present invention first proposes a multimodal nuclear medicine image segmentation model, see Figure 4 As shown, the multimodal nuclear medicine image segmentation model consists of a segmentation model network structure and a missing modality processing structure.

[0092] 1. Segmentation model network structure;

[0093] The segmentation model network structure of the present invention is an improved convolutional neural network structure based on U-Net. Figure 3 As shown in Figure 1, U-Net is a network for biomedical image segmentation, which consists of an Encoder and a Decoder, forming a U-shaped architecture. Figure 5 As shown, the segmentation model network structure of the present invention mainly includes a segmentation encoder module and a segmentation decoder module.

[0094] 1) The segmentation encoder module is a series of convolutional layers and pooling layers for extracting contextual information of the image. The segmentation encoder module includes two segmentation encoding units. One of the segmentation encoding units corresponds to PET input, and is responsible for convolution and pooling the input PET image or PET predicted image to generate a PET pooling feature map. The other segmentation encoding unit corresponds to CT / MRI input, and is responsible for convolution and pooling the input CT / MRI image or CT / MRI predicted image to generate a PET pooling feature map.

[0095] The segmentation encoder unit of the present invention includes a convolution layer, a nonlinear activation function, a pooling layer, a normalization layer, and a skip connection part. The segmentation encoder unit extracts features from the image through a series of convolution layers. Each convolution layer usually includes a convolution operation, which extracts specific features such as edges, textures, etc. through a convolution kernel (or filter). The convolution layer is usually followed by a nonlinear activation function, such as ReLU (Rectified Linear Unit), to increase the nonlinear ability of the model. After one or more layers of convolution, the segmentation encoder unit uses a pooling layer (such as maximum pooling) to reduce the spatial dimensions (height and width) of the feature map, which helps to reduce the amount of calculation and the number of parameters of the model while retaining important feature information. The pooling layer also helps to increase the robustness of the model to small changes in the input image. Sometimes a normalization layer, such as batch normalization, is added after the convolution layer and the nonlinear activation function to stabilize the training process, accelerate the convergence rate, and improve the generalization ability of the model. Although strictly speaking, skip connections connect the segmentation encoder unit and the segmentation decoder unit, they also play an important role in the design of the segmentation encoder unit. The jump connection directly transfers the feature map in the segmentation encoder unit to the corresponding layer of the segmentation decoder unit, which helps the segmentation decoder unit to better reconstruct the detailed information of the image. Because in traditional convolutional neural networks, as the layers deepen, the detailed information of the image is often lost. The segmentation encoder module of the segmentation model network structure of the present invention uses two segmentation encoder units with the same structure, corresponding to the image input of the two modalities of PET and CT / MRI, respectively, so that two different information sources can be used at the same time to improve the accuracy and robustness of segmentation.

[0096] 2) The segmentation decoder module is a series of upsampling layers and convolutional layers for restoring the resolution and precise positioning of the image. The segmentation decoder module contains two segmentation decoding units. One of the segmentation decoding units corresponds to the organ segmentation output, is responsible for upsampling and refining the features of the input PET pooling feature map and CT / MRI pooling feature map, and converting them into organ segmentation images. The other segmentation decoding unit corresponds to the tumor segmentation output, is responsible for upsampling and refining the features of the input PET pooling feature map and CT / MRI pooling feature map, and converting them into tumor segmentation images.

[0097] The segmentation decoder unit of the present invention includes an upsampling layer, a convolution layer, a skip connection part, a nonlinear activation function, a normalization layer, and an output layer. The segmentation decoder unit gradually increases the spatial dimension (i.e., resolution) of the feature map through a series of upsampling operations to reconstruct the detailed structure of the image. Upsampling can be achieved by different methods, including nearest neighbor interpolation, bilinear interpolation, transposed convolution (sometimes called deconvolution), etc. The upsampled feature map is usually further processed by a convolution layer to refine the features and reduce artifacts that may be introduced during the upsampling process. These convolution layers also help the model learn how to reconstruct image details based on compressed feature information. A notable feature of UNet is the skip connection between its encoder and decoder. These connections connect the feature maps in the segmentation encoder unit directly to the corresponding layers in the segmentation decoder unit, helping the segmentation decoder unit to use the detailed contextual information captured by the segmentation encoder unit, which is crucial for reconstructing lost image details and structures. The convolution layer in the segmentation decoder unit is usually followed by a nonlinear activation function, such as ReLU, to introduce nonlinearity and help the network learn complex image reconstruction mappings. Similar to the segmentation encoder unit, a normalization layer, such as batch normalization, can also be added after the convolution layer of the segmentation decoder unit to improve the training process, accelerate the convergence speed and improve the generalization ability of the model. At the end of the segmentation decoder unit, a convolution layer is used to convert the final feature map into a segmentation map of the target. The number of convolution kernels in this layer usually matches the number of target categories, and a sigmoid or softmax activation function is usually used to assign a category probability to each pixel. The segmentation decoder module of the segmentation model network structure of the present invention uses two segmentation decoder units with the same structure, corresponding to the two segmentation outputs of organ segmentation and tumor segmentation, respectively, so that multi-task learning can be achieved, different results can be generated at the same time, and the function and flexibility of the network are increased.

[0098] 2. Missing modal processing structure;

[0099] The missing modality processing structure of the present invention has an architecture similar to that of an autoencoder, see Figure 6 As shown, it includes a synthesis encoder module, a multi-sequence fusion module and a synthesis decoder module.

[0100] 1) The synthetic encoder module is an encoder based on a recurrent neural network (RNN), and the synthetic encoder module includes two synthetic encoder units. One of the synthetic encoder units corresponds to CT / MRI input, is responsible for receiving multiple sequences available for CT / MRI images, and converts the multiple sequences into corresponding features 1. The other synthetic encoder unit corresponds to PET input, is responsible for receiving multiple sequences available for PET images, and converts the multiple sequences into corresponding features 2.

[0101] The main task of the synthetic encoder unit of the present invention is to receive an input sequence and convert it into a context vector (also called a hidden state or encoding vector) of a fixed size, which contains the semantic information of the input sequence and is intended to capture the key information in the input sequence.

[0102] The workflow of the synthetic encoder unit is as follows:

[0103] Given an input sequence , the synthetic encoder unit updates its hidden state h at each time step t t .

[0104] (1)

[0105] In formula (1), f is the activation function (such as tanh or ReLU), W hx and W hh is the weight matrix, b h is the bias. The last hidden state of the synthesis encoder unit is usually passed to the synthesis decoder unit as the context vector for the entire input sequence.

[0106] 2) The multi-sequence fusion module is responsible for fusing feature 1 or feature 2 from multiple input sequences by giving a task-specific code, which uses binary codes to identify the source and the target, and finally generates a CT / MRI multi-sequence fusion feature or a PET multi-sequence fusion feature.

[0107] 3) The synthesis decoder module is a decoder based on a recurrent neural network (RNN), and the synthesis decoder module includes two synthesis decoder units. One of the synthesis decoder units corresponds to the PET prediction output, is responsible for decoding the CT / MRI multi-sequence fusion features, and converts them into the corresponding PET prediction image. The other synthesis decoder unit corresponds to the CT / MRI prediction output, is responsible for decoding the PET multi-sequence fusion features, and converts them into the corresponding CT / MRI prediction image.

[0108] The main task of the synthetic decoder unit of the present invention is to gradually generate an output sequence based on the context vector (or hidden state) generated by the synthetic encoder unit. The synthetic decoder unit accepts the output of the previous time step and the current hidden state at each time step to generate a new output.

[0109] The computation process of the synthesis decoder unit at time step t can be expressed as:

[0110] (2)

[0111] In formula (2), st is the hidden state of the decoder at time step t, y t-1 is the output of the previous time step, c is the context vector provided by the encoder, and W ys , W ss and W cs is the weight matrix, b s is the bias and f is the activation function (such as tanh or ReLU).

[0112] The output generation can be expressed as:

[0113] (3)

[0114] In formula (3), y t is the output of the decoder at time step t, W sy is the weight matrix, b y is the bias and g is the output activation function (such as softmax).

[0115] To exploit the shared information between sequences, arbitrary sequence synthesis is integrated into a single decoder and encoder, which can reduce the distance between different sequences at the feature level to help more stable fusion.

[0116] The present invention also proposes a training strategy for the above-mentioned multimodal nuclear medicine image segmentation model, the specific contents of which are as follows:

[0117] 1) Dataset collection;

[0118] The method of data collection usually depends on the specific application scenario and task requirements. Common data collection methods include manual annotation, semi-automatic annotation, and using open source datasets.

[0119] Manual labeling is one of the most common ways of data collection. Objects in images or videos are manually labeled by professionals or crowdsourcing platforms. This method requires a lot of human resources, but it can ensure the accuracy and quality of the labeling.

[0120] Semi-automatic annotation is a process where professionals use computer-assisted tools, such as annotation software or annotation platforms, to assist humans in annotating objects in images or videos. This method can improve the efficiency of annotation, but it still requires human participation.

[0121] Open source datasets use publicly released open source datasets, such as Amos2022, AutoPET, etc., for model training and evaluation. This method can save the cost of data collection, but attention should be paid to the quality and applicability of the dataset.

[0122] The datasets used in the development of this invention are composed of three parties: the public dataset AutoPET PET-CT dataset (such as Figure 2 As shown in the figure), breast cancer PET-CT dataset from the Fifth Affiliated Hospital of Sun Yat-sen University, MRI dataset from the public dataset Amos2022, PET-CT dataset from Zhejiang Cancer Hospital, and MRI dataset from Shenzhen People's Hospital. These datasets are classified and stored according to different modalities and whether they are tumors or organs.

[0123] 2) Dataset preprocessing;

[0124] For the use of the entire model, data preprocessing is a very important step, which can help improve the training effect and generalization ability of the model. The present invention provides a series of functions in data preprocessing to help users prepare and enhance medical image data, including intensity normalization, cropping and padding, and data enhancement. Intensity normalization is to standardize the intensity range of the image to between 0 and 1 so that the model can better handle images of different intensities. Cropping and padding is to crop or pad the image to the same size for input into the model. Typically, the size of the cropping and padding is a power of 2 to maintain consistency during downsampling and upsampling. Data enhancement is the application of data enhancement techniques such as random rotation, flipping, and scaling to increase the generalization ability of the model.

[0125] 3) Model training;

[0126] After completing data collection and data preprocessing, save the preprocessed data in the specified format and file path, and then use code instructions to let the model perform inference.

[0127] The paired PET and CT / MRI modality data are used as two inputs, and the segmentation model network structure directly performs segmentation processing on the input PET and CT / MRI modality data, and outputs a medical image containing both organ segmentation and tumor segmentation.

[0128] In addition to the above-mentioned conventional training, the present invention introduces a special operation during training to force one of the input values ​​to become -1, which can simulate the missing mode, allowing the model to learn how to recover and predict the missing mode from another mode, and enhance the generalization ability and robustness of the model. The specific operation is that when the entire convolutional neural network reads data, it will generate a matrix full of -1 according to the relevant information of another corresponding modality data, such as the length, width and height of the three-dimensional data matrix and the batch size of the input data, and then fill in the corresponding values ​​to achieve the effect of forcibly becoming -1.

[0129] When the PET modality data is used as the only input, an input value set to -1 is used as another input to simulate the missing CT / MR modality data paired with the PET modality data. The missing modality processing structure first uses the PET modality data to perform missing modality synthesis processing to predict the missing CT / MRI modality data, and then reconstructs the original input PET modality data and the predicted CT / MRI modality data into a new data set, and the reconstructed data set is input into the segmentation model network structure for segmentation processing, and a medical image containing both organ segmentation and tumor segmentation is output.

[0130] When the CT / MRI modality data is used as the only input, an input value set to -1 is used as another input to simulate the missing PET modality data paired with the CT / MRI modality data. The missing modality processing structure first uses the CT / MRI modality data to perform missing modality synthesis processing to predict the missing PET modality data, and then reconstructs the original input CT / MRI modality data and the predicted PET modality data into a new data set, and the reconstructed data set is input into the segmentation model network structure for segmentation processing, and a medical image containing both organ segmentation and tumor segmentation is output.

[0131] Before inferring the missing modality, it is necessary to use real PET-CT / MRI data to input into the missing modality processing structure for training. The two synthetic encoder units in the missing modality processing structure receive CT / MRI and PET modality data respectively, and convert them into their own features 1 and 2, and concatenate the features 1 and 2 of the two modalities; then, for different features 1 and 2, feature 1 and feature 2 are fused through the given specific task code; finally, the new features 1 and new features 2 after fusion are respectively output through the two synthetic decoder units in the missing modality processing structure to output the predicted PET modality or CT / MRI modality.

[0132] The optimization training strategy proposed in the present invention has two main advantages. First, it can increase the generalization ability of the model, that is, the model can work normally under different input conditions without being affected by missing data. This is very important for actual application scenarios, because in real scenarios it is impossible to guarantee that every patient has complete PET and CT or PET and MRI images. Second, it can improve the stability of segmentation, that is, the model can maintain consistent segmentation effects under different input conditions without obvious differences or deviations. This is very beneficial for ensuring the accuracy and reliability of segmentation, because it is not desirable for the model to give different segmentation results under different input conditions.

[0133] Moreover, the optimization training strategy proposed in the present invention enables the model to process missing modal sequences. The model can automatically learn effective feature representations from the input data, which makes the model more robust. In some cases, the input data may have missing parts (for example, in multimodal data, data of a certain modality may be unavailable). This architecture allows the model to still effectively learn and predict even when some information is missing.

[0134] 4) Model enhancement;

[0135] The limitation of the traditional U-Net structure is that it can only process single-modal input and output. In other words, it can only accept images of one modality as input and output segmentation results of one modality. For example, if the input is a PET image, the output is the segmentation result of the PET image. If the input is a CT image, the output is the segmentation result of the CT image. Such a structure cannot make full use of multi-modal information, nor can it adapt to multi-modal application scenarios.

[0136] The present invention proposes a method to enhance the model's tumor segmentation capability, that is, to use the information of the segmented organs to assist in tumor segmentation. It can handle multi-modal input and output. In other words, it can accept images of multiple modalities as input and output the segmentation results of one modality. For example,

[0137] Recombining the medical image data containing organ segmentation obtained during training with the original input PET modality data and the original input CT / MRI modality data into a new data set, and inputting the combined data set into the segmentation model network structure again for deep segmentation processing, only containing the medical image of tumor segmentation;

[0138] Recombining the medical image data containing organ segmentation obtained during training with the original input PET modality data and the predicted missing CT / MRI modality data into a new data set, and inputting the combined data set into the segmentation model network structure again for deep segmentation processing, only containing the medical image of tumor segmentation;

[0139] The medical image data containing organ segmentation obtained during training is reorganized into a new data set together with the original input CT / MRI modality data and the predicted missing PET modality data, and the combined data set is again input into the segmentation model network structure for deep segmentation processing, only containing medical images with tumor segmentation.

[0140] The model structure of the present invention can make full use of multimodal information to improve the accuracy and robustness of segmentation, and it can use the information of the segmented organs to assist in the segmentation of tumors. In other words, it can use the image of the segmented organs as part of the encoder, combine its features with the features of PET and CT / MRI, and then pass it to the decoder to output the segmentation result of the tumor. Such a structure can use the information of the position and shape of the organs to constrain and guide the segmentation of the tumor. This can improve the accuracy and efficiency of segmentation and reduce the risk of mis-segmentation.

[0141] 5) Model testing;

[0142] For this tumor and organ segmentation task, the general evaluation indicators are as follows:

[0143] Precision and Recall: These two indicators are based on pixel classification, which respectively represent the proportion of pixels correctly classified as consistent with the label to all pixels predicted to be consistent with the label, and the proportion of pixels correctly classified as consistent with the label to all pixels that are actually consistent with the label.

[0144] The formula for calculating precision is:

[0145] (4)

[0146] The calculation formula for recall is:

[0147] (5)

[0148] In equations (4) and (5), TP (True Positive) represents the number of pixels correctly classified as consistent with the label, FP (False Positive) represents the number of pixels incorrectly classified, and FN (False Negative) represents the number of pixels incorrectly classified as background.

[0149] Dice Coefficient: This indicator is based on the overlap between the predicted and true segmented regions, indicating the ratio of the intersection of the two regions to their union. In deep learning segmentation tasks, the Dice coefficient is a commonly used evaluation indicator to measure the similarity and accuracy of segmentation results. The higher the Dice coefficient, the higher the overlap between the segmentation result and the true label, which generally means that the segmentation effect is better. The Dice coefficient ranges from 0 to 1, where 1 indicates a perfect segmentation match and 0 indicates no match. The calculation formula is as follows:

[0150] (6)

[0151] In formula (6), TP (True Positive) represents the number of pixels correctly classified as consistent with the label, FP (False Positive) represents the number of pixels misclassified, and FN (False Negative) represents the number of pixels misclassified as background.

[0152] At present, the model of the present invention has been tested on some public data sets, and the test results are shown in the table below.

[0153]

[0154] See also Figure 1 As shown, based on the above-mentioned trained multimodal nuclear medicine image segmentation model, the present invention also proposes a multimodal nuclear medicine image segmentation method, including:

[0155] Step 1) Acquire PET and / or CT / MRI images of the patient.

[0156] Step 2) Input the acquired PET and / or CT / MRI images into the trained multimodal nuclear medicine image segmentation model.

[0157] Step 3) Processing and analyzing the input PET and / or CT / MRI images using the multimodal nuclear medicine image segmentation model.

[0158] When the PET image and the CT / MRI image are simultaneously input, the multimodal nuclear medicine image segmentation model directly performs segmentation processing on the input PET image and CT image, thereby obtaining a medical image containing organ segmentation and tumor segmentation at the same time. The obtained medical image containing organ segmentation can also be used as input again with the PET image and the CT / MRI image, and the multimodal nuclear medicine image segmentation model performs deep segmentation processing, thereby obtaining a medical image containing only tumor segmentation.

[0159] When only a PET image is used as input, the multimodal nuclear medicine image segmentation model first uses the PET image to perform missing modality synthesis processing to generate a corresponding CT / MRI predicted image. Specifically, multiple available sequences in the input PET image are first encoded as corresponding features 2; then, by giving a task-specific code, multiple features 2 are fused to form a fused feature B; at the same time, by giving a task-specific code, multiple features 2 are fused using an attention mechanism for a specific task to form a fused feature A; finally, the fused feature A is fused with the fused feature B and then decoded to generate a missing CT / MRI image, that is, a CT / MRI predicted image. Then, the PET image and the CT / MRI predicted image are used as input again, and the multimodal nuclear medicine image segmentation model is made to perform segmentation processing, thereby simultaneously obtaining a medical image containing organ segmentation and tumor segmentation. The obtained medical image containing organ segmentation can also be used as input again with the PET image and the CT / MRI predicted image, and the multimodal nuclear medicine image segmentation model is made to perform deep segmentation processing, thereby obtaining a medical image containing only tumor segmentation.

[0160] 3) When only CT / MRI images are used as input, the multimodal nuclear medicine image segmentation model network first uses the CT / MRI images to perform missing modality synthesis processing to generate the corresponding PET prediction image. Specifically, multiple available sequences in the input CT / MRI images are first encoded as corresponding features 1; then, by giving task-specific codes, multiple features 1 are fused to form fusion feature B; at the same time, by giving task-specific codes, multiple features 1 are fused using the attention mechanism for specific tasks to form fusion feature A; finally, fusion feature A is fused with fusion feature B and then decoded to generate the missing PET image, i.e., the PET prediction image. Then, the CT / MRI image and the PET prediction image are used as input again, and the multimodal nuclear medicine image segmentation model is made to perform segmentation processing, thereby obtaining a medical image containing organ segmentation and tumor segmentation at the same time. The obtained medical image containing organ segmentation can also be used as input again with the CT / MRI image and the PET prediction image, and the multimodal nuclear medicine image segmentation model is made to perform deep segmentation processing, thereby obtaining a medical image containing only tumor segmentation.

[0161] Step 4) Generate a medical image containing both organ segmentation and tumor segmentation or only generate a medical image containing tumor segmentation.

[0162] See also Figure 7As shown, taking a data set with only CT images as an example, the data set containing CT images is first input into the multimodal nuclear medicine image segmentation model of the present invention, the missing modality processing structure in the model first performs a missing modality generation process, and uses the input CT image to synthesize the corresponding PET prediction image, and then the input CT image and the synthesized PET prediction image are reconstructed into a mixed data set, and then input into the multimodal nuclear medicine image segmentation model of the present invention again, and finally the segmentation model network structure in the model performs a tumor and organ segmentation process, and uses the input CT image and the PET prediction image to generate a medical image containing tumor segmentation and organ segmentation.

[0163] Of course, the modal nuclear medicine image segmentation model of the present invention can also be applied to mixed data sets with only CT / MRI images, only PET images, and both PET and CT / MRI images. When the mixed data set is input into the modal nuclear medicine image segmentation model of the present invention, the model will automatically distinguish the data type. For data with both PET and CT / MRI images, tumor and organ segmentation will be performed directly. For data with only CT / MRI images or only PET images, the missing modal images will be synthesized first, and then combined into a new data set before tumor and organ segmentation is performed, thereby realizing the self-adaptation of multi-modal nuclear medicine image segmentation.

[0164] The present invention also provides a multimodal nuclear medicine image segmentation system, comprising a data receiving unit, a data processing unit and a data output unit.

[0165] The data receiving unit is used to receive PET and / or CT images. This usually refers to the input method of image data, which may be obtained from a camera, an image file or other data source. In the present invention, the image data is usually input into the model in the form of a tensor.

[0166] The data processing unit is used to process and analyze the PET and / or CT / MRI images according to the multimodal nuclear medicine image segmentation method to obtain corresponding organ segmentation images and / or tumor segmentation images. According to different functions, the data processing unit can also be subdivided into a preprocessing unit, a feature extraction unit, a missing modality processing unit and a segmentation unit.

[0167] The preprocessing unit is used to perform necessary preprocessing on the input image to improve the performance of the model, which may include image scaling, cropping, normalization, denoising and other operations.

[0168] The feature extraction unit is usually composed of convolutional layers in a deep neural network, which is responsible for extracting features from the input image. These features are very important for segmentation tasks because they can help the model understand the semantic information of different regions in the image.

[0169] The missing modality processing unit is usually composed of an encoder, a multi-sequence fusion module, and a decoder. The available multi-sequence modality data are first encoded into features separately. The features from multiple input sequences are then fused by giving a task-specific code, which uses binary codes to identify the source and target. Finally, the fused features are passed through the decoder. In order to exploit the shared information between sequences, arbitrary sequence synthesis is integrated into a single decoder and encoder.

[0170] The segmentation unit is one of the core parts of the whole system, which is usually composed of architectures such as the encoder-decoder structure of a convolutional neural network (CNN). The task of the segmentation unit is to map the feature map (generated by the feature extraction unit) to pixel-level predictions to determine which category or object each pixel belongs to in the image.

[0171] The data output unit is used to display the obtained organ segmentation image and / or tumor segmentation image. The output of the present invention is usually a processed image, in which each pixel is assigned a category label. The data output unit is responsible for converting the output of the model into a form that is easy to understand or subsequently processed, such as a visual segmentation result map or labeled data in a specific format.

[0172] These units are usually implemented on hardware platforms through software frameworks (such as TensorFlow, PyTorch, etc.), and the hardware platforms can be CPUs, GPUs, TPUs, etc. When processing large-scale data, acceleration hardware such as GPUs or TPUs are usually used to increase the speed of training and reasoning. This part mainly involves the calculation of graphics cards and processors.

[0173] The present invention also proposes a computer device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus, and the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the above-mentioned multimodal nuclear medicine image segmentation method.

[0174] The present invention also proposes a computer-readable storage medium, in which at least one executable instruction is stored, and the executable instruction enables a processor to perform operations corresponding to the above-mentioned multimodal nuclear medicine image segmentation method.

[0175] The computer-readable storage medium generally also stores the following data:

[0176] 1) Model parameters: Deep learning models usually contain a large number of parameters, such as the weights and biases of neural networks. These parameters need to be saved on the storage medium during training and loaded into the memory for prediction during inference.

[0177] 2) Training data: Training models usually requires a large amount of data, including input images and corresponding labels (such as pixel-level segmentation annotations). These training data are usually stored on a hard disk or distributed storage system, and loaded into memory batch by batch during the training process for the model to use.

[0178] 3) Validation data: During the training process, validation data is usually used to evaluate the performance of the model and adjust hyperparameters. Validation data is similar to training data and also needs to be stored on a storage medium.

[0179] 4) Model checkpoint: In order to prevent information loss caused by unexpected interruptions during training, model checkpoints are usually saved regularly. These checkpoints contain the parameter status of the current model and training progress information, and can be used to resume training from the last saved state when needed.

[0180] 5) Prediction results: In the inference phase, the model performs segmentation prediction on the input image and generates results. These prediction results usually need to be saved on a storage medium for subsequent analysis, visualization or other applications.

[0181] The computer-readable storage medium of the present invention may be a hard disk, a solid-state drive (SSD), a network storage (such as cloud storage), etc. When training a large-scale model or processing a large-scale data set, it is usually considered to use a high-speed storage medium to improve the efficiency of data reading and writing.

[0182] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multimodal nuclear medicine image segmentation method, characterized in that: include: Step 1) Acquire PET and / or CT / MRI images of the patient; Step 2) inputting the acquired PET and / or CT / MRI images into the trained multimodal nuclear medicine image segmentation model; Step 3) using the multimodal nuclear medicine image segmentation model to process and analyze the input PET and / or CT / MRI images; Step 4) generating a medical image containing both organ segmentation and tumor segmentation or generating a medical image containing only tumor segmentation; The method of generating medical images containing organ segmentation and tumor segmentation at the same time is as follows: 1) When the PET image and the CT / MRI image are simultaneously used as input, the multimodal nuclear medicine image segmentation model directly performs segmentation processing on the input PET image and CT image, thereby simultaneously obtaining a medical image containing organ segmentation and tumor segmentation; 2) When only the PET image is used as input, the multimodal nuclear medicine image segmentation model first uses the PET image to perform missing modality synthesis processing to generate a corresponding CT / MRI prediction image, and then uses the PET image and the CT / MRI prediction image as input again, and the multimodal nuclear medicine image segmentation model performs segmentation processing, thereby simultaneously obtaining a medical image containing organ segmentation and tumor segmentation; 3) When only CT / MRI images are used as input, the multimodal nuclear medicine image segmentation model network first uses the CT / MRI images to perform missing modality synthesis processing to generate a corresponding PET prediction image, and then uses the CT / MRI images and the PET prediction image as input again, and the multimodal nuclear medicine image segmentation model performs segmentation processing, thereby simultaneously obtaining a medical image containing organ segmentation and tumor segmentation; The way to generate a medical image containing only tumor segmentation is as follows: 1) The obtained medical image containing organ segmentation, the PET image, and the CT / MRI image are again used as input, and the multimodal nuclear medical image segmentation model is used to perform deep segmentation processing, thereby obtaining a medical image containing only tumor segmentation; 2) The obtained medical image containing organ segmentation, the PET image, and the CT / MRI predicted image are used as input again, and the multimodal nuclear medical image segmentation model is used to perform deep segmentation processing, thereby obtaining a medical image containing only tumor segmentation; 3) The obtained medical image containing organ segmentation, the CT / MRI image, and the PET prediction image are used as input again, and the multimodal nuclear medicine image segmentation model is made to perform deep segmentation processing, thereby obtaining a medical image containing only tumor segmentation.

2. The multimodal nuclear medicine image segmentation method according to claim 1, characterized in that: The missing mode synthesis processing method is: 1) Prediction of missing PET modalities: First, multiple available sequences in the input CT / MRI image are encoded as corresponding features 1 respectively; then, by giving task-specific codes, multiple features 1 are fused to form fused feature B; at the same time, by giving task-specific codes, multiple features 1 are fused using the attention mechanism for a specific task to form fused feature A; finally, fused feature A is fused with fused feature B and then decoded to generate the missing PET image, i.e., PET predicted image; 2) Prediction of missing CT / MRI modalities: First, multiple available sequences in the input PET image are encoded as corresponding features 2 respectively; then, by giving task-specific codes, multiple features 2 are fused to form fused feature B; at the same time, by giving task-specific codes, multiple features 2 are fused using the attention mechanism for a specific task to form fused feature A; finally, the fused feature A is fused with the fused feature B and then decoded to generate the missing CT / MRI image, that is, the CT / MRI predicted image.

3. The multimodal nuclear medicine image segmentation method according to claim 1, characterized in that: The multimodal nuclear medicine image segmentation model is composed of a segmentation model network structure and a missing modality processing structure; 1) The segmentation model network structure is a convolutional neural network structure based on U-Net, which includes a segmentation encoder module and a segmentation decoder module; The segmentation encoder module is a series of convolutional layers and pooling layers for extracting context information of an image; the segmentation encoder module includes two segmentation encoding units, one of which corresponds to a PET input and is responsible for performing convolution and pooling processing on an input PET image or a PET predicted image to generate a PET pooling feature map, and the other segmentation encoding unit corresponds to a CT / MRI input and is responsible for performing convolution and pooling processing on an input CT / MRI image or a CT / MRI predicted image to generate a PET pooling feature map; The segmentation decoder module is a series of upsampling layers and convolution layers, which are used to restore the resolution and precise positioning of the image; the segmentation decoder module includes two segmentation decoding units, one of which corresponds to the organ segmentation output, is responsible for upsampling and refining the features of the input PET pooling feature map and the CT / MRI pooling feature map, and converting them into organ segmentation images, and the other segmentation decoding unit corresponds to the tumor segmentation output, is responsible for upsampling and refining the features of the input PET pooling feature map and the CT / MRI pooling feature map, and converting them into tumor segmentation images; 2) The missing modality processing structure includes a synthesis encoder module, a multi-sequence fusion module and a synthesis decoder module; The synthetic encoder module is an encoder based on a recurrent neural network, which is used to receive an input sequence and encode it into features; the synthetic encoder module includes two synthetic encoder units, one of which corresponds to CT / MRI input and is responsible for receiving multiple sequences available for CT / MRI images and converting the multiple sequences into corresponding features 1; the other synthetic encoder unit corresponds to PET input and is responsible for receiving multiple sequences available for PET images and converting the multiple sequences into corresponding features 2; The multi-sequence fusion module is responsible for fusing feature 1 or feature 2 from multiple input sequences by giving task-specific codes to generate CT / MRI multi-sequence fusion features or PET multi-sequence fusion features; The synthetic decoder module is a decoder based on a recurrent neural network, which is used to gradually generate an output sequence according to the features generated by the encoder; the synthetic decoder module includes two synthetic decoder units, one of which corresponds to the PET prediction output and is responsible for decoding the CT / MRI multi-sequence fusion features and converting them into the corresponding PET prediction image, and the other synthetic decoder unit corresponds to the CT / MRI prediction output and is responsible for decoding the PET multi-sequence fusion features and converting them into the corresponding CT / MRI prediction image.

4. The multimodal nuclear medicine image segmentation method according to claim 3, characterized in that: The training strategy of the multimodal nuclear medicine image segmentation model is as follows: 1) Dataset collection; Collect a certain number of public PET-CT / MRI datasets, and classify and annotate them according to different modalities and whether they are tumors or organs; 2) Dataset preprocessing; Preprocessing of the collected datasets using techniques including intensity normalization, cropping and padding, or data augmentation; Model training; Taking paired PET and CT / MRI modality data as two inputs, the segmentation model network structure directly performs segmentation processing on the input PET and CT / MRI modality data, and outputs a medical image containing both organ segmentation and tumor segmentation; Taking the PET modality data as an input and the input value set to -1 as another input to simulate the missing CT / MR modality data paired with the PET modality data, the missing modality processing structure first performs missing modality synthesis processing using the PET modality data to predict the missing CT / MRI modality data, and then reconstructs a new data set by combining the original input PET modality data and the predicted CT / MRI modality data, and inputs the reconstructed data set into the segmentation model network structure for segmentation processing, and outputs a medical image containing both organ segmentation and tumor segmentation; Taking CT / MRI modality data as an input and an input value set to -1 as another input to simulate missing PET modality data paired with the CT / MRI modality data, the missing modality processing structure first performs missing modality synthesis processing using the CT / MRI modality data to predict the missing PET modality data, and then reconstructs a new data set by combining the original input CT / MRI modality data and the predicted PET modality data, and inputs the reconstructed data set into the segmentation model network structure for segmentation processing, and outputs a medical image containing both organ segmentation and tumor segmentation; 4) Model enhancement; Recombining the medical image data containing organ segmentation obtained during training with the original input PET modality data and the original input CT / MRI modality data into a new data set, and inputting the combined data set into the segmentation model network structure again for deep segmentation processing, only containing the medical image of tumor segmentation; Recombining the medical image data containing organ segmentation obtained during training with the original input PET modality data and the predicted missing CT / MRI modality data into a new data set, and inputting the combined data set into the segmentation model network structure again for deep segmentation processing, only containing the medical image of tumor segmentation; The medical image data containing organ segmentation obtained during training is reorganized into a new data set together with the original input CT / MRI modality data and the predicted missing PET modality data, and the combined data set is again input into the segmentation model network structure for deep segmentation processing, only containing medical images with tumor segmentation.

5. The multimodal nuclear medicine image segmentation method according to claim 4, characterized in that: The training process of the missing mode processing structure is as follows: Before inferring the missing modality, inputting the real existing PET-CT / MRI data into the missing modality processing structure; The two synthetic encoder units in the missing modality processing structure receive CT / MRI and PET modality data respectively, convert them into their own features 1 and features 2, and concatenate the features 1 and features 2 of the two modalities; Then, for different features 1 and 2, feature 1 and feature 2 are fused by giving specific task codes; Finally, the new features 1 and 2 after fusion are respectively passed through two synthesis decoder units in the missing modality processing structure to output the predicted PET modality or CT / MRI modality.

6. A multimodal nuclear medicine image segmentation system, characterized in that: It includes a data receiving unit, a data processing unit and a data output unit; The data receiving unit is used to receive PET and / or CT images of the patient; The data processing unit is used to process and analyze the PET and / or CT images according to the multimodal nuclear medicine image segmentation method according to any one of claims 1 to 5 to obtain corresponding organ segmentation images and / or tumor segmentation images; The data output unit is used to display the obtained organ segmentation image and / or tumor segmentation image.

7. A computer device, characterized in that: include: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus, and the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the multimodal nuclear medicine image segmentation method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the multimodal nuclear medicine image segmentation method as described in any one of claims 1 to 5.

Images