Image segmentation method, model, device and storage medium based on modality specificity

Abstract

The application relates to image processing technology, and provides an image segmentation method based on modality specificity, which generates modality-specific features corresponding to to-be-segmented images through a generator; according to a segmentation module and the modality-specific features, corresponding to-be-segmented images are segmented to generate segmented images; the segmented images and corresponding labeled images are discriminated through a discriminator; and an image training network is trained through a loss function. In the manner, the modality-specific features of to-be-segmented images of different modalities are taken as the basis to independently segment the to-be-segmented images of different modalities, the image training network is trained according to the loss functions corresponding to different modalities, the segmentation data of the image training network for different modality images is corrected, the segmentation precision of segmented images of different modalities is improved, and high-precision segmented images are output.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a modality-specific image segmentation method, a model and a storage medium. BACKGROUND

[0002] PET-CT scanning equipment is mainly applied to early detection and diagnosis of major diseases such as tumors, brain and heart. PET-CT scanning equipment can scan to obtain dual-modality PET-CT images, CT images provide anatomical information, indicating the location and boundary of normal tissues, PET images provide physiological metabolic information, which is used to determine the location of tumors, and the combination of the two modalities provides more accurate reference for target delineation.

[0003] Although PET-CT images have been widely used in clinical practice, segmentation of lung tumors is still a challenging task in the field of medical image processing. When PET images and CT images display lung tumors, there is a large difference in tumor intensity, the shape and size of tumors in PET-CT images are different, and the registration of the two modalities generated by PET-CT scanning is not perfect, resulting in low contrast between tumors and surrounding tissues, and the identification of tumors is easily disturbed by adjacent organs or tissues, and it is difficult to distinguish the boundaries of tumors, so it is difficult to accurately segment the tumor region in multi-modal images. Therefore, how to solve the low segmentation accuracy of the multi-modal images of the existing PET-CT scanning has become a technical problem to be solved at present. SUMMARY

[0004] The main purpose of the present application is to provide a modality-specific image segmentation method, which aims to solve the technical problem of low segmentation accuracy of multi-modal images of the existing PET-CT scanning.

[0005] To achieve the above-mentioned purpose, the present application provides a modality-specific image segmentation method, which comprises: obtaining a to-be-segmented image of different modal images, generating a modality-specific feature corresponding to the to-be-segmented image by a generator; segmenting the corresponding to-be-segmented image according to the segmentation module corresponding to the different modal images and the modality-specific feature, and generating a segmented image; discriminating the segmented image and the modality-specific feature by a discriminator, comparing the segmented image with a labeled image corresponding to the to-be-segmented image, and discriminating the authenticity of the segmented image based on a modality discriminator in the discriminator; if the result of the discrimination is false, training an image training network formed by the generator and the discriminator through a loss function, and segmenting the to-be-segmented image through the trained network to output a high-precision segmented image.

[0006] In addition, to achieve the above object, the present application also provides a modal-specific image segmentation device, which comprises a modal-specific feature generation module, a segmentation image acquisition module, a segmentation image discrimination module and a network training module.

[0007] In addition, to achieve the above object, the present application also provides a modal-specific image segmentation device, which comprises a modal-specific feature generation module, a segmentation image acquisition module, a segmentation image discrimination module and a network training module.

[0008] In addition, to achieve the above object, the present application also provides a modal-specific image segmentation device, which comprises a modal-specific feature generation module, a segmentation image acquisition module, a segmentation image discrimination module and a network training module.

[0009] The application provides a modal-specific-based image segmentation method, which acquires to-be-segmented images of different modal images, generates modal-specific features corresponding to the to-be-segmented images through a generator, segments the corresponding to-be-segmented images according to segmentation modules corresponding to the different modal images and the modal-specific features, generates segmented images, discriminates the segmented images and the modal-specific features through a discriminator, compares the segmented images with labeled images of the corresponding to-be-segmented images, discriminates the authenticity of the segmented images, trains an image training network formed by the generator and the discriminator through a loss function if the discrimination result is false, segments the to-be-segmented images through the trained network, and outputs high-precision segmented images. In the above manner, the application segments to-be-segmented images of different modalities independently according to modal-specific features of to-be-segmented images of different modalities, trains an image training network according to loss functions corresponding to different modalities, corrects segmentation data of the image training network for different modal images, improves the segmentation precision of segmented images of different modalities, and thus outputs high-precision segmented images, thereby solving the technical problem of low segmentation precision of multi-modal images of PET-CT scanning. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 A hardware structure schematic diagram of a modal-specific-based image segmentation device involved in an embodiment scheme of the application is shown.

[0011] Figure 2 A flowchart of a first embodiment of the modal-specific-based image segmentation method of the application is shown.

[0012] Figure 3 A flowchart of a second embodiment of the modal-specific-based image segmentation method of the application is shown.

[0013] Figure 4 A flowchart of a third embodiment of the modal-specific-based image segmentation method of the application is shown.

[0014] Figure 5 A flowchart of a fourth embodiment of the modal-specific-based image segmentation method of the application is shown.

[0015] Figure 6 A functional module schematic diagram of a first embodiment of the modal-specific-based image segmentation device of the application is shown.

[0016] Figure 7 A flowchart of a specific implementation of the generator provided by the application is shown.

[0017] The implementation, functional features and advantages of the application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0018] It should be understood that the specific embodiments described herein are merely exemplary and do not limit the application.

[0019] The image segmentation method based on modality specificity related in the embodiments of the application is mainly applied to an image segmentation device based on modality specificity, which can be a PC, a portable computer, a mobile terminal or the like having a display and processing function.

[0020] With reference to Figure 1 , Figure 1 The figure is a schematic diagram of the hardware structure of the image segmentation device based on modality specificity related in the embodiments of the application. In the embodiments of the application, the image segmentation device based on modality specificity can include a processor 1001 (for example, a CPU), a communication bus 1002, a user interface 1003, a network interface 1004 and a memory 1005. The communication bus 1002 is used to realize the connection and communication among the components; the user interface 1003 can include a display screen (Display) and an input unit such as a keyboard (Keyboard); the network interface 1004 can optionally include a standard wired interface and a wireless interface (for example, a WI-FI interface); the memory 1005 can be a high-speed RAM memory or a stable memory (non-volatile memory) such as a disk memory, and the memory 1005 can optionally be a storage device independent of the aforementioned processor 1001.

[0021] Those skilled in the art can understand that the hardware structure shown in Figure 1 does not constitute a limitation on the image segmentation device based on modality specificity, and can include more or fewer components than those shown, or combine certain components, or different component arrangements.

[0022] With reference to Figure 1 , Figure 1 The memory 1005 as a computer readable storage medium in the embodiments of the application can include an operating system, a network communication module and an image segmentation program based on modality specificity.

[0023] In the embodiments of the application, the network communication module is mainly used to connect a server and communicate data with the server; and the processor 1001 can call the image segmentation program based on modality specificity stored in the memory 1005 and execute the image segmentation method based on modality specificity provided by the embodiments of the application. Figure 1

[0024] The embodiments of the application provide an image segmentation method based on modality specificity.

[0025] With reference to Figure 2 and​Figure 7 , Figure 2 The flowchart of the first embodiment of the image segmentation method based on the modality specificity of the present application is shown in the figure, Figure 7 The flowchart of the specific embodiment of the generator provided by the present application is shown in the figure.

[0026] In this embodiment, the image segmentation method based on the modality specificity comprises the following steps:

[0027] In step S10, the image to be segmented of different modalities is obtained, and the modality-specific features corresponding to the image to be segmented are generated by the generator.

[0028] In this embodiment, PET images and CT images are used as the images to be segmented. The PET image to be segmented and the CT image to be segmented are subjected to the extraction of modality-specific features by the generator. The generator needs to process the PET image to be segmented and the CT image to be segmented separately, so the generator includes a PET encoder, a CT encoder, a fusion decoder, a PET modality-specific decoder, and a CT modality-specific decoder.

[0029] The PET encoder and the CT encoder are respectively used to generate modality features of different levels for the PET image to be segmented and the CT image to be segmented, and the fusion decoder is used to fuse the two kinds of modality features to generate multi-modality features.

[0030] The PET modality-specific decoder is designed to integrate the modality-specific features of different levels of the PET image, and to fuse the modality-specific features of the PET image with the multi-modality features to generate the modality-specific features of the PET image.

[0031] The CT modality-specific decoder is designed to integrate the modality-specific features of different levels of the CT image, and to fuse the modality-specific features of the CT image with the multi-modality features to generate the modality-specific features of the CT image.

[0032] In step S20, the corresponding image to be segmented is segmented according to the segmentation module corresponding to the different modalities and the modality-specific features, to generate a segmented image.

[0033] In this embodiment, the step S20 specifically comprises:

[0034] According to the PET modality-specific features, the PET image to be segmented is segmented by the PET segmentation module to obtain the segmented image of the PET image.

[0035] According to the CT modality-specific features F CT , the CT image to be segmented is segmented by the CT segmentation module S CT to obtain the segmented image of the CT image.

[0036] In this embodiment, the PET modality-specific segmentation is:

[0037] y PET '=S PET (F PET )(6)

[0038]

[0039] wherein S PET is a 3x3 convolution operation, Cov 3×3 .

[0040] Further, the CT modality-specific segmentation is:

[0041] y CT '=S CT (F CT )(8)

[0042]

[0043] wherein S CT is a 3x3 convolution operation.

[0044] In this embodiment, according to the extracted modality-specific features of the PET image and the modality-specific features of the CT image, the PET image to be segmented is segmented by the PET segmentation module to generate a PET segmentation image; the CT image to be segmented is segmented by the CT segmentation module to generate a CT segmentation image.

[0045] In step S30, the discriminator discriminates the segmentation image and the modality-specific features, compares the segmentation image with the labeled image corresponding to the image to be segmented, and discriminates the authenticity of the segmentation image.

[0046] In this embodiment, the PET labeled image manually labeled and the PET segmentation image are input into the PET discriminator. When the PET labeled image is input into the discriminator, it is determined to be true. The data of the PET labeled image is used as a basis to set preset standard data, and the data of the input PET segmentation image is compared and detected. If the data of the PET segmentation image fails to reach the preset standard data, the PET segmentation image is determined to be false.

[0047] It can be understood that the CT labeled image and the CT segmentation image are input into the CT discriminator. When the CT labeled image is input into the discriminator, it is determined to be true. The data of the CT labeled image is used as a basis to set preset standard data, and the data of the input CT segmentation image is compared and detected. If the data of the CT segmentation image fails to reach the preset standard data, the CT segmentation image is determined to be false.

[0048] If the result of the determination is false, the image training network formed by the generator and the discriminator is trained through a loss function, and the to-be-segmented image is segmented through the trained network to output a high-precision segmented image.

[0049] In this embodiment, the image training network is trained through a loss function. If the number of training cycles of the image training network reaches the set number of cycles, the network can be used to process segmented images that need to be segmented at this time, and the discriminator discriminates the re-segmented images. When the discriminator determines that the segmented image is true, it means that the segmentation data of the segmented image is close to the annotation data of the manually annotated image, and the segmentation accuracy reaches the preset standard. The segmented image is a high-precision segmented image.

[0050] The present application provides a kind of based on modal specificity's image segmentation method, the method obtains the to-be-segmented image of different modal image, generates the modal specific feature corresponding to the to-be-segmented image by generator;According to the segmentation module corresponding to different modal images and the modal specific feature, the corresponding to-be-segmented image is segmented, and segmented image is generated;The discriminator is discriminated to the segmented image and the modal specific feature, the segmented image and the annotation image of corresponding to-be-segmented image are compared, and the true and false of the segmented image is discriminated;If the result of the determination is false, the image training network formed by the generator and the discriminator is trained through a loss function, and the to-be-segmented image is segmented through the trained network to output a high-precision segmented image. Through the above-mentioned mode, the modal specific feature of different modal to-be-segmented image is taken as the basis to independently segment the to-be-segmented image of different modal, and the image training network is trained according to the loss function corresponding to different modal, the segmentation data of the image training network of different modal image is corrected, the segmentation accuracy of different modal segmented image is improved, so as to output high-precision segmented image, solve the technical problem of low segmentation accuracy of multi-modal image of PET-CT scanning at present.

[0051] Reference Figure 3 , Figure 3 It is the flowchart of the second embodiment of the image segmentation method based on modal specificity of the present application.

[0052] Based on the above Figure 2 The embodiment, the step S10 specifically includes:

[0053] Step S11, generate PET image modal features and CT image modal features of different levels through an image encoder;Wherein, the image encoder includes PET encoder and CT encoder;

[0054] In this embodiment, the PET encoder includes 5 convolution modules: conv1_pet, conv2_pet, conv3_pet, conv4_pet, and conv5_pet.

[0055] The formula of the convolution module is:

[0056] DCov 3×3 (F in )=Cov 3×3 (Cov 3×3 (MP(F in ))), (1)

[0057] Cov 3×3 (F in )=ReLU(BN(conv 3×3 (F in ))), (2)

[0058] conv 3×3 denotes a 3x3 convolution operation, BN denotes batch normalization, ReLU denotes an activation function, F in denotes an input feature vector, and MP denotes maximum pooling.

[0059] It can be understood that the CT encoder includes 5 convolution modules: conv1_ct, conv2_ct, conv3_ct, conv4_ct, and conv5_ct, and the formula of the convolution module also uses (1)-(2).

[0060] In this embodiment, the PET encoder generates different levels of modality features for the PET image to be segmented, the CT encoder generates different levels of modality features for the CT image to be segmented, and the dual-stream encoder performs modality feature calculation on the two types of modality features.

[0061] In step S12, the PET image modality features and the CT image modality features are fused by the fusion encoder to generate fusion image multi-modal features.

[0062] In this embodiment, the fusion decoder includes 5 convolution modules: fuse1, fuse2, fuse3, fuse4, and fuse5.

[0063] The formula of the convolution module is:

[0064]

[0065] Up is an upsampling operation of 2 times (l=5, no operation is needed), denotes a feature vector output by the lth convolution block convl_pet of the PET encoder (l=1, 2, …, 5), The convl_ct represents the feature vector output by the l-th convolutional block of the CT encoder, and fusel represents the l-th feature fusion module. This represents the feature vector of the fusel output by the l-th feature fusion module.

[0066] In this embodiment, the modal features of PET images and CT images are simultaneously input into the fusion decoder. The modal features of the two images are fused by the convolution module in the fusion decoder to generate multimodal features.

[0067] Step S13: Integrate modality-specific features of different levels of the corresponding image according to different modality-specific decoders, and fuse the modality-specific features with the multimodal features to generate modality-specific features of the corresponding image;

[0068] In this embodiment, firstly, different modality-specific decoders integrate modality-specific features at different levels of the corresponding image. Then, the different modality-specific decoders fuse the integrated modality-specific features with multimodal features to generate modality-specific features of the corresponding image.

[0069] Furthermore, in this embodiment, step S13 specifically includes:

[0070] Integrate PET modality-specific features at different levels of PET images using a PET modality-specific decoder;

[0071] Integrate CT modality-specific features from different levels of CT images using a CT modality-specific decoder;

[0072] The PET modality-specific features and the multimodal features are fused using the PET modality-specific decoder to generate fused PET modality-specific features;

[0073] The CT modality-specific features and the multimodal features are fused using the CT modality-specific decoder to generate fused CT modality-specific features.

[0074] In this embodiment, the PET modality-specific decoder includes four convolutional modules: deconv1_pet, deconv2_pet, deconv3_pet, and deconv4_pet.

[0075] The formula for the convolution module is as follows:

[0076]

[0077] in, deconvl_pet(l = 1, 2, 3, 4) represents the lth convolutional block of the PET encoder, and outputs a feature vector, convl_pet(l = 1, 2, 3, 4) represents the lth convolutional block of the PET encoder, and outputs a feature vector.

[0078] Further, the CT modality-specific decoder comprises four convolutional modules: deconv1_ct, deconv2_ct, deconv3_ct, and deconv4_ct.

[0079] The formula of the convolutional module is as follows:

[0080]

[0081] wherein, deconvl_ct(l = 1, 2, 3, 4) represents the lth convolutional block of the CT encoder, and outputs a feature vector, convl_ct(l = 1, 2, 3, 4) represents the lth convolutional block of the CT encoder, and outputs a feature vector.

[0082] In this embodiment, the PET image to be segmented and the CT image to be segmented are respectively input into the generator, and the PET image to be segmented and the CT image to be segmented are subjected to modality feature extraction, modality feature fusion, modality-specific extraction, and fusion of modality features and modality-specific features through the different modality encoders and decoders in the generator, so as to generate the modality-specific features corresponding to the PET image to be segmented and the CT image to be segmented, respectively.

[0083] Referring to Figure 4 , Figure 4 FIG. 3 is a flowchart of a third embodiment of the image segmentation method based on modality specificity according to the present application.

[0084] Based on the above Figure 2 described embodiments, in this embodiment, the step S30 comprises:

[0085] In step S31, the PET segmentation discriminator is used to compare the segmentation image corresponding to the PET image to be segmented with the labeled image of the PET image to be segmented, so as to determine the authenticity of the segmentation image.

[0086] In step S32, the CT segmentation discriminator is used to compare the segmentation image corresponding to the CT image to be segmented with the labeled image of the CT image to be segmented, so as to determine the authenticity of the segmentation image.

[0087] In step S33, the modality discriminator is used to discriminate the modality-specific features corresponding to the multiple modality images, so as to determine the authenticity of the modality-specific features.

[0088] In this embodiment, the discriminator takes the different modalities of the labeled images marked by humans as the standard to compare the segmentation image, and judges whether the image or feature input into the discriminator is true.

[0089] In the modal discriminator, the discrimination is based on a modality-specific feature, such as a PET modality-specific feature. When the PET modality-specific feature is input into the modal discriminator, the discrimination result of the modal discriminator is true. When the CT modality-specific feature is output, if there is a large feature difference between the CT modality-specific feature and the PET modality-specific feature, the discrimination result of the modal discriminator is false. At this time, the network needs to be trained through the loss function, and the data of the generator and the discriminator are corrected, and then the segmentation of the to-be-segmented image is performed, until the high-precision segmentation image is output.

[0090] Further, the process of training the generator and the discriminator through the loss function is to continuously correct and learn the generator and the discriminator, so as to improve the segmentation accuracy of the segmentation image generated by the generator.

[0091] It can be understood that the segmentation image generated by the generator is to continuously "deceive" the discriminator, and the discriminator is responsible for discriminating the output information of the generator and judging the authenticity of the output information. In the process of continuously "deceiving" the discriminator, the generator needs to continuously correct the output information so that the output information gradually approaches the true information, so as to successfully "deceive" the discriminator. The process of continuously correcting the generator is the network training process.

[0092] Reference Figure 5 , Figure 5 is a flowchart of a third embodiment of the image segmentation method based on modality specificity of the present application.

[0093] Based on the above Figure 3 embodiment, the step S40 specifically includes:

[0094] Step S41, if the segmentation image and / or the modality-specific feature is false, the loss function is called to train the image network;

[0095] Step S42, the to-be-segmented image is segmented and discriminated by the trained image network;

[0096] Step S43, if the discrimination result is true, the segmentation image is output as the high-precision segmentation image.

[0097] In this embodiment, first, the discriminator discriminates the authenticity of the output image, and based on the manually labeled image data, the segmented image data is detected to determine whether the segmented image data can meet certain accuracy requirements. If it cannot meet the requirements, the segmented image is determined to be false. At this time, the image training network needs to be trained through the loss function, that is, the data of the network is corrected.

[0098] In this embodiment, the loss function includes an adversarial loss L1 loss of PET L1 loss of CT Cross-entropy loss of PET Cross-entropy loss of CT Dice loss of PET Dice loss of CT Modal adversarial loss

[0099] Wherein, the adversarial loss is:

[0100]

[0101] Wherein, D PET represents a PET segmentation discriminator, D CT represents a CT segmentation discriminator, x PET represents a PET image to be segmented, y PET represents a labeled image of the PET image to be segmented, x CT represents a CT image to be segmented, y CT represents a labeled image of the CT image to be segmented, and E represents mathematical expectation.

[0102] Further, the L1 loss of PET

[0103]

[0104] Further, the L1 loss of CT is:

[0105]

[0106] Further, the cross-entropy loss of PET is:

[0107]

[0108] Wherein, H represents the height of the image, W represents the width of the image, y PET ' i represents the predicted value of the i-th pixel in the segmented image y PET 'PETi denotes the annotation map y PET denotes the annotation label of the i-th pixel in the segmentation map y PET ' i ∈ [0, 1] y PETi ∈ [0, 1].

[0109] Further, the cross-entropy loss of CT is:

[0110]

[0111] where y CT ' i denotes the prediction value of the i-th pixel in the segmentation map y CT ' CTi denotes the annotation map y CT denotes the annotation label of the i-th pixel in the annotation map y CT ' i ∈ [0, 1] y CTi ∈ [0, 1].

[0112] Further, the dice loss of PET is:

[0113]

[0114] Further, the dice loss of CT is:

[0115]

[0116] Further, the modality adversarial loss is:

[0117]

[0118] Specifically, the training of the generator S and the discriminators D PET , D CT , D M is a minimax game in terms of minimizing and maximizing the objective function, respectively, as follows:

[0119]

[0120]

[0121] where D M denotes the modality discriminator, and λ1 is a weight parameter.

[0122] Furthermore, to verify the superior performance of this invention, it is necessary to conduct tests using a PET-CT dataset of patients with non-small cell lung cancer to quantitatively evaluate the modality-specific image segmentation method provided in this application. The specific testing method is as follows:

[0123] This application uses the following four commonly used evaluation metrics in medical image segmentation as the evaluation criteria for experimental results: Dice coefficient (DSC), intersection-over-union ratio (IoU), precision, and recall.

[0124] Here, DSC is a function that evaluates the similarity between the predicted result and the manually labeled data. It is obtained by using the ratio of the intersection and union of the predicted result and the manually labeled data, and is defined as:

[0125]

[0126] Where TP (True Positive) represents the number of true positive pixels, FP (False Positive) represents the number of false positive pixels, and FN (False Negative) represents the number of false negative pixels.

[0127]

[0128]

[0129]

[0130] On the same dataset, the loss function and modal adversarial loss of this invention were sequentially added to the network for ablation experiments. The ablation experiments are shown in Table 1. The baseline in Table 1 is generated by generator S (excluding PET modality-specific decoder DE). PET CT modality-specific decoder DE CT ), PET segmentation module, CT segmentation module S CT Discriminator D PET D CT The network is constructed as follows: w / oNL indicates that no tumor-free images are added during training. Then, L1 loss function, cross-entropy loss function, DICE loss function, modal adversarial loss, and the segmentation accuracy of the modal discriminator are added in sequence.

[0131] Table 1 Ablation Experiment

[0132]

[0133] In Table 1, Mo represents the addition of the PET modality-specific decoder DE to the generator S. PET and CT modality-specific decoder DE CT , The modal adversarial loss is a maximum mean discrepancy way to F PET , CT The calculation, The modal adversarial loss is a maximum mean discrepancy way to F PET , CT The calculation, The modal adversarial loss is a gradient reversal layer way to F PET , CT The calculation, The adversarial loss prediction adversarial loss is a discriminator D p The calculation, PET The calculation, CT The calculation.

[0134] The method of the present application is compared with UNet, RCNet, UNet3d, UNet2.5d, VNet3d, nnU-Net, WNet, Co-learning and MSAM. The experimental results of different methods are shown in Table 2.

[0135] Table 2 Comparison of segmentation accuracy of different methods

[0136]

[0137] The method of the present application is significantly better than other methods, and the highest DSC, IoU and Recall are obtained. The t test is performed on DSC, and all p values are less than 0.05, indicating that the statistical significance of the aspect of the present application is higher than that of other methods. Through Figure 2 It can also be found that the method of the present application is better than other methods.

[0138] In addition, the embodiment of the present application also provides an image segmentation device based on modal specificity.

[0139] Referring to Figure 6 , Figure 6 The function module schematic diagram of the first embodiment of the image segmentation model based on modal specificity of the present application.

[0140] In this embodiment, the image segmentation model based on modal specificity comprises:

[0141] The modal specific feature generation module 10 is used for acquiring different modal images to be segmented, and generating modal specific features corresponding to the images to be segmented through a generator;

[0142] The segmented image acquisition module 20 is used for segmenting the corresponding images to be segmented according to the segmentation module corresponding to the different modal images and the modal specific features, and generating segmented images.

[0143] The segmentation image discrimination module 30 is configured to discriminate the segmentation image and the modality-specific feature by the discriminator, compare the segmentation image with the labeled image corresponding to the image to be segmented, and discriminate whether the segmentation image is true or false.

[0144] The network training module 40 is configured to train the image training network formed by the generator and the discriminator by using a loss function if the discrimination result is false, and segment the image to be segmented by using the trained network to output a high-precision segmentation image.

[0145] Further, the modality-specific feature generation module 10 specifically comprises:

[0146] The modality feature generation unit is configured to generate PET image modality features and CT image modality features of different levels by using an image encoder; wherein the image encoder comprises a PET encoder and a CT encoder.

[0147] The multi-modality feature generation unit is configured to fuse the PET image modality features and the CT image modality features by using a fusion encoder to generate fused multi-modality features of images.

[0148] The modality-specific feature generation unit is configured to integrate modality-specific features of different levels of corresponding images according to different modality-specific decoders, fuse the modality-specific features with the multi-modality features, and generate modality-specific features of the corresponding images.

[0149] Further, the modality-specific feature generation unit specifically comprises:

[0150] The PET modality-specific feature integration subunit is configured to integrate PET modality-specific features of different levels of PET images by using a PET modality-specific decoder.

[0151] The CT modality-specific feature integration subunit is configured to integrate CT modality-specific features of different levels of CT images by using a CT modality-specific decoder.

[0152] Further, the modality-specific feature generation unit specifically further comprises:

[0153] The PET modality-specific feature generation subunit is configured to fuse the PET modality-specific features and the multi-modality features by using the PET modality-specific decoder to generate fused PET modality-specific features.

[0154] The CT modality-specific feature generation subunit is configured to fuse the CT modality-specific features and the multi-modality features by using the CT modality-specific decoder to generate fused CT modality-specific features.

[0155] Further, the segmented image acquisition module 20 specifically comprises:

[0156] a PET segmented image acquisition unit, configured to acquire the segmented image of the PET image by performing segmentation on the PET image to be segmented through a PET segmentation module according to the PET modality-specific feature;

[0157] a CT segmented image acquisition unit, configured to acquire the segmented image of the CT image by performing segmentation on the CT image to be segmented through a CT segmentation module according to the CT modality-specific feature.

[0158] Further, the segmented image discrimination module 30 specifically comprises:

[0159] a PET segmented image segmentation discrimination unit, configured to compare the segmented image corresponding to the PET image to be segmented with a labeled image of the PET image to be segmented through a PET segmentation discriminator, and determine the truth or falsity of the segmented image;

[0160] a CT segmented image segmentation discrimination unit, configured to compare the segmented image corresponding to the CT image to be segmented with a labeled image of the CT image to be segmented through a CT segmentation discriminator, and determine the truth or falsity of the segmented image;

[0161] a modality discrimination unit, configured to discriminate the modality-specific feature corresponding to the multiple modality images through a modality discriminator, and determine the truth or falsity of the modality-specific feature.

[0162] Further, the network training module 40 specifically comprises:

[0163] a network training unit, configured to, if the segmented image and / or the modality-specific feature is determined to be false, train the image training network by calling a loss function;

[0164] a to-be-segmented image re-segmentation unit, configured to segment and discriminate the to-be-segmented image through the trained image training network;

[0165] a high-precision segmented image output unit, configured to, if the determination result is true, output the segmented image as the high-precision segmented image.

[0166] Corresponding to each step in the above-mentioned image segmentation method embodiment based on modality specificity, each module in the above-mentioned image segmentation device based on modality specificity has the same function and implementation process, which will not be repeated here.

[0167] In addition, the embodiment of the present application also provides a computer readable storage medium.

[0168] The computer readable storage medium stores a modality-specific image segmentation program, wherein the modality-specific image segmentation program, when executed by a processor, implements the steps of the modality-specific image segmentation method as described above.

[0169] The method implemented when the modality-specific image segmentation program is executed can refer to the embodiments of the modality-specific image segmentation method of the present application, which will not be described herein again.

[0170] It should be noted that, in this document, the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or system that includes a list of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent in such a process, method, article, or system. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article, or system that includes the element.

[0171] The above-mentioned embodiment numbers of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0172] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0173] Those skilled in the art can clearly understand the above-mentioned embodiment method can be realized by means of software and the necessary general hardware platform, of course, can also be through hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application essentially or say the part of the prior art contribution can be embodied in the form of software products, the computer software product is stored in a storage medium (such as ROM / RAM, magnetic disc, optical disc) as described above, including a number of instructions to make a terminal device (may be a mobile phone, computer, server, air conditioner, or network equipment, etc.) executes the method described in various embodiments of the present application.

[0174] The above is only the preferred embodiment of the present application, not therefore limit the patent scope of the present application, any equivalent structure or equivalent flow transformation made by using the content of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for image segmentation based on modality specificity, characterized in that, The steps of the method comprise: acquiring an image to be segmented of different modalities, generating modality-specific features corresponding to the image to be segmented by a generator, comprising: generating PET image modality features and CT image modality features of different levels by an image encoder; wherein the image encoder comprises a PET encoder and a CT encoder; fusing the PET image modality features and the CT image modality features by a fusion encoder to generate fused multi-modal features; integrating PET modality-specific features of different levels of PET images by a PET modality-specific decoder, and integrating CT modality-specific features of different levels of CT images by a CT modality-specific decoder; fusing the PET modality-specific features and the multi-modal features by the PET modality-specific decoder to generate fused PET modality-specific features, and fusing the CT modality-specific features and the multi-modal features by the CT modality-specific decoder to generate fused CT modality-specific features; segmenting the corresponding image to be segmented according to the segmentation module corresponding to the different modalities and the modality-specific features to generate a segmented image; discriminating the segmented image and the modality-specific features by a discriminator, comparing the segmented image with a labeled image corresponding to the image to be segmented, and discriminating the authenticity of the segmented image based on a modality discriminator in the discriminator; if the result of the discrimination is false, training an image training network formed by the generator and the discriminator by a loss function, and segmenting the image to be segmented by the trained network to output a high-precision segmented image.

2. The method of image segmentation based on modality specificity according to claim 1, characterized in that, The segmentation of the corresponding image to be segmented according to the segmentation module corresponding to the different modalities and the modality-specific features to generate a segmented image comprises: segmenting a PET image to be segmented by a PET segmentation module according to the PET modality-specific features to obtain the segmented image of the PET image; segmenting a CT image to be segmented by a CT segmentation module according to the CT modality-specific features to obtain the segmented image of the CT image.

3. The method of image segmentation based on modality specificity of claim 1, wherein, The discrimination of the segmented image and the modality-specific features by the discriminator, the comparison of the segmented image with a labeled image corresponding to the image to be segmented, and the discrimination of the authenticity of the segmented image comprise: comparing the segmented image corresponding to the PET image to be segmented with a labeled image of the PET image to be segmented by a PET segmentation discriminator to determine the authenticity of the segmented image; comparing the segmented image corresponding to the CT image to be segmented with a labeled image of the CT image to be segmented by a CT segmentation discriminator to determine the authenticity of the segmented image; discriminating the modality-specific features corresponding to the multi-modal images by a modality discriminator to determine the authenticity of the modality-specific features.

4. The method of image segmentation based on modality specificity according to claim 3, characterized in that, If the result of the discrimination is false, the image training network formed by the generator and the discriminator is trained through a loss function, and the to-be-segmented image is segmented through the trained network to output a high-precision segmentation image, comprising: If it is determined that the segmentation image and / or the modality-specific feature is false, the image training network is trained through a loss function; Segmenting and discriminating the to-be-segmented image through the trained image training network; If the result of the discrimination is true, the segmentation image is output as the high-precision segmentation image.

5. An image segmentation apparatus based on modality specificity, characterized by, The modality-specific image segmentation model comprises: A modality-specific feature generation module is configured to obtain a to-be-segmented image of different modality images, and generate a modality-specific feature corresponding to the to-be-segmented image through a generator, comprising: Different levels of PET image modality features and CT image modality features are generated through an image encoder; wherein the image encoder comprises a PET encoder and a CT encoder; The PET image modality features and the CT image modality features are fused through a fusion encoder to generate a fused multi-modality feature; Different levels of PET modality-specific features of a PET image are integrated through a PET modality-specific decoder, and different levels of CT modality-specific features of a CT image are integrated through a CT modality-specific decoder; The PET modality-specific features and the multi-modality features are fused through the PET modality-specific decoder to generate a fused PET modality-specific feature, and the CT modality-specific features and the multi-modality features are fused through the CT modality-specific decoder to generate a fused CT modality-specific feature; A segmentation image acquisition module is configured to segment a corresponding to-be-segmented image according to a segmentation module corresponding to different modality images and the modality-specific feature to generate a segmentation image; A segmentation image discrimination module is configured to discriminate the segmentation image and the modality-specific feature through a discriminator, compare the segmentation image with a labeled image of the corresponding to-be-segmented image, and discriminate the authenticity of the segmentation image; A network training module is configured to train the image training network formed by the generator and the discriminator through a loss function if the result of the discrimination is false, and segment the to-be-segmented image through the trained network to output a high-precision segmentation image.

6. An image segmentation apparatus based on modality specificity, characterized by, The modality-specific image segmentation device comprises a processor, a memory, and a modality-specific image segmentation program stored on the memory and executable by the processor, wherein the modality-specific image segmentation program is executed by the processor to implement the steps of the modality-specific image segmentation method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a modality-specific image segmentation program, wherein the modality-specific image segmentation program is executed by the processor to implement the steps of the modality-specific image segmentation method according to any one of claims 1 to 4.

Images