A multimodality-assisted semi-supervised segmentation method for peritumoral edema in the non-dominant modality

Abstract

The present invention relates to the field of medical image processing technology, and discloses a semi-supervised segmentation method for peritumoral edema in a non-dominant modality assisted by multimodality, comprising: constructing a semi-supervised segmentation dataset for peritumoral edema, and representing the inputs of the two modalities as I1 and I2, and when there is a label, I i,L , I when there is no label i,U Two encoders are used to receive the two-modal images I1 and I2, and the initial feature mining is performed through the convolution layer, followed by multi-modal interpretable feature fusion. The fusion is repeated twice. After the encoder completes the third upsampling, the dual-branch output features are sent to the single-output modality fusion module to obtain the output features. After decoding using two identical decoders, the features are input to the consistency calculation module to calculate the consistency loss L. con The semi-supervised segmentation method provided by the present invention utilizes two or more non-dominant modalities in the absence of a dominant modality to form a semi-supervised segmentation with better segmentation effect than a single dominant modality.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a semi-supervised segmentation method for peritumoral edema in a non-dominant modality assisted by multimodality. Background Art

[0002] Brain tumors are one of the most common cancers worldwide, with gliomas being the most common malignant brain tumor, with varying degrees of invasiveness. In clinical practice, pixel-level delineation of various brain tumor regions, such as the tumor core and peritumoral edema, is necessary to aid diagnosis and treatment, in order to grade tumor malignancy during diagnosis, delineate the target area for brain tumors during radiotherapy, determine the extent of invasion of normal brain tissue based on the extent of peritumoral edema, and assess postoperative recovery.

[0003] Magnetic resonance imaging (MRI) is widely used in clinical practice because it has a variety of imaging sequences that provide rich, complementary information for analyzing tumor structure and tissue. Specifically, for gliomas, commonly used MRI sequences are T1-weighted (T1), T1-weighted after contrast (T1Gd), T2-weighted (T2), and T2 fluid-attenuated inversion recovery (T2-FLAIR) images. However, in clinical practice, patients are unable to fully acquire MRI sequences of various modalities, and some lesions are only apparent in specific sequences. Summary of the Invention

[0004] This paper provides a multimodality-assisted semi-supervised segmentation method for peritumoral edema in a non-dominant modality. The method considers the modality with the most prominent imaging feature of a specific lesion as the dominant modality, while the modality with the least distinct imaging feature is considered the non-dominant modality. Therefore, in the absence of a dominant modality, using peritumoral edema segmentation as an example, two or more non-dominant modalities (T1, T1ce, and T2) can be used to achieve a semi-supervised segmentation that outperforms a single dominant modality (Flair).

[0005] The present invention provides a semi-supervised segmentation method for peritumoral edema in a non-dominant modality assisted by multimodality, comprising: S1, constructing a semi-supervised segmentation dataset of peritumoral edema using two or more non-dominant modality images and manual annotations, and expressing the inputs of the two modalities as and , when they have labels they are represented as , when there is no label, it is represented as ;

[0006] S2, using two encoders to receive images of two modalities and After the initial feature mining is performed through the convolutional layer, the multi-modal interpretable feature fusion is performed through the modality fusion module;

[0007] S3, repeat step S2 twice. After the encoder completes the third upsampling, the output features of the two encoders are sent to the modality fusion module to obtain the output features after the first stage processing;

[0008] S4: After the output features processed in the first stage are decoded by two identical decoders, the output feature maps are input into the consistency calculation module to calculate the consistency loss. , completing the second stage of semi-supervised learning.

[0009] Furthermore, for a limited number of labeled After calculating the consistency loss Finally, the supervision loss is calculated using the gold standard .

[0010] Furthermore, a novel CMA-Skip connection is used in both the first and second stages to bridge the gap between multimodality and semi-supervision. First, the output features of the two encoders are concatenated together, and then a convolutional layer is used to enhance the representation. The sizes of the two convolution kernels are 1×1×1 and 3×3×3. Finally, the unprocessed original feature map output by the decoder is processed using an attention probability map ranging from 0 to 1. .

[0011] Furthermore, the mathematical expression of CMA-Skip connection is: in, , Represent the feature maps of different non-dominant modes respectively, CMA represents a series of convolution, normalization and activation functions, represents the unprocessed original feature map output by the decoder, It represents the comprehensive modal feature combined with another modal feature, which is used to be input into the original node again.

[0012] Furthermore, in the first stage, the modality fusion module obtains two inputs, namely, the i-th feature map output from the first encoder and the i-th feature map from the second encoder output ; First, and Add by voxel to get the common feature map , providing rich complementary modality information for subsequent feature fusion; then and as well as and After splicing, convolution operation is performed, and after obtaining the weight map, it is multiplied with the input feature map to obtain the two outputs of the modality fusion module. and ; When aggregation features are needed, only one output of the modality fusion module is needed That's it.

[0013] Furthermore, in step S4, in order to unify the two branches formed by the two decoders, the difference between the two predictions is penalized, and the mutual consistency loss is used as the loss function of the proposed network; for the output PM_1 of the first decoder, the pseudo label PL_2 obtained by another decoder is used to calculate the MSE, and then the consistency loss is obtained ; Using consistency loss As a guide for backpropagation and updating network parameters to make the two outputs similar, after consistency and entropy regularization, the two decoders reach a relative balance and achieve the goal of output consistency.

[0014] Furthermore, the supervision loss and consistency loss The formula is:

[0015]

[0016] in, represents the i-th probability map, Represents the jth pseudo label, L represents a limited number of labels, and the supervision loss is calculated using limited labels After that, the total loss is: .

[0017] The beneficial effects of the present invention are:

[0018] The present invention uses two or more non-dominant modality images and a small amount of manual annotation to construct a semi-supervised segmentation dataset for peritumoral edema, uses two interconnected encoders to perform three-dimensional multimodal feature fusion enhancement, and uses two completely independent decoders (four layers of continuous convolutional downsampling pairs) for consistency supervision, introducing multimodal segmentation in medical image segmentation tasks into semi-supervised segmentation, providing it with rich complementary information; using two or more non-dominant modalities (T1, T1ce, T2), a semi-supervised segmentation with better segmentation effect than a single dominant modality (Flair) is formed. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a schematic diagram of the structure adopted by the semi-supervised segmentation method of the present invention.

[0020] Figure 2 Schematic diagram of the segmentation results of the method of the present invention in a non-dominant mode.

[0021] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0022] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0023] In clinical practice, patients cannot fully acquire MRI sequences of various modalities, and some lesions are only evident in specific sequences. Therefore, in the absence of a dominant modality, taking the segmentation of peritumoral edema as an example, we utilize two or more non-dominant modalities (T1, T1ce, T2) to achieve a semi-supervised segmentation that outperforms a single dominant modality (Flair).

[0024] like Figure 1 As shown, the present invention provides a multimodal-assisted semi-supervised segmentation method for peritumoral edema in a non-dominant modality. It uses two interconnected encoders to perform three-dimensional multimodal feature fusion enhancement, and uses two completely independent decoders (four layers of continuous convolutional downsampling pairs) for consistency supervision. The acquired features are learned to obtain the optimal network parameters, and the segmentation results are evaluated and analyzed using numerical methods (Dice coefficient, Jaccard coefficient, 95% Hausdorff distance, ASD). The semi-supervised segmentation method specifically includes:

[0025] S1. Using two or more non-dominant modality images and a small amount of manual annotation, we construct a semi-supervised segmentation dataset for peritumoral edema, introduce multimodal segmentation in medical image segmentation tasks into semi-supervised segmentation, and provide it with rich complementary information; and represent the input of the two modalities as and , when they have labels they are represented as , when there is no label, it is represented as ;

[0026] S2, using two encoders to receive images of two modalities and After initial feature mining through the convolutional layer, multi-modal interpretable feature fusion is performed through the multi-output modality fusion module;

[0027] S3, repeat step S2 twice. After the encoder completes the third upsampling, the dual-branch output features are sent to the single-output modality fusion module to obtain the output features after the first stage processing;

[0028] S4: After the output features processed in the first stage are decoded by two identical decoders, the output feature maps are input into the consistency calculation module to calculate the consistency loss. , completing the second stage of semi-supervised learning. For a limited number of labeled After calculating the consistency loss Finally, the supervision loss is calculated using the gold standard .

[0029] The input of the two modalities is expressed as and , if they have labels, they are represented as , without a label, is represented as , and the limited labels are represented by G. In the stage of multimodal feature fusion: When the spatial positions are not aligned at the pixel level, because considering all spatial positions or channels equally often cannot achieve the global optimum, traditional multimodal segmentation methods usually use two completely independent encoders to separately learn unified features of different modalities. Such methods can provide rich learnable features, but often cannot capture the dependencies of global patterns.

[0030] Therefore, based on the multimodal pre-registered images, two encoders are first used to receive images of the two modalities. and After initial feature mining through the convolutional layer, the multi-output modal fusion module fuses interpretable multi-modal features. The output of the modal fusion module is not completely identical modal features, but rather a dual-branch feature that is discriminatively enhanced based on the input features. The above step (step S2) is repeated twice. After the encoder completes the third upsampling, the dual-branch output features are fed into the single-output modal fusion module. The processed features serve as the output of the first stage and are fed into the second stage for semi-supervised learning.

[0031] In the semi-supervised learning stage: This stage is based on V-Net and obtains pixel-level segmentation through the decoder. It is essentially a binary classification problem, which consists of two identical decoders and a consistency module. The inherent noise of the network is used to achieve consistency. This balance is also achieved through supervised loss. This is achieved by giving greater weight to the region of interest.

[0032] For all input data, including and ,After decoding, the output feature map will be output to the consistency module to calculate the consistency loss For a limited number of After calculating the consistency loss Finally, the supervision loss will be calculated using the gold standard .

[0033] Considering that shallow features carry more contour and texture information, a novel CMA-Skip connection is adopted in both stages to bridge the gap between multimodality and semi-supervision.

[0034] In order to fuse the shallow bimodal features with the deep discriminative specific features, a symmetrical skip connection, CMA-Skip connection, is used to achieve a complementary symmetrical structure. Skip connections have been shown to help improve the performance of deep neural networks in tasks that require the model to learn long-range dependencies or when the input data resolution is low, because it allows information to bypass certain layers that might otherwise hinder the flow of information or cause the gradient to disappear.

[0035] Since the goal is to fuse multimodal information, a pseudo-loop structure is designed for cross-modal auxiliary skip connections. Inspired by the cross-modal attention mechanism, which selectively emphasizes the dominant target features to obtain modality-independent complementary features, the features of the two branches are first concatenated together, and then a convolutional layer is used to enhance the representation. The sizes of the two convolution kernels are 1×1×1 and 3×3×3. Finally, the original features are processed using an attention probability map ranging from 0 to 1. , for more efficient and interpretable information exchange across different modalities.

[0036] The mathematical expression of CMA-Skip connection is: in, , Represent the feature maps of different non-dominant modes respectively, CMA represents a series of convolution, normalization and activation functions, represents the unprocessed raw feature map output by the node decoder, Represents a comprehensive modal feature that is combined with another modal feature and will be input into the original node again.

[0037] Due to the potential value and feature association between different modalities, a feature fusion module named modality fusion module is designed, which can adaptively aggregate modality-independent features in a learnable manner and flexibly output features with modality advantages or features with modality balance advantages according to the process of feature mining.

[0038] In the early stage of multimodal feature fusion, in the first stage, the modality fusion module obtains two inputs, namely the i-th feature map from the output of the first encoder and from the second encoder output First, and Add by voxel to get the common feature map , providing rich complementary modality information for subsequent feature fusion. and as well as and After concatenation, convolution and other operations are performed. Its powerful feature expression ability makes it possible to further explore the potential feature mapping between different modalities. After obtaining the weight map separately, multiplying it with the input feature map, the two outputs of the multi-output modality fusion module can be obtained. and . In addition, if aggregate features are needed, just output That's it.

[0039] The process of acquiring knowledge of the model set that is unique to a particular modality is formulated as the problem of minimizing the binary classification error on a voxel-by-voxel basis. In order to unify the two decoding branches and penalize the difference between the two predictions, the mutual consistency loss is used as the loss function of the proposed network.

[0040] For the output PM_1 of the first decoder, we calculate the MSE with the pseudo label PL_2 obtained by another decoder, and then get the consistency loss , its mathematical expression is as follows. In order to make the two outputs as similar as possible, the consistency loss is adopted As a guide for backpropagation and updating network parameters. After such consistency and entropy regularization, the network can pay more attention to the challenges of sparse annotations and uncertain areas, that is, decoder 1 and decoder 2 reach a relative balance to achieve the goal of output consistency.

[0041] The supervision loss and consistency loss The formula is: in, represents the i-th probability map, Denotes the jth pseudo label, L denotes a limited number of labels. When using limited labels to calculate the supervision loss After that, the total loss can be written as: The quantitative comparison of the effects of the present invention in the peritumoral edema segmentation task is shown in the following table.

[0042]

[0043] Among them, * is the dominant mode, that is, the peritumoral edema is more obvious in the Flair sequence than in the other sequences.

[0044] The present invention uses two or more non-dominant modality images and a small amount of manual annotation to construct a semi-supervised segmentation dataset for peritumoral edema, uses two interconnected encoders to perform three-dimensional multimodal feature fusion enhancement, and uses two completely independent decoders (four layers of continuous convolutional downsampling pairs) for consistency supervision, introducing multimodal segmentation in medical image segmentation tasks into semi-supervised segmentation, providing it with rich complementary information; using two or more non-dominant modalities (T1, T1ce, T2), a semi-supervised segmentation with better segmentation effect than a single dominant modality (Flair) is formed.

[0045] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, apparatus, article, or method comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, apparatus, article, or method. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, apparatus, article, or method comprising the element.

[0046] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A multimodality-assisted semi-supervised segmentation method for peritumoral edema in a non-dominant modality, characterized by: include: S1. Use two or more non-dominant modality images and manual annotations to construct a semi-supervised segmentation dataset for peritumoral edema, and express the input of the two modalities as and , when they have labels they are represented as , when there is no label, it is represented as ; S2, using two encoders to receive images of two modalities and After the initial feature mining is performed through the convolutional layer, the multi-modal interpretable feature fusion is performed through the modality fusion module; S3, repeat step S2 twice. After the encoder completes the third upsampling, the output features of the two encoders are sent to the modality fusion module to obtain the output features after the first stage processing; S4: After the output features processed in the first stage are decoded by two identical decoders, the output feature maps are input into the consistency calculation module to calculate the consistency loss. , completing the second stage of semi-supervised learning.

2. The method for semi-supervised segmentation of peritumoral edema in a non-dominant modality assisted by multimodality according to claim 1, characterized in that: For a limited number of After calculating the consistency loss Finally, the supervision loss is calculated using the gold standard .

3. The method for semi-supervised segmentation of peritumoral edema in a non-dominant modality assisted by multimodality according to claim 1, characterized in that: In both the first and second stages, a novel CMA-Skip connection is used to bridge the gap between multimodality and semi-supervision. First, the output features of the two encoders are concatenated together, and then a convolutional layer is used to enhance the representation. The sizes of the two convolution kernels are 1×1×1 and 3×3×3. Finally, the unprocessed original feature map output by the decoder is processed using an attention probability map from 0 to 1. .

4. The method for semi-supervised segmentation of peritumoral edema in a non-dominant modality assisted by multimodality according to claim 3, characterized in that: The mathematical expression of CMA-Skip connection is: in, , Represent the feature maps of different non-dominant modes respectively, CMA represents a series of convolution, normalization and activation functions, represents the unprocessed original feature map output by the decoder, It represents the comprehensive modal feature combined with another modal feature, which is used to be input into the original node again.

5. The method for semi-supervised segmentation of peritumoral edema in a non-dominant modality assisted by multimodality according to claim 1, characterized in that: In the first stage, the modality fusion module obtains two inputs, namely the i-th feature map output from the first encoder and the i-th feature map from the second encoder output ; First, and Add by voxel to get the common feature map , providing rich complementary modality information for subsequent feature fusion; then and as well as and After splicing, convolution operation is performed, and after obtaining the weight map, it is multiplied with the input feature map to obtain the two outputs of the modality fusion module. and ; When aggregation features are needed, only one output of the modality fusion module is needed That's it.

6. The method for semi-supervised segmentation of peritumoral edema in a non-dominant modality assisted by multimodality according to claim 1, characterized in that: In step S4, in order to unify the two branches formed by the two decoders, the difference between the two predictions is penalized, and the consistency loss is used as the loss function of the proposed network; For the output PM_1 of the first decoder, the pseudo label PL_2 obtained by another decoder is used to calculate the MSE, and then the consistency loss is obtained ; Using consistency loss As a guide for backpropagation and updating network parameters to make the two outputs similar, after consistency and entropy regularization, the two decoders reach a relative balance and achieve the goal of output consistency.

7. The method for semi-supervised segmentation of peritumoral edema in a non-dominant modality assisted by multimodality according to claim 2, characterized in that: The supervision loss and consistency loss The formula is: in, represents the i-th probability map, Represents the jth pseudo label, L represents a limited number of labels, and the supervision loss is calculated using limited labels After that, the total loss is: .

Images