An adaptive brain glioma segmentation method based on semi-supervised deep learning

Abstract

The application discloses a kind of based on semi-supervised deep learning's adaptive brain glioma segmentation method.The method comprises: constructing deep learning model, the deep learning model includes teacher network, student network and multilayer perceptron;The deep learning model is trained based on the loss function set, wherein the teacher network takes complete multimodal magnetic resonance image as input image, the student network takes modal missing magnetic resonance image as input image, and the geometric structure prior learned by teacher network is used to guide the training of student network, and the multilayer perceptron is used to estimate the modal missing condition of the modal missing magnetic resonance image based on the features extracted by student network;The brain glioma segmentation of target image collected is carried out using the trained student network.The application can realize accurate and effective brain glioma segmentation, and adapt to various modal missing images.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and more specifically, to an adaptive glioma segmentation method based on semi-supervised deep learning. Background Technology

[0002] Gliomas are the most common primary malignant tumors of the neuroepithelial tract. Magnetic resonance imaging (MRI), with its superior multimodal soft tissue imaging contrast, is widely used in clinical glioma detection. Quantitative analysis of MRI images of glioma lesions is crucial for patient diagnosis, treatment, and surgical planning. Due to the complex and variable structure of gliomas, manually marking tumor boundaries is time-consuming and laborious. To assist physicians in achieving accurate quantitative analysis of gliomas, automated glioma segmentation based on brain MRI has become an important research area.

[0003] Currently, automatic glioma segmentation technology mainly employs deep learning methods such as convolutional neural networks. These methods typically involve inputting a large number of multimodal magnetic resonance imaging (MRI) image sequences from glioma patients (e.g., T1 (longitudinal relaxation time-weighted sequence), T1ce (longitudinal relaxation time-enhanced sequence), T2 (lateral relaxation time-weighted sequence), FLAIR (fluid attenuation flip sequence)) into the MRI network, using cross-entropy or Dice segmentation loss to constrain segmentation annotations and model predictions. After model training, in the clinical testing phase, multimodal MRI scan data from cases are directly fed into the model to extract heterogeneous features of tumor regions, ultimately outputting segmentations for different glioma regions. However, due to the complexity of gliomas in shape and texture, existing methods lack the extraction and constraint of spatial geometric information of tumor tissue structure during model training. Furthermore, existing methods require a large amount of expert-annotated training data, the quality of which often varies among experts, and the annotation process is time-consuming and laborious. Therefore, the quality and quantity of labeled data are difficult to guarantee, while a large amount of unlabeled data in clinical settings is usually not used for model training. Furthermore, existing classification models often require complete multimodal magnetic resonance imaging data of the case to be segmented in order to ensure the segmentation performance of the model when used clinically. However, due to issues such as the number of radiologists in hospitals, data storage, and the inapplicability of enhanced sequence scanning to special cases, the completeness of the multimodal magnetic resonance sequence of the case is difficult to guarantee, which significantly reduces the clinical usability of the model.

[0004] In summary, existing convolutional neural network-based glioma segmentation methods are highly dependent on the number of labeled training samples during training and lack the ability to mine and constrain the geometric structure of multimodal data during training. Furthermore, existing glioma segmentation methods require complete multimodal MRI scan data of the case as input during clinical testing. However, clinical limitations such as hospital conditions and patient-specific factors can easily lead to missing modal data, causing discrepancies between the test and training data. These discrepancies can significantly affect the actual segmentation performance of the model. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide an adaptive glioma segmentation method based on semi-supervised deep learning. This method includes the following steps:

[0006] Construct a deep learning model that includes a teacher network, a student network, and a multilayer perceptron;

[0007] The deep learning model is trained based on a defined loss function. The teacher network uses a complete multimodal magnetic resonance imaging (MRI) image as input, which includes both labeled and unlabeled data. The student network uses an MRI image with missing modes as input, which also includes both labeled and unlabeled data. The training of the student network is guided by prior geometric structures learned by the teacher network. The multilayer perceptron is used to estimate the mode-missing status of the MRI image based on features extracted by the student network.

[0008] A trained student network was used to segment gliomas from acquired target images.

[0009] Compared with existing technologies, the advantages of this invention are as follows: it adopts a semi-supervised training strategy, that is, only a portion of labeled data is used during model distillation training; under semi-supervised training conditions, it innovatively constrains a large amount of unlabeled data from the perspective of consistent feature space geometry, effectively avoiding the problem of the high dependence of deep learning segmentation models for gliomas on labeled training data; and to address the performance degradation problem of existing segmentation methods when clinical data modalities are missing, it designs a modality task-adaptive training strategy, which sets task prompt encoding for features under different missing modalities, enabling the segmentation network to achieve relatively robust segmentation of the target tumor region under various modality missing conditions.

[0010] Other features and advantages of the invention will become clear from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings. Attached Figure Description

[0011] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0012] Figure 1 This is a flowchart of an adaptive glioma segmentation method based on semi-supervised deep learning according to an embodiment of the present invention;

[0013] Figure 2 This is a schematic diagram of a semi-supervised deep learning-based adaptive glioma segmentation method according to an embodiment of the present invention. Detailed Implementation

[0014] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0015] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0016] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0017] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0018] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0019] See Figure 1 As shown, the provided adaptive glioma segmentation method based on semi-supervised deep learning includes the following steps:

[0020] Step S110: Construct a semi-supervised deep learning model. This deep learning model adopts a knowledge transfer framework and includes a teacher network, a student network, and a multilayer perceptron.

[0021] See Figure 2As shown, the deep learning model employs a semi-supervised knowledge transfer training framework based on geometric constraints, comprising a teacher network (or teacher model), a student network (or student model), and a multilayer perceptron. The teacher and student networks generally contain encoders and decoders. The teacher network takes a complete multimodal magnetic resonance image as input and outputs a segmented image, which includes both labeled and unlabeled data. The student network takes a magnetic resonance image with missing modalities as input and outputs a segmented image, which also contains both labeled and unlabeled data. The multilayer perceptron is used to estimate the modality missing status of the missing modalities in the magnetic resonance image based on features extracted by the student network.

[0022] In one embodiment, the existing 3d-UNet network is used as the basic framework for the deep learning model network. The modality-deficient images can be obtained by randomly partitioning a complete multimodal magnetic resonance image.

[0023] Step S120: Train the deep learning model based on the set loss function. During the training process, the prior geometric structure learned by the teacher network is used to guide the student network, and task prompt encoding is designed using the modality missing data to enhance the adaptability of the student network to modality missing situations.

[0024] First, in order to help the model learn tumor region information better from unlabeled data and get rid of the dependence on labeled training data when training the model, a semi-supervised training strategy combined with knowledge distillation is adopted on the basis of the existing framework. That is, the student model is trained by distillation using a pre-trained teacher model, and only a small portion of labeled training data is used during distillation training.

[0025] Specifically, the teacher network is first pre-trained using labeled data. Then, during distillation training, both the student and teacher networks are simultaneously fed labeled data and a large amount of unlabeled data. For the large amount of unlabeled training data, a loss constraint based on geometric structure prior is designed. For example, the geometric structure constraint refers to reducing the dimensionality of the deep, high-dimensional spatial geometric features extracted by the encoders of the teacher and student networks during distillation training, and using a comparative loss constraint to account for the voxel-level differences in the spatial structure of the unlabeled training samples in the encoded features of the student and teacher networks. In one embodiment, to more effectively capture the modal information of the teacher model, the weights of the student model are updated using the exponential moving average of the teacher model during distillation training. This design utilizes the geometric structure prior learned by the teacher network to guide the training of the student network, which helps to obtain more accurate segmentation results and compensates for the potential for low accuracy when training with a large amount of unlabeled data.

[0026] In a preferred embodiment, in order to enhance the adaptability to modal missing data, task prompting coding is designed for modal missing data, that is, simulation and task coding are performed for modal missing data.

[0027] Specifically, during distillation training, for data with complete modalities (4 modalities), the samples are randomly divided into 15 equal parts, with each part simulating a corresponding modality missing condition (15 in total). Each modality missing condition is encoded using 0 and 1, and each encoding represents the segmentation task under a modality missing condition. For example, 1 indicates the presence of the corresponding modality, and 0 indicates the absence of the modality. The encoded vector is concatenated with the low-dimensional features of the input image obtained through convolutional dimensionality reduction. The concatenated features are then input into the decoder of the student network. Finally, when the student network decodes, a task classification result is output through a multilayer perceptron and constrained by the loss of the encoding label for the modality missing condition. In this way, gradient backpropagation can effectively constrain the network to recognize different segmentation tasks, achieving adaptive segmentation for modality missing tasks. By simulating modality missing scenarios that may occur in clinical practice and setting task cue codes for each scenario, a glioma segmentation model with high utilization of unlabeled training data and robustness to modality missing conditions is achieved.

[0028] Still combined Figure 2 As shown, in one embodiment, the overall loss function for training the deep learning model includes a spatial geometric feature constraint term, a segmentation loss constraint term, and an encoding prediction loss term. The spatial geometric feature constraint term refers to the loss between the voxel-level spatial geometric features of the unlabeled data extracted by the teacher and student networks. The segmentation loss term reflects the segmentation loss between the labeled data of the segmented images output by the teacher and student networks. The encoding prediction loss term reflects the prediction loss between the modality missing condition encoding estimated by the multilayer perceptron and the set task cue encoding. The overall loss function can be the sum of the above three losses, or a weighted sum.

[0029] Step S130: Use the trained student network to segment the target image for glioma.

[0030] In clinical applications, only a pre-trained student network is used for segmentation prediction. This involves inputting the acquired target image into the trained student network to obtain glioma segmentation results. Because task-encoding prediction loss is used as a constraint during model training, accurate segmentation can still be achieved even when the acquired MRI image is modally missing.

[0031] It should be noted that the model training process involved in this invention can be performed offline on a server or in the cloud. Real-time image segmentation can be achieved by embedding the trained student model into an electronic device or server. The electronic device can be a terminal device or a server. Terminal devices include any terminal device such as mobile phones, tablets, personal digital assistants (PDAs), point-of-sale (POS) terminals, in-vehicle computers, and smart wearable devices (smartwatches, virtual reality glasses, virtual reality headsets, etc.). Servers include, but are not limited to, application servers or web servers, and can be independent servers, cluster servers, or cloud servers.

[0032] In summary, relevant studies have shown that current deep learning models for glioma segmentation are highly dependent on the scale of training data and the modal completeness of test data. During training, these models often require a large number of labeled multimodal MRI images, and for testing, they need to acquire complete multimodal MRI scan sequences consistent with those used in training. This high requirement for data completeness significantly reduces the clinical usability of the model. This invention organically combines semi-supervised training strategies, knowledge distillation training, and modality-deficient adaptive training, resulting in a glioma segmentation model (student network) with high clinical usability. The semi-supervised training strategy reduces the model's dependence on labeled data during training, lowering the workload for physicians and allowing the model to utilize patients' brain MRI image data more efficiently. In knowledge distillation training, deep geometric information from multimodal MRI data is extracted to constrain unlabeled samples for distillation training, ensuring the model's segmentation accuracy while reducing dependence on labeled data. Furthermore, by encoding the modality missing process into a task, the high dependence of deep learning models on the number of labeled sample data and modality completeness when applied to glioma segmentation tasks is avoided, enabling the student network to adapt well to various modality missing data and cope with different data conditions in clinical applications.

[0033] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0034] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0035] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0036] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, Python, etc., and conventional procedural programming languages ​​such as "C" or similar languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0037] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0038] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0039] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0040] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0041] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. An adaptive glioma segmentation method based on semi-supervised deep learning, comprising the following steps: Construct a deep learning model that includes a teacher network, a student network, and a multilayer perceptron; The deep learning model is trained based on a defined loss function. The teacher network uses a complete multimodal magnetic resonance imaging (MRI) image as input, which includes both labeled and unlabeled data. The student network uses an MRI image with missing modes as input, which also includes both labeled and unlabeled data. The training of the student network is guided by prior geometric structures learned by the teacher network. The multilayer perceptron is used to estimate the mode-missing status of the MRI image based on features extracted by the student network. A trained student network was used to segment gliomas from acquired target images.

2. The method according to claim 1, characterized in that, The loss function includes a spatial geometric feature constraint term, a segmentation loss constraint term, and an encoding prediction loss term. The spatial geometric feature constraint term reflects the loss between the voxel-level spatial geometric features of the unlabeled data extracted by the teacher network and the student network. The segmentation loss constraint term reflects the segmentation loss between the labeled data of the segmented images output by the teacher network and the student network. The encoding prediction loss term reflects the loss between the modality missing status encoding estimated by the multilayer perceptron and the set task indication encoding, which is used to indicate the modality missing status of the modality missing magnetic resonance image.

3. The method according to claim 2, characterized in that, The spatial geometric feature constraint term refers to the dimensionality reduction of the deep high-dimensional spatial geometric features extracted by the encoders of the teacher network and the student network, and the design of a contrastive loss to constrain the differences in the voxel level of the spatial structure of the unlabeled training samples in the encoded features of the student network and the teacher network.

4. The method according to claim 1, characterized in that, The magnetic resonance image with missing modes was obtained according to the following steps: For magnetic resonance images with intact modalities, the samples are randomly divided into multiple data sets, with each data set corresponding to a modality missing condition. For each modality missing case, a task indicator code is performed. This task indicator code is used to characterize the modalities contained in the image and the missing modalities. Each code represents a segmentation task under the modality missing case.

5. The method according to claim 4, characterized in that, The modal complete magnetic resonance image includes four modes: T1, T1ce, T2, and FLAIR. The multiple data sets are set to 15 equal parts, and the task indication encoding type uses 0 and 1 encoding.

6. The method according to claim 2, characterized in that, The multilayer perceptron used to estimate the modality missingness of the modality-missing magnetic resonance image based on features extracted by the student network includes the following steps: The task instruction encoding vector is concatenated with the low-dimensional features of the input image after dimensionality reduction through convolution; The concatenated features are input into the decoder of the student network. When the student network decodes, a task classification result is output through the multilayer perceptron. The result is then compared with the encoding label of the missing modality and a loss constraint is applied. Gradient backpropagation is then used to constrain the student network to recognize different segmentation tasks.

7. The method according to claim 1, characterized in that, The teacher network is a model pre-trained with labeled data.

8. The method according to claim 1, characterized in that, During the training of the deep learning model, the weights of the student network are updated using the exponential moving average weights of the teacher network.

9. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

10. A computer device comprising a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

Images