Brain glioma segmentation model establishment method and segmentation method based on target perception

Abstract

The invention discloses a brain glioma segmentation model building method and a segmentation method based on target perception, and belongs to the field of medical image segmentation, and the method comprises the steps: building an initial segmentation model comprising a Swin UNETR network and a target perception network; the target sensing network is used for converting a prompt text into a static task prompt vector, generating 2N + M channel feature weight maps according to the static task prompt vector, respectively acting on a window-based multi-head attention mechanism layer in each coding module in the Swin UNETR network, and outputting features of the multi-head attention mechanism layer based on a sliding window, and each decoding module outputs characteristics. And training an initial segmentation model by using a training data set formed by a multi-modal brain MRI image with a segmentation subtask prompt text and marked with a segmentation result gold standard to obtain a required model. The model established by the invention has a dynamic perception segmentation sub-task, the feature extraction capability of the model for a corresponding tumor region is enhanced, and the brain tumor segmentation precision can be improved.

Description

Technical Field

[0001] The present invention belongs to the field of medical image segmentation, and more specifically, relates to a method for establishing a brain glioma segmentation model based on target perception and a segmentation method. Background Art

[0002] Glioma is the most common invasive brain tumor and poses a serious threat to human life. Multimodal Magnetic Resonance Imaging (MRI) is often used to diagnose brain tumors. T1-weighted (T1), T2-weighted (T2), contrast enhanced T1-weighted (T1ce) and fluid attenuation inversion recovery (FLAIR) are four commonly used data modalities, providing unique visual appearances for different brain structures. The segmentation of brain tumors mainly involves three target regions, namely Enhanced Tumor (ET), Tumor Core (TC) containing ET and necrotic core, and Whole Tumor (WT) containing TC and edema. These three sub-regions have different pathological characteristics, such as Figure 1 As shown in the figure, (a), (b), (c), and (d) represent MRI images of four modalities, namely T1, T2, T1ce, and FLAIR, respectively, and (e) represents the segmentation results of the three target regions. The glioma segmentation result is composed of the segmentation results of these three target regions. Generally speaking, these tumor regions are manually annotated by clinicians based on their professional knowledge, which is a complex, time-consuming, and error-prone process. The inherent characteristics of gliomas, such as variable shapes and sizes, unpredictable locations, and blurred edges, make their segmentation tasks extremely challenging.

[0003] Convolutional Neural Network (CNN) has been widely used in brain tumor segmentation tasks. However, due to the limited receptive field of the convolution operation, CNN-based methods often have difficulty capturing long-distance dependencies and may ignore global contextual information. Although some work has used attention mechanisms to deal with this problem, this inherent limitation may still lead to insufficient feature learning of global information between brain tumors and surrounding tissues.

[0004] Compared with convolutional neural networks, Vision Transformer (ViT) can effectively model long-distance dependencies and has attracted considerable attention in the field of image processing in recent years. Some ViT-based architectures, such as SwinUNETR (Swin Transformer for Brain Tumor Segmentation), use a structure that combines ViT with CNN to achieve effective extraction of global features; NestedFormer (Nested Modality-Aware Transformer) proposes a nested structure that considers single-modal and cross-modal feature fusion through a modality-aware feature aggregation module; S2CA Net (Shape-Scale Co-Awareness Network) considers both the shape and scale characteristics of brain tumors, showing great potential in brain tumor segmentation. Currently, many methods have proposed various feature fusion strategies to enhance the feature learning of multimodal images. Most current research methods focus on the full utilization and information mining of multimodal MRI data. For example Figure 1 As shown in the figure, each sub-region has its own unique pathological characteristics, has different visual manifestations in multimodal MRI images, and has a spatial inclusion relationship. However, in the current ViT-based method, the unique characteristics of the three sub-regions (WT, TC, ET) and their spatial correlation are rarely noticed, which limits the accuracy of brain glioma segmentation.

[0005] Although there are some works that have explored the unique features of these three sub-target areas, they are often implemented through multiple parallel or serial decoder branches, such as E1D3-UNet (One-encoder, Three-decoder Fully-Convolutional Neural etwork) using three parallel decoding branches, TSBTS-Net (Exploring Task Structure for Brain Tumor Segmentation) considering the spatial position relationship between different sub-regions from the inside to the outside and designing and modeling the serial network structure, MimicNet (MimicNet: Mimicking manual delineation of human expert for brain tumor segmentation.) sequentially segmenting the three sub-regions according to the order in which clinicians manually annotate different sub-regions. Although these methods can take into account the characteristics of different target sub-regions, the network needs to process three sub-tasks at the same time. In order to take into account the segmentation accuracy of each of the three segmentation sub-tasks, the overall modeling capability of the network will be statically dispersed into three parts, and each sub-task can only get a part (one-third) of the attention of the network's total capability, which will not be able to give full play to the network's feature extraction capabilities and is not conducive to improving accuracy. Summary of the invention

[0006] In view of the defects of the prior art and the need for improvement, the present invention provides a method for establishing a brain glioma segmentation model and a segmentation method based on target perception, the purpose of which is to enable the model to have the ability to dynamically perceive segmentation subtasks and dynamically enhance the model's feature extraction capability for the corresponding tumor area, thereby improving the accuracy of brain tumor segmentation.

[0007] To achieve the above object, according to one aspect of the present invention, a method for establishing a brain glioma segmentation model based on target perception is provided, comprising:

[0008] Step S1: Establish an initial segmentation model; the initial segmentation model includes:

[0009] Swin UNETR network is used to perform the glioma segmentation subtask on multimodal brain MRI images. The number of encoding modules in its encoding structure is N, and the number of decoding modules in its decoding structure is M. The segmentation subtasks include: full tumor segmentation, tumor core segmentation, and enhanced tumor segmentation.

[0010] And target perception network; the target perception network includes:

[0011] The STP generation module is used to convert the prompt text into a one-dimensional static task prompt vector; the prompt text is used to indicate the type of the segmentation subtask;

[0012] The channel feature weight map generation module is used to generate 2N+M channel feature weight maps according to the static task prompt vector;

[0013] and 2N+M channel feature enhancement modules, which are respectively arranged after the window-based multi-head attention mechanism layer, after the sliding window-based multi-head attention mechanism layer, and after each decoding module in each encoding module of the Swin UNETR network, and are used to apply the 2N+M channel feature weight maps to the corresponding features to achieve channel feature enhancement;

[0014] Step S2: The initial segmentation model is trained using the training data set. After the training is completed, a brain glioma segmentation model based on target perception is obtained;

[0015] The training data in the training dataset includes prompt text to indicate the type of segmentation subtask, and multimodal brain MRI images annotated with the gold standard of segmentation results.

[0016] Furthermore, the target perception network also includes: a spatial feature weight map generation module and N spatial feature enhancement modules;

[0017] The spatial feature weight map generation module is used to generate N spatial feature weight maps according to the static task prompt vector;

[0018] N spatial feature enhancement modules are respectively set after each encoding module in the Swin UNETR network, and are used to act on the N spatial feature maps on the corresponding features to achieve spatial feature enhancement.

[0019] Furthermore, the initial segmentation model further includes: a self-region prompt network; the self-region prompt network includes: a downsampling module and a region prompt fusion module;

[0020] The downsampling module is used to downsample the segmentation results of the previous segmentation subtask to the level consistent with the deepest image feature f image The same scale, get the regional feature f region ; Image feature f image Image features output by the encoding structure in the Swin UNETR network;

[0021] The regional hint fusion module is set between the encoding structure and the decoding structure in the Swin UNETR network to integrate the image features f image and regional characteristics f region After fusion, it is used as the input of the decoding structure;

[0022] Furthermore, the training data in the training data set also includes the segmentation results of the previous segmentation subtask;

[0023] Among them, the segmentation result of the front-stage segmentation subtask of tumor core segmentation is the segmentation result of the whole tumor; the segmentation result of the front-stage segmentation subtask of enhanced tumor segmentation is spliced ​​by the whole tumor segmentation result and the tumor core segmentation result.

[0024] Furthermore, the regional hint fusion module includes:

[0025] The first linear layer is used to transform the regional features f region Projection as feature Q mask ;

[0026] The second linear layer is used to transform the image features f image Projection to feature K image ;

[0027] The third linear layer is used to transform the image features f image Projection as feature V image ;

[0028] Cross-attention layer, used to Generate cross attention map A cross ;

[0029] The fusion layer is used to transform the image features f image and the cross attention map A cross Concatenate in the channel dimension to get feature f fusion ;

[0030] Residual convolution block, used for feature f fusion After feature extraction and image feature f image Addition;

[0031] And the convolution block is used to extract the features of the residual convolution block output to obtain the image feature f image and regional characteristics f region The fused features

[0032] Furthermore, the self-region prompt network further includes: a partitioning module and an element addition module;

[0033] The partitioning module is used to divide the segmentation results of the previous segmentation subtask into non-overlapping feature maps;

[0034] The element addition module is set between the Patch Partition layer and the encoding structure in the Swin UNETR network. It is used to add the non-overlapping feature maps obtained by dividing the multimodal brain MRI image by the Patch Partition layer and the feature maps obtained by the partition module at an element scale as the input of the encoding structure.

[0035] Furthermore, the channel feature weight map generation module includes 2N+M multi-layer perceptrons, which are respectively used to generate 2N+M channel feature weight maps according to the static task prompt vector;

[0036] The spatial feature weight map generation module includes N multi-layer perceptrons, which are used to generate N spatial feature weight maps according to static task prompt vectors.

[0037] According to another aspect of the present invention, a method for segmenting brain glioma based on target perception is provided, comprising:

[0038] After generating prompt texts for indicating the type of segmentation subtasks for brain glioma, they are combined with the multimodal brain MRI images to be segmented to obtain the model inputs corresponding to each segmentation subtask; the segmentation subtasks include: whole tumor segmentation, tumor core segmentation, and enhanced tumor segmentation;

[0039] The model inputs corresponding to the three segmentation subtasks are respectively input into the target-aware glioma segmentation model to obtain the segmentation results of the three segmentation subtasks and complete the glioma segmentation.

[0040] Among them, the target perception-based glioma segmentation model is established by the above-mentioned target perception-based glioma segmentation model establishment method provided by the present invention.

[0041] According to another aspect of the present invention, a method for segmenting brain glioma based on target perception and self-region prompts is provided, comprising:

[0042] Generate prompt text for indicating the type of segmentation subtask of brain glioma; the segmentation subtasks include: whole tumor segmentation, tumor core segmentation and enhanced tumor segmentation;

[0043] Input the prompt text of the full tumor segmentation subtask and the multimodal brain MRI image to be segmented into the target-aware brain glioma segmentation model to obtain the segmentation result of the full tumor segmentation subtask;

[0044] Input the prompt text of the tumor core segmentation subtask, the multimodal brain MRI image to be segmented, and the segmentation result of the whole tumor segmentation subtask into the target-aware brain glioma segmentation model to obtain the segmentation result of the tumor core segmentation subtask;

[0045] After the segmentation results of the whole tumor segmentation subtask are spliced ​​with the segmentation results of the tumor core segmentation subtask, they are input into the target-aware glioma segmentation model together with the prompt text of the enhanced tumor segmentation subtask and the multimodal brain MRI image to be segmented, and the segmentation results of the enhanced tumor segmentation subtask are obtained, thus completing the glioma segmentation.

[0046] Among them, the target perception-based glioma segmentation model is established by the above-mentioned target perception-based glioma segmentation model establishment method provided by the present invention.

[0047] According to another aspect of the present invention, a computer program product is provided, including a computer program. When the computer program is executed by a processor, it implements the above-mentioned method for establishing a glioma segmentation model based on target perception provided by the present invention, or the above-mentioned method for establishing a glioma segmentation model based on target perception provided by the present invention, or the above-mentioned method for establishing a glioma segmentation model based on target perception provided by the present invention, or the above-mentioned method for establishing a glioma segmentation model based on target perception and self-region cues provided by the present invention.

[0048] According to another aspect of the present invention, a computer-readable storage medium is provided, comprising a stored computer program. When the computer program is executed by a processor, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned method for establishing a glioma segmentation model based on target perception provided by the present invention, or the above-mentioned method for establishing a glioma segmentation model based on target perception provided by the present invention, or the above-mentioned method for segmenting a glioma based on target perception and self-region prompting provided by the present invention.

[0049] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

[0050] (1) The Swin UNETR network combines the context modeling capability of Swin Transformer and the pixel-level prediction capability of U-Net, and has good semantic segmentation performance. Based on the Swin UNETR network, the present invention uses prompt text as trigger information and introduces a target perception network to convert the prompt text used to indicate the segmentation subtask into a channel feature weight map, and uses the channel feature weight map to enhance the channel features of key features when the Swin UNETR network performs the segmentation task, thereby realizing the combination of prompt learning and brain glioma segmentation, so that the model has the ability to perceive the target subtask, and dynamically adjusts all modeling capabilities to specific segmentation subtasks, while enhancing the feature extraction capability of the corresponding tumor area, effectively improving the segmentation accuracy of the segmentation subtask, and ultimately improving the accuracy of brain glioma segmentation.

[0051] (2) In the preferred embodiment of the present invention, the introduced target perception network will also convert the prompt text into a spatial feature weight map, and use the spatial feature weight map to perform spatial feature enhancement on key features when the Swin UNETR network performs the segmentation task, thereby further enhancing the model's feature extraction capability for specific segmentation subtasks and further improving the accuracy of brain glioma segmentation.

[0052] (3) In the preferred embodiment of the present invention, a self-region hint network is further introduced on the basis of the Swin UNETR network to generate corresponding region hint information according to the cross-attention mechanism and the segmentation results of the previous segmentation subtask, and the region hint information is applied to the deepest image features when the Swin UNETR network performs the segmentation task, so as to guide the image features by the regional features. In this way, the spatial position relationship among the three sub-regions of the whole tumor (WT), tumor core (TC) and enhanced tumor (ET) can be fully utilized, so as to further enhance the model's attention to the local region information corresponding to the current segmentation subtask and further improve the segmentation accuracy.

[0053] (4) In the preferred embodiment of the present invention, the self-region hint network also integrates the segmentation results of the previous segmentation subtask into the input of the encoding structure of the Swin UNETR network, thereby enhancing the model's perception of the local area of ​​the input image and further improving the accuracy of brain glioma segmentation.

[0054] (5) In the preferred embodiment of the present invention, a multi-layer perceptron is used to generate a channel feature weight map and a spatial feature weight map according to a static task prompt vector, so that the attention weight of the local area can be learned more accurately, and the computational complexity is low and the model training difficulty is small. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 Schematic diagram of MRI images and corresponding brain glioma segmentation results provided in an embodiment of the present invention; wherein (a) is an MRI image in T1 mode, (b) is an MRI image in T2 mode, (c) is an MRI image in T1ce mode, (d) is an MRI image in FLAIR mode, and (e) is a brain glioma segmentation result;

[0056] Figure 2 Schematic diagram of the existing Swin UNETR network and the encoding module therein; wherein, (a) is a schematic diagram of the Swin UNETR network structure, and (b) is a schematic diagram of the structure of the encoding module Swin Transformer Block in the Swin UNETR network;

[0057] Figure 3 A schematic diagram of training and testing a target-aware brain glioma segmentation model provided in an embodiment of the present invention;

[0058] Figure 4 A schematic diagram of a brain glioma segmentation model based on target perception provided by an embodiment of the present invention;

[0059] Figure 5 A schematic diagram of static task prompt vector generation and feature enhancement provided by an embodiment of the present invention;

[0060] Figure 6 A schematic diagram of a regional prompt fusion module provided in an embodiment of the present invention;

[0061] Figure 7 A comparison chart of the quantitative results of the brain glioma segmentation method provided in an embodiment of the present invention and other existing brain tumor segmentation methods on the BraTS2021 dataset;

[0062] Figure 8 A comparison chart of the visualization results of the brain glioma segmentation method provided in the embodiment of the method of the present invention and other existing brain tumor segmentation methods on the BraTS2021 dataset;

[0063] Fig. 9 A comparison chart of the quantitative results of the brain glioma segmentation method provided in the embodiment of the method of the present invention and other existing brain tumor segmentation methods on the BraTS2020 dataset;

[0064] Fig.10 A comparison chart of the visualization results of the brain glioma segmentation method provided in an embodiment of the method of the present invention and other existing brain tumor segmentation methods on the BraTS2020 dataset. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0066] In the present invention, the terms "first", "second", etc. (if any) in the present invention and the accompanying drawings are used to distinguish similar objects but not necessarily to describe a specific order or sequence.

[0067] like Figure 1As shown, each sub-region has its own unique pathological characteristics, has different visual manifestations in multimodal MRI images, and has a spatial position inclusion relationship. Considering the differences and relationships between different sub-regions may help the model capture more effective unique features for each target and may improve performance accordingly. Based on the above analysis, the present invention proposes that the three sub-target regions of WT, TC and ET should be processed separately while retaining their spatial correlation. To achieve this, the present invention proposes a method based on trigger information for segmentation, which aims to instruct the network to dynamically adjust all modeling capabilities to each sub-target according to the trigger information. Specifically, the learning based on text prompts is integrated into the segmentation model as a static task prompt (Static Task Prompt, STP), and STP is used as a trigger condition to instruct the network to dynamically segment a specific sub-region, while enhancing the network's feature extraction capability for the sub-region during image encoding and decoding, thereby assisting the segmentation model in dynamically applying all modeling capabilities to each sub-target in the brain glioma segmentation task.

[0068] In addition, since there is a mutual inclusion relationship between the three sub-regions, the present invention intends to integrate this spatial inclusion relationship into the network modeling process and develop a self-region prompt (SRP) mechanism to enhance the information extraction of local regions. Specifically, the three sub-tasks are segmented in order from easy to difficult, divided into three stages, that is, first segmenting the WT target region, then segmenting the TC target region, and finally segmenting the ET target region. On the basis of this sequential segmentation, the implementation idea of ​​the SRP mechanism is that when segmenting the TC target region, the segmentation result of the WT target region is used as a region prompt. Since the TC target region is included in the WT target region, the model can better focus on the local region where the TC target region is located with the help of the WT target region segmentation result; when segmenting the ET target region, the segmentation results of the WT target region and the TC target region are used as region prompts. Similarly, since the ET target region is simultaneously included in the TC target region and the WT region, the model can better focus on the local region where the ET target region is located with the help of the TC target region segmentation result and the WT target region segmentation result. In general, the self-region hint mechanism can use the segmentation results of all previous stages as region hints based on the static task hint mechanism to enhance the local spatial information of each stage.

[0069] The Swin UNETR network combines the context modeling capability of Swin Transformer and the pixel-level prediction capability of U-Net, and has good semantic segmentation performance. In order to enable the model to have the ability to dynamically perceive segmentation subtasks and dynamically enhance the model's feature extraction capability for the corresponding tumor area, thereby improving the accuracy of brain tumor segmentation, the present invention improves the model structure based on the SwinUNETR network and proposes a new brain glioma segmentation model.

[0070] Before explaining the technical solution of the present invention in detail, the structure and related principles of the Swin UNETR network are explained.

[0071] like Figure 2 As shown in (a) of the figure, the core structure of the Swin UNETR network is the encoder-decoder structure. Before the encoding structure, it includes the first patch partition layer, which is used to convert the multimodal input image into Divide into non-overlapping image blocks, and get Wherein, H, W and D represent the height, width and depth of the image respectively. Optionally, in the following embodiments, H, W and D are all 128. Input image I image It is composed of brain MRI images of four modalities spliced ​​in the channel dimension.

[0072] Then the input coding structure is used to implement feature coding. Usually, the coding structure specifically includes four existing Swin Transformer coding modules. After coding, multi-scale image features are obtained, which are recorded as Among them C i =2 i ×C, C represents the number of feature channels. Optionally, in the following embodiments, C is 48. The structure of the encoding module is as follows Figure 2 As shown in (b), the specific operation of each encoding process is as follows: input feature f i en It will first pass through a LayerNorm (LN) layer, and then through a window-based multi-head self-attention mechanism layer (Window-based Multi-head Self-Attention, W-MSA) to obtain Then it goes through LN and two fully connected layers, a GELU nonlinear activation function layer, and then another LN layer and a sliding window multi-head attention mechanism layer (Shifted Window-based Multi-head Self-Attention, SW-MSA), and we get After that, it passes through LN, two fully connected layers, and a GELU nonlinear activation function layer, and then through the patch merging layer to reduce the dimension of the feature scale to obtain the output of the encoding stage.

[0073] The deepest image features encoded by the encoding structure will be input into the decoding structure. Usually, the decoding structure includes 5 decoding modules. For the decoding process, the decoding of each scale feature is implemented by the CNN structure to obtain multi-level decoding features. In each CNN-based decoding stage, the input features are After passing through a deconvolution layer, the features in the encoding process The jump connection is realized through two convolutional layers, and then two convolutional layers are used to process the fusion features after the jump connection to obtain

[0074] On the basis of the above-mentioned Swin UNETR network, the present invention introduces a target perception network to realize the above-mentioned static task prompt mechanism; on this basis, a self-region prompt network is further introduced to realize the above-mentioned self-region prompt mechanism.

[0075] The following are examples.

[0076] Embodiment 1:

[0077] A method for establishing a glioma segmentation model based on target perception, such as Figure 3 As shown, including:

[0078] Step S0: constructing a training data set for a brain glioma segmentation model through data preprocessing;

[0079] Step S1: Establish an initial segmentation model;

[0080] Step S2: The initial segmentation model is trained using the training data set. After the training, a target-aware brain glioma segmentation model is obtained.

[0081] The specific implementation of each step is further explained below.

[0082] In step S0, the preprocessing specifically includes:

[0083] The multimodal MRI data, namely T1, T2, T1ce, and FLAIR data, were resampled to 1mm×1mm×1mm to align the tumor areas in the images of different modalities. The black background area was then removed to remove data irrelevant to the segmentation of gliomas, reduce the amount of calculation, and save computing resources. Next, each image of different modalities was mean normalized:

[0084]

[0085] Among them, I′ n Represents the normalized image matrix of the nth image, I n represents the pixel matrix of the nth image, Represents the average value of the image pixel matrix, I std Represents the standard deviation of the image pixel matrix.

[0086] The preprocessing operation also includes: randomly cropping the normalized input image into image blocks of size 128×128×128; performing data augmentation operations, including making a random intensity offset in the range of (-0.1, 0.1) on the pixel matrix and randomly mirror-flipping each direction axis with a probability of 0.5.

[0087] On the basis of the above-mentioned preprocessed image, the gold standard of the segmentation result corresponding to the segmentation subtask is marked in the image, and a prompt text for indicating the segmentation subtask is generated to obtain a training data set.

[0088] The segmentation subtasks include: whole tumor segmentation, tumor core segmentation and enhanced tumor segmentation; optionally, in this embodiment, a prompt text in the form of "The task is {task}" is used as a static prompt input. {task}∈{wholetumor,tumor core,enhanced tumor}, "whole tumor", "tumor core", "enhanced tumor" correspond to whole tumor segmentation, tumor core segmentation and enhanced tumor segmentation respectively. It is easy to understand that the prompt text can also be in other forms, as long as the text form is unified in the model training stage and the subsequent segmentation stage.

[0089] It is easy to understand that the above preprocessing operations and related parameters are only exemplary and should not be understood as the sole limitation of the present invention. In practical applications, multimodal brain MRI images containing text prompts of the segmentation subtask and the gold standard of the segmentation results can also be obtained by other means to construct a training data set.

[0090] The structure of the initial segmentation model established in step S1 of this embodiment is as follows: Figure 4 As shown, it includes: Swin UNETR network, target perception network and self-region prompt network.

[0091] The Swin UNETR network is used to perform the glioma segmentation subtask on multimodal brain MRI images. The number of encoding modules in its encoding structure is 4, and the number of decoding modules in its decoding structure is 5.

[0092] The target-aware network is used to implement static text prompts, such as Figure 4 and Figure 5 As shown, the target perception network includes:

[0093] STP generation module, used to convert prompt text into a one-dimensional static task prompt vector In this embodiment, L is set to 20;

[0094] A channel feature weight map generation module is used to generate 2×4+5=13 channel feature weight maps according to the static task prompt vector;

[0095] The spatial feature weight map generation module is used to generate 5 spatial feature weight maps based on the static task prompt vector.

[0096] Alternatively, if Figure 5 As shown, in this embodiment, each channel feature weight map and each spatial feature weight map are generated by a multi-layer perceptron (MLP), and the channel feature weight map is represented as Channel feature enhancement is used at each stage of the encoding and decoding process to extract key semantic information from multimodal input information; the spatial feature weight map is represented as Used for spatial feature enhancement in the encoding process. In some other embodiments of the present invention, the channel feature weight map and the spatial feature weight map can also be implemented through other structures such as a fully connected layer and CNN.

[0097] In order to use the generated channel feature weight map to enhance the channel features in the encoding and decoding process of the Swin UNETR network, in this embodiment, the target perception network also includes 13 channel feature enhancement modules, such as Figure 4 and Figure 5 As shown, they are respectively arranged after the window-based multi-head attention mechanism layer, after the sliding window-based multi-head attention mechanism layer, and after each decoding module in each encoding module of the Swin UNETR network, and are used to apply the 13 channel feature weight maps to the corresponding features to achieve channel feature enhancement.

[0098] That is to say, for each feature f output by the window-based multi-head attention mechanism layer in the encoding process i WSMA The corresponding channel feature weight map will be used Perform channel dimension feature enhancement on it, such as Figure 5 As shown, if the feature map enhanced by channel information is recorded as Then we have:

[0099]

[0100] Where m = 2 × (i-1), stands for broadcast multiplication, LRelu is the leaky-relu activation layer, and Conv stands for the convolution operation.

[0101] Similarly, for each feature f output by the sliding window-based multi-head attention mechanism layer in the encoding process i SWSMA , will use the corresponding channel feature weight map To enhance the channel dimension features, such as Figure 5 As shown, if the feature map enhanced by channel information is recorded as Then we have:

[0102]

[0103] Where m = 2 × i-1, stands for broadcast multiplication, LRelu is the leaky-relu activation layer, and Conv stands for the convolution operation.

[0104] Similarly, for each feature map f output by the decoding module j , the corresponding channel feature map will also be used to enhance the channel dimension. If the feature map obtained after channel feature enhancement is recorded as Then we have:

[0105]

[0106] Where m = 12-j, stands for broadcast multiplication, LRelu is the leaky-relu activation layer, and Conv stands for the convolution operation.

[0107] In order to utilize the generated spatial feature weight map to enhance the spatial features in the encoding process of the Swin UNETR network, in this embodiment, the target perception network further includes: 5 spatial feature enhancement modules, such as Figure 4 and Figure 5 As shown in FIG. 1 , they are respectively set after each encoding module in the Swin UNETR network, and are used to act on the five spatial feature maps to the corresponding features to achieve spatial feature enhancement. In other words, for each feature f output by the encoding module i en , the corresponding spatial feature weight map will be used to enhance its spatial information. Represents the enhanced feature map, then:

[0108]

[0109] Where n = i, Represents broadcast multiplication.

[0110] It should be noted that this embodiment generates a channel feature weight map and a spatial feature weight map based on a static task prompt vector, and acts on the features in the Swin UNETR network in the above manner. It can effectively achieve channel feature enhancement and spatial feature enhancement under specific subtasks without affecting the original segmentation performance of the Swin UNETR network, enhance the model's feature extraction capability for specific sub-targets, and thereby improve the accuracy of brain glioma segmentation.

[0111] In this embodiment, the self-region hint network in the initial segmentation model is used to implement the self-region hint mechanism to enhance the network's attention to local region information. Specifically, it is divided into three stages. First, in the first stage, WT is segmented without any region hint, and the segmentation result of WT is used as the region information input when segmenting TC, which acts on the input stage and the intermediate feature layer of the network respectively, aiming to enhance the network's ability to capture local region information and obtain the segmentation result of TC. After that, WT and TC are used together as the region hint input in the segmentation ET stage to segment ET.

[0112] like Figure 4 As shown in the figure, in this embodiment, the implementation of the self-region prompt mechanism is divided into two parts. In the first part, at the input stage, the segmentation result of the previous segmentation subtask is divided into non-overlapping feature maps through the patch partition operation, and the feature maps are combined with the input image I image The result of the patch partition operation (i.e. ) to add the element scales, thereby enhancing the network's perception of the local area of ​​the input image. In the second part, in order to more effectively realize the guiding role of local area information on high-dimensional image features, the deepest image feature is recorded as f image , the segmentation result of the previous segmentation subtask is used as the region prompt, and the regional feature f is obtained by downsampling region , further f image With f region The guidance of regional features on image features is achieved through the fusion of cross-attention mechanisms.

[0113] Corresponding to the above mechanism, in this embodiment, the self-region hint network includes: a partitioning module, an element addition module, a downsampling module and a region hint fusion module;

[0114] The partitioning module is used to divide the segmentation results of the previous segmentation subtask into non-overlapping feature maps through the patch partition operation;

[0115] The element addition module is set between the Patch Partition layer and the encoding structure in the Swin UNETR network, and is used to add the non-overlapping feature maps obtained by dividing the multimodal brain MRI image by the Patch Partition layer and the feature maps obtained by the partition module at an element scale as the input of the encoding structure;

[0116] The downsampling module is used to downsample the segmentation results of the previous segmentation subtask to the level consistent with the deepest image feature f image The same scale, get the regional feature f region ; Image feature f image Image features output by the encoding structure in the Swin UNETR network;

[0117] The regional hint fusion module is set between the encoding structure and the decoding structure in the Swin UNETR network to integrate the image features f image and regional characteristics f region After fusion, it is used as the input of the decoding structure;

[0118] Furthermore, the training data in the training data set also includes the segmentation results of the previous segmentation subtask;

[0119] Among them, the segmentation result of the front-stage segmentation subtask of tumor core segmentation is the segmentation result of the whole tumor; the segmentation result of the front-stage segmentation subtask of enhanced tumor segmentation is spliced ​​by the whole tumor segmentation result and the tumor core segmentation result.

[0120] Optionally, in this embodiment, the downsampling module is composed of five CNNs connected in sequence; the specific structure of the regional hint fusion module is as follows: Figure 6 As shown, including:

[0121] The first linear layer is used to transform the regional features f region Projection as feature Q mask ;

[0122] The second linear layer is used to transform the image features f image Projection to feature K image ;

[0123] The third linear layer is used to transform the image features f image Projection as feature V image ;

[0124] Cross-attention layer, used to Generate cross attention map A cross ;

[0125] The fusion layer is used to transform the image features f image and the cross attention map A crossConcatenate in the channel dimension to get feature f fusion ;

[0126] Residual convolution block, used for feature f fusion After feature extraction and image feature f image Addition;

[0127] And the convolution block is used to extract the features of the residual convolution block output to obtain the image feature f image and regional characteristics f region The fused features

[0128] In step S2 of this embodiment, when the initial segmentation model is trained using the training data, for each piece of training data, the multimodal brain MRI image, the prompt text, and the segmentation result of the previous segmentation subtask in the training data are specifically input into the initial segmentation model, and for the same multimodal brain MRI image, the segmentation is performed in the order of WT, TC, and ET, respectively, to enhance the network's ability to extract unique features of the corresponding sub-target areas, and at the same time, the spatial inclusion relationship between them is explicitly modeled to obtain the final segmentation result of the network.

[0129] In order to better constrain the network learning process, this embodiment adopts the loss function based on Dice and cross entropy, which are respectively denoted as and

[0130]

[0131] Among them, N V represents the total number of voxels, y n and are the gold standard value and prediction result of the nth voxel respectively. The overall training loss function of the network for:

[0132]

[0133] Optionally, the AdamW optimizer is used in the training to update the network model weights, with an initial learning rate of 1e-4 and a total of 300 rounds of iterative training. The parameters of the Swin UNETR network, the target perception network, and the self-region prompt network in the model will be continuously updated and optimized during the training process.

[0134] During training, it will be determined in real time whether the network has reached the maximum number of iterations. If so, it will enter the testing process; if not, the training data will be repeatedly input to train the initial segmentation model.

[0135] During the test, the multimodal brain MRI images, prompt texts and segmentation results of the previous segmentation subtask are input into the network respectively to obtain the segmentation results of each sub-stage, and then the final segmentation results are obtained to achieve more accurate segmentation, thereby assisting doctors in determining surgical plans and implementing radiation and treatment plans.

[0136] In general, this embodiment combines the cue-based learning technology with the challenging task of brain tumor segmentation. Unlike existing brain glioma segmentation methods, it changes the way of processing three segmentation subtasks at a time. The model can be activated by different static task cue information, thereby dynamically enhancing the model's perception of each tumor sub-region; the target perception network in the model will generate a feature weight map for a specific segmentation subtask, and adaptively enhance the channel and spatial features of a specific target during feature encoding and decoding, which enables the model to dynamically focus on each tumor sub-region triggered by static text cue information; the model's self-region cue network uses the region cue information generated from continuous prediction results to explicitly model the spatial correlation between the three sub-targets, enhance the model's attention to local features, and help the model better understand the ongoing target segmentation subtask.

[0137] Embodiment 2:

[0138] A method for establishing a brain glioma segmentation model based on target perception. This embodiment is similar to the above-mentioned embodiment 1, except that, in this embodiment, the initial segmentation model established only includes the Swin UNETR network and the target perception network, but does not include the self-region prompt network. Accordingly, the data in the training data set only includes multimodal brain MRI images and the prompt text and segmentation result of the segmentation subtask, but does not include the segmentation result of the previous segmentation subtask.

[0139] In this embodiment, when the initial segmentation network established is trained using the training data set, there is no requirement for the execution order of the segmentation subtasks corresponding to the same multimodal brain MRI image.

[0140] In this embodiment, the specific implementation of each module can refer to the description in the above embodiment 1 and will not be repeated here.

[0141] This embodiment can also generate a feature weight map for a specific segmentation subtask by introducing a target perception network into the model, and adaptively enhance the channel and spatial features of a specific target during the feature encoding and decoding process, so that the model can dynamically focus on each tumor sub-region triggered by static text prompt information, ultimately effectively improving the segmentation accuracy of the model.

[0142] Embodiment 3:

[0143] A method for brain glioma segmentation based on target perception and self-region cues, comprising:

[0144] Generate prompt text for indicating the type of segmentation subtask of brain glioma; the segmentation subtasks include: whole tumor segmentation, tumor core segmentation and enhanced tumor segmentation;

[0145] Input the prompt text of the full tumor segmentation subtask and the multimodal brain MRI image to be segmented into the target-aware brain glioma segmentation model to obtain the segmentation result of the full tumor segmentation subtask;

[0146] Input the prompt text of the tumor core segmentation subtask, the multimodal brain MRI image to be segmented, and the segmentation result of the whole tumor segmentation subtask into the target-aware brain glioma segmentation model to obtain the segmentation result of the tumor core segmentation subtask;

[0147] After the segmentation results of the whole tumor segmentation subtask are spliced ​​with the segmentation results of the tumor core segmentation subtask, they are input into the target-aware glioma segmentation model together with the prompt text of the enhanced tumor segmentation subtask and the multimodal brain MRI image to be segmented, and the segmentation results of the enhanced tumor segmentation subtask are obtained, thus completing the glioma segmentation.

[0148] Among them, the target perception-based glioma segmentation model is established by the target perception-based glioma segmentation model establishment method provided in the above-mentioned embodiment 1.

[0149] Embodiment 4:

[0150] A method for brain glioma segmentation based on target perception and self-region cues, comprising:

[0151] After generating prompt texts for indicating the type of segmentation subtasks for brain glioma, they are combined with the multimodal brain MRI images to be segmented to obtain the model inputs corresponding to each segmentation subtask; the segmentation subtasks include: whole tumor segmentation, tumor core segmentation, and enhanced tumor segmentation;

[0152] The model inputs corresponding to the three segmentation subtasks are respectively input into the target-aware glioma segmentation model to obtain the segmentation results of the three segmentation subtasks and complete the glioma segmentation.

[0153] Among them, the target perception-based glioma segmentation model is established by the target perception-based glioma segmentation model establishment method provided in the above-mentioned embodiment 2.

[0154] Embodiment 5:

[0155] A computer program product includes a computer program. When the computer program is executed by a processor, it implements the method for establishing a glioma segmentation model based on target perception provided in the above-mentioned embodiment 1 or 2, or the method for establishing a glioma segmentation model based on target perception and self-region prompting provided in the above-mentioned embodiment 3, or the method for establishing a glioma segmentation model based on target perception provided in the above-mentioned embodiment 4.

[0156] Embodiment 6:

[0157] A computer-readable storage medium includes a stored computer program. When the computer program is executed by a processor, the device where the computer-readable storage medium is located is controlled to execute the method for establishing a glioma segmentation model based on target perception provided in the above-mentioned embodiment 1 or 2, or the method for establishing a glioma segmentation model based on target perception and self-region prompting provided in the above-mentioned embodiment 3, or the method for establishing a glioma segmentation model based on target perception provided in the above-mentioned embodiment 4.

[0158] The following is a further analysis and verification of the beneficial effects that can be achieved by the present invention in combination with the segmentation results on the public dataset.

[0159] Specifically, eleven existing advanced methods were selected for comparison, and experimental verification was performed on two public datasets BraTS2021 and BraTS2020. The methods selected for comparison are: CA-Net, E1D3_Unet, nnU-Net, ACMINet, EoFormer, F2Net, ETUNet, NestedFormer, Swin UNETR, nnFormer, S2CA_Net. For ease of presentation, in the following segmentation results, the brain glioma segmentation model established by the present invention is abbreviated as TSRNet.

[0160] The quantitative indicators used are the Dice Similarity Coefficient (DSC) and 95% Hausdorff Distance (HD95), which are commonly used in segmentation tasks. The definitions of the two indicators are:

[0161]

[0162] Among them I pred and I GT Represent the prediction result and the gold standard image respectively. d(p,q) is the 95th percentile of the Euclidean distances of all points in the prediction image that are closest to the gold standard image. Similarly, d(p,q) is the 95th percentile of the Euclidean distances of all points in the gold standard image that are closest to the prediction image.

[0163] Figure 7For the BraTS2021 dataset, the segmentation results of the three sub-regions obtained based on TSRNet are compared with the quantitative results of other existing methods. The first four are methods based on the CNN architecture, and the last seven are methods based on the hybrid architecture of ViT and CNN. The bold font in the figure indicates the best result, and the underline indicates the suboptimal result. Each comparison method is subjected to a t-test significance test with TSRNet. The "*" mark indicates that the p-value of the test result is less than 0.05, that is, there is a significant difference. From the results, we can see that:

[0164] First, TSRNet achieves the highest average DSC of 91.24%, which is at least 0.78% higher than other methods. At the same time, TSRNet's average HD95 value of 5.19mm is lower than other methods. Compared with the best CNN-based method ACMINet, which aims to exploit the interaction of cross-modal features, TSRNet improves the DSC of WT, TC, and ET by 0.44%, 0.69%, and 1.57%, respectively, and reduces the HD95 by 0.18mm, 2.28mm, and 5.58mm, respectively. Compared with the best ViT-based method S2CA-Net, which focuses on shape-aware and scale-aware features, TSRNet improves the DSC of WT, TC, and ET by 0.09%, 0.84%, and 1.40%, respectively, and reduces the HD95 by 0.21mm, 1.42mm, and 4.47mm, respectively. These results show that TSRNet can effectively enhance target-specific features, thereby improving the overall performance.

[0165] Secondly, compared with existing methods, TSRNet shows significant improvements of at least 0.69% and 1.40% on DSC of highly entangled targets TC and ET. Accurate TC and ET segmentation results can better assist neurosurgeons in clinical treatment, such as surgical planning, monitoring tumor growth, and radiotherapy, indicating the effectiveness and superiority of the proposed method.

[0166] Figure 8 This is a visualization comparison of the segmentation results obtained based on TSRNet and other existing methods for the BraTS2021 dataset. The five methods with the best results were selected for comparison. Among them, "Axial", "Saggital" and "Coronal" represent a slice image on the three axes respectively, and "3D" represents a three-dimensional rendering. From the visualization results, it can be seen that the segmentation results obtained based on TSRNet are more accurate. This is because the proposed STP mechanism enhances the key features of specific targets, and the designed SRP helps to understand the relationship between targets.

[0167] The same verification method was used to conduct experimental verification on the public dataset BraTS2020. The comparison of the quantitative results and the visualization results are shown in the following table. Fig. 9 and Fig.10 As shown. From the results, we can see that:

[0168] First, if Fig. 9 As shown in the figure, the segmentation results based on TSRNet are consistently better than other existing methods in all indicators. The highest average DSC of 89.97% was achieved, which is at least 1.10% higher than other methods, and the average HD95 value of our method is 4.00mm lower than other methods. Compared with the second-best method ACMINet, the glioma segmentation method based on TSRNet increased WT, TC and ET by 0.77%, 0.94% and 1.59% on DSC, and reduced 0.52mm, 0.16mm and 0.58mm on HD95, respectively. Fig.10 The segmentation visualization results in Figure 3 show that the TSRNet-based glioma segmentation method produces fewer false positive results than other methods. This also proves the effectiveness of the method of the present invention.

[0169] In summary, the present invention proposes for the first time a strategy for dynamic enhancement in the brain glioma segmentation task using cue-based learning technology. Compared with the existing brain tumor segmentation algorithms, the brain glioma segmentation results obtained based on the method of the present invention are not only better in quantitative evaluation, but also better in visualization results, which fully demonstrates the effectiveness and superiority of the method of the present invention.

[0170] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for establishing a brain glioma segmentation model based on target perception, characterized in that: include: Step S1: Establish an initial segmentation model; The initial segmentation model includes: Swin UNETR network is used to perform the glioma segmentation subtask on multimodal brain MRI images. The number of encoding modules in its encoding structure is N, and the number of decoding modules in its decoding structure is M. The segmentation subtasks include: full tumor segmentation, tumor core segmentation, and enhanced tumor segmentation. And a target perception network; the target perception network includes: An STP generation module is used to convert the prompt text into a one-dimensional static task prompt vector; the prompt text is used to indicate the type of the segmentation subtask; The channel feature weight map generation module is used to generate 2N+M channel feature weight maps according to the static task prompt vector; and 2N+M channel feature enhancement modules, which are respectively arranged after the window-based multi-head attention mechanism layer, after the sliding window-based multi-head attention mechanism layer, and after each decoding module in each encoding module of the Swin UNETR network, and are respectively used to act on the 2N+M channel feature weight maps on the corresponding features to achieve channel feature enhancement; Step S2: training the initial segmentation model using a training data set, and after the training is completed, obtaining a brain glioma segmentation model based on target perception; The training data in the training data set includes prompt text for indicating the type of segmentation subtask, and multimodal brain MRI images annotated with the gold standard of segmentation results.

2. The method for establishing a brain glioma segmentation model based on target perception according to claim 1, characterized in that: The target perception network also includes: a spatial feature weight map generation module and N spatial feature enhancement modules; The spatial feature weight map generation module is used to generate N spatial feature weight maps according to the static task prompt vector; The N spatial feature enhancement modules are respectively arranged after each encoding module in the Swin UNETR network, and are respectively used to act on the N spatial feature maps on corresponding features to achieve spatial feature enhancement.

3. The method for establishing a brain glioma segmentation model based on target perception according to claim 1 or 2, characterized in that: The initial segmentation model also includes: a self-region prompt network; the self-region prompt network includes: a downsampling module and a region prompt fusion module; The downsampling module is used to downsample the segmentation result of the previous segmentation subtask to a level consistent with the deepest image feature f image The same scale, get the regional feature f region ; Image feature f image The image features output by the encoding structure in the Swin UNETR network; The regional hint fusion module is set between the encoding structure and the decoding structure in the Swin UNETR network, and is used to integrate the image features f image and regional characteristics f region After fusion, it is used as the input of the decoding structure; Furthermore, the training data in the training data set also includes the segmentation results of the previous segmentation subtask; Among them, the segmentation result of the front-stage segmentation subtask of tumor core segmentation is the segmentation result of the whole tumor; the segmentation result of the front-stage segmentation subtask of enhanced tumor segmentation is spliced ​​by the whole tumor segmentation result and the tumor core segmentation result.

4. The method for establishing a brain glioma segmentation model based on target perception as claimed in claim 3, characterized in that: The regional hint fusion module comprises: The first linear layer is used to transform the regional features f region Projection as feature Q mask ; The second linear layer is used to transform the image features f image Projection to feature K image ; The third linear layer is used to transform the image features f image Projection as feature V image ; Cross-attention layer, used to Generate cross attention map A cross ; The fusion layer is used to transform the image features f image and the cross attention map A cross Concatenate in the channel dimension to get the feature f fusion ; Residual convolution block, used for feature f fusion After feature extraction and image feature f image Addition; And the convolution block is used to extract the features of the residual convolution block output to obtain the image feature f image and regional characteristics f region The fused features 5. The method for establishing a brain glioma segmentation model based on target perception as claimed in claim 3, characterized in that: The self-region prompt network also includes: a partitioning module and an element addition module; The partitioning module is used to divide the segmentation result of the previous segmentation subtask into non-overlapping feature maps; The element addition module is arranged between the PatchPartition layer and the encoding structure in the SwinUNETR network, and is used to perform element-wise addition of the non-overlapping feature maps obtained by dividing the multimodal brain MRI image by the Patch Partition layer and the feature maps obtained by dividing by the partition module, as the input of the encoding structure.

6. The method for establishing a brain glioma segmentation model based on target perception according to claim 1 or 2, characterized in that: The channel feature weight map generation module includes 2N+M multi-layer perceptrons, which are respectively used to generate 2N+M channel feature weight maps according to the static task prompt vector; The spatial feature weight map generation module includes N multi-layer perceptrons, which are respectively used to generate N spatial feature weight maps according to static task prompt vectors.

7. A method for segmenting brain glioma based on target perception, characterized in that: include: After generating prompt texts for indicating the type of segmentation subtasks of brain glioma, they are respectively combined with the multimodal brain MRI images to be segmented to obtain model inputs corresponding to each segmentation subtask; The segmentation subtasks include: whole tumor segmentation, tumor core segmentation, and enhanced tumor segmentation; The model inputs corresponding to the three segmentation subtasks are respectively input into the target-aware glioma segmentation model to obtain the segmentation results of the three segmentation subtasks and complete the glioma segmentation. Wherein, the target perception-based glioma segmentation model is established by the target perception-based glioma segmentation model establishment method described in claim 1, 2 or 6.

8. A method for brain glioma segmentation based on target perception and self-region hints, characterized in that: include: Generate a hint text for indicating the type of the segmentation subtask of brain glioma; The segmentation subtasks include: whole tumor segmentation, tumor core segmentation, and enhanced tumor segmentation; Input the prompt text of the whole tumor segmentation subtask and the multimodal brain MRI image to be segmented into the target-aware brain glioma segmentation model to obtain the segmentation result of the whole tumor segmentation subtask; Inputting the prompt text of the tumor core segmentation subtask, the multimodal brain MRI image to be segmented, and the segmentation result of the whole tumor segmentation subtask into the target-aware brain glioma segmentation model to obtain the segmentation result of the tumor core segmentation subtask; After splicing the segmentation result of the whole tumor segmentation subtask with the segmentation result of the tumor core segmentation subtask, the segmentation result of the whole tumor segmentation subtask is input into the target-aware glioma segmentation model together with the prompt text of the enhanced tumor segmentation subtask and the multimodal brain MRI image to be segmented, so as to obtain the segmentation result of the enhanced tumor segmentation subtask and complete the glioma segmentation; The target perception-based glioma segmentation model is established by the target perception-based glioma segmentation model establishment method according to any one of claims 3 to 6.

9. A computer program product, characterized in that It includes a computer program, which, when executed by a processor, implements the method for establishing a glioma segmentation model based on target perception as described in any one of claims 1 to 6, or the method for establishing a glioma segmentation model based on target perception as described in claim 7, or the method for establishing a glioma segmentation model based on target perception and self-region cues as described in claim 8.

10. A computer-readable storage medium, characterized in that: It includes a stored computer program, which, when executed by a processor, controls the device where the computer-readable storage medium is located to execute the method for establishing a glioma segmentation model based on target perception according to any one of claims 1 to 6, or the method for establishing a glioma segmentation model based on target perception according to claim 7, or the method for establishing a glioma segmentation based on target perception and self-regional cues according to claim 8.

Images