Target perception-based glioma segmentation model establishment method and segmentation method

Abstract

The application discloses a brain glioma segmentation model establishment method and a segmentation method based on target perception, belongs to the medical image segmentation field, and comprises the following steps: establishing an initial segmentation model comprising a Swin UNETR network and a target perception network; the target perception network is used for converting prompt text into a static task prompt vector, and generating 2N+M channel feature weight maps according to the static task prompt vector, which are respectively used for a window-based multi-head attention mechanism layer in each encoding module of the Swin UNETR network, features output by a sliding window-based multi-head attention mechanism layer, and features output by each decoding module; and the initial segmentation model is trained by using a training data set composed of multi-modal brain MRI images with segmentation sub-task prompt text and labeled segmentation result gold standards, so as to obtain a required model. The model established by the application has the advantages of dynamic perception of segmentation sub-tasks and enhanced feature extraction capability of the model for corresponding tumor regions, and can improve the brain tumor segmentation precision.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of medical image segmentation, and more particularly relates to a brain glioma segmentation model establishment method and segmentation method based on target perception. BACKGROUND

[0002] Glioma is the most common invasive brain tumor, which seriously threatens human life safety. Multi-modal magnetic resonance imaging (Magnetic Resonance Imaging, MRI) is often used for the diagnosis of brain tumors. T1-weighted (T1), T2-weighted (T2), contrast-enhanced T1-weighted (contrast enhanced T1-weighted, T1ce) and fluid attenuation inversion recovery (Fluid Attenuation Inversion Recovery, FLAIR) are four commonly used data modalities, which provide unique visual appearances for different structures of the brain. The segmentation of brain tumors mainly involves three target regions, namely enhanced tumor (Enhanced Tumor, ET), tumor core (Tumor Core, TC) containing ET and necrotic core, and whole tumor (Whole Tumor, WT) containing TC and edema, which have different pathological characteristics, such as Figure 1 are shown in FIG. 1, where (a), (b), (c), and (d) represent the MRI images of the four modalities of T1, T2, T1ce and FLAIR, respectively, and (e) represents the segmentation results of the three target regions. The brain glioma segmentation result is composed of the segmentation results of the three target regions. Generally, these tumor regions are manually annotated by clinicians according to their professional knowledge, which is complex, time-consuming and prone to errors. The inherent characteristics of glioma, such as variable shape and size, unpredictable location and blurred edge, make the segmentation task extremely challenging.

[0003] Convolutional Neural Network (CNN) has been widely used in brain tumor segmentation tasks. However, due to the limited receptive field of convolution operation, CNN-based methods often have difficulty in capturing long-distance dependencies and may ignore global context information, although there have been works using attention mechanisms to address this issue, but this inherent limitation may still lead to insufficient feature learning of global information between brain tumors and surrounding tissues.

[0004] Compared with convolutional neural networks, vision transformers (ViT) can effectively model long-distance dependencies and have attracted considerable attention in the field of image processing in recent years. Some ViT-based architectures, such as SwinUNETR (Swin Transformer for Brain Tumor Segmentation), use a structure that combines ViT with CNN to achieve effective extraction of global features; NestedFormer (Nested Modality-Aware Transformer) proposes a nested structure that considers single-modal and cross-modal feature fusion through a modality-aware feature aggregation module; S2CA Net (Shape-Scale Co-Awareness Network) considers both the shape and scale characteristics of brain tumors, showing great potential in brain tumor segmentation. Currently, many methods have proposed various feature fusion strategies to enhance the feature learning of multimodal images. Most current research methods focus on the full utilization and information mining of multimodal MRI data. For example Figure 1 As shown in the figure, each sub-region has its own unique pathological characteristics, different visual manifestations in multimodal MRI images, and an inclusion relationship in spatial position. However, in the current ViT-based method, the unique characteristics of the three sub-regions (WT, TC, ET) and their spatial correlation are rarely noticed, which limits the accuracy of brain glioma segmentation.

[0005] Although some current works have explored the unique characteristics of the three sub-target regions, they are often implemented through multiple parallel or serial decoder branches. For example, E1D3-UNet (One-encoder, Three-decoder Fully-Convolutional Neural etwork) uses three parallel decoding branches, TSBTS-Net (Exploring Task Structure for Brain Tumor Segmentation) considers the spatial location relationship from inside to outside between different sub-regions and designs and models a serial network structure, and MimicNet (MimicNet: Mimicking manual delineation of human expert for brain tumor segmentation.) sequentially segments the three sub-regions according to the order of the manual labeling of the different sub-regions by the clinician. Although these methods can consider the characteristics of different target sub-regions, the network needs to process three sub-tasks at the same time. In order to balance the segmentation accuracy of the three segmentation sub-tasks, the overall modeling capability of the network will be statically divided into three parts, and each sub-task can only get one-third of the attention of the network's overall capability, which will not fully exert the network's feature extraction capability and is not conducive to the improvement of the accuracy. SUMMARY

[0006] In view of the defects of the prior art and the need for improvement, the present application provides a brain glioma segmentation model establishment method and segmentation method based on target perception, which aims to enable the model to dynamically perceive the segmentation sub-tasks and dynamically enhance the model's feature extraction capability for the corresponding tumor region, thereby improving the brain tumor segmentation accuracy.

[0007] To achieve the above-mentioned purpose, according to one aspect of the present application, a brain glioma segmentation model establishment method based on target perception is provided, comprising:

[0008] Step S1: establishing an initial segmentation model; the initial segmentation model comprises:

[0009] a Swin UNETR network for performing brain glioma segmentation sub-tasks on multi-modal brain MRI images, the number of encoding modules in the encoding structure being N, and the number of decoding modules in the decoding structure being M; the segmentation sub-tasks including: whole tumor segmentation, tumor core segmentation, and enhanced tumor segmentation;

[0010] and a target perception network; the target perception network comprises:

[0011] The STP generation module is configured to convert the prompt text into a one-dimensional static task prompt vector; the prompt text is used to indicate the type of the segmentation subtask.

[0012] The channel feature weight map generation module is configured to generate 2N+M channel feature weight maps according to the static task prompt vector.

[0013] The 2N+M channel feature enhancement modules are respectively arranged after the window-based multi-head attention mechanism layer, the sliding window-based multi-head attention mechanism layer in each encoding module of the Swin UNETR network, and after each decoding module, and are respectively configured to apply the 2N+M channel feature weight maps to the corresponding features to realize channel feature enhancement.

[0014] Step S2: training the initial segmentation model using the training data set, and obtaining the brain glioma segmentation model based on target perception after the training is completed.

[0015] The training data in the training data set includes prompt text used to indicate the type of the segmentation subtask, and a multi-modal brain MRI image with a segmentation result gold standard labeled.

[0016] Further, the target perception network further includes a spatial feature weight map generation module and N spatial feature enhancement modules.

[0017] The spatial feature weight map generation module is configured to generate N spatial feature weight maps according to the static task prompt vector.

[0018] The N spatial feature enhancement modules are respectively arranged after each encoding module of the Swin UNETR network, and are respectively configured to apply the N spatial feature maps to the corresponding features to realize spatial feature enhancement.

[0019] Further, the initial segmentation model further includes a self-region prompt network; the self-region prompt network includes a down-sampling module and a region prompt fusion module.

[0020] The down-sampling module is configured to down-sample the segmentation result of the previous segmentation subtask to the same scale as the deepest image feature f image , to obtain a region feature f region ; the image feature f image is an image feature output by the encoding structure of the Swin UNETR network.

[0021] The region prompt fusion module is arranged between the encoding structure and the decoding structure of the Swin UNETR network, and is configured to fuse the image feature f image and the region feature f region through cross-attention mechanism and take the fusion result as the input of the decoding structure.

[0022] Moreover, the training data in the training dataset also includes the segmentation results of the previous segmentation subtask;

[0023] Among them, the segmentation result of the front-stage segmentation subtask of tumor core segmentation is the segmentation result of the entire tumor; the segmentation result of the front-stage segmentation subtask of enhanced tumor segmentation is spliced ​​by the whole tumor segmentation result and the tumor core segmentation result.

[0024] Furthermore, the regional hint fusion module includes:

[0025] The first linear layer is used to transform the regional features f region Projected as feature Q mask ;

[0026] The second linear layer is used to transform the image feature f image Projected as feature K image ;

[0027] The third linear layer is used to transform the image feature f image Projection as feature V image ;

[0028] Cross attention layer, used to follow Generate cross attention map A cross ;

[0029] Fusion layer, used to combine image features f image and cross attention map A cross Splicing in the channel dimension, we get the feature f fusion ;

[0030] Residual convolution block, used to perform feature f fusion After feature extraction and image feature f image Addition;

[0031] And the convolution block is used to extract the features output by the residual convolution block to obtain the image features f image and regional characteristics f region The fused features

[0032] Furthermore, the self-region hint network further includes: a partitioning module and an element addition module;

[0033] The partitioning module is used to divide the segmentation results of the previous segmentation subtask into non-overlapping feature maps;

[0034] The element addition module is arranged between a Patch Partition layer and an encoding structure in the Swin UNETR network, and is configured to add element scales of feature maps obtained by the Patch Partition layer and feature maps obtained by the partition module, and input the added feature maps to the encoding structure.

[0035] Further, the channel feature weight map generation module includes 2N+M multi-layer perceptrons, respectively configured to generate 2N+M channel feature weight maps according to the static task prompt vector.

[0036] The spatial feature weight map generation module includes N multi-layer perceptrons, respectively configured to generate N spatial feature weight maps according to the static task prompt vector.

[0037] According to another aspect of the present application, a target perception-based glioma segmentation method is provided, comprising:

[0038] After generating the prompt text for indicating the segmentation sub-task type of the glioma, the prompt text is combined with the multi-modal brain MRI image to be segmented to obtain model input corresponding to each segmentation sub-task; the segmentation sub-task includes: whole tumor segmentation, tumor core segmentation and enhanced tumor segmentation.

[0039] The model input corresponding to the three segmentation sub-tasks is input into the target perception-based glioma segmentation model to obtain the segmentation results of the three segmentation sub-tasks, and the glioma segmentation is completed.

[0040] The target perception-based glioma segmentation model is established by the above-mentioned target perception-based glioma segmentation model establishment method provided by the present application.

[0041] According to another aspect of the present application, a target perception and self-region prompt-based glioma segmentation method is provided, comprising:

[0042] Generating a prompt text for indicating the segmentation sub-task type of the glioma; the segmentation sub-task includes: whole tumor segmentation, tumor core segmentation and enhanced tumor segmentation.

[0043] Inputting the prompt text of the whole tumor segmentation sub-task and the multi-modal brain MRI image to be segmented into the target perception-based glioma segmentation model to obtain the segmentation result of the whole tumor segmentation sub-task.

[0044] Inputting the prompt text of the tumor core segmentation sub-task, the multi-modal brain MRI image to be segmented and the segmentation result of the whole tumor segmentation sub-task into the target perception-based glioma segmentation model to obtain the segmentation result of the tumor core segmentation sub-task.

[0045] The segmentation result of the whole tumor segmentation subtask is spliced with the segmentation result of the tumor core segmentation subtask, and then the segmentation result of the enhanced tumor segmentation subtask is obtained by inputting the segmentation result, the prompt text of the enhanced tumor segmentation subtask, and the multi-modal brain MRI image to be segmented into the target perception-based glioma segmentation model, and the glioma segmentation is completed.

[0046] The target perception-based glioma segmentation model is established by the target perception-based glioma segmentation model establishment method provided by the present application.

[0047] According to another aspect of the present application, a computer program product is provided, which comprises a computer program, and the computer program is executed by a processor to implement the target perception-based glioma segmentation model establishment method provided by the present application, or the target perception-based glioma segmentation method provided by the present application, or the target perception and self-region prompting-based glioma segmentation method provided by the present application.

[0048] According to another aspect of the present application, a computer readable storage medium is provided, which comprises a stored computer program, and the computer program is executed by a processor to control the device where the computer readable storage medium is located to execute the target perception-based glioma segmentation model establishment method provided by the present application, or the target perception-based glioma segmentation method provided by the present application, or the target perception and self-region prompting-based glioma segmentation method provided by the present application.

[0049] Overall, the above technical solutions conceived by the present application can achieve the following beneficial effects:

[0050] (1) The Swin UNETR network combines the context modeling capability of Swin Transformer and the pixel-level prediction capability of U-Net, and has good semantic segmentation performance. The present application uses the prompt text as the trigger information on the basis of the Swin UNETR network, and introduces the target perception network to convert the prompt text indicating the segmentation subtask into a channel feature weight map, and utilize the channel feature weight map to enhance the channel features of the key features when the Swin UNETR network performs the segmentation task, realizing the combination of prompt learning and glioma segmentation, enabling the model to have the ability of target subtask perception, dynamically adjusting all modeling capabilities to specific segmentation subtasks, and enhancing the feature extraction capability of the corresponding tumor region, effectively improving the segmentation accuracy of the segmentation subtask, and ultimately improving the accuracy of glioma segmentation.

[0051] (2) In the preferred scheme of the present application, the introduced target perception network also converts the prompt text into a spatial feature weight map, and utilizes the spatial feature weight map to perform spatial feature enhancement on the key features when the Swin UNETR network performs the segmentation task, thereby further enhancing the feature extraction capability of the model for specific segmentation subtasks and further improving the brain glioma segmentation accuracy.

[0052] (3) In the preferred scheme of the present application, a self-region prompt network is further introduced on the basis of the Swin UNETR network to generate corresponding region prompt information according to the cross-attention mechanism and the segmentation results of the previous segmentation subtasks, and the region prompt information is applied to the deepest image features when the Swin UNETR network performs the segmentation task, realizing the guidance of the region features to the image features, thereby fully utilizing the spatial position relationship among the three sub-regions of the whole tumor (WT), the tumor core (TC) and the enhanced tumor (ET), further enhancing the attention degree of the model to the local region information corresponding to the current segmentation subtask, and further improving the segmentation accuracy.

[0053] (4) In the preferred scheme of the present application, the self-region prompt network also fuses the segmentation results of the previous segmentation subtasks into the input of the encoding structure of the Swin UNETR network, thereby enhancing the perception ability of the model to the local region of the input image and further improving the brain glioma segmentation accuracy.

[0054] (5) In the preferred scheme of the present application, the multi-layer perception is utilized to generate the channel feature weight map and the spatial feature weight map according to the static task prompt vector, which can accurately learn the attention weight of the local region, has low computational complexity, and has small model training difficulty. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 The MRI image and the corresponding brain glioma segmentation result provided by the embodiment of the present application are shown in the figure; wherein (a) is the MRI image of T1 mode, (b) is the MRI image of T2 mode, (c) is the MRI image of T1ce mode, (d) is the MRI image of FLAIR mode, and (e) is the brain glioma segmentation result;

[0056] Figure 2 The existing Swin UNETR network and the encoding module therein are shown in the figure; wherein (a) is a structure diagram of the Swin UNETR network, and (b) is a structure diagram of the encoding module Swin Transformer Block in the Swin UNETR network;

[0057] Figure 3 The training and testing of the brain glioma segmentation model based on target perception provided by the embodiment of the present application are shown in the figure;

[0058] Figure 4 A target perception-based brain glioma segmentation model schematic diagram provided for an embodiment of the present application;

[0059] Figure 5 A static task prompt vector generation and feature enhancement schematic diagram provided for an embodiment of the present application;

[0060] Figure 6 A regional prompt fusion module schematic diagram provided for an embodiment of the present application;

[0061] Figure 7 A comparison chart of quantitative results of a brain glioma segmentation method provided for an embodiment of the present application and other existing brain tumor segmentation methods on the BraTS2021 dataset;

[0062] Figure 8 A comparison chart of visual results of a brain glioma segmentation method provided for an embodiment of the present application and other existing brain tumor segmentation methods on the BraTS2021 dataset;

[0063] Figure 9 A comparison chart of quantitative results of a brain glioma segmentation method provided for an embodiment of the present application and other existing brain tumor segmentation methods on the BraTS2020 dataset;

[0064] Figure 10 A comparison chart of visual results of a brain glioma segmentation method provided for an embodiment of the present application and other existing brain tumor segmentation methods on the BraTS2020 dataset. DETAILED DESCRIPTION

[0065] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as there is no conflict.

[0066] In the present application, the terms "first", "second", etc. (if any) in the present application and the accompanying drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0067] As Figure 1As shown, each sub-region has its unique pathological characteristics, which have different visual manifestations in multi-modal MRI images and have a spatial position containing relationship. Considering the differences and relationships between different sub-regions may help the model to capture more effective specific features for each target and may correspondingly improve the performance. Based on the above analysis, the present application proposes that the three sub-target regions of WT, TC and ET should be processed separately under the premise of retaining their spatial correlation. To achieve this, the present application proposes a segmentation method based on trigger information, aiming to instruct the network to dynamically adjust all modeling capabilities to each sub-target according to the trigger information. Specifically, the text prompt-based learning is integrated into the segmentation model as a static task prompt (STP), and the STP is used as a trigger condition to instruct the network to dynamically segment a specific sub-region, while enhancing the network's feature extraction ability for the sub-region during image encoding and decoding, thereby assisting the segmentation model in applying all modeling capabilities dynamically to each sub-target in the brain glioma segmentation task.

[0068] In addition, since there is a mutual containing relationship between the three sub-regions, the present application intends to integrate this spatial containing relationship into the network modeling process and develop a self region prompt (SRP) mechanism to enhance the information extraction of local regions. Specifically, the three sub-tasks are segmented in order from easy to difficult, divided into three stages, i.e., first segmenting the WT target region, then segmenting the TC target region, and finally segmenting the ET target region. Based on this order of segmentation, the implementation idea of the SRP mechanism is to use the segmentation results of the WT target region as a region prompt when segmenting the TC target region. Since the TC target region is contained in the WT target region, the model can better focus on the local region where the TC target region is located with the help of the WT target region segmentation result. When segmenting the ET target region, the segmentation results of the WT target region and the TC target region are used as region prompts. Similarly, since the ET target region is contained in both the TC target region and the WT region, the model can better focus on the local region where the ET target region is located with the help of the TC target region segmentation result and the WT target region segmentation result. In general, the self region prompt mechanism can use all previous stage segmentation results as region prompts on the basis of the static task prompt mechanism to enhance the local spatial information of each stage.

[0069] The Swin UNETR network combines the contextual modeling capabilities of the Swin Transformer with the pixel-level prediction capabilities of the U-Net, resulting in excellent semantic segmentation performance. To enable the model to dynamically perceive segmentation subtasks and dynamically enhance the model's feature extraction capabilities for corresponding tumor regions, thereby improving brain tumor segmentation accuracy, this paper improves the model structure based on the Swin UNETR network and proposes a new brain glioma segmentation model.

[0070] Before explaining the technical solution of the present invention in detail, the structure and related principles of the Swin UNETR network are first explained.

[0071] like Figure 2 As shown in (a), the core structure of the Swin UNETR network is the encoder-decoder structure. Before the encoding structure, it includes the first layer of patch partition layer, which is used to convert the multimodal input image into Divide into non-overlapping image blocks, and get Wherein, H, W and D represent the height, width and depth of the image respectively. Optionally, in the following embodiment, H, W and D are all 128. Input image I image It is composed of four modalities of brain MRI images spliced ​​together in the channel dimension.

[0072] Then the input encoding structure is used to implement feature encoding. Usually, the encoding structure specifically includes four existing Swin Transformer encoding modules. After encoding, multi-scale image features are obtained, which are recorded as Among them C i =2 i ×C, C represents the number of feature channels. Optionally, in the following embodiment, C is 48. The structure of the encoding module is as follows Figure 2 As shown in (b), the specific operation of each encoding process is as follows: input feature f i en It will first pass through a LayerNorm (LN) layer, and then pass through a window-based multi-head self-attention mechanism layer (Window-based Multi-head Self-Attention, W-MSA) to obtain Then it passes through LN and two fully connected layers, a GELU nonlinear activation function layer, and then another LN layer and a sliding window multi-head attention mechanism layer (Shifted Window-based Multi-head Self-Attention, SW-MSA), and we get After passing through the LN and two fully connected layers and a GELU nonlinear activation function layer, the feature scale is reduced through the patch merging layer to obtain the output of the encoding stage

[0073] The deepest image features obtained by the encoded structure encoding are input into the decoding structure, which usually includes five decoding modules. For the decoding process, the decoding of each scale feature is realized by the CNN structure to obtain multi-level decoding features In each CNN-based decoding stage, the input feature is After passing through an inverse convolution layer, the feature is combined with the feature in the encoding process The two convolution layers realize the jump connection, and then two convolution layers are used to process the fused feature after the jump connection to obtain

[0074] On the basis of the above Swin UNETR network, the target perception network is introduced to realize the above static task prompt mechanism; on this basis, the self-region prompt network is further introduced to realize the above self-region prompt mechanism.

[0075] The following is an example.

[0076] Example 1:

[0077] A brain glioma segmentation model based on target perception is established, as shown in Figure 3 , comprising:

[0078] Step S0: constructing a training data set of the brain glioma segmentation model through data preprocessing;

[0079] Step S1: establishing an initial segmentation model;

[0080] Step S2: training the initial segmentation model using the training data set, and obtaining the brain glioma segmentation model based on target perception after the training is completed.

[0081] The specific implementation of each step is further explained and described below.

[0082] In step S0, the preprocessing specifically includes:

[0083] Resample the multi-modal MRI data, i.e. T1, T2, T1ce, and FLAIR data, to 1mmx1mmx1mm to align the tumor regions in different modal images; then remove the black background area to remove data unrelated to brain glioma segmentation and reduce computational load and save computing resources; then, each image of different modalities is subjected to mean normalization:

[0084]

[0085] wherein I' represents the normalized image matrix of the nth image, I n represents the pixel matrix of the nth image, I n represents the average value of the image pixel matrix, I std represents the standard deviation of the image pixel matrix.

[0086] The preprocessing operation further includes: randomly cropping the normalized input image into an image block with a size of 128x128x128; and performing a data enhancement operation, including performing a random intensity offset on the pixel matrix in a range of (-0.1, 0.1), and performing a random mirror flip on each direction axis with a probability of 0.5.

[0087] On the basis of the preprocessed image, a segmentation result gold standard corresponding to a segmentation subtask is labeled in the image, and a prompt text indicating the segmentation subtask is generated, to obtain a training data set.

[0088] The segmentation subtask includes: whole tumor segmentation, tumor core segmentation, and enhanced tumor segmentation; optionally, in the embodiment, a prompt text in the form of "The task is{task}" is used as a static prompt input.{task}∈{wholetumor,tumor core,enhanced tumor}, "whole tumor", "tumor core", and "enhanced tumor" correspond to whole tumor segmentation, tumor core segmentation, and enhanced tumor segmentation, respectively. It is easy to understand that the prompt text can also take other forms, as long as the text form is uniform in the model training stage and the subsequent segmentation stage.

[0089] It is easy to understand that the above preprocessing operation and related parameters are only exemplary descriptions and should not be understood as the only limitation of the present application. In actual application, the multi-modal brain MRI image containing the text prompt of the segmentation subtask and the segmentation result gold standard can also be obtained through other ways to construct the training data set.

[0090] The structure of the initial segmentation model established in step S1 of the embodiment is shown in Figure 4 , which includes a Swin UNETR network, a target perception network, and a self-region prompt network.

[0091] The Swin UNETR network is used to perform the segmentation subtask of brain glioma on the multi-modal brain MRI image, and the number of encoding modules in the encoding structure is 4, and the number of decoding modules in the decoding structure is 5.

[0092] The target perception network is used to realize a static text prompt, such as​Figure 4 and Figure 5 As shown, the target perception network includes:

[0093] STP generation module, used to convert prompt text into a one-dimensional static task prompt vector In this embodiment, L is set to 20;

[0094] Channel feature weight map generation module, used to generate 2×4+5=13 channel feature weight maps based on the static task prompt vector;

[0095] The spatial feature weight map generation module is used to generate five spatial feature weight maps based on the static task prompt vector.

[0096] Alternatively, as Figure 5 As shown, in this embodiment, each channel feature weight map and each spatial feature weight map are generated by a multi-layer perceptron (MLP), and the channel feature weight map is represented as Channel feature enhancement is used in each stage of the encoding and decoding process to extract key semantic information from multimodal input information; the spatial feature weight map is represented as Used for spatial feature enhancement in the encoding process. In some other embodiments of the present invention, the channel feature weight map and the spatial feature weight map can also be implemented through other structures such as a fully connected layer and a CNN.

[0097] In order to utilize the generated channel feature weight map to enhance the channel features in the encoding and decoding process of the Swin UNETR network, in this embodiment, the target perception network also includes 13 channel feature enhancement modules, such as Figure 4 and Figure 5 As shown, they are respectively set after the window-based multi-head attention mechanism layer, after the sliding window-based multi-head attention mechanism layer, and after each decoding module in each encoding module of the Swin UNETR network, and are used to act on the 13 channel feature weight maps to the corresponding features to achieve channel feature enhancement.

[0098] That is to say, for each encoding process, the feature f output by the window-based multi-head attention mechanism layer i WSMA The corresponding channel feature weight map will be used Perform channel dimension feature enhancement on it, such as Figure 5 As shown, if the feature map enhanced by channel information is recorded as Then we have:

[0099]

[0100] Where m = 2 × (i-1), where denotes broadcast multiplication, LRelu is a Leaky-relu activation layer, and Conv denotes a convolution operation.

[0101] Similarly, for each feature f i output by the multi-head attention mechanism layer based on the sliding window in the encoding process SWSMA , a corresponding channel feature weight map is used to enhance the channel dimension of the feature, as shown in Figure 5 If the feature map after channel information enhancement is denoted as , then:

[0102]

[0103] where m = 2 x i - 1, denotes broadcast multiplication, LRelu is a Leaky-relu activation layer, and Conv denotes a convolution operation.

[0104] Similarly, for each feature f j output by the decoding module, a corresponding channel feature map is used to enhance the channel dimension of the feature. If the feature map after channel feature enhancement is denoted as , then:

[0105]

[0106] where m = 12 - j, denotes broadcast multiplication, LRelu is a Leaky-relu activation layer, and Conv denotes a convolution operation.

[0107] In order to use the generated spatial feature weight map to implement spatial feature enhancement for the encoding process of the Swin UNETR network, in the embodiment, the target perception network further includes: 5 spatial feature enhancement modules, as shown in Figure 4 and Figure 5 , which are respectively arranged after each encoding module in the Swin UNETR network and are respectively used to apply the 5 spatial feature maps to the corresponding features to implement spatial feature enhancement. That is, for each feature f i en output by the encoding module, a corresponding spatial feature weight map is used to enhance the spatial information. If the enhanced feature map is denoted as , then:

[0108]

[0109] where n = i, denotes broadcast multiplication.

[0110] It should be noted that, in the embodiment, the channel feature weight map and the spatial feature weight map are generated based on the static task prompt vector, and the features in the Swin UNETR network are acted on by the above-mentioned manner, which can effectively realize the channel feature enhancement and the spatial feature enhancement under a specific subtask without affecting the original segmentation performance of the Swin UNETR network, enhance the feature extraction capability of the model for a specific sub-target, and further improve the segmentation accuracy of the brain glioma.

[0111] In the embodiment, the self-region prompt network in the initial segmentation model is used to realize the self-region prompt mechanism to enhance the attention of the network to local region information. Specifically, it is divided into three stages. First, in the first stage, WT is segmented without any region prompt. The segmentation result of WT is used as the region information input when segmenting TC, and is used to act on the input stage and the intermediate feature layer of the network respectively, aiming to enhance the local region information capture ability of the network, and obtain the segmentation result of TC. Then, WT and TC are used as the region prompt input of the segmentation of ET stage, and ET is segmented.

[0112] As shown in Figure 4 , in the embodiment, the implementation of the self-region prompt mechanism is divided into two parts. The first part is in the input stage. The segmentation result of the previous segmentation subtask is divided into non-overlapping feature maps by the patch partition operation, and is added to the input image I image after the patch partition operation (i.e. ) in the element scale, so as to enhance the perception ability of the network to the local region of the input image. The second part is to more effectively realize the guiding effect of the local region information on the high-dimensional image features. The deepest image feature is denoted as f image . The segmentation result of the previous segmentation subtask is used as the region prompt, and the region feature f region is obtained by downsampling. f image and f region are further fused by the cross-attention mechanism to realize the guidance of the region feature to the image feature.

[0113] Corresponding to the above-mentioned mechanism, in the embodiment, the self-region prompt network includes a partition module, an element addition module, a downsampling module and a region prompt fusion module.

[0114] The partition module is used to divide the segmentation result of the previous segmentation subtask into non-overlapping feature maps by the patch partition operation.

[0115] The element-wise addition module is set between the Patch Partition layer and the encoding structure in the Swin UNETR network. It is used to add the non-overlapping feature maps of the multimodal brain MRI image obtained by the Patch Partition layer and the feature maps obtained by the partition module at an element-wise scale as the input of the encoding structure.

[0116] The downsampling module is used to downsample the segmentation results of the previous segmentation subtask to the level consistent with the deepest image feature f image The same scale, get the regional feature f region ; Image feature f image Image features output by the encoding structure in the Swin UNETR network;

[0117] The regional hint fusion module is set between the encoding structure and the decoding structure in the Swin UNETR network, and is used to integrate the image features f through the cross attention mechanism. image and regional characteristics f region After fusion, it serves as the input of the decoding structure;

[0118] Moreover, the training data in the training dataset also includes the segmentation results of the previous segmentation subtask;

[0119] Among them, the segmentation result of the front-stage segmentation subtask of tumor core segmentation is the segmentation result of the entire tumor; the segmentation result of the front-stage segmentation subtask of enhanced tumor segmentation is spliced ​​by the whole tumor segmentation result and the tumor core segmentation result.

[0120] Optionally, in this embodiment, the downsampling module is composed of five CNNs connected in sequence; the specific structure of the regional hint fusion module is as follows: Figure 6 Shown, including:

[0121] The first linear layer is used to transform the regional features f region Projected as feature Q mask ;

[0122] The second linear layer is used to transform the image feature f image Projected as feature K image ;

[0123] The third linear layer is used to transform the image feature f image Projection as feature V image ;

[0124] Cross attention layer, used to follow Generate cross attention map A cross ;

[0125] Fusion layer, used to combine image features f image and cross attention map A crossIn the channel dimension splicing, the feature f is obtained fusion ;

[0126] The residual convolution block is used for feature extraction on the feature f fusion After the feature extraction, the feature is added to the image feature f image .

[0127] The convolution block is used for feature extraction on the feature output by the residual convolution block, so as to obtain the image feature f image and the region feature f region after fusion

[0128] In step S2 of the embodiment, when the initial segmentation model is trained by using the training data, for each piece of training data, the multi-modal brain MRI image, the prompt text and the segmentation result of the previous segmentation subtask in the training data are input into the initial segmentation model, and for the same multi-modal brain MRI image, the segmentation is sequentially performed in the order of WT, TC and ET, the extraction ability of the network for the unique features of the corresponding sub-target regions is respectively enhanced, and the spatial inclusion relationship among them is simultaneously modeled, so as to obtain the final segmentation result of the network.

[0129] In order to better constrain the network learning process, the loss function based on Dice and cross entropy is used in the embodiment, which is respectively denoted as and

[0130]

[0131] Wherein, N V represents the total number of voxels, y n and are the standard value and the prediction result of the nth voxel, respectively. The overall training loss function of the network is :

[0132]

[0133] Optionally, the AdamW optimizer is used to update the network model weight in the training, the initial learning rate is 1e-4, and the total iteration training is 300 rounds. The parameters in the Swin UNETR network, the target perception network and the self-region prompt network in the model will be continuously updated and optimized in the training process.

[0134] During the training, it is judged in real time whether the network reaches the maximum iteration number, if yes, the testing process is entered, and if not, the initial segmentation model is repeatedly trained by inputting the training data.

[0135] During testing, the multi-modal brain MRI image, the prompt text and the segmentation result of the previous segmentation subtask are input into the network to obtain the segmentation result of each substage, and then the final segmentation result is obtained, so that more accurate segmentation is realized, thereby assisting doctors in determining surgical plans, implementing radiotherapy and chemotherapy plans, etc.

[0136] Overall, the present embodiment combines the technology based on prompt learning with the challenging task of brain tumor segmentation. Unlike existing methods for glioma segmentation, the present embodiment changes the way of processing three segmentation subtasks at a time. The model can be activated by different static task prompt information, thereby dynamically enhancing the perception ability of the model for each tumor subregion. The target perception network in the model will generate a feature weight map for a specific segmentation subtask, and adaptively enhance the channel and spatial features of a specific target during feature encoding and decoding, which enables the model to dynamically focus on each tumor subregion triggered by static text prompt information. The self-region prompt network of the model uses the region prompt information generated from the continuous prediction result to explicitly model the spatial correlation between the three sub-targets, enhances the attention of the model to local features, and helps the model better understand the ongoing target segmentation subtask.

[0137] Embodiment 2

[0138] A method for establishing a brain glioma segmentation model based on target perception, similar to the above-mentioned embodiment 1, the difference is that in the present embodiment, the established initial segmentation model only includes the Swin UNETR network and the target perception network, but does not include the self-region prompt network, and accordingly, the data in the training data set only includes the multi-modal brain MRI image and the segmentation subtask prompt text and segmentation result standard, but does not include the segmentation result of the previous segmentation subtask.

[0139] In the present embodiment, when training the established initial segmentation network using the training data set, there is no requirement for the execution order of the segmentation subtasks corresponding to the same multi-modal brain MRI image.

[0140] In the present embodiment, the specific implementation of each module can refer to the description in the above-mentioned embodiment 1, which will not be repeated here.

[0141] The present embodiment can also generate a feature weight map for a specific segmentation subtask by introducing a target perception network into the model, adaptively enhance the channel and spatial features of a specific target during feature encoding and decoding, and enable the model to dynamically focus on each tumor subregion triggered by static text prompt information, thereby effectively improving the segmentation accuracy of the model.

[0142] Embodiment 3

[0143] A brain glioma segmentation method based on target perception and self-region prompt, comprising:

[0144] Generating prompt text for indicating the segmentation sub-task type of brain glioma; the segmentation sub-tasks include: whole tumor segmentation, tumor core segmentation and enhanced tumor segmentation;

[0145] Input the prompt text of the whole tumor segmentation sub-task and the multi-modal brain MRI image to be segmented into the brain glioma segmentation model based on target perception, to obtain the segmentation result of the whole tumor segmentation sub-task;

[0146] Input the prompt text of the tumor core segmentation sub-task, the multi-modal brain MRI image to be segmented, and the segmentation result of the whole tumor segmentation sub-task into the brain glioma segmentation model based on target perception, to obtain the segmentation result of the tumor core segmentation sub-task;

[0147] After splicing the segmentation result of the whole tumor segmentation sub-task and the segmentation result of the tumor core segmentation sub-task, together with the prompt text of the enhanced tumor segmentation sub-task and the multi-modal brain MRI image to be segmented, input into the brain glioma segmentation model based on target perception, to obtain the segmentation result of the enhanced tumor segmentation sub-task, and complete the brain glioma segmentation;

[0148] Wherein, the brain glioma segmentation model based on target perception is established by the brain glioma segmentation model based on target perception establishment method provided in the above embodiment 1.

[0149] Embodiment 4:

[0150] A brain glioma segmentation method based on target perception and self-region prompt, comprising:

[0151] After generating the prompt text for indicating the segmentation sub-task type of brain glioma, combine it with the multi-modal brain MRI image to be segmented respectively, to obtain the model input corresponding to each segmentation sub-task; the segmentation sub-tasks include: whole tumor segmentation, tumor core segmentation and enhanced tumor segmentation;

[0152] Input the model input corresponding to the three segmentation sub-tasks into the brain glioma segmentation model based on target perception respectively, to obtain the segmentation results of the three segmentation sub-tasks, and complete the brain glioma segmentation;

[0153] Wherein, the brain glioma segmentation model based on target perception is established by the brain glioma segmentation model based on target perception establishment method provided in the above embodiment 2.

[0154] Embodiment 5:

[0155] A computer program product comprising a computer program which, when executed by a processor, implements the method for establishing a target perception based glioma segmentation model provided in the above-mentioned embodiment 1 or 2, or the target perception and self-region prompt based glioma segmentation method provided in the above-mentioned embodiment 3, or the target perception based glioma segmentation method provided in the above-mentioned embodiment 4.

[0156] Embodiment 6:

[0157] A computer readable storage medium comprising a stored computer program which, when executed by a processor, controls a device in which the computer readable storage medium is located to perform the method for establishing a target perception based glioma segmentation model provided in the above-mentioned embodiment 1 or 2, or the target perception and self-region prompt based glioma segmentation method provided in the above-mentioned embodiment 3, or the target perception based glioma segmentation method provided in the above-mentioned embodiment 4.

[0158] The following further analyzes and verifies the beneficial effects that can be achieved by the present application based on the segmentation results on public data sets.

[0159] Specifically, eleven existing advanced methods are selected for comparison, and experiments are conducted on two public data sets BraTS2021 and BraTS2020. The selected methods for comparison are: CA-Net, E1D3_Unet, nnU-Net, ACMINet, EoFormer, F2Net, ETUNet, NestedFormer, Swin UNETR, nnFormer, and S2CA_Net. For ease of display, the glioma segmentation model established by the present application is simply denoted as TSRNet in the following segmentation results.

[0160] The quantitative indicators used are the commonly used quantitative indicators in segmentation tasks, namely the Dice coefficient (Dice Similarity Coefficient, DSC) and the 95% Hausdorff distance (95% Hausdorff Distance, HD95). The definitions of the two indicators are as follows:

[0161]

[0162] where I pred and I GT represent the prediction result and the gold standard image respectively. d(p, q) is the 95% of the Euclidean distance of all points in the prediction image to the gold standard image, and similarly, d(p, q) is the 95% of the Euclidean distance of all points in the gold standard image to the prediction image.

[0163] Figure 7is the quantitative comparison of the segmentation results of three sub-regions based on TSRNet and other existing methods for the BraTS2021 dataset. The first four are methods based on CNN architecture, and the last seven are methods based on the hybrid architecture of ViT and CNN. The bold font in the figure represents the optimal result, and the underlined font represents the sub-optimal result. The t-test significance test is performed between each comparison method and TSRNet, and the "*" mark indicates that the p-value of the test result is less than 0.05, that is, there is a significant difference. From the results, it can be seen that:

[0164] Firstly, TSRNet achieves the highest average DSC of 91.24%, which is at least 0.78% higher than other methods, and the average HD95 value of TSRNet is 5.19mm lower than other methods. Compared with the best CNN-based method ACMINet which aims to utilize cross-modality feature interaction, TSRNet improves the DSC of WT, TC and ET by 0.44%, 0.69% and 1.57% respectively, and reduces the HD95 by 0.18mm, 2.28mm and 5.58mm respectively. Compared with the best ViT-based method S2CA-Net which focuses on shape perception and scale perception features, TSRNet improves the DSC of WT, TC and ET by 0.09%, 0.84% and 1.40% respectively, and reduces the HD95 by 0.21mm, 1.42mm and 4.47mm respectively. These results show that TSRNet can effectively enhance target-specific features and thus improve overall performance.

[0165] Secondly, compared with existing methods, TSRNet shows at least 0.69% and 1.40% significant improvement in DSC for highly entangled targets TC and ET. Accurate TC and ET segmentation results can better help neurosurgeons in clinical treatment, such as surgical planning, monitoring tumor growth and radiotherapy, indicating the effectiveness and superiority of the method.

[0166] Figure 8 is the visualization comparison of the segmentation results based on TSRNet and other existing methods for the BraTS2021 dataset. The five methods with the best results are compared, and "Axial", "Saggital" and "Coronal" represent a slice in the three axial directions, and "3D" represents a three-dimensional rendering. From the visualization results, it can be seen that the segmentation results based on TSRNet are more accurate. This is because the proposed STP mechanism enhances the key features of specific targets, and the designed SRP helps to understand the relationship between targets.

[0167] The same verification method is used to conduct experimental verification on the public dataset BraTS2020. The quantitative result comparison and visualization result areFigure 9 and Figure 10 As shown. From the results we can see that:

[0168] First, if Figure 9 As shown, TSRNet-based segmentation results consistently outperformed existing methods across all metrics. We achieved the highest average DSC of 89.97%, at least 1.10% higher than other methods, and our method's average HD95 value of 4.00 mm was lower than that of other methods. Compared to the next-best method, ACMINet, the TSRNet-based glioma segmentation method increased WT, TC, and ET by 0.77%, 0.94%, and 1.59% in DSC, respectively, and reduced HD95 by 0.52 mm, 0.16 mm, and 0.58 mm, respectively. Figure 10 The segmentation visualization results in Figure 3 show that the TSRNet-based glioma segmentation method produces fewer false positive results than other methods, which also proves the effectiveness of the proposed method.

[0169] In summary, the present invention proposes for the first time a strategy for dynamic enhancement in the glioma segmentation task using cue-based learning technology. Compared with existing brain tumor segmentation algorithms, the glioma segmentation results obtained based on the method of the present invention are not only better in quantitative evaluation, but also in visualization results, which fully demonstrates the effectiveness and superiority of the method of the present invention.

[0170] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for establishing a glioma segmentation model based on target perception, characterized in that: include: Step S1: Establish an initial segmentation model; The initial segmentation model includes: The Swin UNETR network is used to perform the glioma segmentation subtask on multimodal brain MRI images. The number of encoding modules in its encoding structure is N, and the number of decoding modules in its decoding structure is M. The segmentation subtasks include: full tumor segmentation, tumor core segmentation, and enhanced tumor segmentation. And a target perception network; the target perception network includes: An STP generation module is used to convert the prompt text into a one-dimensional static task prompt vector; the prompt text is used to indicate the type of the segmentation subtask; Channel feature weight map generation module, used to generate 2N+M channel feature weight maps based on the static task prompt vector; and 2N+M channel feature enhancement modules, which are respectively arranged after the window-based multi-head attention mechanism layer, after the sliding window-based multi-head attention mechanism layer, and after each decoding module in each encoding module of the Swin UNETR network, and are respectively used to apply the 2N+M channel feature weight maps to the corresponding features to achieve channel feature enhancement; Step S2: using the training data set to train the initial segmentation model, and after the training is completed, a target-aware brain glioma segmentation model is obtained; The training data in the training dataset includes prompt text for indicating the type of segmentation subtask and multimodal brain MRI images annotated with a gold standard of segmentation results.

2. The method for establishing a glioma segmentation model based on target perception according to claim 1, wherein: The target perception network further includes: a spatial feature weight map generation module and N spatial feature enhancement modules; The spatial feature weight map generation module is used to generate N spatial feature weight maps according to the static task prompt vector; The N spatial feature enhancement modules are respectively arranged after each encoding module in the Swin UNETR network, and are respectively used to act on the N spatial feature maps to the corresponding features to achieve spatial feature enhancement.

3. The method for establishing a glioma segmentation model based on target perception according to claim 1 or 2, characterized in that: The initial segmentation model further includes: a self-region prompt network; the self-region prompt network includes: a downsampling module and a region prompt fusion module; The downsampling module is used to downsample the segmentation results of the previous segmentation subtask to a level that is consistent with the deepest image feature f image The same scale, get the regional feature f region ; Image feature f image Image features output by the encoding structure in the Swin UNETR network; The regional hint fusion module is set between the encoding structure and the decoding structure in the Swin UNETR network, and is used to integrate the image features f image and regional characteristics f region After fusion, it is used as the input of the decoding structure; Furthermore, the training data in the training data set also includes the segmentation results of the previous segmentation subtask; Among them, the segmentation result of the front-stage segmentation subtask of tumor core segmentation is the segmentation result of the entire tumor; the segmentation result of the front-stage segmentation subtask of enhanced tumor segmentation is spliced ​​by the whole tumor segmentation result and the tumor core segmentation result.

4. The method for establishing a glioma segmentation model based on target perception according to claim 3, wherein: The regional prompt fusion module includes: The first linear layer is used to transform the regional features f region Projected as feature Q mask ; The second linear layer is used to transform the image feature f image Projected as feature K image ; The third linear layer is used to transform the image feature f image Projection as feature V image ; Cross attention layer, used to follow Generate cross attention map A cross ; Fusion layer, used to combine image features f image and cross attention map A cross Splicing in the channel dimension, we get the feature f fusion ; Residual convolution block, used to perform feature f fusion After feature extraction and image feature f image Addition; And the convolution block is used to extract the features output by the residual convolution block to obtain the image features f image and regional characteristics f region The fused features 5. The method for establishing a glioma segmentation model based on target perception according to claim 3, wherein: The self-region prompt network also includes: a partitioning module and an element addition module; The partitioning module is used to divide the segmentation results of the previous segmentation subtask into non-overlapping feature maps; The element addition module is arranged between the PatchPartition layer and the encoding structure in the SwinUNETR network, and is used to add the non-overlapping feature maps obtained by dividing the multimodal brain MRI image by the Patch Partition layer and the feature maps obtained by the partitioning module at an element scale as the input of the encoding structure.

6. The method for establishing a glioma segmentation model based on target perception according to claim 1 or 2, wherein: The channel feature weight map generation module includes 2N+M multi-layer perceptrons, each used to generate 2N+M channel feature weight maps according to a static task prompt vector; The spatial feature weight map generation module includes N multi-layer perceptrons, which are respectively used to generate N spatial feature weight maps according to the static task prompt vector.

7. A method for brain glioma segmentation based on target perception, characterized in that: include: After generating prompt text indicating the type of brain glioma segmentation subtask, it is combined with the multimodal brain MRI image to be segmented to obtain the model input corresponding to each segmentation subtask; Segmentation subtasks include: whole tumor segmentation, tumor core segmentation, and enhanced tumor segmentation; The model inputs corresponding to the three segmentation subtasks are respectively input into the target-aware glioma segmentation model to obtain the segmentation results of the three segmentation subtasks and complete the glioma segmentation. Wherein, the target perception-based glioma segmentation model is established by the target perception-based glioma segmentation model establishment method described in claim 1, 2 or 6.

8. A method for brain glioma segmentation based on target perception and self-region hints, characterized in that: include: Generate hint text for indicating the type of glioma segmentation subtask; Segmentation subtasks include: whole tumor segmentation, tumor core segmentation, and enhanced tumor segmentation; Input the prompt text of the full tumor segmentation subtask and the multimodal brain MRI image to be segmented into the target-aware glioma segmentation model to obtain the segmentation result of the full tumor segmentation subtask; Inputting the prompt text of the tumor core segmentation subtask, the multimodal brain MRI image to be segmented, and the segmentation result of the full tumor segmentation subtask into the target-aware brain glioma segmentation model to obtain the segmentation result of the tumor core segmentation subtask; After concatenating the segmentation results of the full tumor segmentation subtask with the segmentation results of the tumor core segmentation subtask, the results are input into the target-aware glioma segmentation model together with the prompt text of the enhanced tumor segmentation subtask and the multimodal brain MRI image to be segmented, thereby obtaining the segmentation results of the enhanced tumor segmentation subtask and completing the glioma segmentation. The target-aware brain glioma segmentation model is established by the target-aware brain glioma segmentation model establishment method according to any one of claims 3 to 6.

9. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the method for establishing a glioma segmentation model based on target perception according to any one of claims 1 to 6, or the method for establishing a glioma segmentation model based on target perception according to claim 7, or the method for establishing a glioma segmentation model based on target perception and self-region hints according to claim 8.

10. A computer-readable storage medium, characterized in that The invention comprises a stored computer program, which, when executed by a processor, controls the device where the computer-readable storage medium is located to execute the method for establishing a glioma segmentation model based on target perception according to any one of claims 1 to 6, or the method for establishing a glioma segmentation model based on target perception according to claim 7, or the method for establishing a glioma segmentation based on target perception and self-region prompting according to claim 8.

Images