A multi-modal image segmentation method

Abstract

The application provides a multi-modal image segmentation method, which comprises the following steps: extracting an initial feature map of a multi-modal image; decoupling different regions of the initial feature map by using an anatomy-induced region decoupler to obtain a decoupled boundary feature map and a decoupled semantic feature map; within each anatomy sub-region structure, coupling the decoupled boundary feature map and the decoupled semantic feature map by using a boundary-assisted intra-region coupler to obtain an updated boundary feature map and an updated semantic feature map with complementary information; based on the updated semantic feature map in each anatomy sub-region structure, using a cross-region coupler between different anatomy sub-region structures to enable complementary learning of features between different sub-regions to obtain a final feature map; and taking the final feature map as an input of a decoder and generating a final segmentation result through a plurality of deconvolution blocks. The scheme combines the complementary advantages of information between multi-modalities, establishes an effective segmentation model, and can reduce the calculation cost and improve the segmentation ability of the model.

Description

Technical Field

[0001] This invention belongs to the field of image segmentation technology, and specifically relates to a multimodal image segmentation method. Background Technology

[0002] Precise image segmentation and quantitative assessment of tumor regions are crucial for treatment planning, disease progression monitoring, and postoperative tumor prediction. Brain tumor segmentation primarily includes the following parts: enhanced tumor (ET), peritumoral edema / infiltrative tissue (ED), and necrotic and non-enhancing tumor core (NCR). Accurate localization and resection of different sites based on postoperative progression helps effectively prevent tumor spread and recurrence. Multimodal magnetic resonance imaging (MRI) images with varying contrast are used to segment brain tumor regions. Different modalities provide different imaging views of the brain tumor, and these multimodal imaging techniques have been widely used to determine and analyze the structure and pathology of brain tumors. Examples include T1-weighted (T1), contrast-enhanced T1-weighted (T1c), T2-weighted (T2), and fluid attenuation inversion recovery (FLAIR).

[0003] With the development of computer-aided diagnostic techniques based on multimodal MRI, numerous methods for brain tumor segmentation have emerged. Convolutional neural network (CNN)-based methods utilize successive multi-layer convolutions to learn local features about the tumor region. Furthermore, specific modality or modal shared attention models, preprocessing methods, and multi-scale frameworks are constantly being proposed to enrich local feature representations. However, due to the limitations of the local receptive field, its inherent global information is less extracted, leading to CNNs easily reaching performance saturation. Visual Transformer-based methods have demonstrated good performance in tumor segmentation tasks. This is because of the long-range dependency of features and the acquisition of effective global contextual information through stacked self-attention.

[0004] While these research methods can significantly improve brain tumor segmentation performance by utilizing complementary information from multimodal images, their limitations are equally prominent. Most existing multimodal brain tumor segmentation methods treat it as a general semantic segmentation problem. In other words, features of all different tumor regions are learned indiscriminately, which easily leads to biased feature acquisition. For example, the ED region exhibits extensive edema and tissue fluid exudation, causing large areas of edema to attract the model's primary attention, while neglecting ET and NCR. Furthermore, tumors on MRI also exhibit sparsely scattered, small, and isolated NCR regions surrounded by adjacent large regions, making it easy for the model to overlook these small, overlapping areas. Summary of the Invention

[0005] The purpose of the embodiments in this specification is to provide a multimodal image segmentation method.

[0006] To solve the above-mentioned technical problems, the embodiments of this application are implemented in the following ways:

[0007] This application provides a multimodal image segmentation method, the method comprising:

[0008] Extract the initial feature map from the multimodal image;

[0009] An anatomy-induced region decoupler is used to decouple different regions of the initial feature map, resulting in a decoupling boundary feature map and a decoupling semantic feature map.

[0010] Within each anatomical sub-region structure, a boundary-assisted intra-region coupler is used to couple and decouple the boundary feature map and the decoupled semantic feature map respectively, resulting in updated boundary feature maps and updated semantic feature maps with complementary information.

[0011] Based on the updated semantic feature map in each anatomical sub-region structure, a cross-region coupler is used between different anatomical sub-region structures to enable complementary feature learning between different sub-regions, resulting in the final feature map.

[0012] The final feature map is used as input to the decoder and is used to generate the final segmentation result through multiple deconvolution blocks.

[0013] In one embodiment, an anatomy-induced region decoupler is used to decouple different regions of the initial feature map, resulting in a decoupled boundary feature map and a decoupled semantic feature map, including:

[0014] The initial semantic feature map of the entire region of the multimodal image is learned using the high-level semantic feature map in the initial feature map;

[0015] The initial boundary feature map of the multimodal image is learned using the low-level feature map in the initial feature map;

[0016] The initial semantic feature map is used to generate a decoupled semantic feature map using the first convolutional layer;

[0017] The initial boundary feature map is used to generate a decoupled boundary feature map using the second convolutional layer.

[0018] In one embodiment, the boundary-assisted regional coupler employs multiple Transformer layers that couple information responses in parallel.

[0019] Within each anatomical sub-region structure, a boundary-assisted intra-region coupler is used to couple and decouple the boundary feature map and the semantic feature map, respectively, to obtain updated boundary feature maps and updated semantic feature maps with complementary information, including:

[0020] Within each anatomical sub-region structure, the decoupled boundary feature map, the decoupled semantic feature map, and the corresponding boundary category-related representation and semantic category-related representation are input into the Transformer layer of the coupled information response to obtain the updated boundary feature map and the updated semantic feature map.

[0021] In one embodiment, the anatomical subregion structure includes a first anatomical subregion and a second anatomical subregion;

[0022] Based on the updated semantic feature maps in each anatomical sub-region structure, a cross-region coupler is used between different anatomical sub-region structures to enable complementary feature learning between different sub-regions, resulting in the final feature maps, including:

[0023] The first updated semantic feature map of the first anatomical sub-region and its corresponding first updated semantic category related representation, and the second updated semantic feature map of the second anatomical sub-region and its corresponding second updated semantic category related representation are coupled using a Transformer layer with coupled information response to obtain the first coupled semantic feature map and the first coupled semantic category related representation of the first anatomical sub-region and the second coupled semantic feature map and the second coupled semantic category related representation of the second anatomical sub-region.

[0024] The product of the first coupled semantic feature map and the first coupled semantic category related representation is concatenated with the product of the second coupled semantic feature map and the second coupled semantic category related representation to obtain the concatenation result between sub-regions;

[0025] The stitching results between any two anatomical subregions are stitched together to obtain the stitched result;

[0026] Based on the splicing results, the final feature map is determined.

[0027] In one embodiment, the final feature map is determined based on the splicing result, including:

[0028] The convolution operation is performed on the spliced ​​result to obtain the final feature map.

[0029] In one embodiment, the Transformer layer that couples the information response includes layer normalization, attention to the information response, and a feedforward layer.

[0030] In one embodiment, the Transformer layer that couples the information response is input with a first feature map and its corresponding first category-related representation, and a second feature map and its corresponding second category-related representation.

[0031] Based on the first feature map, the first category-related representation, the second feature map, and the second category-related representation, the first splicing feature, the second splicing feature, the third splicing feature, and the fourth splicing feature are determined.

[0032] Layer normalization is performed on the first splicing feature, the second splicing feature, the third splicing feature, and the fourth splicing feature to obtain the normalized first splicing feature, the normalized second splicing feature, the normalized third splicing feature, and the normalized fourth splicing feature.

[0033] Matrix transformations are performed on the normalized first concatenated feature, normalized second concatenated feature, normalized third concatenated feature, and normalized fourth concatenated feature to obtain the first keyword and first feature value of the normalized first concatenated feature, the second query value and second keyword of the normalized second concatenated feature, the third query value and third feature value of the normalized third concatenated feature, and the fourth keyword and fourth feature value of the normalized fourth concatenated feature.

[0034] The first keyword, the first feature value, the second query value, and the second keyword are processed by the attention of the coupled information response to obtain the second coupled feature map after the first feature map is supplemented to the second feature map, and the corresponding second coupled category-related representation is obtained.

[0035] The third query value, the third feature value, the fourth keyword, and the fourth feature value are processed by the attention of the coupled information response to obtain the second feature map, which is then supplemented to the first feature map to obtain the first coupled feature map and the corresponding first coupled category related representation.

[0036] The second coupled feature map and its corresponding second coupled category-related representation, the first coupled feature map and its corresponding first coupled category-related representation are sequentially subjected to layer normalization and feedforward layers to obtain the updated first feature map and its corresponding updated first category-related representation, and the updated second feature map and its corresponding updated second category-related representation.

[0037] In one embodiment, based on the first feature map, the first category-related representation, the second feature map, and the second category-related representation, a first splicing feature, a second splicing feature, a third splicing feature, and a fourth splicing feature are determined, including:

[0038] Based on the first feature map and the second category related representation, the first most relevant feature is determined. The first most relevant feature is the first set of preset features in the first feature map that are most relevant to the second category related representation.

[0039] Based on the second feature map and the second category-related representation, the second most relevant feature is determined. The second most relevant feature is the top two preset features in the second feature map that are most relevant to the second category-related representation.

[0040] Based on the first feature map and the first category-related representation, the third most relevant feature is determined. The third most relevant feature is the top three preset features in the first feature map that are most relevant to the first category-related representation.

[0041] Based on the second feature map and the first category related representation, the fourth most relevant feature is determined. The fourth most relevant feature is the top four preset features in the second feature map that are most relevant to the first type related representation.

[0042] Based on the first most relevant feature, the second most relevant feature, the third most relevant feature, the fourth most relevant feature, the position code, the first category relevance representation, and the second category relevance representation, the first splicing feature, the second splicing feature, the third splicing feature, and the fourth splicing feature are determined.

[0043] In one embodiment, the first concatenation feature, the second concatenation feature, the third concatenation feature, and the fourth concatenation feature are determined based on the first most relevant feature, the second most relevant feature, the third most relevant feature, the fourth most relevant feature, the position code, the first category relevance representation, and the second category relevance representation, including:

[0044] The first most relevant feature, after being added with positional encoding, is concatenated with the first category-related representation to obtain the first concatenated feature;

[0045] The second most relevant feature, after being added with positional encoding, is concatenated with the second category-related representation to obtain the second concatenated feature;

[0046] The third most relevant feature, after being added with positional encoding, is concatenated with the first category-related representation to obtain the third concatenated feature;

[0047] The fourth most relevant feature, after being encoded by location, is concatenated with the second category-related representation to obtain the fourth concatenated feature.

[0048] In one embodiment, the most relevant features are determined by the top-k algorithm, wherein the most relevant features include a first most relevant feature, a second most relevant feature, a third most relevant feature, and a fourth most relevant feature.

[0049] As can be seen from the technical solutions provided in the embodiments of this specification above, this solution combines the advantages of information complementarity between multiple modalities to establish an effective segmentation model, which can reduce computational costs and improve the model's segmentation capabilities. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 A flowchart illustrating the multimodal image segmentation method provided in this application;

[0052] Figure 2 A schematic diagram of the Transformer layer structure for the coupled information response provided in this application. Detailed Implementation

[0053] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0054] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0055] Various modifications and variations can be made to the specific embodiments described in this application without departing from the scope or spirit of this application, as will be apparent to those skilled in the art. Other embodiments derived from this application will be obvious to those skilled in the art. This application specification and embodiments are merely exemplary.

[0056] The terms “include,” “including,” “have,” “contain,” etc., used in this article are all open-ended terms, meaning that they include but are not limited to.

[0057] Unless otherwise specified, "parts" in this application refers to parts by weight.

[0058] A related technique is the medical image segmentation framework based on convolutional neural networks (CNNs). Before the widespread application of Transformer-based models in computer vision, CNN-based methods achieved significant progress in medical image segmentation tasks due to their inherent ability to extract local features. For example, Ding et al. proposed a region-sensitive fusion network for brain tumor segmentation to adaptively learn probabilistic maps for multimodal data fusion. Zhao et al. proposed a novel coarse-to-fine attention network to perform multi-lesion segmentation. He et al. introduced a search framework for differentiable neural network topologies for large-scale 3D medical image segmentation, which effectively reduces GPU memory usage. Furthermore, some studies have incorporated boundary spatial information to aid in lesion target region identification. Lee et al. proposed a boundary-preserving semantic segmentation framework based on boundary keypoint maps to improve the prediction accuracy of lesion regions. Huang et al. proposed a boundary-sensitive semi-supervised medical image segmentation model that utilizes the co-occurrence relationship between target regions and corresponding boundaries to improve segmentation performance. However, due to the limitation of convolutional kernel size, these models cannot fully construct the long-distance spatial relationships between pixels in the image.

[0059] Another related technology, Transformer, has been applied to multimodal brain tumor segmentation, with the visual Transformer demonstrating excellent performance in brain tumor segmentation tasks. Wang et al. proposed combining Transformer and 3DUNet for MRI brain tumor segmentation. However, this simple integration cannot effectively learn features within and between tumor regions. Tang et al. proposed a Transformer-based self-supervised learning framework that can be customized for specific medical image segmentation agent tasks and extract more contextual information. Hatamizadeh et al. proposed using a Transformer encoder and a skip connection decoder to effectively capture long-distance relationships. Zhang et al. proposed a framework with an encoder within four mixed modalities and a decoder between modalities to capture local and global contextual information of brain tumor lesion regions. However, these studies use dense self-attention mechanisms to stack multiple Transformer layers to encode multimodal 3D images, resulting in excessive computational and storage costs. Furthermore, the accumulation and redundancy of a large amount of invalid information during Transformer-based multimodal token fusion limits the improvement of these models' segmentation performance.

[0060] The aforementioned prior art has the following drawbacks:

[0061] 1) Most existing models based on convolutional neural networks and Transformers tend to employ dense attention mechanisms, which leads to excessive computational and storage costs. Due to the limitations of the local receptive field of convolutional neural networks, less of their inherent global information is extracted, making them prone to performance saturation.

[0062] 2) The tumor appears as small and isolated NCR regions on MRI, which are sparsely scattered and surrounded by adjacent large regions. This makes it easy for existing models to overlook the existence of these small and intersecting regions, thus resulting in the inability to accurately segment the brain tumor region.

[0063] 3) Most existing convolutional neural networks and Transformer models do not consider the impact of brain tumor anatomy on segmentation performance. Features of all different tumor regions are learned indiscriminately, which easily leads to biased features.

[0064] To address the aforementioned issues, this application proposes a multimodal image segmentation method for multimodal brain tumor image segmentation. This method is based on anatomically guided decoupling-coupling learning, combining the advantages of complementary information between multiple modalities to establish an effective segmentation model, reducing computational costs and improving model segmentation capabilities.

[0065] The multimodal image segmentation method proposed in this application uses three main components: an anatomically guided region decoupler, a boundary-assisted intra-regional coupler, and a cross-regional coupler to segment lesion regions. The anatomically guided region decoupler decouples the tumor region into anatomically category-specific representations. The boundary-assisted intra-regional coupler is designed to couple boundary features and segmentation semantic features within each anatomical sub-region structure. Inter-regional complementary learning is performed by coupling complementary features between different sub-regions through the cross-regional coupler. Addressing the issue that existing models often extract limited global information and convolutional neural networks are prone to performance saturation, this invention employs a Transformer layer with coupled information response in these two couplers to transfer key contextual information between modalities, thereby dynamically masking invalid redundant information and avoiding complex computations.

[0066] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0067] Reference Figure 1 It shows a flowchart of the multimodal image segmentation method applicable to the embodiments of this application.

[0068] like Figure 1 As shown, a multimodal image segmentation method may include:

[0069] S110. Extract the initial feature map of the multimodal image.

[0070] Specifically, multimodal images can be tumor images, such as brain tumor images.

[0071] An encoder with convolutional blocks can be used to extract initial feature maps of multimodal images, such as the initial feature maps {F1, F2, F3, F4} extracted by the UNet encoder.

[0072] S120. Using an anatomically induced region decoupling device to decouple different regions of the initial feature map, resulting in a decoupling boundary feature map and a decoupling semantic feature map, which may include:

[0073] The initial semantic feature map of the entire region of the multimodal image is learned using the high-level semantic feature map in the initial feature map;

[0074] The initial boundary feature map of the multimodal image is learned using the low-level feature map in the initial feature map;

[0075] The initial semantic feature map is used to generate a decoupled semantic feature map using the first convolutional layer;

[0076] The initial boundary feature map is used to generate a decoupled boundary feature map using the second convolutional layer.

[0077] Specifically, to learn the distinctive features of different tumor regions, following the anatomical view, an anatomically guided region decoupler (or simply decoupler) decouples the features of different tumor regions and encourages the model to focus on learning the anatomical features of each internal region, avoiding interference from surrounding regions. Furthermore, considering that boundary information can help accurately locate the tumor target region, the decoupler decouples from both the boundary and semantic views.

[0078] After obtaining the initial feature maps {F1, F2, F3, F4}, use F S Low-level features from the shallow encoding stage of the network are used to create the initial boundary feature map F. E The system learns from the high-level semantic feature map to learn the initial semantic feature map for the entire region, as shown in the following formula:

[0079]

[0080] Where Up() is the upsampling operation, || is the feature map concatenation, F() is the convolution operation, and θ E and θ S These are the parameters in the respective convolutional layers of the two convolutional operations.

[0081] The obtained initial boundary feature map and initial semantic feature map are decoupled using two sets of different convolutional layers (i.e., the second convolutional layer and the first convolutional layer) to generate a decoupled boundary feature map. and decoupling semantic feature map The formula for generating the formula is shown below:

[0082]

[0083] in, and These are the parameters in the second convolutional layer and the first convolutional layer, respectively.

[0084] Understandably, to achieve better segmentation results, intermediate decoupling supervision can be added to ensure that each decoupled feature... and Only contextual information specific to the i-th anatomical sub-region is captured. Specifically, features of each sub-region are used to predict their respective boundaries and segmentation regions, forcing the model to focus on learning the anatomical features within each sub-region.

[0085] S130. Within each anatomical sub-region structure, a boundary-assisted intra-region coupler is used to couple and decouple the boundary feature map and the decoupled semantic feature map respectively, resulting in updated boundary feature maps and updated semantic feature maps with complementary information, including:

[0086] Within each anatomical sub-region structure, the decoupled boundary feature map, the decoupled semantic feature map, and the corresponding boundary category-related representation and semantic category-related representation are input into the Transformer layer of the coupled information response to obtain the updated boundary feature map and the updated semantic feature map.

[0087] Specifically, the boundary-assisted intra-region coupler can clearly distinguish the features of the entire tumor region. This coupler couples the boundary feature map and semantic feature map of each anatomical sub-region separately, unaffected by interference from other semantic categories. The coupler uses multiple parallel Transformer layers (or simply Transformer layers) that couple information responses in parallel, enabling the entire model to not only effectively extract non-locally dependent features within each sub-region through the Transformer layers, but also simultaneously capture fine-grained details with boundary priors.

[0088] This application introduces category-related tokens. and (i.e., boundary category-related representation and semantic category-related representation respectively), class-related tokens and The features output by the decoupler (i.e., the decoupling boundary feature map and the decoupling semantic feature map) are used as input to the Transformer layer of the coupling information response to obtain updated boundary feature maps and updated semantic feature maps with complementary information, specifically:

[0089]

[0090] CAformer() is a Transformer layer operation. These are parameters in the Transformer layer.

[0091] Understandably, this application further optimizes the updated semantic feature map generated by the Transformer layer for coupled information response and the updated boundary feature map of each sub-region under decoupling supervision, which also further enhances the decoupling effect.

[0092] S140. Based on the updated semantic feature map in each anatomical sub-region structure, a cross-region coupler is used between different anatomical sub-region structures to enable complementary feature learning between different sub-regions, resulting in the final feature map.

[0093] After obtaining the anatomical semantic features of each tumor subregion, a cross-region coupler based on coupling information response is further applied to achieve complementary learning of features between different subregions. Specifically, we reuse the coupling information response Transformer layer to achieve coupling between regions.

[0094] The anatomical sub-region structure includes a first anatomical sub-region and a second anatomical sub-region; S140, based on the updated semantic feature map in each anatomical sub-region structure, uses a cross-region coupler between different anatomical sub-region structures to enable complementary feature learning between different sub-regions, obtaining a final feature map, which may include:

[0095] The first updated semantic feature map of the first anatomical sub-region and its corresponding first updated semantic category related representation, and the second updated semantic feature map of the second anatomical sub-region and its corresponding second updated semantic category related representation are coupled using a Transformer layer with coupled information response to obtain the first coupled semantic feature map and the first coupled semantic category related representation of the first anatomical sub-region and the second coupled semantic feature map and the second coupled semantic category related representation of the second anatomical sub-region.

[0096] The product of the first coupled semantic feature map and the first coupled semantic category related representation is concatenated with the product of the second coupled semantic feature map and the second coupled semantic category related representation to obtain the concatenation result between sub-regions;

[0097] The stitching results between any two anatomical subregions are stitched together to obtain the stitched result;

[0098] Based on the concatenation result, the final feature map is determined. Optionally, a convolution operation can be performed on the concatenation result to obtain the final feature map.

[0099] Specifically, a Transformer with coupled information response is used for pairwise integration between sub-regions, and its coupling formula is defined as follows:

[0100]

[0101] For the semantic features of the same sub-region, we further integrate the element-wise product of each sequence and the corresponding category token through concatenation and convolution operations. This process can be defined as follows:

[0102]

[0103] S150: The final feature map is used as input to the decoder and generates the final segmentation result through multiple deconvolution blocks.

[0104] For example, the final feature map F″′ is used as input to the UNet decoder, and the final segmentation result is generated through multiple deconvolution blocks.

[0105] During the coupling process, reducing information uncertainty and computational complexity, dynamically detecting invalid tokens, and selectively learning valuable complementary information from different modalities are crucial. To adaptively achieve contextual information communication between tokens belonging to different sub-regions and categories (e.g., intra-regional semantics and boundaries in a boundary-assisted intra-regional coupler, and cross-regional semantic context in a cross-regional coupler), embodiments of this application provide a Transformer layer for coupling information responses, which includes layer normalization (corresponding to Layer Norm in the figure), attention for coupling information responses (corresponding to Dot-poduct Attention in the figure), and a feedforward layer (corresponding to FFN in the figure), as follows... Figure 2 As shown.

[0106] In one embodiment, the first feature map T is input to the Transformer layer that couples the information response. r and the corresponding first-category related representation t r Second feature map T z and the corresponding second-category related representation t z ;

[0107] Based on the first feature map, the first category-related representation, the second feature map, and the second category-related representation, the first splicing feature τ is determined. r Second splicing feature τ z Third splicing feature τ′ r Fourth splicing feature τ′ z ,include:

[0108] Based on the first feature map and the second category of related representations, determine the first most relevant feature. The first most relevant features are the top few features in the first feature map that are most relevant to the second category.

[0109] Based on the second feature map and the second category of correlation representation, determine the second most relevant feature. The second most relevant features are the top two preset features in the second feature map that are most relevant to the second category.

[0110] Based on the first feature map and the first category-related representation, determine the third most relevant feature. The third most relevant feature is the top three preset features in the first feature map that are most relevant to the first category.

[0111] Based on the second feature map and the first category of related representations, the fourth most relevant feature is determined. The fourth most relevant feature is the top four preset features in the second feature map that are most relevant to the first type of related representation;

[0112] Based on the first most relevant feature, the second most relevant feature, the third most relevant feature, the fourth most relevant feature, the position code PE, the first category correlation representation, and the second category correlation representation, the first splicing feature, the second splicing feature, the third splicing feature, and the fourth splicing feature are determined, which may include:

[0113] The first most relevant feature, after being added with positional encoding, is concatenated with the first category-related representation to obtain the first concatenated feature;

[0114] The second most relevant feature, after being added with positional encoding, is concatenated with the second category-related representation to obtain the second concatenated feature;

[0115] The third most relevant feature, after being added with positional encoding, is concatenated with the first category-related representation to obtain the third concatenated feature;

[0116] The fourth most relevant feature, after being encoded by location, is concatenated with the second category-related representation to obtain the fourth concatenated feature.

[0117] Layer normalization is performed on the first splicing feature, the second splicing feature, the third splicing feature, and the fourth splicing feature to obtain the normalized first splicing feature, the normalized second splicing feature, the normalized third splicing feature, and the normalized fourth splicing feature.

[0118] Perform matrix transformations on the normalized first concatenated feature, the normalized second concatenated feature, the normalized third concatenated feature, and the normalized fourth concatenated feature to obtain the first keyword K of the normalized first concatenated feature. r and the first eigenvalue V r The second query value Q of the normalized second concatenation feature z Second keyword K zThe third query value Q′ of the normalized third concatenation feature r and the third eigenvalue V′ r The fourth keyword K′ of the fourth splicing feature after normalization z and the fourth eigenvalue V′ z ;

[0119] The first keyword, the first feature value, the second query value, and the second keyword, after being coupled with the attention of the information response, result in a second coupled feature map obtained by supplementing the first feature map with the second feature map, and the corresponding second coupled category-related representation.

[0120] The third query value, the third feature value, the fourth keyword, and the fourth feature value, after being processed by the attention of the coupled information response, result in the second feature map, which is then added to the first feature map to obtain the first coupled feature map and the corresponding first coupled category-related representation.

[0121] The second coupled feature map and its corresponding second coupled category related representation, and the first coupled feature map and its corresponding first coupled category related representation are sequentially subjected to layer normalization and feedforward layers to obtain the updated first feature map T′. r and the corresponding updated first-category related representation t′ r and the updated second feature map T′ z and the corresponding updated second-category related representation t′ z .

[0122] The most relevant features can be determined using the top-k algorithm. These most relevant features include the first, second, third, and fourth most relevant features.

[0123] Specifically, the first, second, third, and fourth preset numbers can be set according to actual needs. The preset numbers can be equal or unequal, and there are no restrictions here.

[0124] Considering that not all tokens in a sequence can provide effective auxiliary information, this model needs to identify tokens from the current sequence that are effective for prediction and can provide supplementary clues from other sequences. Without loss of generality, given two distinct components T... r and T z Given the input token sequence, we introduce two corresponding learnable category-related tokens t. r and t z This is used to selectively filter out invalid tokens and predict the results. The calculation formula is as follows:

[0125]

[0126]

[0127]

[0128]

[0129] Where d() is the metric function used to measure the similarity between the two; PE is the positional encoding of the Transformer layer (corresponding to...). Figure 2 (middle position); the top-k() algorithm is for finding the k largest numbers, and || is a concatenation algorithm.

[0130] For [τ] r , τ z ] and [τ′ r ,τ′ z Perform layer normalization on [τ]. r , τ z ] and [τ′ r ,τ′ z ] Perform matrix transformations to obtain τ r The first query value Q r First keyword K r First eigenvalue V r , τ z The second query value Q z Second keyword K z Second eigenvalue V z τ′ r The third query value is Q′ r The third keyword K′ r The third eigenvalue V′ r ,τ′ z The fourth query value Q′ z The fourth keyword K′ z The fourth eigenvalue V′ z .

[0131]

[0132]

[0133]

[0134]

[0135] Based on this, this application designs an attention mechanism that couples information responses. and The adaptive bidirectional information flow fusion processing is performed as follows:

[0136]

[0137]

[0138] in, This refers to the second coupled feature map obtained by supplementing the first feature map with the second feature map, and the corresponding second coupled category-related representation. This refers to the first coupled feature map obtained after the second feature map is added to the first feature map, and the corresponding first coupled category-related representation.

[0139] Received and Layer normalization and feedforward layers are performed sequentially to obtain the updated first feature map T′. r and the corresponding updated first-category related representation t′ r and the updated second feature map T′ z and the corresponding updated second-category related representation t′ z .

[0140] An attention mechanism that couples information responses can fuse the boundaries and semantics within and between tumor regions to prepare for tumor region localization and improve tumor segmentation performance.

[0141] This application presents a simple and effective Transformer-based framework for segmenting multimodal brain tumors through decoupling and coupling strategies. The anatomically guided region decoupler forces the model to learn the unique features of each tumor region from an anatomical perspective. The boundary-assisted intra-region coupler couples information about intra-region boundaries into semantic features, while the cross-region coupler further effectively promotes complementary learning of features between regions.

[0142] In addition, the embodiments of this application design a coupling-sensitive Transformer layer and a coupling-sensitive attention mechanism to supplement key contextual information between different modalities, while dynamically filtering out invalid token sequences to avoid computational complexity.

[0143] The multimodal image segmentation method provided in this application embodiment shows excellent performance in segmenting small tumor regions, far exceeding other methods.

[0144] Understandably, the method of this application can be further integrated with semi-supervised, weakly supervised, or self-supervised learning techniques to more effectively utilize multimodal medical data to segment lesion regions.

[0145] It should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0146] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

Claims

1. A multimodal image segmentation method, characterized in that, The method includes: Extract the initial feature map from the multimodal image; An anatomically induced region decoupler is used to decouple different regions of the initial feature map to obtain a decoupling boundary feature map and a decoupling semantic feature map. Within each anatomical sub-region structure, a boundary-assisted intra-region coupler is used to couple the decoupled boundary feature map and the decoupled semantic feature map respectively, to obtain an updated boundary feature map and an updated semantic feature map with complementary information. Based on the updated semantic feature map in each anatomical sub-region structure, a cross-region coupler is used between different anatomical sub-region structures to enable complementary feature learning between different sub-regions, resulting in the final feature map. The final feature map serves as the input to the decoder and generates the final segmentation result through multiple deconvolution blocks.

2. The method according to claim 1, characterized in that, The method employs an anatomically induced region decoupling device to decouple different regions of the initial feature map, resulting in a decoupling boundary feature map and a decoupling semantic feature map, including: The high-level semantic feature map in the initial feature map is used to learn the initial semantic feature map for the entire region of the multimodal image; The initial boundary feature map of the multimodal image is learned using the low-level feature map in the initial feature map; The initial semantic feature map is used to generate the decoupled semantic feature map using the first convolutional layer; The initial boundary feature map is used to generate the decoupled boundary feature map using a second convolutional layer.

3. The method according to claim 1, characterized in that, The boundary-assisted regional coupler employs multiple parallel coupled information response Transformer layers; Within each anatomical sub-region structure, a boundary-assisted regional coupler couples the decoupled boundary feature map and the decoupled semantic feature map respectively, to obtain updated boundary feature maps and updated semantic feature maps with complementary information, including: Within each anatomical sub-region structure, the decoupled boundary feature map, the decoupled semantic feature map, and the corresponding boundary category-related representation and semantic category-related representation are input into the Transformer layer of the coupling information response to obtain the updated boundary feature map and the updated semantic feature map.

4. The method according to claim 1, characterized in that, The anatomical subregion structure includes a first anatomical subregion and a second anatomical subregion; The updated semantic feature map based on the structure of each anatomical sub-region uses a cross-region coupler between different anatomical sub-region structures to enable complementary feature learning between different sub-regions, resulting in a final feature map, including: The first updated semantic feature map and its corresponding first updated semantic category related representation of the first anatomical sub-region, and the second updated semantic feature map and its corresponding second updated semantic category related representation of the second anatomical sub-region are coupled using a Transformer layer with coupled information response to obtain the first coupled semantic feature map and its first coupled semantic category related representation of the first anatomical sub-region, and the second coupled semantic feature map and its second coupled semantic category related representation of the second anatomical sub-region. The product of the first coupled semantic feature map and the first coupled semantic category related representation is concatenated with the product of the second coupled semantic feature map and the second coupled semantic category related representation to obtain the concatenation result between sub-regions; The stitching results between any two anatomical subregions are stitched together to obtain the stitched result; Based on the splicing result, the final feature map is determined.

5. The method according to claim 4, characterized in that, The step of determining the final feature map based on the splicing result includes: The convolution operation is performed on the spliced ​​result to obtain the final feature map.

6. The method according to claim 3 or 4, characterized in that, The coupled information response Transformer layer includes layer normalization, coupled information response attention, and a feedforward layer.

7. The method according to claim 6, characterized in that, The inputs of the Transformer layer of the coupled information response are a first feature map and its corresponding first category-related representation, and a second feature map and its corresponding second category-related representation. Based on the first feature map, the first category-related representation, the second feature map, and the second category-related representation, the first splicing feature, the second splicing feature, the third splicing feature, and the fourth splicing feature are determined. The first splicing feature, the second splicing feature, the third splicing feature, and the fourth splicing feature are subjected to layer normalization to obtain the normalized first splicing feature, the normalized second splicing feature, the normalized third splicing feature, and the normalized fourth splicing feature. Matrix transformation is performed on the normalized first concatenation feature, normalized second concatenation feature, normalized third concatenation feature, and normalized fourth concatenation feature to obtain the first keyword and first feature value of the normalized first concatenation feature, the second query value and second keyword of the normalized second concatenation feature, the third query value and third feature value of the normalized third concatenation feature, and the fourth keyword and fourth feature value of the normalized fourth concatenation feature. The first keyword, the first feature value, the second query value, and the second keyword are processed by the attention of the coupling information response to obtain the second coupled feature map after the first feature map is supplemented to the second feature map, and the corresponding second coupled category related representation is obtained. The third query value, the third feature value, the fourth keyword, and the fourth feature value are processed by the attention of the coupling information response to obtain the first coupled feature map after the second feature map is supplemented to the first feature map, and the corresponding first coupled category related representation is obtained. The second coupled feature map and its corresponding second coupled category related representation, the first coupled feature map and its corresponding first coupled category related representation are sequentially subjected to layer normalization and feedforward layer to obtain the updated first feature map and its corresponding updated first category related representation, and the updated second feature map and its corresponding updated second category related representation.

8. The method according to claim 7, characterized in that, Based on the first feature map, the first category-related representation, the second feature map, and the second category-related representation, a first splicing feature, a second splicing feature, a third splicing feature, and a fourth splicing feature are determined, including: Based on the first feature map and the second category-related representation, a first most relevant feature is determined. The first most relevant feature is the first set of preset features in the first feature map that are most relevant to the second category-related representation. Based on the second feature map and the second category-related representation, the second most relevant feature is determined. The second most relevant feature is the top second preset number of features in the second feature map that are most relevant to the second category-related representation. Based on the first feature map and the first category-related representation, the third most relevant feature is determined. The third most relevant feature is the top third preset number of features in the first feature map that are most relevant to the first category-related representation. Based on the second feature map and the first category-related representation, the fourth most relevant feature is determined. The fourth most relevant feature is the fourth preset number of features in the second feature map that are most relevant to the first category-related representation. Based on the first most relevant feature, the second most relevant feature, the third most relevant feature, the fourth most relevant feature, the position code, the first category-related representation, and the second category-related representation, the first splicing feature, the second splicing feature, the third splicing feature, and the fourth splicing feature are determined.

9. The method according to claim 8, characterized in that, The step of determining the first splicing feature, the second splicing feature, the third splicing feature, and the fourth splicing feature based on the first most relevant feature, the second most relevant feature, the third most relevant feature, the fourth most relevant feature, the position code, the first category-related representation, and the second category-related representation includes: The first most relevant feature is combined with a positional encoding and then concatenated with the first category-related representation to obtain the first concatenated feature; The second most relevant feature, after being added to the positional encoding, is concatenated with the second category-related representation to obtain the second concatenated feature; The third most relevant feature, after being added to the positional encoding, is concatenated with the first category-related representation to obtain the third concatenated feature; The fourth most relevant feature, after being added to the positional encoding, is concatenated with the second category-related representation to obtain the fourth concatenated feature.

10. The method according to claim 8, characterized in that, The most relevant features are determined by the top-k algorithm, wherein the most relevant features include the first most relevant feature, the second most relevant feature, the third most relevant feature, and the fourth most relevant feature.

Images